From patchwork Sat Feb 15 00:09:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975778 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C283F513 for ; Sat, 15 Feb 2025 00:10:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578205; cv=none; b=KT860TL9SUfnnsrDbTC7dZqd1vi9E2BPeABnxH48vXi61xxSjl7G57+2tkDujdiFYa2OyVsFyfNLeWc1NdGP8/5T82cNIrDl7BOCRr9LQ64QFirQUuna66NzKLezscXVBt65aZmS9AhjMb1jd/s692bsU9ThVwMDEzkwKcdyjSs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578205; c=relaxed/simple; bh=zj4GIYtz6NKFAa0klaWO1xlf9wkU4S05aJWPNtEFybc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=vAQXCSTQzU1MdMq3d66whPpmRFOnfK0HioumcVP2O293/2orgPkhYyvJiDhrGLI1jNsRdned1QlMyWSzjrV/UAx/nnV31RgSeAiPW1KU7fC3kQfxY+UQ940+U7o3/9tgNq4yIBYF09fzZvTPNFVDLYkq4UYlDSSSXsw9cPYfesQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=YWU+hyBK; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="YWU+hyBK" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21f7f1e1194so66264435ad.2 for ; Fri, 14 Feb 2025 16:10:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578200; x=1740183000; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6DdRdY/fbRokL7PssPrbn76GM3fdK2ghRVLK0jFvN2Q=; b=YWU+hyBKf0KVJMxFcygkPyNwh/Twn1sLyWri9XWJK1k7pSWXMzd+W64f19bo+FLYkv kRiVjNhvaVq/vvM0wQ931aKncf6al6l5mv/WfuiyqgYSJV94H+iGD15k0lUr+SxJDxry GzbRNdxxdKyPV8lXpq6mdDVwzH+hQn/Smn+pUdi5RemdYVjVQbLtpIHnmqOnQo/iOh4a DtGSZKQuDOfTDNqk+bKH/5BKLPvLRqfhffPwQsuZWL9dzmVT6ws6Fzy7+oNL55RKpBYI Mdcz5o2X8UdLLMRkB3UienMn0KB9stjkI+7wGl9k/iDbDf8jomXFLEU2/hgBeF9U4+fz NrAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578200; x=1740183000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6DdRdY/fbRokL7PssPrbn76GM3fdK2ghRVLK0jFvN2Q=; b=T8gAXzeXfqraJM4vWlf+g/OgVIM97F5Co4/xUvWQrI+T+WnX8nsbTx1kWLVRhtSIfr H6q1rCOi6+Ja3XKzi3JSstbxABzNDWgY1oGI/EmYmhA+KfvlIHaeYk825Vuxl16MCVx7 WUk1zi2JdgMwW2DEBFzoF5DUhDUKnuMrlzRl7FPIS2YRlAsiLnXdzHjVsTOpY1+aCTyz BWshMHot/0JhwrDuvM3T6JcA1avd50SbceZz2WpW2/Hzw4rKm43irZKIU7CH9/YtQW1p W8/KQeKkA9aDRqzntjC6NZX+r0j3OvJD9h5GXk4WmrVhUmtOmuVbxfNwbc+2FPGIGdIq YWrA== X-Forwarded-Encrypted: i=1; AJvYcCWd/6jWD725M+aJ9ZApZM+UZavQYKO92Qm2SOT9cKZBR0afCNp868a5w2S0mT++WJkzx9GWuCQ=@vger.kernel.org X-Gm-Message-State: AOJu0YzD9exnE95blix31HqvASQtGT22gPMwdsl1sRIGNBMHCpx2o837 lbpOsCMjat72Pd0rii1vY2jucOV6nnZprl5jwHCT2IEFDsMe5BdT+gkyBsdOcBg= X-Gm-Gg: ASbGncsaF22sGEKCt/NK32Qi/yEchlmCxAgoeqAGL+fuIhaD6OOOgOlMKwSdow1Rm/T sKYNZDNkJu4ga2cs5BvQiYvIJEdASBb80ED0GR+4sap1T3ufqv7y+iQMMCHJjBYEQ2kUQV7d28r xUweRxytvcXh3yMkl96BEqrZN+IKD9c0hHU4qeivko0XM4IfiqX7VaCfxeqg4rlLXRKUEzPwXe6 XV2Bz97jNGEVP186DfjsOJb+Kw55fd6GbJ4V8dv7Vyz5dvqRu1hK9psM7Zj++ffoPrmr+vZIL+u X-Google-Smtp-Source: AGHT+IEjXR8JuecXB8phxrG+7IVzT1j9Bq/W/118P/wDfrs5eMaRRA+B3/yxv+UqTQXZwLX2R/zrAA== X-Received: by 2002:a17:903:22cd:b0:21f:49f2:e33f with SMTP id d9443c01a7336-221040389ddmr16899085ad.21.1739578200552; Fri, 14 Feb 2025 16:10:00 -0800 (PST) Received: from localhost ([2a03:2880:ff:42::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545cf8fsm34144255ad.152.2025.02.14.16.10.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:00 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 01/11] io_uring/zcrx: add interface queue and refill queue Date: Fri, 14 Feb 2025 16:09:36 -0800 Message-ID: <20250215000947.789731-2-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a new object called an interface queue (ifq) that represents a net rx queue that has been configured for zero copy. Each ifq is registered using a new registration opcode IORING_REGISTER_ZCRX_IFQ. The refill queue is allocated by the kernel and mapped by userspace using a new offset IORING_OFF_RQ_RING, in a similar fashion to the main SQ/CQ. It is used by userspace to return buffers that it is done with, which will then be re-used by the netdev again. The main CQ ring is used to notify userspace of received data by using the upper 16 bytes of a big CQE as a new struct io_uring_zcrx_cqe. Each entry contains the offset + len to the data. For now, each io_uring instance only has a single ifq. Reviewed-by: Jens Axboe Signed-off-by: David Wei --- Kconfig | 2 + include/linux/io_uring_types.h | 6 ++ include/uapi/linux/io_uring.h | 43 +++++++++- io_uring/KConfig | 10 +++ io_uring/Makefile | 1 + io_uring/io_uring.c | 7 ++ io_uring/memmap.h | 1 + io_uring/register.c | 7 ++ io_uring/zcrx.c | 149 +++++++++++++++++++++++++++++++++ io_uring/zcrx.h | 35 ++++++++ 10 files changed, 260 insertions(+), 1 deletion(-) create mode 100644 io_uring/KConfig create mode 100644 io_uring/zcrx.c create mode 100644 io_uring/zcrx.h diff --git a/Kconfig b/Kconfig index 745bc773f567..529ea7694ba9 100644 --- a/Kconfig +++ b/Kconfig @@ -30,3 +30,5 @@ source "lib/Kconfig" source "lib/Kconfig.debug" source "Documentation/Kconfig" + +source "io_uring/KConfig" diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index e2fef264ff8b..7bf478727a31 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -40,6 +40,8 @@ enum io_uring_cmd_flags { IO_URING_F_TASK_DEAD = (1 << 13), }; +struct io_zcrx_ifq; + struct io_wq_work_node { struct io_wq_work_node *next; }; @@ -382,6 +384,8 @@ struct io_ring_ctx { struct wait_queue_head poll_wq; struct io_restriction restrictions; + struct io_zcrx_ifq *ifq; + u32 pers_next; struct xarray personalities; @@ -434,6 +438,8 @@ struct io_ring_ctx { struct io_mapped_region ring_region; /* used for optimised request parameter and wait argument passing */ struct io_mapped_region param_region; + /* just one zcrx per ring for now, will move to io_zcrx_ifq eventually */ + struct io_mapped_region zcrx_region; }; struct io_tw_state { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index e11c82638527..6a1632d0fba1 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -639,7 +639,8 @@ enum io_uring_register_op { /* send MSG_RING without having a ring */ IORING_REGISTER_SEND_MSG_RING = 31, - /* 32 reserved for zc rx */ + /* register a netdev hw rx queue for zerocopy */ + IORING_REGISTER_ZCRX_IFQ = 32, /* resize CQ ring */ IORING_REGISTER_RESIZE_RINGS = 33, @@ -956,6 +957,46 @@ enum io_uring_socket_op { SOCKET_URING_OP_SETSOCKOPT, }; +/* Zero copy receive refill queue entry */ +struct io_uring_zcrx_rqe { + __u64 off; + __u32 len; + __u32 __pad; +}; + +struct io_uring_zcrx_cqe { + __u64 off; + __u64 __pad; +}; + +/* The bit from which area id is encoded into offsets */ +#define IORING_ZCRX_AREA_SHIFT 48 +#define IORING_ZCRX_AREA_MASK (~(((__u64)1 << IORING_ZCRX_AREA_SHIFT) - 1)) + +struct io_uring_zcrx_offsets { + __u32 head; + __u32 tail; + __u32 rqes; + __u32 __resv2; + __u64 __resv[2]; +}; + +/* + * Argument for IORING_REGISTER_ZCRX_IFQ + */ +struct io_uring_zcrx_ifq_reg { + __u32 if_idx; + __u32 if_rxq; + __u32 rq_entries; + __u32 flags; + + __u64 area_ptr; /* pointer to struct io_uring_zcrx_area_reg */ + __u64 region_ptr; /* struct io_uring_region_desc * */ + + struct io_uring_zcrx_offsets offsets; + __u64 __resv[4]; +}; + #ifdef __cplusplus } #endif diff --git a/io_uring/KConfig b/io_uring/KConfig new file mode 100644 index 000000000000..9e2a4beba1ef --- /dev/null +++ b/io_uring/KConfig @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# io_uring configuration +# + +config IO_URING_ZCRX + def_bool y + depends on PAGE_POOL + depends on INET + depends on NET_RX_BUSY_POLL diff --git a/io_uring/Makefile b/io_uring/Makefile index d695b60dba4f..98e48339d84d 100644 --- a/io_uring/Makefile +++ b/io_uring/Makefile @@ -14,6 +14,7 @@ obj-$(CONFIG_IO_URING) += io_uring.o opdef.o kbuf.o rsrc.o notif.o \ epoll.o statx.o timeout.o fdinfo.o \ cancel.o waitid.o register.o \ truncate.o memmap.o alloc_cache.o +obj-$(CONFIG_IO_URING_ZCRX) += zcrx.o obj-$(CONFIG_IO_WQ) += io-wq.o obj-$(CONFIG_FUTEX) += futex.o obj-$(CONFIG_NET_RX_BUSY_POLL) += napi.o diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ec98a0ec6f34..7c34b5459e2d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -97,6 +97,7 @@ #include "uring_cmd.h" #include "msg_ring.h" #include "memmap.h" +#include "zcrx.h" #include "timeout.h" #include "poll.h" @@ -2700,6 +2701,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) mutex_lock(&ctx->uring_lock); io_sqe_buffers_unregister(ctx); io_sqe_files_unregister(ctx); + io_unregister_zcrx_ifqs(ctx); io_cqring_overflow_kill(ctx); io_eventfd_unregister(ctx); io_free_alloc_caches(ctx); @@ -2859,6 +2861,11 @@ static __cold void io_ring_exit_work(struct work_struct *work) io_cqring_overflow_kill(ctx); mutex_unlock(&ctx->uring_lock); } + if (ctx->ifq) { + mutex_lock(&ctx->uring_lock); + io_shutdown_zcrx_ifqs(ctx); + mutex_unlock(&ctx->uring_lock); + } if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) io_move_task_work_from_local(ctx); diff --git a/io_uring/memmap.h b/io_uring/memmap.h index c898dcba2b4e..dad0aa5b1b45 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -2,6 +2,7 @@ #define IO_URING_MEMMAP_H #define IORING_MAP_OFF_PARAM_REGION 0x20000000ULL +#define IORING_MAP_OFF_ZCRX_REGION 0x30000000ULL struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); diff --git a/io_uring/register.c b/io_uring/register.c index 9a4d2fbce4ae..cc23a4c205cd 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -30,6 +30,7 @@ #include "eventfd.h" #include "msg_ring.h" #include "memmap.h" +#include "zcrx.h" #define IORING_MAX_RESTRICTIONS (IORING_RESTRICTION_LAST + \ IORING_REGISTER_LAST + IORING_OP_LAST) @@ -813,6 +814,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_clone_buffers(ctx, arg); break; + case IORING_REGISTER_ZCRX_IFQ: + ret = -EINVAL; + if (!arg || nr_args != 1) + break; + ret = io_register_zcrx_ifq(ctx, arg); + break; case IORING_REGISTER_RESIZE_RINGS: ret = -EINVAL; if (!arg || nr_args != 1) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c new file mode 100644 index 000000000000..f3ace7e8264d --- /dev/null +++ b/io_uring/zcrx.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include + +#include "io_uring.h" +#include "kbuf.h" +#include "memmap.h" +#include "zcrx.h" + +#define IO_RQ_MAX_ENTRIES 32768 + +static int io_allocate_rbuf_ring(struct io_zcrx_ifq *ifq, + struct io_uring_zcrx_ifq_reg *reg, + struct io_uring_region_desc *rd) +{ + size_t off, size; + void *ptr; + int ret; + + off = sizeof(struct io_uring); + size = off + sizeof(struct io_uring_zcrx_rqe) * reg->rq_entries; + if (size > rd->size) + return -EINVAL; + + ret = io_create_region_mmap_safe(ifq->ctx, &ifq->ctx->zcrx_region, rd, + IORING_MAP_OFF_ZCRX_REGION); + if (ret < 0) + return ret; + + ptr = io_region_get_ptr(&ifq->ctx->zcrx_region); + ifq->rq_ring = (struct io_uring *)ptr; + ifq->rqes = (struct io_uring_zcrx_rqe *)(ptr + off); + return 0; +} + +static void io_free_rbuf_ring(struct io_zcrx_ifq *ifq) +{ + io_free_region(ifq->ctx, &ifq->ctx->zcrx_region); + ifq->rq_ring = NULL; + ifq->rqes = NULL; +} + +static struct io_zcrx_ifq *io_zcrx_ifq_alloc(struct io_ring_ctx *ctx) +{ + struct io_zcrx_ifq *ifq; + + ifq = kzalloc(sizeof(*ifq), GFP_KERNEL); + if (!ifq) + return NULL; + + ifq->if_rxq = -1; + ifq->ctx = ctx; + return ifq; +} + +static void io_zcrx_ifq_free(struct io_zcrx_ifq *ifq) +{ + io_free_rbuf_ring(ifq); + kfree(ifq); +} + +int io_register_zcrx_ifq(struct io_ring_ctx *ctx, + struct io_uring_zcrx_ifq_reg __user *arg) +{ + struct io_uring_zcrx_ifq_reg reg; + struct io_uring_region_desc rd; + struct io_zcrx_ifq *ifq; + int ret; + + /* + * 1. Interface queue allocation. + * 2. It can observe data destined for sockets of other tasks. + */ + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + + /* mandatory io_uring features for zc rx */ + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN && + ctx->flags & IORING_SETUP_CQE32)) + return -EINVAL; + if (ctx->ifq) + return -EBUSY; + if (copy_from_user(®, arg, sizeof(reg))) + return -EFAULT; + if (copy_from_user(&rd, u64_to_user_ptr(reg.region_ptr), sizeof(rd))) + return -EFAULT; + if (memchr_inv(®.__resv, 0, sizeof(reg.__resv))) + return -EINVAL; + if (reg.if_rxq == -1 || !reg.rq_entries || reg.flags) + return -EINVAL; + if (reg.rq_entries > IO_RQ_MAX_ENTRIES) { + if (!(ctx->flags & IORING_SETUP_CLAMP)) + return -EINVAL; + reg.rq_entries = IO_RQ_MAX_ENTRIES; + } + reg.rq_entries = roundup_pow_of_two(reg.rq_entries); + + if (!reg.area_ptr) + return -EFAULT; + + ifq = io_zcrx_ifq_alloc(ctx); + if (!ifq) + return -ENOMEM; + + ret = io_allocate_rbuf_ring(ifq, ®, &rd); + if (ret) + goto err; + + ifq->rq_entries = reg.rq_entries; + ifq->if_rxq = reg.if_rxq; + + reg.offsets.rqes = sizeof(struct io_uring); + reg.offsets.head = offsetof(struct io_uring, head); + reg.offsets.tail = offsetof(struct io_uring, tail); + + if (copy_to_user(arg, ®, sizeof(reg)) || + copy_to_user(u64_to_user_ptr(reg.region_ptr), &rd, sizeof(rd))) { + ret = -EFAULT; + goto err; + } + + ctx->ifq = ifq; + return 0; +err: + io_zcrx_ifq_free(ifq); + return ret; +} + +void io_unregister_zcrx_ifqs(struct io_ring_ctx *ctx) +{ + struct io_zcrx_ifq *ifq = ctx->ifq; + + lockdep_assert_held(&ctx->uring_lock); + + if (!ifq) + return; + + ctx->ifq = NULL; + io_zcrx_ifq_free(ifq); +} + +void io_shutdown_zcrx_ifqs(struct io_ring_ctx *ctx) +{ + lockdep_assert_held(&ctx->uring_lock); +} diff --git a/io_uring/zcrx.h b/io_uring/zcrx.h new file mode 100644 index 000000000000..58e4ab6c6083 --- /dev/null +++ b/io_uring/zcrx.h @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef IOU_ZC_RX_H +#define IOU_ZC_RX_H + +#include + +struct io_zcrx_ifq { + struct io_ring_ctx *ctx; + struct io_uring *rq_ring; + struct io_uring_zcrx_rqe *rqes; + u32 rq_entries; + + u32 if_rxq; +}; + +#if defined(CONFIG_IO_URING_ZCRX) +int io_register_zcrx_ifq(struct io_ring_ctx *ctx, + struct io_uring_zcrx_ifq_reg __user *arg); +void io_unregister_zcrx_ifqs(struct io_ring_ctx *ctx); +void io_shutdown_zcrx_ifqs(struct io_ring_ctx *ctx); +#else +static inline int io_register_zcrx_ifq(struct io_ring_ctx *ctx, + struct io_uring_zcrx_ifq_reg __user *arg) +{ + return -EOPNOTSUPP; +} +static inline void io_unregister_zcrx_ifqs(struct io_ring_ctx *ctx) +{ +} +static inline void io_shutdown_zcrx_ifqs(struct io_ring_ctx *ctx) +{ +} +#endif + +#endif From patchwork Sat Feb 15 00:09:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975777 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AAB8DF60 for ; Sat, 15 Feb 2025 00:10:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578204; cv=none; b=k0u0B7dhqfrJzqdFLh4u3j+0haQr4xlXGFl0Xc4DFzd6SEGNZrFM2/WB6hXl6PebY0J6ElMSgZVg4Uj2obNtcC6PwPmnl6uPJ1q16NP7ctjhCwiBGVLX50HRf8oc6pPVK06IRazJa0MN+EFoUzjqrA9KSOJ7EQj3RAzT8KtRvnU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578204; c=relaxed/simple; bh=xV+HbjoPZhjUAGGHGU8YbI0mz3uuDMqf8qpq252dPvg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AlXlfykKDFU9Uz9GyAReSla27RgbODYIswCYxvOrbiJ1ufpSnVy9DB1ndGscLG+z+jv55Ud9XelhpZ70etySD+xyC7JQJc5vSP2PWz0mORBpgDYACkWNQIjvYZYekCZJWmcvl1JX7riVrgYEJbmGxpAG+J5JJ1RwW479zyiuJ20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=w+WpDSfH; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="w+WpDSfH" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21f61b01630so59820535ad.1 for ; Fri, 14 Feb 2025 16:10:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578202; x=1740183002; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ayGmHinChkh4euhmmLNAbFxNHBewzdvRepp7HKsmfMg=; b=w+WpDSfHoXl68xFW8BquntvFpXPeGGr0CE43sCjhb76kRGukvQwPuPJJMJpRaSEdCG d4a2W9Yhcvxb6YxFb0r/EabPwC2dWPEAv8x4MwukuqicudgbOzsNRlJzhdKcMARWQG/m MKwfz0iJtpSnzyzRRpJ756Ivlw0nMtu6p/eKymCJxNSu4QVC1mSzFWyr0q+FeM7QFYnY CAE+9+YqcqSLNOt6ifaEOBj4oQJxbnoZ8bJYrmpmlq3apX6QA3i0KtmT5WofbPzWMMIe L7g2OhvVPGFXDQ6mo+5SpwjU/o4Yq7ke9CSbwuPLDzDX/SQBxmXyNopY4a1R8fxn00oW YQhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578202; x=1740183002; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ayGmHinChkh4euhmmLNAbFxNHBewzdvRepp7HKsmfMg=; b=AZVQksltCxz9ydxQshJ6pp315pckrJdCI0x/pBzyjLHYZr2qzlFxFwEKmKyHJd7fJ5 4785xgYg7N/ZSaEEfebfj/McIjHyVh4IV3O5/4J1c5J169pZfAH4AmGO1DlVR7al9ZV7 gIau+Jq/wzkpFJcysWG8V10VwWeuEWA7IgfyG73sfQQtS7Mob0moW19vIrBoh2Zzo8XW +2b8b6bMk205OPwnH+eTOyx0MF9D3FuJFx+FH+HAIrDzMLpLVnyrJFjjhQVzfMl/CqfT zVDV9E+7GJkfSdHkxP7RGOFzXtKhiPWxKR5KjGS4bXxJ7uLkt+tvYDjgAdlU4nGAqK0J hLBw== X-Forwarded-Encrypted: i=1; AJvYcCXYldnCijSNd4tJGNoKiGvZZ7GJ0JSm+8ZsJa+aXF0GhF28AgRSI6925GlStIipALJ09syjuJA=@vger.kernel.org X-Gm-Message-State: AOJu0YyTsG1pPOzq9D2h+jhqSZsdGKAr2EJMpivl235DCrpusZZ4aF/g /sueEESsORtG6qTyqIzh18lB8ZhCTg/UTUfaX4/aXUANwotzgx5gqKQc2U48jFQ= X-Gm-Gg: ASbGncsxsOjkkVr1I1Ax89S5M/MGn8iidnL49w7clQsNSCv8NHU8eKqdQzZqNLVCS5I gBCwnmEzNn0ar2sv9UOuQy6AxzNk4BolMenui7izwqBHX9tjQwuitgL7yrrVe+52LFdH/cDXVXp X0QVRs+tBI1IKo/OxTWwRkFV0jmgs+tsjwV6gcxvltA0RcudX0CpmsIYFsq9aRXYJbNbD9wEckM XCYGmASQx0pwltTJz39YIEvUan0tgryebGCuaujmyC0zXwm/A58yY4hYCZgHR91oz8xJ6s4HyBX X-Google-Smtp-Source: AGHT+IHVx0wjxAiXmdCr0Fn7Fi85LqJv6+vxgpdoI89LKMlUUCt/oZixCJVVOa0VWBe6546UoRSrdw== X-Received: by 2002:a05:6a20:2584:b0:1e1:a449:ff71 with SMTP id adf61e73a8af0-1ee8d67ef96mr1645068637.1.1739578201719; Fri, 14 Feb 2025 16:10:01 -0800 (PST) Received: from localhost ([2a03:2880:ff:10::]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7324f64a39asm2160299b3a.69.2025.02.14.16.10.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:01 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 02/11] io_uring/zcrx: add io_zcrx_area Date: Fri, 14 Feb 2025 16:09:37 -0800 Message-ID: <20250215000947.789731-3-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add io_zcrx_area that represents a region of userspace memory that is used for zero copy. During ifq registration, userspace passes in the uaddr and len of userspace memory, which is then pinned by the kernel. Each net_iov is mapped to one of these pages. The freelist is a spinlock protected list that keeps track of all the net_iovs/pages that aren't used. For now, there is only one area per ifq and area registration happens implicitly as part of ifq registration. There is no API for adding/removing areas yet. The struct for area registration is there for future extensibility once we support multiple areas and TCP devmem. Reviewed-by: Jens Axboe Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- include/uapi/linux/io_uring.h | 9 ++++ io_uring/rsrc.c | 2 +- io_uring/rsrc.h | 1 + io_uring/zcrx.c | 89 ++++++++++++++++++++++++++++++++++- io_uring/zcrx.h | 16 +++++++ 5 files changed, 114 insertions(+), 3 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 6a1632d0fba1..44844707d327 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -981,6 +981,15 @@ struct io_uring_zcrx_offsets { __u64 __resv[2]; }; +struct io_uring_zcrx_area_reg { + __u64 addr; + __u64 len; + __u64 rq_area_token; + __u32 flags; + __u32 __resv1; + __u64 __resv2[2]; +}; + /* * Argument for IORING_REGISTER_ZCRX_IFQ */ diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index af39b69eb4fd..20b884c84e55 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -77,7 +77,7 @@ static int io_account_mem(struct io_ring_ctx *ctx, unsigned long nr_pages) return 0; } -static int io_buffer_validate(struct iovec *iov) +int io_buffer_validate(struct iovec *iov) { unsigned long tmp, acct_len = iov->iov_len + (PAGE_SIZE - 1); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 190f7ee45de9..0f8c20246a74 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -68,6 +68,7 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, unsigned size, unsigned type); int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); +int io_buffer_validate(struct iovec *iov); bool io_check_coalesce_buffer(struct page **page_array, int nr_pages, struct io_imu_folio_data *data); diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index f3ace7e8264d..04883a3ae80c 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -10,6 +10,7 @@ #include "kbuf.h" #include "memmap.h" #include "zcrx.h" +#include "rsrc.h" #define IO_RQ_MAX_ENTRIES 32768 @@ -44,6 +45,79 @@ static void io_free_rbuf_ring(struct io_zcrx_ifq *ifq) ifq->rqes = NULL; } +static void io_zcrx_free_area(struct io_zcrx_area *area) +{ + kvfree(area->freelist); + kvfree(area->nia.niovs); + if (area->pages) { + unpin_user_pages(area->pages, area->nia.num_niovs); + kvfree(area->pages); + } + kfree(area); +} + +static int io_zcrx_create_area(struct io_zcrx_ifq *ifq, + struct io_zcrx_area **res, + struct io_uring_zcrx_area_reg *area_reg) +{ + struct io_zcrx_area *area; + int i, ret, nr_pages; + struct iovec iov; + + if (area_reg->flags || area_reg->rq_area_token) + return -EINVAL; + if (area_reg->__resv1 || area_reg->__resv2[0] || area_reg->__resv2[1]) + return -EINVAL; + if (area_reg->addr & ~PAGE_MASK || area_reg->len & ~PAGE_MASK) + return -EINVAL; + + iov.iov_base = u64_to_user_ptr(area_reg->addr); + iov.iov_len = area_reg->len; + ret = io_buffer_validate(&iov); + if (ret) + return ret; + + ret = -ENOMEM; + area = kzalloc(sizeof(*area), GFP_KERNEL); + if (!area) + goto err; + + area->pages = io_pin_pages((unsigned long)area_reg->addr, area_reg->len, + &nr_pages); + if (IS_ERR(area->pages)) { + ret = PTR_ERR(area->pages); + area->pages = NULL; + goto err; + } + area->nia.num_niovs = nr_pages; + + area->nia.niovs = kvmalloc_array(nr_pages, sizeof(area->nia.niovs[0]), + GFP_KERNEL | __GFP_ZERO); + if (!area->nia.niovs) + goto err; + + area->freelist = kvmalloc_array(nr_pages, sizeof(area->freelist[0]), + GFP_KERNEL | __GFP_ZERO); + if (!area->freelist) + goto err; + + for (i = 0; i < nr_pages; i++) + area->freelist[i] = i; + + area->free_count = nr_pages; + area->ifq = ifq; + /* we're only supporting one area per ifq for now */ + area->area_id = 0; + area_reg->rq_area_token = (u64)area->area_id << IORING_ZCRX_AREA_SHIFT; + spin_lock_init(&area->freelist_lock); + *res = area; + return 0; +err: + if (area) + io_zcrx_free_area(area); + return ret; +} + static struct io_zcrx_ifq *io_zcrx_ifq_alloc(struct io_ring_ctx *ctx) { struct io_zcrx_ifq *ifq; @@ -59,6 +133,9 @@ static struct io_zcrx_ifq *io_zcrx_ifq_alloc(struct io_ring_ctx *ctx) static void io_zcrx_ifq_free(struct io_zcrx_ifq *ifq) { + if (ifq->area) + io_zcrx_free_area(ifq->area); + io_free_rbuf_ring(ifq); kfree(ifq); } @@ -66,6 +143,7 @@ static void io_zcrx_ifq_free(struct io_zcrx_ifq *ifq) int io_register_zcrx_ifq(struct io_ring_ctx *ctx, struct io_uring_zcrx_ifq_reg __user *arg) { + struct io_uring_zcrx_area_reg area; struct io_uring_zcrx_ifq_reg reg; struct io_uring_region_desc rd; struct io_zcrx_ifq *ifq; @@ -99,7 +177,7 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, } reg.rq_entries = roundup_pow_of_two(reg.rq_entries); - if (!reg.area_ptr) + if (copy_from_user(&area, u64_to_user_ptr(reg.area_ptr), sizeof(area))) return -EFAULT; ifq = io_zcrx_ifq_alloc(ctx); @@ -110,6 +188,10 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, if (ret) goto err; + ret = io_zcrx_create_area(ifq, &ifq->area, &area); + if (ret) + goto err; + ifq->rq_entries = reg.rq_entries; ifq->if_rxq = reg.if_rxq; @@ -122,7 +204,10 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, ret = -EFAULT; goto err; } - + if (copy_to_user(u64_to_user_ptr(reg.area_ptr), &area, sizeof(area))) { + ret = -EFAULT; + goto err; + } ctx->ifq = ifq; return 0; err: diff --git a/io_uring/zcrx.h b/io_uring/zcrx.h index 58e4ab6c6083..53fd94b65b38 100644 --- a/io_uring/zcrx.h +++ b/io_uring/zcrx.h @@ -3,9 +3,25 @@ #define IOU_ZC_RX_H #include +#include + +struct io_zcrx_area { + struct net_iov_area nia; + struct io_zcrx_ifq *ifq; + + u16 area_id; + struct page **pages; + + /* freelist */ + spinlock_t freelist_lock ____cacheline_aligned_in_smp; + u32 free_count; + u32 *freelist; +}; struct io_zcrx_ifq { struct io_ring_ctx *ctx; + struct io_zcrx_area *area; + struct io_uring *rq_ring; struct io_uring_zcrx_rqe *rqes; u32 rq_entries; From patchwork Sat Feb 15 00:09:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975779 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC1EC14F98 for ; Sat, 15 Feb 2025 00:10:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578205; cv=none; b=Zn2Ix1vIWt8DQhFJ9jQLQBNIpvNaNZoIarherHpZSoLF6pbT0ASkmCzTZSpEnFBpt7BvVABFw+7pNRyR4ZRuZ9GeQL5tSeKJdEs7VU4MwY59bsnt8WPYZykMW9WcFjcabk3pp7SybvTm8jnvNvVOdhW2e9fDjBGssDHu7IjSexI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578205; c=relaxed/simple; bh=moxT/9tlqEGGqkvg89jEl7qnqkLxWddMi6wsJk9KyOo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ILIBK+Ub5FtFQJJZFJlC2h1XtFJuDmtA0tMj3jhezSwWsSOhCIClvufdGTImT/0aKXIu6juZdB1yElEtRmq/DxOEEUiSYo8ruEP/aXt4oazU7ZqtnEIdy4HNaonGKNz86rYR2M0SVEd/bxcXF8vrGsQxnLcfhh6Qwzm0T/RH3RE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=Hv2y3sGr; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="Hv2y3sGr" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-220c8cf98bbso52232425ad.1 for ; Fri, 14 Feb 2025 16:10:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578203; x=1740183003; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6McXPFUQHYSYEvRYSRp9lVm8DJpoafPAg8IZnzA+YUI=; b=Hv2y3sGrOOkWb/VZEERGqAG1vj1LIOJtaU+Du3+TlUbZG+TRBGgoPBY8poq9aU0MC2 F0xUvANubrw+8DOTH4WkW6QPXAbR+BTgGdxBc/FW/Uwk4SgENLhR6kULK+SPHJHc2V6B XZ0eIlzXxrJyXxVHKPuwcYwCRcnMkxyA0D3ad1GpuZ4f++Zl/zyu18R1KS/+16XoeYyu Y9S/h24CLzFCWid8xyQmKhh6pSV01lIZBsFqN4jbZRGBIcvn1OMejYs9pmpSBO8Ucq+9 xZERpSHyM1jubM5rU333XtgJQQAsJrD9jLiZdCx7nafq2BK3GCU70uaciqgzeiG6aEaR 1Yyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578203; x=1740183003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6McXPFUQHYSYEvRYSRp9lVm8DJpoafPAg8IZnzA+YUI=; b=GBLfdwAvzvFkPcAAabyF5YZpyirja7I6TKKRJ5+VkWlnDt9ATolGGM04SiyKwlKHhg 0yFcx19roKxIotaN+nf1JCrd10czfQzUwRq6/bEPhjw+aUKJEAjF/ig1SJ3BQgfW+YLg dCAEmJNoBnk+TnUilrKzKVGo2LdfNxXkZ8YBk6Gw/paVJ13lTF6RppZ9kCfEo2MGvonJ kY1As/MRdE8On6+1BgL94Tagi3e1kAwDNF1u6guKmJ3QCGbWTxqAo5Z58YGD46Fp/Y2z sQeaM38zjWQ6obPKddN9zKGxWPeHXl5xrb677d6fOK+OzLhj00Jdbb44lu7+/zrlV33d 7h3A== X-Forwarded-Encrypted: i=1; AJvYcCWt83kmasNKhI0iec6npw7uSOO94gmVQ1S9vbrpNDs0dw6rnu1ffghoqsTzXd/zPs557nypFQU=@vger.kernel.org X-Gm-Message-State: AOJu0YwUgMVpSm8ph0OMVdYcocj/mpI5QvfE9F+PXRjudgr2knL5FqzP F8uhh5ScNnYttAPYN61D5mE9BR21wN8Kf71kikU94YAYPo6YL1gDTOpHiQp4XNs= X-Gm-Gg: ASbGncv5Rj3HE6+G7A+RiS8acBT/CxftEcRfM35Zor28tFALJzvTUWNH6hnd+gXe9CL WK50P1rGwYlQh+j13h4ej0Tq/qumw986tU+5H5M/P7r3nzNNxnWonElAbLxnmbzoutIbuxDjFo9 nwsCObBSHbdCPxYmFs67zwDj2tQVpeHl0QBN+dVa098G0Swg8eRsX+g2KDKZ98bwaKYphKpYkF/ O6zqbyEQDd7/yQYgHqkW4UMIhBdPm6cogdrga89n2w8D55BoERRcVCUHro5kKziwwRKNAVME6A= X-Google-Smtp-Source: AGHT+IFevbVH5i83MykE+u/7tPFQvttGF3Y4A0qYLAhpnjloJAQZ2GzCkWB/VjCLdy7ev7yQpcrcKg== X-Received: by 2002:a17:902:c411:b0:220:d078:eb33 with SMTP id d9443c01a7336-221040ac0a7mr18352845ad.36.1739578203042; Fri, 14 Feb 2025 16:10:03 -0800 (PST) Received: from localhost ([2a03:2880:ff:f::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d53643d2sm34229195ad.57.2025.02.14.16.10.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:02 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 03/11] io_uring/zcrx: grab a net device Date: Fri, 14 Feb 2025 16:09:38 -0800 Message-ID: <20250215000947.789731-4-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Pavel Begunkov Zerocopy receive needs a net device to bind to its rx queue and dma map buffers. As a preparation to following patches, resolve a net device from the if_idx parameter with no functional changes otherwise. Reviewed-by: Jens Axboe Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- io_uring/zcrx.c | 28 ++++++++++++++++++++++++++++ io_uring/zcrx.h | 5 +++++ 2 files changed, 33 insertions(+) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index 04883a3ae80c..435cd634f91c 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -3,6 +3,8 @@ #include #include #include +#include +#include #include @@ -128,13 +130,28 @@ static struct io_zcrx_ifq *io_zcrx_ifq_alloc(struct io_ring_ctx *ctx) ifq->if_rxq = -1; ifq->ctx = ctx; + spin_lock_init(&ifq->lock); return ifq; } +static void io_zcrx_drop_netdev(struct io_zcrx_ifq *ifq) +{ + spin_lock(&ifq->lock); + if (ifq->netdev) { + netdev_put(ifq->netdev, &ifq->netdev_tracker); + ifq->netdev = NULL; + } + spin_unlock(&ifq->lock); +} + static void io_zcrx_ifq_free(struct io_zcrx_ifq *ifq) { + io_zcrx_drop_netdev(ifq); + if (ifq->area) io_zcrx_free_area(ifq->area); + if (ifq->dev) + put_device(ifq->dev); io_free_rbuf_ring(ifq); kfree(ifq); @@ -195,6 +212,17 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, ifq->rq_entries = reg.rq_entries; ifq->if_rxq = reg.if_rxq; + ret = -ENODEV; + ifq->netdev = netdev_get_by_index(current->nsproxy->net_ns, reg.if_idx, + &ifq->netdev_tracker, GFP_KERNEL); + if (!ifq->netdev) + goto err; + + ifq->dev = ifq->netdev->dev.parent; + if (!ifq->dev) + return -EOPNOTSUPP; + get_device(ifq->dev); + reg.offsets.rqes = sizeof(struct io_uring); reg.offsets.head = offsetof(struct io_uring, head); reg.offsets.tail = offsetof(struct io_uring, tail); diff --git a/io_uring/zcrx.h b/io_uring/zcrx.h index 53fd94b65b38..595bca0001d2 100644 --- a/io_uring/zcrx.h +++ b/io_uring/zcrx.h @@ -4,6 +4,7 @@ #include #include +#include struct io_zcrx_area { struct net_iov_area nia; @@ -27,6 +28,10 @@ struct io_zcrx_ifq { u32 rq_entries; u32 if_rxq; + struct device *dev; + struct net_device *netdev; + netdevice_tracker netdev_tracker; + spinlock_t lock; }; #if defined(CONFIG_IO_URING_ZCRX) From patchwork Sat Feb 15 00:09:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975780 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0846412E5D for ; Sat, 15 Feb 2025 00:10:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578206; cv=none; b=OHcptrJdrSsiIT47NgNib6MVdbFDATwVpDfqvSDYSDWV1s8dkb9Skzdup1wltxTMJcdGCHdRqNe8SUTQfQa9xvlCgrzQq6JSBg3JJI4FKHmyCPgeJj/oY/RYJN+JCTbVYMr3dSfHoD1M9ISngC4BsTRNpd85PII7qF/stnaBvyA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578206; c=relaxed/simple; bh=YJds1tZitfGyTje8MB2OGm/pFeRAuOhChMxLf5uotj4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t8tICKDFf5rHHlB7wyYTtvATdu4BahuZhtTElohA0BrIowiOprQF7+r84+YnuNxl53Rl49PFQXOFRxPOiZtt0LAyra4+YQOx3kuJlcaWuwIbYwfpW+E9Enk6C5VP6++rtwH4YxV4JBclCkKLxCLSXaFeLmjEX88XMlXVZHlgfkc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=n9mjuDgj; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="n9mjuDgj" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-220bfdfb3f4so54727475ad.2 for ; Fri, 14 Feb 2025 16:10:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578204; x=1740183004; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PiXV3d0VTqH63q2mQrRj8B/2tlw/jhipl7dwMOEsW14=; b=n9mjuDgjv9j2T/yeKB98c0btkPhTn7hcCeWs+0GTzBA1hu2yaqMZsOO/YT3DCFXWNM UBY5yP2aaL1fwOx7iHtf4RGTtZjR/uYeSZ4DCBqWG2FfcNdXkl7d7dQtfMFcqT6Rh3h8 l0M8DhU+AeesqP/BAOTzFYZ77iWa+T1QOj9jFaUXsNzQ9DvviwFSvHzRmztpbW33mDyn grHJtCyHYZtcvRPVPq72OX5ItbAJZUsGb2zTP6+5utkV4fh0QpqyNsogLIqMShh5OFUf 2Egl5nCvqTLsOsceOY+nC6/UYQxB5371WA+Qdx3E3ByinIpulL+Kx773KhgI3tBABUXs CfoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578204; x=1740183004; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PiXV3d0VTqH63q2mQrRj8B/2tlw/jhipl7dwMOEsW14=; b=AsptDNHecoDrOW5lHeUcSgtarPzjbR+xmktpGdKjhIBLKRRL0iSev2H4jVOviF5P5T 24p1d8iMWh3XySMNfT1aU7jKGSyMFWX2AVrr2RXZ//cFAsTL7euXr0juXCL+IitSb1c6 WC9EqHH+uL/kGplIlEKgsQfCqL298yftVfX8hTqIh1fu33m2vuH/1pDAApjl3AwCGFIM DtEaPsnEyEbpVB7OOXLbvl8Ix6OaxbXdFxu74APBBOGUiJsuvDH8kHlUjDL2gN5N4oeV TK4vyYxsZu8ysseJYqp1bxAgF0nPPg4n+XLyOeHGKoEUgK7vAW7uZGFoHLE21StPYUr1 F2iw== X-Forwarded-Encrypted: i=1; AJvYcCV+Jb73QlLDCrhFUpItF9nquyfM80P4lu1gmGJEJU/Dh64j5XG5KCOnlRdNAA38I8IaaOElHtI=@vger.kernel.org X-Gm-Message-State: AOJu0YydKBXc2Rv4kWv5OucMTDGLzk5NBDoMdw1Gx2rBom295MxdUEF+ jQoJk3TJMf0Era+iPShS5TMaax1/UxuNnBD9QSZXGYrmxRkEbpi3R1dQlFkhxeg= X-Gm-Gg: ASbGncvmzKiVMlQ+JIY2Asz2dOeoc61KMGWGQS5OvUgfwiEF/o4DdpmkYYvuBF4ogkn WOKKh8NvyG14pKXWLeGXHUHYNTqFNDljHBzLBU1UIzpagKzumhVWSK9At2P97/xgjlgOW6NoLeg tWNWXLsCgVsSl/lJfRe49Bt3pMX8EthJ0U5BHNb1/eH4gUr6bNZDWeLXkGIrj0tTbJw0ymHvCiG Rr+0XiguzhTJ9o7nIZugaMXxtDlPBr9DIOlKarrMPXykNRwbgiA+8f1KdXtkmkm1oBwIdRC+OA= X-Google-Smtp-Source: AGHT+IHOq8EVNpG5Sf97eQZmapotxKP6T77dYDjx/EMSGRDm3SnXRx9J4FymyLNtkPRdZ9fNZimBAw== X-Received: by 2002:a17:902:dac5:b0:215:431f:268f with SMTP id d9443c01a7336-22103efc119mr21819935ad.10.1739578204227; Fri, 14 Feb 2025 16:10:04 -0800 (PST) Received: from localhost ([2a03:2880:ff:3::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d5596133sm34302175ad.259.2025.02.14.16.10.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:03 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 04/11] io_uring/zcrx: implement zerocopy receive pp memory provider Date: Fri, 14 Feb 2025 16:09:39 -0800 Message-ID: <20250215000947.789731-5-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Pavel Begunkov Implement a page pool memory provider for io_uring to receieve in a zero copy fashion. For that, the provider allocates user pages wrapped around into struct net_iovs, that are stored in a previously registered struct net_iov_area. Unlike the traditional receive, that frees pages and returns them back to the page pool right after data was copied to the user, e.g. inside recv(2), we extend the lifetime until the user space confirms that it's done processing the data. That's done by taking a net_iov reference. When the user is done with the buffer, it must return it back to the kernel by posting an entry into the refill ring, which is usually polled off the io_uring memory provider callback in the page pool's netmem allocation path. There is also a separate set of per net_iov "user" references accounting whether a buffer is currently given to the user (including possible fragmentation). Reviewed-by: Jens Axboe Reviewed-by: Mina Almasry Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- io_uring/zcrx.c | 274 ++++++++++++++++++++++++++++++++++++++++++++++++ io_uring/zcrx.h | 3 + 2 files changed, 277 insertions(+) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index 435cd634f91c..f9e924cfa829 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -2,10 +2,17 @@ #include #include #include +#include #include #include #include +#include +#include +#include + +#include + #include #include "io_uring.h" @@ -16,6 +23,33 @@ #define IO_RQ_MAX_ENTRIES 32768 +__maybe_unused +static const struct memory_provider_ops io_uring_pp_zc_ops; + +static inline struct io_zcrx_area *io_zcrx_iov_to_area(const struct net_iov *niov) +{ + struct net_iov_area *owner = net_iov_owner(niov); + + return container_of(owner, struct io_zcrx_area, nia); +} + +static inline atomic_t *io_get_user_counter(struct net_iov *niov) +{ + struct io_zcrx_area *area = io_zcrx_iov_to_area(niov); + + return &area->user_refs[net_iov_idx(niov)]; +} + +static bool io_zcrx_put_niov_uref(struct net_iov *niov) +{ + atomic_t *uref = io_get_user_counter(niov); + + if (unlikely(!atomic_read(uref))) + return false; + atomic_dec(uref); + return true; +} + static int io_allocate_rbuf_ring(struct io_zcrx_ifq *ifq, struct io_uring_zcrx_ifq_reg *reg, struct io_uring_region_desc *rd) @@ -51,6 +85,7 @@ static void io_zcrx_free_area(struct io_zcrx_area *area) { kvfree(area->freelist); kvfree(area->nia.niovs); + kvfree(area->user_refs); if (area->pages) { unpin_user_pages(area->pages, area->nia.num_niovs); kvfree(area->pages); @@ -106,6 +141,19 @@ static int io_zcrx_create_area(struct io_zcrx_ifq *ifq, for (i = 0; i < nr_pages; i++) area->freelist[i] = i; + area->user_refs = kvmalloc_array(nr_pages, sizeof(area->user_refs[0]), + GFP_KERNEL | __GFP_ZERO); + if (!area->user_refs) + goto err; + + for (i = 0; i < nr_pages; i++) { + struct net_iov *niov = &area->nia.niovs[i]; + + niov->owner = &area->nia; + area->freelist[i] = i; + atomic_set(&area->user_refs[i], 0); + } + area->free_count = nr_pages; area->ifq = ifq; /* we're only supporting one area per ifq for now */ @@ -131,6 +179,7 @@ static struct io_zcrx_ifq *io_zcrx_ifq_alloc(struct io_ring_ctx *ctx) ifq->if_rxq = -1; ifq->ctx = ctx; spin_lock_init(&ifq->lock); + spin_lock_init(&ifq->rq_lock); return ifq; } @@ -256,7 +305,232 @@ void io_unregister_zcrx_ifqs(struct io_ring_ctx *ctx) io_zcrx_ifq_free(ifq); } +static struct net_iov *__io_zcrx_get_free_niov(struct io_zcrx_area *area) +{ + unsigned niov_idx; + + lockdep_assert_held(&area->freelist_lock); + + niov_idx = area->freelist[--area->free_count]; + return &area->nia.niovs[niov_idx]; +} + +static void io_zcrx_return_niov_freelist(struct net_iov *niov) +{ + struct io_zcrx_area *area = io_zcrx_iov_to_area(niov); + + spin_lock_bh(&area->freelist_lock); + area->freelist[area->free_count++] = net_iov_idx(niov); + spin_unlock_bh(&area->freelist_lock); +} + +static void io_zcrx_return_niov(struct net_iov *niov) +{ + netmem_ref netmem = net_iov_to_netmem(niov); + + page_pool_put_unrefed_netmem(niov->pp, netmem, -1, false); +} + +static void io_zcrx_scrub(struct io_zcrx_ifq *ifq) +{ + struct io_zcrx_area *area = ifq->area; + int i; + + if (!area) + return; + + /* Reclaim back all buffers given to the user space. */ + for (i = 0; i < area->nia.num_niovs; i++) { + struct net_iov *niov = &area->nia.niovs[i]; + int nr; + + if (!atomic_read(io_get_user_counter(niov))) + continue; + nr = atomic_xchg(io_get_user_counter(niov), 0); + if (nr && !page_pool_unref_netmem(net_iov_to_netmem(niov), nr)) + io_zcrx_return_niov(niov); + } +} + void io_shutdown_zcrx_ifqs(struct io_ring_ctx *ctx) { lockdep_assert_held(&ctx->uring_lock); + + if (ctx->ifq) + io_zcrx_scrub(ctx->ifq); +} + +static inline u32 io_zcrx_rqring_entries(struct io_zcrx_ifq *ifq) +{ + u32 entries; + + entries = smp_load_acquire(&ifq->rq_ring->tail) - ifq->cached_rq_head; + return min(entries, ifq->rq_entries); } + +static struct io_uring_zcrx_rqe *io_zcrx_get_rqe(struct io_zcrx_ifq *ifq, + unsigned mask) +{ + unsigned int idx = ifq->cached_rq_head++ & mask; + + return &ifq->rqes[idx]; +} + +static void io_zcrx_ring_refill(struct page_pool *pp, + struct io_zcrx_ifq *ifq) +{ + unsigned int mask = ifq->rq_entries - 1; + unsigned int entries; + netmem_ref netmem; + + spin_lock_bh(&ifq->rq_lock); + + entries = io_zcrx_rqring_entries(ifq); + entries = min_t(unsigned, entries, PP_ALLOC_CACHE_REFILL - pp->alloc.count); + if (unlikely(!entries)) { + spin_unlock_bh(&ifq->rq_lock); + return; + } + + do { + struct io_uring_zcrx_rqe *rqe = io_zcrx_get_rqe(ifq, mask); + struct io_zcrx_area *area; + struct net_iov *niov; + unsigned niov_idx, area_idx; + + area_idx = rqe->off >> IORING_ZCRX_AREA_SHIFT; + niov_idx = (rqe->off & ~IORING_ZCRX_AREA_MASK) >> PAGE_SHIFT; + + if (unlikely(rqe->__pad || area_idx)) + continue; + area = ifq->area; + + if (unlikely(niov_idx >= area->nia.num_niovs)) + continue; + niov_idx = array_index_nospec(niov_idx, area->nia.num_niovs); + + niov = &area->nia.niovs[niov_idx]; + if (!io_zcrx_put_niov_uref(niov)) + continue; + + netmem = net_iov_to_netmem(niov); + if (page_pool_unref_netmem(netmem, 1) != 0) + continue; + + if (unlikely(niov->pp != pp)) { + io_zcrx_return_niov(niov); + continue; + } + + net_mp_netmem_place_in_cache(pp, netmem); + } while (--entries); + + smp_store_release(&ifq->rq_ring->head, ifq->cached_rq_head); + spin_unlock_bh(&ifq->rq_lock); +} + +static void io_zcrx_refill_slow(struct page_pool *pp, struct io_zcrx_ifq *ifq) +{ + struct io_zcrx_area *area = ifq->area; + + spin_lock_bh(&area->freelist_lock); + while (area->free_count && pp->alloc.count < PP_ALLOC_CACHE_REFILL) { + struct net_iov *niov = __io_zcrx_get_free_niov(area); + netmem_ref netmem = net_iov_to_netmem(niov); + + net_mp_niov_set_page_pool(pp, niov); + net_mp_netmem_place_in_cache(pp, netmem); + } + spin_unlock_bh(&area->freelist_lock); +} + +static netmem_ref io_pp_zc_alloc_netmems(struct page_pool *pp, gfp_t gfp) +{ + struct io_zcrx_ifq *ifq = pp->mp_priv; + + /* pp should already be ensuring that */ + if (unlikely(pp->alloc.count)) + goto out_return; + + io_zcrx_ring_refill(pp, ifq); + if (likely(pp->alloc.count)) + goto out_return; + + io_zcrx_refill_slow(pp, ifq); + if (!pp->alloc.count) + return 0; +out_return: + return pp->alloc.cache[--pp->alloc.count]; +} + +static bool io_pp_zc_release_netmem(struct page_pool *pp, netmem_ref netmem) +{ + struct net_iov *niov; + + if (WARN_ON_ONCE(!netmem_is_net_iov(netmem))) + return false; + + niov = netmem_to_net_iov(netmem); + net_mp_niov_clear_page_pool(niov); + io_zcrx_return_niov_freelist(niov); + return false; +} + +static int io_pp_zc_init(struct page_pool *pp) +{ + struct io_zcrx_ifq *ifq = pp->mp_priv; + + if (WARN_ON_ONCE(!ifq)) + return -EINVAL; + if (pp->dma_map) + return -EOPNOTSUPP; + if (pp->p.order != 0) + return -EOPNOTSUPP; + + percpu_ref_get(&ifq->ctx->refs); + return 0; +} + +static void io_pp_zc_destroy(struct page_pool *pp) +{ + struct io_zcrx_ifq *ifq = pp->mp_priv; + struct io_zcrx_area *area = ifq->area; + + if (WARN_ON_ONCE(area->free_count != area->nia.num_niovs)) + return; + percpu_ref_put(&ifq->ctx->refs); +} + +static int io_pp_nl_fill(void *mp_priv, struct sk_buff *rsp, + struct netdev_rx_queue *rxq) +{ + struct nlattr *nest; + int type; + + type = rxq ? NETDEV_A_QUEUE_IO_URING : NETDEV_A_PAGE_POOL_IO_URING; + nest = nla_nest_start(rsp, type); + if (!nest) + return -EMSGSIZE; + nla_nest_end(rsp, nest); + + return 0; +} + +static void io_pp_uninstall(void *mp_priv, struct netdev_rx_queue *rxq) +{ + struct pp_memory_provider_params *p = &rxq->mp_params; + struct io_zcrx_ifq *ifq = mp_priv; + + io_zcrx_drop_netdev(ifq); + p->mp_ops = NULL; + p->mp_priv = NULL; +} + +static const struct memory_provider_ops io_uring_pp_zc_ops = { + .alloc_netmems = io_pp_zc_alloc_netmems, + .release_netmem = io_pp_zc_release_netmem, + .init = io_pp_zc_init, + .destroy = io_pp_zc_destroy, + .nl_fill = io_pp_nl_fill, + .uninstall = io_pp_uninstall, +}; diff --git a/io_uring/zcrx.h b/io_uring/zcrx.h index 595bca0001d2..6c808240ac91 100644 --- a/io_uring/zcrx.h +++ b/io_uring/zcrx.h @@ -9,6 +9,7 @@ struct io_zcrx_area { struct net_iov_area nia; struct io_zcrx_ifq *ifq; + atomic_t *user_refs; u16 area_id; struct page **pages; @@ -26,6 +27,8 @@ struct io_zcrx_ifq { struct io_uring *rq_ring; struct io_uring_zcrx_rqe *rqes; u32 rq_entries; + u32 cached_rq_head; + spinlock_t rq_lock; u32 if_rxq; struct device *dev; From patchwork Sat Feb 15 00:09:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975781 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 354CD13C80E for ; Sat, 15 Feb 2025 00:10:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578207; cv=none; b=ErPJISfqLWinHlZL+DGoPgEBAsu4pQXOI99Vk2ka6ZidNBJ1qhlJ1wT1LQJAJNBR9q30nZCNmzMESLESNDAxfGDomVfl8ACOQ5wdHJBG73zwD2e1ySmMtBbKOfWWD8yKun83ck5x1SUY1vym7WxWnHGA7J/z8jqungi6sR36bxo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578207; c=relaxed/simple; bh=giWr7QZLp39SzXcwZ1dGleY3dSlDZX/Jpi5o0vExGlQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BwhvAWygV5EybvkE+GM6zwZLddLLHOthjYkVVq3vO5UzuEydcMGGurlWjQETqQv04Cu0VW6pqH8qkGbiOyZJMZ5IOcx22fCO6835DBHrTS+KT0CPNKujqlnD3JxnuTvz+atWamBdezRQyT9kfrviEmPJspCVEVKrIREclM3H60o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=ARONV0OQ; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="ARONV0OQ" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-220d28c215eso37387385ad.1 for ; Fri, 14 Feb 2025 16:10:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578205; x=1740183005; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W1w09fUZIhM8twL945gEZguDCFzB5iap37OBCII+bFw=; b=ARONV0OQ/iHUb4uvioFARLG8590ta4B7jd7gISo8bXKld4VTqOqXvck00qd4xAhOiF gUc43JExBEbNZUbsnXCrh3M+t84GTG3vka8MO2CNrkak7hIjbeTlJSULo7pkfrQNG4dq ooumaphDZePpPbR+v8qtWytMZz5WH6LqrrBdsr/7F2elWRekO1cIzY6HwpslERZ7y2RG UsH6GPA/kCxwdV2lo+LkixWSh1j93zDEEuyXwcIcy14qLG8IL9mLVmiW+qvhmR6Hw/um Nwgaq0igYxIluxnEhKg3JfBmfxNM3H6k2ucnvbHxIYLqKXT8UBh2a2UAX79IP7Lbq2di Tzlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578205; x=1740183005; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W1w09fUZIhM8twL945gEZguDCFzB5iap37OBCII+bFw=; b=qfjtJKWQKonKEjpssRh4Y/FnoXQdOigtK2G+z+UVsm8ZsXN0enym/bRNCav5qbw6uE f8E+WowHF6nnisdF/aHK0vBmqoI9j1V93YgIoGLKr4JZwN5uT2mW4oCp8oHc6PP8vKOS k4QnGvnzDMAoYj6euD+LGBEoiApY2scvDHg1lq4ek+1HOoVU+se8K4vhjvGA96yhINAb s1llLCs9q06KyVpqF1jGhUyJtdlfPEHxME51vn0ln03vTbcwomwlPigwacz2Lttu3Y0g dMkvPQ7dkPaYbbz9pgihHNuosPFNRD8D9DYhQeTXlWnSKqzeaF1CUKd/mY7l0rTq6ygn DcIw== X-Forwarded-Encrypted: i=1; AJvYcCVKeqg/oUOoMq8oUgyH0Nm8U5h5S8VLTUuiQ2RPIVXcX/5q0yf0Pm29jduNLQx4MI2Ae4S51PI=@vger.kernel.org X-Gm-Message-State: AOJu0Ywh6Jrpupjwx+D6ThkdR4Zyc4bDGCfOVduW0PYo2djWajLuqvIv kqwtoyqxk2Yj0hMX9EM9CWRMAJqkbK9GrEsjoClWe0Ydc4x2vWe2/5ypMojbqvk= X-Gm-Gg: ASbGnctfGgBAF/grZ1q7+FMJaE91hRgLs39pHMeNwdO/lqXs39pI6wk1yxKMkq49ruj HrY4eXFhmRWUdQI8ClADX1rvuL8tM5lTJJE8eAQm5tD14b1bfd4dxNy08Bl3TbYC7VNZcUe2SW+ 9MYbDlS8nASjFHQ1H7fBd3TTWetO1Xv3a5JHJ9Vq06O5rbMuGWlLjv8g76C4jeEO8Idh3C9Lqh3 nmjQuU+oEHWSOq/1Q5roqdJaFvowJHbOQ6lJAMFUzjMQ2FG10JY0IunSmreTxQ+CG66BrwbRVOh X-Google-Smtp-Source: AGHT+IEFOVk9QNBd+yOxmobpINJ/GuGuqTolBk2X3X0YiBskwBeihaEYlhV6o6r62ODmskPpuf1NDA== X-Received: by 2002:a05:6a21:7889:b0:1ee:6567:135c with SMTP id adf61e73a8af0-1ee8cacbf33mr2682877637.6.1739578205624; Fri, 14 Feb 2025 16:10:05 -0800 (PST) Received: from localhost ([2a03:2880:ff:16::]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7324244d5bbsm3690796b3a.0.2025.02.14.16.10.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:05 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 05/11] io_uring/zcrx: dma-map area for the device Date: Fri, 14 Feb 2025 16:09:40 -0800 Message-ID: <20250215000947.789731-6-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Pavel Begunkov Setup DMA mappings for the area into which we intend to receive data later on. We know the device we want to attach to even before we get a page pool and can pre-map in advance. All net_iov are synchronised for device when allocated, see page_pool_mp_return_in_cache(). Reviewed-by: Jens Axboe Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- io_uring/zcrx.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++- io_uring/zcrx.h | 1 + 2 files changed, 82 insertions(+), 1 deletion(-) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index f9e924cfa829..9d14fdf7a568 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include #include #include @@ -21,6 +22,73 @@ #include "zcrx.h" #include "rsrc.h" +#define IO_DMA_ATTR (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING) + +static void __io_zcrx_unmap_area(struct io_zcrx_ifq *ifq, + struct io_zcrx_area *area, int nr_mapped) +{ + int i; + + for (i = 0; i < nr_mapped; i++) { + struct net_iov *niov = &area->nia.niovs[i]; + dma_addr_t dma; + + dma = page_pool_get_dma_addr_netmem(net_iov_to_netmem(niov)); + dma_unmap_page_attrs(ifq->dev, dma, PAGE_SIZE, + DMA_FROM_DEVICE, IO_DMA_ATTR); + net_mp_niov_set_dma_addr(niov, 0); + } +} + +static void io_zcrx_unmap_area(struct io_zcrx_ifq *ifq, struct io_zcrx_area *area) +{ + if (area->is_mapped) + __io_zcrx_unmap_area(ifq, area, area->nia.num_niovs); +} + +static int io_zcrx_map_area(struct io_zcrx_ifq *ifq, struct io_zcrx_area *area) +{ + int i; + + for (i = 0; i < area->nia.num_niovs; i++) { + struct net_iov *niov = &area->nia.niovs[i]; + dma_addr_t dma; + + dma = dma_map_page_attrs(ifq->dev, area->pages[i], 0, PAGE_SIZE, + DMA_FROM_DEVICE, IO_DMA_ATTR); + if (dma_mapping_error(ifq->dev, dma)) + break; + if (net_mp_niov_set_dma_addr(niov, dma)) { + dma_unmap_page_attrs(ifq->dev, dma, PAGE_SIZE, + DMA_FROM_DEVICE, IO_DMA_ATTR); + break; + } + } + + if (i != area->nia.num_niovs) { + __io_zcrx_unmap_area(ifq, area, i); + return -EINVAL; + } + + area->is_mapped = true; + return 0; +} + +static void io_zcrx_sync_for_device(const struct page_pool *pool, + struct net_iov *niov) +{ +#if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) + dma_addr_t dma_addr; + + if (!dma_dev_need_sync(pool->p.dev)) + return; + + dma_addr = page_pool_get_dma_addr_netmem(net_iov_to_netmem(niov)); + __dma_sync_single_for_device(pool->p.dev, dma_addr + pool->p.offset, + PAGE_SIZE, pool->p.dma_dir); +#endif +} + #define IO_RQ_MAX_ENTRIES 32768 __maybe_unused @@ -83,6 +151,8 @@ static void io_free_rbuf_ring(struct io_zcrx_ifq *ifq) static void io_zcrx_free_area(struct io_zcrx_area *area) { + io_zcrx_unmap_area(area->ifq, area); + kvfree(area->freelist); kvfree(area->nia.niovs); kvfree(area->user_refs); @@ -272,6 +342,10 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, return -EOPNOTSUPP; get_device(ifq->dev); + ret = io_zcrx_map_area(ifq, ifq->area); + if (ret) + goto err; + reg.offsets.rqes = sizeof(struct io_uring); reg.offsets.head = offsetof(struct io_uring, head); reg.offsets.tail = offsetof(struct io_uring, tail); @@ -422,6 +496,7 @@ static void io_zcrx_ring_refill(struct page_pool *pp, continue; } + io_zcrx_sync_for_device(pp, niov); net_mp_netmem_place_in_cache(pp, netmem); } while (--entries); @@ -439,6 +514,7 @@ static void io_zcrx_refill_slow(struct page_pool *pp, struct io_zcrx_ifq *ifq) netmem_ref netmem = net_iov_to_netmem(niov); net_mp_niov_set_page_pool(pp, niov); + io_zcrx_sync_for_device(pp, niov); net_mp_netmem_place_in_cache(pp, netmem); } spin_unlock_bh(&area->freelist_lock); @@ -482,10 +558,14 @@ static int io_pp_zc_init(struct page_pool *pp) if (WARN_ON_ONCE(!ifq)) return -EINVAL; - if (pp->dma_map) + if (WARN_ON_ONCE(ifq->dev != pp->p.dev)) + return -EINVAL; + if (WARN_ON_ONCE(!pp->dma_map)) return -EOPNOTSUPP; if (pp->p.order != 0) return -EOPNOTSUPP; + if (pp->p.dma_dir != DMA_FROM_DEVICE) + return -EOPNOTSUPP; percpu_ref_get(&ifq->ctx->refs); return 0; diff --git a/io_uring/zcrx.h b/io_uring/zcrx.h index 6c808240ac91..1b6363591f72 100644 --- a/io_uring/zcrx.h +++ b/io_uring/zcrx.h @@ -11,6 +11,7 @@ struct io_zcrx_area { struct io_zcrx_ifq *ifq; atomic_t *user_refs; + bool is_mapped; u16 area_id; struct page **pages; From patchwork Sat Feb 15 00:09:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975782 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79AA11465AD for ; Sat, 15 Feb 2025 00:10:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578209; cv=none; b=tCtFuhJOavGZRucit7xLupT0pU68UXd+SPRScEKsM9o97cfz+47F1QY6hqbNe+yonu5LEGb1cvoZ+9FiZQ9VaxgNwOz4WptRVieSW07zdt843Et9913ZJmNbY/i7Q5BUjLceF1iF4bwiJ/QUrrzUzb1aEtlogrVqjqjZ0YzVdzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578209; c=relaxed/simple; bh=jbz0kLV8lQ5dY9o4FCwcNnHh6d7cOcCpHu65hnrbSbE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d3aH/dyGdJkqp6dRRG0fRLSeH/aodEGvgrx8GM4gpU+eRm/PHasYl5XTV3NyZkWglbxSY37R7XRGTMXR1KAk3++z5eAaAej8mkET28Bj/yEDCuZWcpoVVlAfyRITaqHbKFgasDLzs8CJUDCRmfhRFSPvbiekI0YxEpUsAK8zop8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=QgkI/4mv; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="QgkI/4mv" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-220c8eb195aso53225855ad.0 for ; Fri, 14 Feb 2025 16:10:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578207; x=1740183007; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oxoCVAFRuxvSLMPvN/NwgU5d2P2x91U8VeDCmuD/ImY=; b=QgkI/4mvTTrJwsWvZXo3YVASmLxNl1x6M1z7lJYUSwoVg2ISqX5WPRIiaHIphvkpgX LuJBg6E80vjyy/5WYZPRMM5rEj5mLEC1JIU4tJx54AaLpmn+Cd3coIn6MS6267EtejB9 wsXxQvgDIvtvnIkeIPPJFyTrKV/QM6bHis7esBviZKa+iQn0pnIHedl8sQuP3ECjpKZd UBq2FU29uTyb8kyPKVd9BpYT3cR8TUu/H/6ftHjd/8bw5fKRNmvHQhAVgzm6joIaqn0h qqiU4FM0O4FsYgLtJxw8NYnZjhWnKjlzFpAsIk4IE5SXb8cUhqsdWvf7AiLIlGVWMO0R sxQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578207; x=1740183007; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oxoCVAFRuxvSLMPvN/NwgU5d2P2x91U8VeDCmuD/ImY=; b=bQ0HbB26yO8pvI4Bcu4cfUEULvzDsfgHNt7yYv3Rterp2Mhxw6bJZb6/B6qIrDmIQw EGOYR8IGtA3HsaXLbLWqAPPRlnvjvbVaNf0V1UfZH8D8puYCDEQ7m7/Z2wPesTBx8lY1 zFCFcyDUuinP5jusgaTPTrGSI/nTGi9euhZrBsn+oywNqGLfCJdu6hDX6kLOR2N9KObz YZwtIYKPELTm9rXU5NYhKTTBwOE0LWapMQvaE6n0+ioBoWJYp9F4EaY4cd6SGoD9G3M9 1i3NggiNrSUUPyV8FKeUq/TYdM7FWne910Xjs4BvlBy3Yy95jrZgQbA6NFzDme7EqPMT ZPnQ== X-Forwarded-Encrypted: i=1; AJvYcCV1x0eSQoakQSXpJaNw/D8riXTag06xzseBmfuHkDDyE30D2JazCLvsE6/5iIhN0MneRZ36Hu4=@vger.kernel.org X-Gm-Message-State: AOJu0YwPWNvRH57BpD69mmaXMbNC1sCcLkRlJfJzCCBojdLRxoYuV2pc a3aaZf+ig+4fd4EN4zEJEVle85UZ8lK74rYOLHydvUp00WEyBt0oP+hTXMpcCgk= X-Gm-Gg: ASbGncv3iRBtMDDiKcmZObYAsKY/hAHw3JGC5l4zcFpKSDYeOPZJGFkh93Ao4BMfFmL Mrgo2ECaqueC3EQu1vFtB8drBzLmpkiIxnuWzpcpRXaW9T7hN4YH+ee/+gGpwJHh4s6PgM4C437 uqkCcLEDhuhZyDqYKEG9Q8KyzBHVy1jjou6WYxQ8lJc/Lb9kwrm+psJVeJeqFjrCHrpBi1azZdP 60OGGKyi81wVFX2vvzOGjZkCiUCo/R6mkgpMdgA7NLZWsdojuPfa5RKLRGRpYArEJcmaMrhV+kG X-Google-Smtp-Source: AGHT+IFfR/rmDAstmQoiZ7KDaAznriJXIr4TbiNH+OmcDqWd2iL3r6/ekU+9TP2QkTsEpKYMpBMWbg== X-Received: by 2002:a17:902:fc4e:b0:216:4169:f9d7 with SMTP id d9443c01a7336-22103efb5cbmr19016195ad.2.1739578206718; Fri, 14 Feb 2025 16:10:06 -0800 (PST) Received: from localhost ([2a03:2880:ff:16::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d5584dfbsm33878055ad.213.2025.02.14.16.10.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:06 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 06/11] io_uring/zcrx: add io_recvzc request Date: Fri, 14 Feb 2025 16:09:41 -0800 Message-ID: <20250215000947.789731-7-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add io_uring opcode OP_RECV_ZC for doing zero copy reads out of a socket. Only the connection should be land on the specific rx queue set up for zero copy, and the socket must be handled by the io_uring instance that the rx queue was registered for zero copy with. That's because neither net_iovs / buffers from our queue can be read by outside applications, nor zero copy is possible if traffic for the zero copy connection goes to another queue. This coordination is outside of the scope of this patch series. Also, any traffic directed to the zero copy enabled queue is immediately visible to the application, which is why CAP_NET_ADMIN is required at the registration step. Of course, no data is actually read out of the socket, it has already been copied by the netdev into userspace memory via DMA. OP_RECV_ZC reads skbs out of the socket and checks that its frags are indeed net_iovs that belong to io_uring. A cqe is queued for each one of these frags. Recall that each cqe is a big cqe, with the top half being an io_uring_zcrx_cqe. The cqe res field contains the len or error. The lower IORING_ZCRX_AREA_SHIFT bits of the struct io_uring_zcrx_cqe::off field contain the offset relative to the start of the zero copy area. The upper part of the off field is trivially zero, and will be used to carry the area id. For now, there is no limit as to how much work each OP_RECV_ZC request does. It will attempt to drain a socket of all available data. This request always operates in multishot mode. Reviewed-by: Jens Axboe Signed-off-by: David Wei --- include/uapi/linux/io_uring.h | 2 + io_uring/io_uring.h | 10 ++ io_uring/net.c | 72 +++++++++++++ io_uring/opdef.c | 16 +++ io_uring/zcrx.c | 190 +++++++++++++++++++++++++++++++++- io_uring/zcrx.h | 13 +++ 6 files changed, 302 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 44844707d327..05d6255b0f6a 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -87,6 +87,7 @@ struct io_uring_sqe { union { __s32 splice_fd_in; __u32 file_index; + __u32 zcrx_ifq_idx; __u32 optlen; struct { __u16 addr_len; @@ -278,6 +279,7 @@ enum io_uring_op { IORING_OP_FTRUNCATE, IORING_OP_BIND, IORING_OP_LISTEN, + IORING_OP_RECV_ZC, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 85bc8f76ca19..fd0dbe7b0c9a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -185,6 +185,16 @@ static inline bool io_get_cqe(struct io_ring_ctx *ctx, struct io_uring_cqe **ret return io_get_cqe_overflow(ctx, ret, false); } +static inline bool io_defer_get_uncommited_cqe(struct io_ring_ctx *ctx, + struct io_uring_cqe **cqe_ret) +{ + io_lockdep_assert_cq_locked(ctx); + + ctx->cq_extra++; + ctx->submit_state.cq_flush = true; + return io_get_cqe(ctx, cqe_ret); +} + static __always_inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, struct io_kiocb *req) { diff --git a/io_uring/net.c b/io_uring/net.c index 10344b3a6d89..260eb73a5854 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -16,6 +16,7 @@ #include "net.h" #include "notif.h" #include "rsrc.h" +#include "zcrx.h" #if defined(CONFIG_NET) struct io_shutdown { @@ -89,6 +90,13 @@ struct io_sr_msg { */ #define MULTISHOT_MAX_RETRY 32 +struct io_recvzc { + struct file *file; + unsigned msg_flags; + u16 flags; + struct io_zcrx_ifq *ifq; +}; + int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_shutdown *shutdown = io_kiocb_to_cmd(req, struct io_shutdown); @@ -1227,6 +1235,70 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) return ret; } +int io_recvzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_recvzc *zc = io_kiocb_to_cmd(req, struct io_recvzc); + unsigned ifq_idx; + + if (unlikely(sqe->file_index || sqe->addr2 || sqe->addr || + sqe->len || sqe->addr3)) + return -EINVAL; + + ifq_idx = READ_ONCE(sqe->zcrx_ifq_idx); + if (ifq_idx != 0) + return -EINVAL; + zc->ifq = req->ctx->ifq; + if (!zc->ifq) + return -EINVAL; + + zc->flags = READ_ONCE(sqe->ioprio); + zc->msg_flags = READ_ONCE(sqe->msg_flags); + if (zc->msg_flags) + return -EINVAL; + if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST | IORING_RECV_MULTISHOT)) + return -EINVAL; + /* multishot required */ + if (!(zc->flags & IORING_RECV_MULTISHOT)) + return -EINVAL; + /* All data completions are posted as aux CQEs. */ + req->flags |= REQ_F_APOLL_MULTISHOT; + + return 0; +} + +int io_recvzc(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_recvzc *zc = io_kiocb_to_cmd(req, struct io_recvzc); + struct socket *sock; + int ret; + + if (!(req->flags & REQ_F_POLLED) && + (zc->flags & IORING_RECVSEND_POLL_FIRST)) + return -EAGAIN; + + sock = sock_from_file(req->file); + if (unlikely(!sock)) + return -ENOTSOCK; + + ret = io_zcrx_recv(req, zc->ifq, sock, zc->msg_flags | MSG_DONTWAIT, + issue_flags); + if (unlikely(ret <= 0) && ret != -EAGAIN) { + if (ret == -ERESTARTSYS) + ret = -EINTR; + + req_set_fail(req); + io_req_set_res(req, ret, 0); + + if (issue_flags & IO_URING_F_MULTISHOT) + return IOU_STOP_MULTISHOT; + return IOU_OK; + } + + if (issue_flags & IO_URING_F_MULTISHOT) + return IOU_ISSUE_SKIP_COMPLETE; + return -EAGAIN; +} + void io_send_zc_cleanup(struct io_kiocb *req) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index e8baef4e5146..89f50ecadeaf 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -37,6 +37,7 @@ #include "waitid.h" #include "futex.h" #include "truncate.h" +#include "zcrx.h" static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags) { @@ -514,6 +515,18 @@ const struct io_issue_def io_issue_defs[] = { .async_size = sizeof(struct io_async_msghdr), #else .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_RECV_ZC] = { + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .ioprio = 1, +#if defined(CONFIG_NET) + .prep = io_recvzc_prep, + .issue = io_recvzc, +#else + .prep = io_eopnotsupp_prep, #endif }, }; @@ -745,6 +758,9 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_LISTEN] = { .name = "LISTEN", }, + [IORING_OP_RECV_ZC] = { + .name = "RECV_ZC", + }, }; const char *io_uring_get_opcode(u8 opcode) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index 9d14fdf7a568..8833879d94ba 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -13,6 +13,8 @@ #include #include +#include +#include #include @@ -91,7 +93,12 @@ static void io_zcrx_sync_for_device(const struct page_pool *pool, #define IO_RQ_MAX_ENTRIES 32768 -__maybe_unused +struct io_zcrx_args { + struct io_kiocb *req; + struct io_zcrx_ifq *ifq; + struct socket *sock; +}; + static const struct memory_provider_ops io_uring_pp_zc_ops; static inline struct io_zcrx_area *io_zcrx_iov_to_area(const struct net_iov *niov) @@ -118,6 +125,11 @@ static bool io_zcrx_put_niov_uref(struct net_iov *niov) return true; } +static void io_zcrx_get_niov_uref(struct net_iov *niov) +{ + atomic_inc(io_get_user_counter(niov)); +} + static int io_allocate_rbuf_ring(struct io_zcrx_ifq *ifq, struct io_uring_zcrx_ifq_reg *reg, struct io_uring_region_desc *rd) @@ -614,3 +626,179 @@ static const struct memory_provider_ops io_uring_pp_zc_ops = { .nl_fill = io_pp_nl_fill, .uninstall = io_pp_uninstall, }; + +static bool io_zcrx_queue_cqe(struct io_kiocb *req, struct net_iov *niov, + struct io_zcrx_ifq *ifq, int off, int len) +{ + struct io_uring_zcrx_cqe *rcqe; + struct io_zcrx_area *area; + struct io_uring_cqe *cqe; + u64 offset; + + if (!io_defer_get_uncommited_cqe(req->ctx, &cqe)) + return false; + + cqe->user_data = req->cqe.user_data; + cqe->res = len; + cqe->flags = IORING_CQE_F_MORE; + + area = io_zcrx_iov_to_area(niov); + offset = off + (net_iov_idx(niov) << PAGE_SHIFT); + rcqe = (struct io_uring_zcrx_cqe *)(cqe + 1); + rcqe->off = offset + ((u64)area->area_id << IORING_ZCRX_AREA_SHIFT); + rcqe->__pad = 0; + return true; +} + +static int io_zcrx_recv_frag(struct io_kiocb *req, struct io_zcrx_ifq *ifq, + const skb_frag_t *frag, int off, int len) +{ + struct net_iov *niov; + + if (unlikely(!skb_frag_is_net_iov(frag))) + return -EOPNOTSUPP; + + niov = netmem_to_net_iov(frag->netmem); + if (niov->pp->mp_ops != &io_uring_pp_zc_ops || + niov->pp->mp_priv != ifq) + return -EFAULT; + + if (!io_zcrx_queue_cqe(req, niov, ifq, off + skb_frag_off(frag), len)) + return -ENOSPC; + + /* + * Prevent it from being recycled while user is accessing it. + * It has to be done before grabbing a user reference. + */ + page_pool_ref_netmem(net_iov_to_netmem(niov)); + io_zcrx_get_niov_uref(niov); + return len; +} + +static int +io_zcrx_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, + unsigned int offset, size_t len) +{ + struct io_zcrx_args *args = desc->arg.data; + struct io_zcrx_ifq *ifq = args->ifq; + struct io_kiocb *req = args->req; + struct sk_buff *frag_iter; + unsigned start, start_off; + int i, copy, end, off; + int ret = 0; + + start = skb_headlen(skb); + start_off = offset; + + if (offset < start) + return -EOPNOTSUPP; + + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { + const skb_frag_t *frag; + + if (WARN_ON(start > offset + len)) + return -EFAULT; + + frag = &skb_shinfo(skb)->frags[i]; + end = start + skb_frag_size(frag); + + if (offset < end) { + copy = end - offset; + if (copy > len) + copy = len; + + off = offset - start; + ret = io_zcrx_recv_frag(req, ifq, frag, off, copy); + if (ret < 0) + goto out; + + offset += ret; + len -= ret; + if (len == 0 || ret != copy) + goto out; + } + start = end; + } + + skb_walk_frags(skb, frag_iter) { + if (WARN_ON(start > offset + len)) + return -EFAULT; + + end = start + frag_iter->len; + if (offset < end) { + copy = end - offset; + if (copy > len) + copy = len; + + off = offset - start; + ret = io_zcrx_recv_skb(desc, frag_iter, off, copy); + if (ret < 0) + goto out; + + offset += ret; + len -= ret; + if (len == 0 || ret != copy) + goto out; + } + start = end; + } + +out: + if (offset == start_off) + return ret; + return offset - start_off; +} + +static int io_zcrx_tcp_recvmsg(struct io_kiocb *req, struct io_zcrx_ifq *ifq, + struct sock *sk, int flags, + unsigned issue_flags) +{ + struct io_zcrx_args args = { + .req = req, + .ifq = ifq, + .sock = sk->sk_socket, + }; + read_descriptor_t rd_desc = { + .count = 1, + .arg.data = &args, + }; + int ret; + + lock_sock(sk); + ret = tcp_read_sock(sk, &rd_desc, io_zcrx_recv_skb); + if (ret <= 0) { + if (ret < 0 || sock_flag(sk, SOCK_DONE)) + goto out; + if (sk->sk_err) + ret = sock_error(sk); + else if (sk->sk_shutdown & RCV_SHUTDOWN) + goto out; + else if (sk->sk_state == TCP_CLOSE) + ret = -ENOTCONN; + else + ret = -EAGAIN; + } else if (sock_flag(sk, SOCK_DONE)) { + /* Make it to retry until it finally gets 0. */ + if (issue_flags & IO_URING_F_MULTISHOT) + ret = IOU_REQUEUE; + else + ret = -EAGAIN; + } +out: + release_sock(sk); + return ret; +} + +int io_zcrx_recv(struct io_kiocb *req, struct io_zcrx_ifq *ifq, + struct socket *sock, unsigned int flags, + unsigned issue_flags) +{ + struct sock *sk = sock->sk; + const struct proto *prot = READ_ONCE(sk->sk_prot); + + if (prot->recvmsg != tcp_recvmsg) + return -EPROTONOSUPPORT; + + sock_rps_record_flow(sk); + return io_zcrx_tcp_recvmsg(req, ifq, sk, flags, issue_flags); +} diff --git a/io_uring/zcrx.h b/io_uring/zcrx.h index 1b6363591f72..a16bdd921f03 100644 --- a/io_uring/zcrx.h +++ b/io_uring/zcrx.h @@ -3,6 +3,7 @@ #define IOU_ZC_RX_H #include +#include #include #include @@ -43,6 +44,9 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, struct io_uring_zcrx_ifq_reg __user *arg); void io_unregister_zcrx_ifqs(struct io_ring_ctx *ctx); void io_shutdown_zcrx_ifqs(struct io_ring_ctx *ctx); +int io_zcrx_recv(struct io_kiocb *req, struct io_zcrx_ifq *ifq, + struct socket *sock, unsigned int flags, + unsigned issue_flags); #else static inline int io_register_zcrx_ifq(struct io_ring_ctx *ctx, struct io_uring_zcrx_ifq_reg __user *arg) @@ -55,6 +59,15 @@ static inline void io_unregister_zcrx_ifqs(struct io_ring_ctx *ctx) static inline void io_shutdown_zcrx_ifqs(struct io_ring_ctx *ctx) { } +static inline int io_zcrx_recv(struct io_kiocb *req, struct io_zcrx_ifq *ifq, + struct socket *sock, unsigned int flags, + unsigned issue_flags) +{ + return -EOPNOTSUPP; +} #endif +int io_recvzc(struct io_kiocb *req, unsigned int issue_flags); +int io_recvzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); + #endif From patchwork Sat Feb 15 00:09:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975783 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9687151985 for ; Sat, 15 Feb 2025 00:10:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578210; cv=none; b=Lju9wttFPaQ3ivBGjyaL/bR1abboYO7B9kVLzCQ1ncoyaIcxHPnZPccWtjyF/JeW/zxzln0WbWS18Vv185/bE6Exvf8d0OojhEROaoxOS7pNTSctoz5sgPChhJxnUEyqexH5d923nZ2ZMbgrY34UqzSc+n4nq+8o4OZxFd117ws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578210; c=relaxed/simple; bh=yv828TZcRCYjgOUYA40GgMIAhXfiazwYmHkQmnj1NWA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j71HZNBKEBQXQeuxqznRA0DpDGvDOrQvwNfkGDlMCJ1qkqKZ9yLT+LonFyKIcU8iB+VeQHnyx9HoFFZPm0nQGgM6bXbqHQlV9Bk54cS4vyghCSMqEPxj8Vx2QW8LsIFHjnp2q2lStRHyTsl6SNKqaHDx2U8bNPWKUr/5z/CeAhg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=fYI9mQWG; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="fYI9mQWG" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21f78b1fb7dso45801585ad.3 for ; Fri, 14 Feb 2025 16:10:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578208; x=1740183008; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cUvMDJY3LulrovKGYmLf0HXuAW/efdEv7GgYFLavCrU=; b=fYI9mQWGJiHSWJmS13WxAtZAm+uWMW6Bt3Dk53fj7nCmPzbRffb6XMe0NO0wxPjLNd sKT5HEVGskZLFcR6IwZ6BiA7kURyi5FFiH1h85rnHOO1u9EAj7pkurm1FZTTCN15HS7v qrnAlwbNV5KeWLj9JUbk6FLmsdLXqk3Wdt7rFU+kAlddRBuynSs9xjhe4npm+uq3+Jlo xqfvaKYH05ZbIKCgxqRQ2JXgknU4SBCCSzQhE68Y3FjyXq282K6Pi5C+BhOdrt8HnzA8 FuxL7O11wALHkmcYoH5d1eAjIquUQ1yhEcaAk8ZtGeyo/PL8TmtTbIbjdKoi049a7mMf kt9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578208; x=1740183008; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cUvMDJY3LulrovKGYmLf0HXuAW/efdEv7GgYFLavCrU=; b=gu8oAu3fNW807D0c6L9bDBH9IsnfYPyi9KpzCzNFMA6UZ44WXJ0QDqDCh/dilp5w70 zcuWaIvR/PPoj2sphAXmUfVf6fDVP95gfAeX4ZLmwlQfeuANhx+axJA6LpFnqcA32fTt coKTxNQd37GxC3ACXGYXbrmqC9xiF7NQarIqxKm8iVMTyc5L63qqjRMTChVyMr5PV04b 1tWyogCHuPmuItx7PhpkyWHxiEpVYdzpc6mY5YiphHJkhQPMJ+pHjU1edTescBbDCs45 BtsQ5k7AfTfmsoFjZjk3KDEZzLEbrGv7eV2Li56ANImi36bq9mKAzI0Tcy9gzfw+K/Tq N0Xg== X-Forwarded-Encrypted: i=1; AJvYcCX3NCAIjpZ04dJR+xeMts7E3Lz4rgv/Rr/y7g8lal2XFGGHU3gg9zVqM9DmoD9+ASY4yVl/NPs=@vger.kernel.org X-Gm-Message-State: AOJu0Yy0ZbABvAz/K5QTF7y+WlwAWM38bniKwLY0nBBZuj21FPDvOjFt uzaKUbdhyEBlb8yVu9ZEk1DnKngDGPtCZoFiX+SOtPL2TualpkCbPzU+SY3MLB19CyIkMyKXjGl / X-Gm-Gg: ASbGncskfiHeQ93u/q8aTB8sdhlVI14U39HoOjQUx6vm/UnpqGzjpM8Uua7VmIx5BSS 892V1RLmgqLJGk1OFCg9kaXobpHayBowQ7Is39x9EmyO5Is5xFX3LDjrMYG6Rhj8valL3ElVGEq 0NyHIm+TtUuP1FOonSqWfQMACzNHZBKR6u/zoGdXDvPhwdQWohByo6/sQdbym+oUCSG8/Q5LILF VQTof8EVfarqm2qivISwcYwGIJGMRSx1tR953YtyqOtOGYs7xloDEr8xWVeALLwLK/VJKvnpQg= X-Google-Smtp-Source: AGHT+IFCyZhH/nkjfrONygHuHLVOUuRCt5s+VsXaryqRUIqav3y3If8MKHA+uOTyAwE5xWM7Sc7Wgw== X-Received: by 2002:a17:902:e80e:b0:220:e362:9b1a with SMTP id d9443c01a7336-221040623acmr18942415ad.25.1739578207917; Fri, 14 Feb 2025 16:10:07 -0800 (PST) Received: from localhost ([2a03:2880:ff:c::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d07dsm33772755ad.138.2025.02.14.16.10.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:07 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 07/11] io_uring/zcrx: set pp memory provider for an rx queue Date: Fri, 14 Feb 2025 16:09:42 -0800 Message-ID: <20250215000947.789731-8-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Set the page pool memory provider for the rx queue configured for zero copy to io_uring. Then the rx queue is reset using netdev_rx_queue_restart() and netdev core + page pool will take care of filling the rx queue from the io_uring zero copy memory provider. For now, there is only one ifq so its destruction happens implicitly during io_uring cleanup. Reviewed-by: Jens Axboe Signed-off-by: David Wei --- io_uring/zcrx.c | 49 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 8 deletions(-) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index 8833879d94ba..7d24fc98b306 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -11,11 +11,12 @@ #include #include #include - -#include +#include #include #include +#include + #include #include "io_uring.h" @@ -275,8 +276,34 @@ static void io_zcrx_drop_netdev(struct io_zcrx_ifq *ifq) spin_unlock(&ifq->lock); } +static void io_close_queue(struct io_zcrx_ifq *ifq) +{ + struct net_device *netdev; + netdevice_tracker netdev_tracker; + struct pp_memory_provider_params p = { + .mp_ops = &io_uring_pp_zc_ops, + .mp_priv = ifq, + }; + + if (ifq->if_rxq == -1) + return; + + spin_lock(&ifq->lock); + netdev = ifq->netdev; + netdev_tracker = ifq->netdev_tracker; + ifq->netdev = NULL; + spin_unlock(&ifq->lock); + + if (netdev) { + net_mp_close_rxq(netdev, ifq->if_rxq, &p); + netdev_put(netdev, &netdev_tracker); + } + ifq->if_rxq = -1; +} + static void io_zcrx_ifq_free(struct io_zcrx_ifq *ifq) { + io_close_queue(ifq); io_zcrx_drop_netdev(ifq); if (ifq->area) @@ -291,6 +318,7 @@ static void io_zcrx_ifq_free(struct io_zcrx_ifq *ifq) int io_register_zcrx_ifq(struct io_ring_ctx *ctx, struct io_uring_zcrx_ifq_reg __user *arg) { + struct pp_memory_provider_params mp_param = {}; struct io_uring_zcrx_area_reg area; struct io_uring_zcrx_ifq_reg reg; struct io_uring_region_desc rd; @@ -341,7 +369,6 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, goto err; ifq->rq_entries = reg.rq_entries; - ifq->if_rxq = reg.if_rxq; ret = -ENODEV; ifq->netdev = netdev_get_by_index(current->nsproxy->net_ns, reg.if_idx, @@ -358,16 +385,20 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, if (ret) goto err; + mp_param.mp_ops = &io_uring_pp_zc_ops; + mp_param.mp_priv = ifq; + ret = net_mp_open_rxq(ifq->netdev, reg.if_rxq, &mp_param); + if (ret) + goto err; + ifq->if_rxq = reg.if_rxq; + reg.offsets.rqes = sizeof(struct io_uring); reg.offsets.head = offsetof(struct io_uring, head); reg.offsets.tail = offsetof(struct io_uring, tail); if (copy_to_user(arg, ®, sizeof(reg)) || - copy_to_user(u64_to_user_ptr(reg.region_ptr), &rd, sizeof(rd))) { - ret = -EFAULT; - goto err; - } - if (copy_to_user(u64_to_user_ptr(reg.area_ptr), &area, sizeof(area))) { + copy_to_user(u64_to_user_ptr(reg.region_ptr), &rd, sizeof(rd)) || + copy_to_user(u64_to_user_ptr(reg.area_ptr), &area, sizeof(area))) { ret = -EFAULT; goto err; } @@ -444,6 +475,8 @@ void io_shutdown_zcrx_ifqs(struct io_ring_ctx *ctx) if (ctx->ifq) io_zcrx_scrub(ctx->ifq); + + io_close_queue(ctx->ifq); } static inline u32 io_zcrx_rqring_entries(struct io_zcrx_ifq *ifq) From patchwork Sat Feb 15 00:09:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975784 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00BAA152E12 for ; Sat, 15 Feb 2025 00:10:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578212; cv=none; b=hwdLUY5t6UU+xC9dZTqmbPr9vkEPb9IVBwb34lCKvu8nD4sKphksl7dPw75+f4yOJGxDjvbR88srrgh8ieEBZpv/+dg6WCkqHnJLQ9kiH2Bz8Z7iD6mGVYfG0dJw/NrYyoXBuUkqNmgvRlDwhXx59S9tXX9SQ/V9jbMFqRkG8EY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578212; c=relaxed/simple; bh=0MhYOdt8cjrVmKwEWCLWnrpFJTfV5rsb6X7z7fMlHBk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DO6D5nHy9daBI9yEqL7ORrKoGDVMCUVl95kuHC70cuPWMNg9VJrffZ5lGfBBXVjnT0/T8rHD4/i1Y9ObLwOkC5oZ+m5DFhikkxpgZupVLL/2M95BCkOs5DSMrHSgC+OndHydi7q0IXm8CutKxIDb+x7jshCdnqTneiQyi2LDMZk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=ZC7ByNfX; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="ZC7ByNfX" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2fc292b3570so2587538a91.1 for ; Fri, 14 Feb 2025 16:10:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578209; x=1740183009; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VsiC6df/8ep3kbOMlvv6NHk6oNEYLUzPa8XU1Wwcwfk=; b=ZC7ByNfXE3l7MFiPD1cXGbUoNNyT3v1Cwbj5IZEtVEFSoHXC+d3NAskgvmJDiJlOt7 kKuViUW87q+NiMIL849Bf2AF+SOnGE5A8qJIpneiTNj/3I6z+dsJTC6IHgDbjKZCAIeR OgI0CabA6DRf+e+3O6TAlgC0BCp5LQjnD1RuslssaX2BWWYyT4R6Bq0y4OWRvJTai5RU 7WYz++e5EGaF9E58XLmVNefU65a9AeU6KdFNwi6VntqqsrKfLHVORgp8NJDgaeILK9gG 3D3CKLdN1zFqfU/m4YQBDyJ6exxG87V72hWYdr+bLdrrLALHPpCZN9gm9H8fpjfOcAQU 5veA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578209; x=1740183009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VsiC6df/8ep3kbOMlvv6NHk6oNEYLUzPa8XU1Wwcwfk=; b=RESq6oix4qw87CF2ISsMwP/Edsw4PfC5ilfKS79z9dgn5jsep5teKIqUlt/jZLBIvC ejXzB9tyu40oAlJjflK3DxUamGzHiGLIIUklp6u3T+uNfLZWSte/oUFBV8eMROFCDHSa Fn5/my1zYws4kU6UatwsTl5YSh4EZAK72JWvyFD6x0dVc5gglzOUSVYDBw1yb8hRdO2u 0R42w8lj7J7Py8R60+nSXWj9PpMtnQ1Miz+DoNH23SpKVwojer/tM9cbjLJBaLh39nJg cQbuSNsE6fo1EjEaLWV79yP8hRjXe+1V+nFnB6Pjf0yeFPqmU3EPP1zNKxhG9RdAT7zb fr7g== X-Forwarded-Encrypted: i=1; AJvYcCUMXpRHBabj3xrfe/9auzhYShism7WUi7ewV0A888YyWCiPIbdc3lGR3Pm6aEqVa7VrSP+dSGs=@vger.kernel.org X-Gm-Message-State: AOJu0YzgoCADSoaWNogWWoOkjIOAsIGinzhHYrWYBOWH4ikYKxZlxX5Q jZspd3kQdpC2M3kMuukpILiZAe2/V1Xof2h65TSf3CGiT9hYM8RFA/0REuPN7VU= X-Gm-Gg: ASbGnctTw3tHJEGeO38sTTyYl5LNHxnQITFULN+ThqUUDh75HRWpUPlOCtIZgDivfrk HNc1eiOAfYo5lyZa+OnyEeMD/3wz6NlUkL2uQIulWbv3gyjae5RNIYBG6RR3hbp/6vlVSJsPcSx u6DEfQMXiJrUDASTw1vPD5V+2Oa76re8lvTmDQMVaqE21jvU3jPv1xE98fSioG6xLIHF3ernEW4 i7mvYBUu2RiOkmHq14ortyDAW6KqPT8Vj+OyhGzDCjUMrtfwEJ6sRg2nYKEGJXa76/OHS9We7o= X-Google-Smtp-Source: AGHT+IHugWOSLZs+TvP13HCCC7S3wQ76eEDjldwt9oMOrHcveDSC9AhwnXDIS5a7f4CumTvw/f2SSg== X-Received: by 2002:a17:90a:d44c:b0:2ee:bf84:4fe8 with SMTP id 98e67ed59e1d1-2fc41153f71mr1530599a91.30.1739578209221; Fri, 14 Feb 2025 16:10:09 -0800 (PST) Received: from localhost ([2a03:2880:ff:7::]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2fc13aafebesm3690961a91.4.2025.02.14.16.10.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:08 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 08/11] io_uring/zcrx: throttle receive requests Date: Fri, 14 Feb 2025 16:09:43 -0800 Message-ID: <20250215000947.789731-9-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Pavel Begunkov io_zc_rx_tcp_recvmsg() continues until it fails or there is nothing to receive. If the other side sends fast enough, we might get stuck in io_zc_rx_tcp_recvmsg() producing more and more CQEs but not letting the user to handle them leading to unbound latencies. Break out of it based on an arbitrarily chosen limit, the upper layer will either return to userspace or requeue the request. Reviewed-by: Jens Axboe Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- io_uring/net.c | 2 ++ io_uring/zcrx.c | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/io_uring/net.c b/io_uring/net.c index 260eb73a5854..000dc70d08d0 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1285,6 +1285,8 @@ int io_recvzc(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret <= 0) && ret != -EAGAIN) { if (ret == -ERESTARTSYS) ret = -EINTR; + if (ret == IOU_REQUEUE) + return IOU_REQUEUE; req_set_fail(req); io_req_set_res(req, ret, 0); diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index 7d24fc98b306..7e0cba1e0f39 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -94,10 +94,13 @@ static void io_zcrx_sync_for_device(const struct page_pool *pool, #define IO_RQ_MAX_ENTRIES 32768 +#define IO_SKBS_PER_CALL_LIMIT 20 + struct io_zcrx_args { struct io_kiocb *req; struct io_zcrx_ifq *ifq; struct socket *sock; + unsigned nr_skbs; }; static const struct memory_provider_ops io_uring_pp_zc_ops; @@ -720,6 +723,9 @@ io_zcrx_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, int i, copy, end, off; int ret = 0; + if (unlikely(args->nr_skbs++ > IO_SKBS_PER_CALL_LIMIT)) + return -EAGAIN; + start = skb_headlen(skb); start_off = offset; @@ -810,6 +816,9 @@ static int io_zcrx_tcp_recvmsg(struct io_kiocb *req, struct io_zcrx_ifq *ifq, ret = -ENOTCONN; else ret = -EAGAIN; + } else if (unlikely(args.nr_skbs > IO_SKBS_PER_CALL_LIMIT) && + (issue_flags & IO_URING_F_MULTISHOT)) { + ret = IOU_REQUEUE; } else if (sock_flag(sk, SOCK_DONE)) { /* Make it to retry until it finally gets 0. */ if (issue_flags & IO_URING_F_MULTISHOT) From patchwork Sat Feb 15 00:09:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975785 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C8601624CB for ; Sat, 15 Feb 2025 00:10:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578212; cv=none; b=MzlvMvHyxVCNV/sf4PbsJTS2WCrmA0Kk9ce7wOCoSuTZHcDcpP1xQsDJccS/EJZMZJbrbpbciA7R3ySM6yTHdtJc4niu5tMvntyMgFxfZuTe24pJja2k3G8hVQaDUxkRw85C/uyIndyudNUySru336Vsiiz9Kr5wrbL6RpOi6jg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578212; c=relaxed/simple; bh=waZElP+hnGci0XgG5ol4AhH/YpGYtuPpjvD6W8bnmQ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p7Jq03a4uC9hCmCZETuRauHxHb2uFEULuOsKaXkIYi/yecT5/xXRM1mxWC1WagB3GbJKDbnv8CDtBvKEZ9UbfXXtzROttFKjyg8zECYlpbtm+wVOTyaM2d6BKa94ccHcm/vYo1j7anuoOP0SPODqvAgd+kxIqiXJ6KaaHs5KSNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=XV08BNqg; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="XV08BNqg" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2fbfe16cc39so4882097a91.3 for ; Fri, 14 Feb 2025 16:10:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578210; x=1740183010; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FmNIyCXXoWBh40mpSPtcuAkEk/iQTmfVKkUBNv+QvTM=; b=XV08BNqgAmqqQQTAO2EC9tMpEjcuQVi1emyTROZeUuxdKtdJ+n+tPM+MRI8KvkYjER IHMj33JNAItNHoIRPw7ZCyVDyr+lXEzAV2L8NV/p08JR0LjmI2UwkRWHUiB5OeOmcEkL TAjLeoYwsQNRO87obRphraHw9XKxMOzGwIEhmK67S2EcvYymoOpHs9qad49qcFZKEbFt c1ryKhSeFsb9IPWzeO3CbaFOGk/ple5CsZuEcsCQDfjBK3eLOQ5B0cLbz+Xwhbnd/6eZ VEZoK5NxlhTSSPtFHsGEB+ThptdSSe32kz3m3tXzQMyJ4qc2DpzGDy9IAvOrUFEfYnQ0 L8gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578210; x=1740183010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FmNIyCXXoWBh40mpSPtcuAkEk/iQTmfVKkUBNv+QvTM=; b=nAINh4vooXGXN4pyweQ6Q+4+V1NUWWBM7/WVeW9DiSLsupp/OYowxvNTsnqh5laOC4 Wzjp+a33ia/CtstuajxhGnQqjyYXrqpY8EBg0pKuur9c7cJcDUMAGMR2QpTUzPONx0o6 CqxysnsXaH1ALNQmY2h1Yl42oAxyAKkWA1Ob585+G4tRrszOCg0DwrV9+sW02NfRg798 R6chaXZ0aGa82Z0dY1CK0RO9cEhL0UgQgiso++vzgKq+t3dSm7qYzTvTfNSLBnSV3iWI JUQgCdeAHXS53zPpcf8ShsGAnSZ4swtlsHHg+1Ull66PrroAqkmUI8kBs9b+CJvoUMyj lK8A== X-Forwarded-Encrypted: i=1; AJvYcCXiBp65RHQBfVO6L1WwtVDCvY7B80+1m8jnhDxrPaVnzV1UZO6n1V8u9Nm0uhGGGVFh+w4gWH4=@vger.kernel.org X-Gm-Message-State: AOJu0YxuNyMIpCsN0gFP7F0HI+5FJKwsahz3GzAplBGzucNi5KmlVK4I HNuEL+PC10pXQpc83cJmyVDY7QGcRkMsgVpjuDumyGIAsFGMKzxsBsLNeDGS3RZopFFLpmINClG Z X-Gm-Gg: ASbGncsprDQkfNoywrI1/nJEgkqFlcetyeH5G7Oki19T3pQZ9oI212y7dJvkDKEvq21 rDgShE5rt9gyl0E1D1LEN+vORSlPM3ur7vlW+QP3/bPjHechWtZtWUUhDI4JHihSDIk17kiVSED ISQhq4mpLjwQ3YAT/D9O10fMdY05vK+JAF+O3BEyvqqAcx1bX48B1u/tVx/f3T1e5SZGhVLQL2j LRrPtzOMNUa9ejDNR7NcIpO5v7JjeqA4bwYMGUf75zDKl/uB0XjjXXx+QOefNkKWS8uQqetwYA= X-Google-Smtp-Source: AGHT+IEinmI6YzYKXk3Kc5KlfdAL6uHZcNBPKDAy0rcbD2dGEYI7Nt2Eoa6T6eOa2HTjIdHra2zZ2Q== X-Received: by 2002:a05:6a00:1391:b0:72d:9cbc:730d with SMTP id d2e1a72fcca58-732617c4b97mr2151318b3a.11.1739578210384; Fri, 14 Feb 2025 16:10:10 -0800 (PST) Received: from localhost ([2a03:2880:ff:d::]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-732621c4762sm342097b3a.172.2025.02.14.16.10.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:09 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 09/11] io_uring/zcrx: add copy fallback Date: Fri, 14 Feb 2025 16:09:44 -0800 Message-ID: <20250215000947.789731-10-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Pavel Begunkov There are scenarios in which the zerocopy path can get a kernel buffer instead of a net_iov and needs to copy it to the user, whether it is because of mis-steering or simply getting an skb with the linear part. In this case, grab a net_iov, copy into it and return it to the user as normally. At the moment the user doesn't get any indication whether there was a copy or not, which is left for follow up work. Reviewed-by: Jens Axboe Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- io_uring/zcrx.c | 120 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 114 insertions(+), 6 deletions(-) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index 7e0cba1e0f39..026efb8dd381 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -134,6 +135,13 @@ static void io_zcrx_get_niov_uref(struct net_iov *niov) atomic_inc(io_get_user_counter(niov)); } +static inline struct page *io_zcrx_iov_page(const struct net_iov *niov) +{ + struct io_zcrx_area *area = io_zcrx_iov_to_area(niov); + + return area->pages[net_iov_idx(niov)]; +} + static int io_allocate_rbuf_ring(struct io_zcrx_ifq *ifq, struct io_uring_zcrx_ifq_reg *reg, struct io_uring_region_desc *rd) @@ -448,6 +456,11 @@ static void io_zcrx_return_niov(struct net_iov *niov) { netmem_ref netmem = net_iov_to_netmem(niov); + if (!niov->pp) { + /* copy fallback allocated niovs */ + io_zcrx_return_niov_freelist(niov); + return; + } page_pool_put_unrefed_netmem(niov->pp, netmem, -1, false); } @@ -686,13 +699,93 @@ static bool io_zcrx_queue_cqe(struct io_kiocb *req, struct net_iov *niov, return true; } +static struct net_iov *io_zcrx_alloc_fallback(struct io_zcrx_area *area) +{ + struct net_iov *niov = NULL; + + spin_lock_bh(&area->freelist_lock); + if (area->free_count) + niov = __io_zcrx_get_free_niov(area); + spin_unlock_bh(&area->freelist_lock); + + if (niov) + page_pool_fragment_netmem(net_iov_to_netmem(niov), 1); + return niov; +} + +static ssize_t io_zcrx_copy_chunk(struct io_kiocb *req, struct io_zcrx_ifq *ifq, + void *src_base, struct page *src_page, + unsigned int src_offset, size_t len) +{ + struct io_zcrx_area *area = ifq->area; + size_t copied = 0; + int ret = 0; + + while (len) { + size_t copy_size = min_t(size_t, PAGE_SIZE, len); + const int dst_off = 0; + struct net_iov *niov; + struct page *dst_page; + void *dst_addr; + + niov = io_zcrx_alloc_fallback(area); + if (!niov) { + ret = -ENOMEM; + break; + } + + dst_page = io_zcrx_iov_page(niov); + dst_addr = kmap_local_page(dst_page); + if (src_page) + src_base = kmap_local_page(src_page); + + memcpy(dst_addr, src_base + src_offset, copy_size); + + if (src_page) + kunmap_local(src_base); + kunmap_local(dst_addr); + + if (!io_zcrx_queue_cqe(req, niov, ifq, dst_off, copy_size)) { + io_zcrx_return_niov(niov); + ret = -ENOSPC; + break; + } + + io_zcrx_get_niov_uref(niov); + src_offset += copy_size; + len -= copy_size; + copied += copy_size; + } + + return copied ? copied : ret; +} + +static int io_zcrx_copy_frag(struct io_kiocb *req, struct io_zcrx_ifq *ifq, + const skb_frag_t *frag, int off, int len) +{ + struct page *page = skb_frag_page(frag); + u32 p_off, p_len, t, copied = 0; + int ret = 0; + + off += skb_frag_off(frag); + + skb_frag_foreach_page(frag, off, len, + page, p_off, p_len, t) { + ret = io_zcrx_copy_chunk(req, ifq, NULL, page, p_off, p_len); + if (ret < 0) + return copied ? copied : ret; + copied += ret; + } + return copied; +} + static int io_zcrx_recv_frag(struct io_kiocb *req, struct io_zcrx_ifq *ifq, const skb_frag_t *frag, int off, int len) { struct net_iov *niov; if (unlikely(!skb_frag_is_net_iov(frag))) - return -EOPNOTSUPP; + return io_zcrx_copy_frag(req, ifq, frag, off, len); niov = netmem_to_net_iov(frag->netmem); if (niov->pp->mp_ops != &io_uring_pp_zc_ops || @@ -719,18 +812,33 @@ io_zcrx_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, struct io_zcrx_ifq *ifq = args->ifq; struct io_kiocb *req = args->req; struct sk_buff *frag_iter; - unsigned start, start_off; + unsigned start, start_off = offset; int i, copy, end, off; int ret = 0; if (unlikely(args->nr_skbs++ > IO_SKBS_PER_CALL_LIMIT)) return -EAGAIN; - start = skb_headlen(skb); - start_off = offset; + if (unlikely(offset < skb_headlen(skb))) { + ssize_t copied; + size_t to_copy; - if (offset < start) - return -EOPNOTSUPP; + to_copy = min_t(size_t, skb_headlen(skb) - offset, len); + copied = io_zcrx_copy_chunk(req, ifq, skb->data, NULL, + offset, to_copy); + if (copied < 0) { + ret = copied; + goto out; + } + offset += copied; + len -= copied; + if (!len) + goto out; + if (offset != skb_headlen(skb)) + goto out; + } + + start = skb_headlen(skb); for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { const skb_frag_t *frag; From patchwork Sat Feb 15 00:09:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975786 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51E3F2943F for ; Sat, 15 Feb 2025 00:10:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578213; cv=none; b=h9o2Kz4pXFsVRKAI2KNdP8fxwyVoDC9TZDvOrnNBM/qp92GViZzE7qFGxBNpwUJwadyVmIZ/PFBjh0HV8W4EqSbyZtwwNgjMurmto1S4UytaqvdP6BPYOs5VKMb8wZkvnOuS+SsfFuOUTQTVeTmtbPzBjzLZHpuqTcbvDvOp9ds= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578213; c=relaxed/simple; bh=9pTdmByBVpWL3oRo72V1P/Qnfm3p1w1EAQ8TvZwzW4E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CsPyNLXBBWAsiBwSG0vFjvMpj+4CRbkKFV7PQSmS5YBY25XRy/prY01Yx4n7CiNfHhW+Wo5aNZwreloWosdtJuQuJeI353PP7jYyF0wpg0EJqu/ch0Bum6bqxt5KqKmSVKQ/OaRg5HkHq7DVJqQRj5GJlFatIgkpJYNwDwioTC8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=sCXWpXYC; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="sCXWpXYC" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-220c92c857aso42490435ad.0 for ; Fri, 14 Feb 2025 16:10:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578211; x=1740183011; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NHWoGsRhiFc/ooKo3TvF3R3d8ADOTW+DffRpnDLhJ48=; b=sCXWpXYCkPr4AIgT85SWKKDZgQpmOYI1AlUlPutZhDv9ApQnXaa2Z2BwJBMPMv19KC EaLI1lDprh7TUTYfByVEtSgDwxKWvyfywAibiWDyBwplB3CUpsHTPZT78ncgxkZ8nXHv Q7zpkAZ3gKvGFBj5EMHX8Ft+n8sHvCDMuUMe8/2hHRlSRExtqqepa1ZAjpIQCekbrE5L n/N2g3NctFgTy59DeQhWGiEQ8PHN5DKkgT54mEjSHuy1KkLLOwnsRQlvYP30gi6/cpWb K17YE/wTgwJGV9xtXcmj5jtM1tSABWZ46wRWaf+ikX4Ym5VOFUDSwJDLA1KliDwqW+Ek tBRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578211; x=1740183011; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NHWoGsRhiFc/ooKo3TvF3R3d8ADOTW+DffRpnDLhJ48=; b=PqezGmv3K4GYVzzN9jnI2ShKiYp3XA5GSZ6tEorKpugyHjpbrZNduGm4sZ+ex493bR Rgl/Voc3JDxBkTmUTnmnkoP02yym1ueNyZ6EqY9WAmThrDQ501Vk8WRvoQHjhk+OMaB5 cme2gcGOnIw6t4101GGeRgVsGoXCkuIzpM6XsKLYASh3uWuh3Q+082fq7XDQ8PO3iM8T di2pBCYBny3MKJ0sHmeG5A8IYkl8j30Gf1HW5j/ROUqfjrbpsiZ7Poul2uCa4Jw97CVv p45mbOuY/klyNNqQut4PgcJE+0AWHFACRLeGvxqgaEFyI0KLSKJ5A4SfuMzJ87J666Fz fTeQ== X-Forwarded-Encrypted: i=1; AJvYcCVmbPvEeXnQWwE/dMxOkDNx57jQK/Da3cNYfaDAa8NVXzv2oonlPTHTeFNJBJsOqpptaTsU45Q=@vger.kernel.org X-Gm-Message-State: AOJu0Yy6SNXCtHlWNxooLLubJtwxCmL2tTg42Q7BSGFuRLg8aP/S5vNb JOQyAixQykuEY9lT0Re2ZVuz1L0h7AdL/eBofzJFhuCpU8rFTs+rDcTo++Qd/iA= X-Gm-Gg: ASbGnctZMQtKSW8hOshkmkY22l1Xwr4nXmkp/NDQp24QfIkkrf1ZXZfM79HJbh2gRpi isITZa1vUFv1ts6q8jTzpc9Y35bdVLHh0k6FtK1QshxD8MgBzPDVHr8nA3ry9hOjM62DyIbSPwq otbY7A4SPDszzyUoaH+zbyY0ImGp672nlbbR28epEZcD2OI0RCrhOmzXLAMGzfoSv/NfmwZyOJs LglpcmYOzXv+0iZa2b7+9jqmJumq2AtDMOoJQ/uqzhKgXYDq6rBD752knTa6fljzjQrdziWH9FI X-Google-Smtp-Source: AGHT+IGB+0BtJEKjVmh9epCDeysVRSF5mdDq27M8L2mrfMjVlA01YXkfPFySEiuWNB9RVj09mjVQCA== X-Received: by 2002:a17:903:94d:b0:21f:ba77:c49a with SMTP id d9443c01a7336-220d33a50a3mr143708735ad.4.1739578211445; Fri, 14 Feb 2025 16:10:11 -0800 (PST) Received: from localhost ([2a03:2880:ff:71::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d556d473sm34116265ad.166.2025.02.14.16.10.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:11 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 10/11] net: add documentation for io_uring zcrx Date: Fri, 14 Feb 2025 16:09:45 -0800 Message-ID: <20250215000947.789731-11-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Add documentation for io_uring zero copy Rx that explains requirements and the user API. Signed-off-by: David Wei --- Documentation/networking/index.rst | 1 + Documentation/networking/iou-zcrx.rst | 202 ++++++++++++++++++++++++++ 2 files changed, 203 insertions(+) create mode 100644 Documentation/networking/iou-zcrx.rst diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst index 058193ed2eeb..c64133d309bf 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -63,6 +63,7 @@ Contents: gtp ila ioam6-sysctl + iou-zcrx ip_dynaddr ipsec ip-sysctl diff --git a/Documentation/networking/iou-zcrx.rst b/Documentation/networking/iou-zcrx.rst new file mode 100644 index 000000000000..0127319b30bb --- /dev/null +++ b/Documentation/networking/iou-zcrx.rst @@ -0,0 +1,202 @@ +.. SPDX-License-Identifier: GPL-2.0 + +===================== +io_uring zero copy Rx +===================== + +Introduction +============ + +io_uring zero copy Rx (ZC Rx) is a feature that removes kernel-to-user copy on +the network receive path, allowing packet data to be received directly into +userspace memory. This feature is different to TCP_ZEROCOPY_RECEIVE in that +there are no strict alignment requirements and no need to mmap()/munmap(). +Compared to kernel bypass solutions such as e.g. DPDK, the packet headers are +processed by the kernel TCP stack as normal. + +NIC HW Requirements +=================== + +Several NIC HW features are required for io_uring ZC Rx to work. For now the +kernel API does not configure the NIC and it must be done by the user. + +Header/data split +----------------- + +Required to split packets at the L4 boundary into a header and a payload. +Headers are received into kernel memory as normal and processed by the TCP +stack as normal. Payloads are received into userspace memory directly. + +Flow steering +------------- + +Specific HW Rx queues are configured for this feature, but modern NICs +typically distribute flows across all HW Rx queues. Flow steering is required +to ensure that only desired flows are directed towards HW queues that are +configured for io_uring ZC Rx. + +RSS +--- + +In addition to flow steering above, RSS is required to steer all other non-zero +copy flows away from queues that are configured for io_uring ZC Rx. + +Usage +===== + +Setup NIC +--------- + +Must be done out of band for now. + +Ensure there are at least two queues:: + + ethtool -L eth0 combined 2 + +Enable header/data split:: + + ethtool -G eth0 tcp-data-split on + +Carve out half of the HW Rx queues for zero copy using RSS:: + + ethtool -X eth0 equal 1 + +Set up flow steering, bearing in mind that queues are 0-indexed:: + + ethtool -N eth0 flow-type tcp6 ... action 1 + +Setup io_uring +-------------- + +This section describes the low level io_uring kernel API. Please refer to +liburing documentation for how to use the higher level API. + +Create an io_uring instance with the following required setup flags:: + + IORING_SETUP_SINGLE_ISSUER + IORING_SETUP_DEFER_TASKRUN + IORING_SETUP_CQE32 + +Create memory area +------------------ + +Allocate userspace memory area for receiving zero copy data:: + + void *area_ptr = mmap(NULL, area_size, + PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, + 0, 0); + +Create refill ring +------------------ + +Allocate memory for a shared ringbuf used for returning consumed buffers:: + + void *ring_ptr = mmap(NULL, ring_size, + PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, + 0, 0); + +This refill ring consists of some space for the header, followed by an array of +``struct io_uring_zcrx_rqe``:: + + size_t rq_entries = 4096; + size_t ring_size = rq_entries * sizeof(struct io_uring_zcrx_rqe) + PAGE_SIZE; + /* align to page size */ + ring_size = (ring_size + (PAGE_SIZE - 1)) & ~(PAGE_SIZE - 1); + +Register ZC Rx +-------------- + +Fill in registration structs:: + + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area_ptr, + .len = area_size, + .flags = 0, + }; + + struct io_uring_region_desc region_reg = { + .user_addr = (__u64)(unsigned long)ring_ptr, + .size = ring_size, + .flags = IORING_MEM_REGION_TYPE_USER, + }; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = if_nametoindex("eth0"), + /* this is the HW queue with desired flow steered into it */ + .if_rxq = 1, + .rq_entries = rq_entries, + .area_ptr = (__u64)(unsigned long)&area_reg, + .region_ptr = (__u64)(unsigned long)®ion_reg, + }; + +Register with kernel:: + + io_uring_register_ifq(ring, ®); + +Map refill ring +--------------- + +The kernel fills in fields for the refill ring in the registration ``struct +io_uring_zcrx_ifq_reg``. Map it into userspace:: + + struct io_uring_zcrx_rq refill_ring; + + refill_ring.khead = (unsigned *)((char *)ring_ptr + reg.offsets.head); + refill_ring.khead = (unsigned *)((char *)ring_ptr + reg.offsets.tail); + refill_ring.rqes = + (struct io_uring_zcrx_rqe *)((char *)ring_ptr + reg.offsets.rqes); + refill_ring.rq_tail = 0; + refill_ring.ring_ptr = ring_ptr; + +Receiving data +-------------- + +Prepare a zero copy recv request:: + + struct io_uring_sqe *sqe; + + sqe = io_uring_get_sqe(ring); + io_uring_prep_rw(IORING_OP_RECV_ZC, sqe, fd, NULL, 0, 0); + sqe->ioprio |= IORING_RECV_MULTISHOT; + +Now, submit and wait:: + + io_uring_submit_and_wait(ring, 1); + +Finally, process completions:: + + struct io_uring_cqe *cqe; + unsigned int count = 0; + unsigned int head; + + io_uring_for_each_cqe(ring, head, cqe) { + struct io_uring_zcrx_cqe *rcqe = (struct io_uring_zcrx_cqe *)(cqe + 1); + + unsigned long mask = (1ULL << IORING_ZCRX_AREA_SHIFT) - 1; + unsigned char *data = area_ptr + (rcqe->off & mask); + /* do something with the data */ + + count++; + } + io_uring_cq_advance(ring, count); + +Recycling buffers +----------------- + +Return buffers back to the kernel to be used again:: + + struct io_uring_zcrx_rqe *rqe; + unsigned mask = refill_ring.ring_entries - 1; + rqe = &refill_ring.rqes[refill_ring.rq_tail & mask]; + + unsigned long area_offset = rcqe->off & ~IORING_ZCRX_AREA_MASK; + rqe->off = area_offset | area_reg.rq_area_token; + rqe->len = cqe->res; + IO_URING_WRITE_ONCE(*refill_ring.ktail, ++refill_ring.rq_tail); + +Testing +======= + +See ``tools/testing/selftests/drivers/net/hw/iou-zcrx.c`` From patchwork Sat Feb 15 00:09:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13975787 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BBD61624F4 for ; Sat, 15 Feb 2025 00:10:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578216; cv=none; b=jzQ0d4KNcPL4932p299cLaZzPwC86Bq/9zOVhiEtS3cRK6TMEHamHC7uyj+UUshR5sM839pgDh8N8UjUtIsN6osxq7uBdK5dCRddN4VXN2c3OUjQWoX+9bCyT0Zpa/7TUoBSEXH2QLstIClzKagvIdrC+HLV3HKuHj98SND6vNo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739578216; c=relaxed/simple; bh=cug8gWZNB+fVPqIM3eSb0lRttWPY03cy3UltoOm88oM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C0GGvAEo+6t6lUN4ny/pabdNG+0a0QLO9nncq3C2pWcWU7GUtD4Cl/UNe3aBPvVkcJg+rt9R2Sfhj9ONiyCVj7r5IwfuyM4eh2M6rSyGkHAlNgrCDhN/XSsH2CR7n9KTz9gTCLvqaH8eC0ntT8mVG0bZaDFofG/ymSo2Oo7qWRo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=bPZy76ll; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="bPZy76ll" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21f78b1fb7dso45802195ad.3 for ; Fri, 14 Feb 2025 16:10:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1739578213; x=1740183013; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZV9MbmHy+OETbAKxN4PX54lk+MlnQr4pOuqlO0aDqtA=; b=bPZy76lludetFjSw3IOfbKs4XZAj36jr8bVKYbvIr6Kf9zCd8vKwsi+BhN5QCq2xKs RoOtVlkBEi6m7FhonZZminht8p5xn2Qf3qnwc+zP55NAx8lkUSOLkWXDfn14v6chcZzi xm811ZcFbNuK3Jmar7R6fVZBQX+dEW5qiYxJE5Veg5XdyFccW1yzFVVpbbZkboJhk0X9 Cz9JTwzsgvreZPqktH0fFYUXHffHiPNcsYG4uL+ThpdOYC+k+U/MJGVpWMjJKLMFEEhZ uuP/5e/Zx+1hlCTnHe8kE4NafiS+roylkvIvSeCFpR6Sbq7zJwSGKfOdWJrHeXUvOqD7 h8Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739578213; x=1740183013; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZV9MbmHy+OETbAKxN4PX54lk+MlnQr4pOuqlO0aDqtA=; b=VK+oAGY6OFJAawN7Gdb8uFTOcQOOuRrIO+W37lpkykaiUS5BT/82h2ZiFveVdCtr+Z BsNzRmMyQEc6LBIO4v/9vF662Wy54sSseV4fVL7MIK21v/FgG+XjMSlfcn6IDSlQX4aA 6xKwGGq7yFstxiEEAmdzYC5ILZ9V2X0U/p1QVRg4fkjfDJsqhsu4QcTyTEcFbA3/J/5g DcbI8gd/qwMliCKK5s8bTHB0UlnKt5qRyVuOmpFnb0lBSYa+L6oVMcIWz2XRaxr9fv2J oq6y3V41hGw1L0evwHYZwOVLctuf1vc+nI4EpNLAfVHuu4naUtOwQOH4H0O/5uDpkD2h IBRQ== X-Forwarded-Encrypted: i=1; AJvYcCURyLRL43WlHAtsBI0q8LoCB5IyJE85jqJOKtJMIiOIGlqb423FxO793CF5VaK8356UtWqm6ng=@vger.kernel.org X-Gm-Message-State: AOJu0Yw7IXMYtBv4+8gzNw+B2K7Neyr//Q5flt1iIrr145MibVVuugpf kxK0TbJzNnNGCIqIJv9eFyUnekgai5z3olEMFchnaWbKecQe7TkVlXuN54OEqxw= X-Gm-Gg: ASbGncuVRVHJRGvXOjtTn5MPeLTK4t62SzDwjkk29ecfoPV6dSyh/n1rBoTGSdZK+oG DozZkhfhvZQMmBWUvdBPX/PB0OcS+568Z9aqjWxB1Wm2MfA9RjqGzshXm3OhUCepL9Qvvk59+py wTFEznzcJYzbLlwonLDhzbPYpqCuRTywLQmiUq8fNk9fRfo8nQV8r0gSMiM8GFi5Pi7dUoMcb7J ++kyeBVx0rl3YvnSrJh6OOj6UIxXyoCa0NrFU5znJlMf7ZnK3J5XFbLNT129u73quLppnDNOg+W X-Google-Smtp-Source: AGHT+IEGnaOoh/T9o1CgyylYlUdBn5i9SkiVlipSNdsE5ifqIaO2FURIjeIDWlqXEioTBikpLUUwwA== X-Received: by 2002:a17:903:2b07:b0:215:b8c6:338a with SMTP id d9443c01a7336-22103f038e6mr20410055ad.4.1739578212650; Fri, 14 Feb 2025 16:10:12 -0800 (PST) Received: from localhost ([2a03:2880:ff:40::]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d051sm34057945ad.108.2025.02.14.16.10.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 16:10:12 -0800 (PST) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela , lizetao Subject: [PATCH v14 11/11] io_uring/zcrx: add selftest Date: Fri, 14 Feb 2025 16:09:46 -0800 Message-ID: <20250215000947.789731-12-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20250215000947.789731-1-dw@davidwei.uk> References: <20250215000947.789731-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a selftest for io_uring zero copy Rx. This test cannot run locally and requires a remote host to be configured in net.config. The remote host must have hardware support for zero copy Rx as listed in the documentation page. The test will restore the NIC config back to before the test and is idempotent. liburing is required to compile the test and be installed on the remote host running the test. Signed-off-by: David Wei --- .../selftests/drivers/net/hw/.gitignore | 2 + .../testing/selftests/drivers/net/hw/Makefile | 5 + .../selftests/drivers/net/hw/iou-zcrx.c | 426 ++++++++++++++++++ .../selftests/drivers/net/hw/iou-zcrx.py | 64 +++ 4 files changed, 497 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/hw/iou-zcrx.c create mode 100755 tools/testing/selftests/drivers/net/hw/iou-zcrx.py diff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore index e9fe6ede681a..6942bf575497 100644 --- a/tools/testing/selftests/drivers/net/hw/.gitignore +++ b/tools/testing/selftests/drivers/net/hw/.gitignore @@ -1 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +iou-zcrx ncdevmem diff --git a/tools/testing/selftests/drivers/net/hw/Makefile b/tools/testing/selftests/drivers/net/hw/Makefile index 21ba64ce1e34..7efc47c89463 100644 --- a/tools/testing/selftests/drivers/net/hw/Makefile +++ b/tools/testing/selftests/drivers/net/hw/Makefile @@ -1,5 +1,7 @@ # SPDX-License-Identifier: GPL-2.0+ OR MIT +TEST_GEN_FILES = iou-zcrx + TEST_PROGS = \ csum.py \ devlink_port_split.py \ @@ -10,6 +12,7 @@ TEST_PROGS = \ ethtool_rmon.sh \ hw_stats_l3.sh \ hw_stats_l3_gre.sh \ + iou-zcrx.py \ loopback.sh \ nic_link_layer.py \ nic_performance.py \ @@ -38,3 +41,5 @@ include ../../../lib.mk # YNL build YNL_GENS := ethtool netdev include ../../../net/ynl.mk + +$(OUTPUT)/iou-zcrx: LDLIBS += -luring diff --git a/tools/testing/selftests/drivers/net/hw/iou-zcrx.c b/tools/testing/selftests/drivers/net/hw/iou-zcrx.c new file mode 100644 index 000000000000..5d04dd55ae55 --- /dev/null +++ b/tools/testing/selftests/drivers/net/hw/iou-zcrx.c @@ -0,0 +1,426 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#define PAGE_SIZE (4096) +#define AREA_SIZE (8192 * PAGE_SIZE) +#define SEND_SIZE (512 * 4096) +#define min(a, b) \ + ({ \ + typeof(a) _a = (a); \ + typeof(b) _b = (b); \ + _a < _b ? _a : _b; \ + }) +#define min_t(t, a, b) \ + ({ \ + t _ta = (a); \ + t _tb = (b); \ + min(_ta, _tb); \ + }) + +#define ALIGN_UP(v, align) (((v) + (align) - 1) & ~((align) - 1)) + +static int cfg_server; +static int cfg_client; +static int cfg_port = 8000; +static int cfg_payload_len; +static const char *cfg_ifname; +static int cfg_queue_id = -1; +static struct sockaddr_in6 cfg_addr; + +static char payload[SEND_SIZE] __attribute__((aligned(PAGE_SIZE))); +static void *area_ptr; +static void *ring_ptr; +static size_t ring_size; +static struct io_uring_zcrx_rq rq_ring; +static unsigned long area_token; +static int connfd; +static bool stop; +static size_t received; + +static unsigned long gettimeofday_ms(void) +{ + struct timeval tv; + + gettimeofday(&tv, NULL); + return (tv.tv_sec * 1000) + (tv.tv_usec / 1000); +} + +static int parse_address(const char *str, int port, struct sockaddr_in6 *sin6) +{ + int ret; + + sin6->sin6_family = AF_INET6; + sin6->sin6_port = htons(port); + + ret = inet_pton(sin6->sin6_family, str, &sin6->sin6_addr); + if (ret != 1) { + /* fallback to plain IPv4 */ + ret = inet_pton(AF_INET, str, &sin6->sin6_addr.s6_addr32[3]); + if (ret != 1) + return -1; + + /* add ::ffff prefix */ + sin6->sin6_addr.s6_addr32[0] = 0; + sin6->sin6_addr.s6_addr32[1] = 0; + sin6->sin6_addr.s6_addr16[4] = 0; + sin6->sin6_addr.s6_addr16[5] = 0xffff; + } + + return 0; +} + +static inline size_t get_refill_ring_size(unsigned int rq_entries) +{ + size_t size; + + ring_size = rq_entries * sizeof(struct io_uring_zcrx_rqe); + /* add space for the header (head/tail/etc.) */ + ring_size += PAGE_SIZE; + return ALIGN_UP(ring_size, 4096); +} + +static void setup_zcrx(struct io_uring *ring) +{ + unsigned int ifindex; + unsigned int rq_entries = 4096; + int ret; + + ifindex = if_nametoindex(cfg_ifname); + if (!ifindex) + error(1, 0, "bad interface name: %s", cfg_ifname); + + area_ptr = mmap(NULL, + AREA_SIZE, + PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, + 0, + 0); + if (area_ptr == MAP_FAILED) + error(1, 0, "mmap(): zero copy area"); + + ring_size = get_refill_ring_size(rq_entries); + ring_ptr = mmap(NULL, + ring_size, + PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, + 0, + 0); + + struct io_uring_region_desc region_reg = { + .size = ring_size, + .user_addr = (__u64)(unsigned long)ring_ptr, + .flags = IORING_MEM_REGION_TYPE_USER, + }; + + struct io_uring_zcrx_area_reg area_reg = { + .addr = (__u64)(unsigned long)area_ptr, + .len = AREA_SIZE, + .flags = 0, + }; + + struct io_uring_zcrx_ifq_reg reg = { + .if_idx = ifindex, + .if_rxq = cfg_queue_id, + .rq_entries = rq_entries, + .area_ptr = (__u64)(unsigned long)&area_reg, + .region_ptr = (__u64)(unsigned long)®ion_reg, + }; + + ret = io_uring_register_ifq(ring, ®); + if (ret) + error(1, 0, "io_uring_register_ifq(): %d", ret); + + rq_ring.khead = (unsigned int *)((char *)ring_ptr + reg.offsets.head); + rq_ring.ktail = (unsigned int *)((char *)ring_ptr + reg.offsets.tail); + rq_ring.rqes = (struct io_uring_zcrx_rqe *)((char *)ring_ptr + reg.offsets.rqes); + rq_ring.rq_tail = 0; + rq_ring.ring_entries = reg.rq_entries; + + area_token = area_reg.rq_area_token; +} + +static void add_accept(struct io_uring *ring, int sockfd) +{ + struct io_uring_sqe *sqe; + + sqe = io_uring_get_sqe(ring); + + io_uring_prep_accept(sqe, sockfd, NULL, NULL, 0); + sqe->user_data = 1; +} + +static void add_recvzc(struct io_uring *ring, int sockfd) +{ + struct io_uring_sqe *sqe; + + sqe = io_uring_get_sqe(ring); + + io_uring_prep_rw(IORING_OP_RECV_ZC, sqe, sockfd, NULL, 0, 0); + sqe->ioprio |= IORING_RECV_MULTISHOT; + sqe->user_data = 2; +} + +static void process_accept(struct io_uring *ring, struct io_uring_cqe *cqe) +{ + if (cqe->res < 0) + error(1, 0, "accept()"); + if (connfd) + error(1, 0, "Unexpected second connection"); + + connfd = cqe->res; + add_recvzc(ring, connfd); +} + +static void process_recvzc(struct io_uring *ring, struct io_uring_cqe *cqe) +{ + unsigned rq_mask = rq_ring.ring_entries - 1; + struct io_uring_zcrx_cqe *rcqe; + struct io_uring_zcrx_rqe *rqe; + struct io_uring_sqe *sqe; + uint64_t mask; + char *data; + ssize_t n; + int i; + + if (cqe->res == 0 && cqe->flags == 0) { + stop = true; + return; + } + + if (cqe->res < 0) + error(1, 0, "recvzc(): %d", cqe->res); + + if (!(cqe->flags & IORING_CQE_F_MORE)) + add_recvzc(ring, connfd); + + rcqe = (struct io_uring_zcrx_cqe *)(cqe + 1); + + n = cqe->res; + mask = (1ULL << IORING_ZCRX_AREA_SHIFT) - 1; + data = (char *)area_ptr + (rcqe->off & mask); + + for (i = 0; i < n; i++) { + if (*(data + i) != payload[(received + i)]) + error(1, 0, "payload mismatch"); + } + received += n; + + rqe = &rq_ring.rqes[(rq_ring.rq_tail & rq_mask)]; + rqe->off = (rcqe->off & ~IORING_ZCRX_AREA_MASK) | area_token; + rqe->len = cqe->res; + io_uring_smp_store_release(rq_ring.ktail, ++rq_ring.rq_tail); +} + +static void server_loop(struct io_uring *ring) +{ + struct io_uring_cqe *cqe; + unsigned int count = 0; + unsigned int head; + int i, ret; + + io_uring_submit_and_wait(ring, 1); + + io_uring_for_each_cqe(ring, head, cqe) { + if (cqe->user_data == 1) + process_accept(ring, cqe); + else if (cqe->user_data == 2) + process_recvzc(ring, cqe); + else + error(1, 0, "unknown cqe"); + count++; + } + io_uring_cq_advance(ring, count); +} + +static void run_server(void) +{ + unsigned int flags = 0; + struct io_uring ring; + int fd, enable, ret; + uint64_t tstop; + + fd = socket(AF_INET6, SOCK_STREAM, 0); + if (fd == -1) + error(1, 0, "socket()"); + + enable = 1; + ret = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(int)); + if (ret < 0) + error(1, 0, "setsockopt(SO_REUSEADDR)"); + + ret = bind(fd, (struct sockaddr *)&cfg_addr, sizeof(cfg_addr)); + if (ret < 0) + error(1, 0, "bind()"); + + if (listen(fd, 1024) < 0) + error(1, 0, "listen()"); + + flags |= IORING_SETUP_COOP_TASKRUN; + flags |= IORING_SETUP_SINGLE_ISSUER; + flags |= IORING_SETUP_DEFER_TASKRUN; + flags |= IORING_SETUP_SUBMIT_ALL; + flags |= IORING_SETUP_CQE32; + + io_uring_queue_init(512, &ring, flags); + + setup_zcrx(&ring); + + add_accept(&ring, fd); + + tstop = gettimeofday_ms() + 5000; + while (!stop && gettimeofday_ms() < tstop) + server_loop(&ring); + + if (!stop) + error(1, 0, "test failed\n"); +} + +static void run_client(void) +{ + ssize_t to_send = SEND_SIZE; + ssize_t sent = 0; + ssize_t chunk, res; + int fd; + + fd = socket(AF_INET6, SOCK_STREAM, 0); + if (fd == -1) + error(1, 0, "socket()"); + + if (connect(fd, (struct sockaddr *)&cfg_addr, sizeof(cfg_addr))) + error(1, 0, "connect()"); + + while (to_send) { + void *src = &payload[sent]; + + chunk = min_t(ssize_t, cfg_payload_len, to_send); + res = send(fd, src, chunk, 0); + if (res < 0) + error(1, 0, "send(): %d", sent); + sent += res; + to_send -= res; + } + + close(fd); +} + +static void usage(const char *filepath) +{ + error(1, 0, "Usage: %s (-4|-6) (-s|-c) -h -p " + "-l -i -q", filepath); +} + +static void parse_opts(int argc, char **argv) +{ + const int max_payload_len = sizeof(payload) - + sizeof(struct ipv6hdr) - + sizeof(struct tcphdr) - + 40 /* max tcp options */; + struct sockaddr_in6 *addr6 = (void *) &cfg_addr; + char *addr = NULL; + int ret; + int c; + + if (argc <= 1) + usage(argv[0]); + cfg_payload_len = max_payload_len; + + while ((c = getopt(argc, argv, "46sch:p:l:i:q:")) != -1) { + switch (c) { + case 's': + if (cfg_client) + error(1, 0, "Pass one of -s or -c"); + cfg_server = 1; + break; + case 'c': + if (cfg_server) + error(1, 0, "Pass one of -s or -c"); + cfg_client = 1; + break; + case 'h': + addr = optarg; + break; + case 'p': + cfg_port = strtoul(optarg, NULL, 0); + break; + case 'l': + cfg_payload_len = strtoul(optarg, NULL, 0); + break; + case 'i': + cfg_ifname = optarg; + break; + case 'q': + cfg_queue_id = strtoul(optarg, NULL, 0); + break; + } + } + + if (cfg_server && addr) + error(1, 0, "Receiver cannot have -h specified"); + + memset(addr6, 0, sizeof(*addr6)); + addr6->sin6_family = AF_INET6; + addr6->sin6_port = htons(cfg_port); + addr6->sin6_addr = in6addr_any; + if (addr) { + ret = parse_address(addr, cfg_port, addr6); + if (ret) + error(1, 0, "receiver address parse error: %s", addr); + } + + if (cfg_payload_len > max_payload_len) + error(1, 0, "-l: payload exceeds max (%d)", max_payload_len); +} + +int main(int argc, char **argv) +{ + const char *cfg_test = argv[argc - 1]; + int i; + + parse_opts(argc, argv); + + for (i = 0; i < SEND_SIZE; i++) + payload[i] = 'a' + (i % 26); + + if (cfg_server) + run_server(); + else if (cfg_client) + run_client(); + + return 0; +} diff --git a/tools/testing/selftests/drivers/net/hw/iou-zcrx.py b/tools/testing/selftests/drivers/net/hw/iou-zcrx.py new file mode 100755 index 000000000000..ea0a346c3eff --- /dev/null +++ b/tools/testing/selftests/drivers/net/hw/iou-zcrx.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import re +from os import path +from lib.py import ksft_run, ksft_exit +from lib.py import NetDrvEpEnv +from lib.py import bkg, cmd, ethtool, wait_port_listen + + +def _get_rx_ring_entries(cfg): + output = ethtool(f"-g {cfg.ifname}", host=cfg.remote).stdout + values = re.findall(r'RX:\s+(\d+)', output) + return int(values[1]) + + +def _get_combined_channels(cfg): + output = ethtool(f"-l {cfg.ifname}", host=cfg.remote).stdout + values = re.findall(r'Combined:\s+(\d+)', output) + return int(values[1]) + + +def _set_flow_rule(cfg, chan): + output = ethtool(f"-N {cfg.ifname} flow-type tcp6 dst-port 9999 action {chan}", host=cfg.remote).stdout + values = re.search(r'ID (\d+)', output).group(1) + return int(values) + + +def test_zcrx(cfg) -> None: + cfg.require_v6() + + combined_chans = _get_combined_channels(cfg) + if combined_chans < 2: + raise KsftSkipEx('at least 2 combined channels required') + rx_ring = _get_rx_ring_entries(cfg) + + rx_cmd = f"{cfg.bin_remote} -s -p 9999 -i {cfg.ifname} -q {combined_chans - 1}" + tx_cmd = f"{cfg.bin_local} -c -h {cfg.remote_v6} -p 9999 -l 12840" + + try: + ethtool(f"-G {cfg.ifname} rx 64", host=cfg.remote) + ethtool(f"-X {cfg.ifname} equal {combined_chans - 1}", host=cfg.remote) + flow_rule_id = _set_flow_rule(cfg, combined_chans - 1) + + with bkg(rx_cmd, host=cfg.remote, exit_wait=True): + wait_port_listen(9999, proto="tcp", host=cfg.remote) + cmd(tx_cmd) + finally: + ethtool(f"-N {cfg.ifname} delete {flow_rule_id}", host=cfg.remote) + ethtool(f"-X {cfg.ifname} default", host=cfg.remote) + ethtool(f"-G {cfg.ifname} rx {rx_ring}", host=cfg.remote) + + +def main() -> None: + with NetDrvEpEnv(__file__) as cfg: + cfg.bin_local = path.abspath(path.dirname(__file__) + "/../../../drivers/net/hw/iou-zcrx") + cfg.bin_remote = cfg.remote.deploy(cfg.bin_local) + + ksft_run(globs=globals(), case_pfx={"test_"}, args=(cfg, )) + ksft_exit() + + +if __name__ == "__main__": + main()