From patchwork Fri Mar 7 16:00:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006692 Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DB7B22156A for ; Fri, 7 Mar 2025 15:59:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363189; cv=none; b=KDnxADcDClj2aDhDNLbHiXliB9V2xfVwU1+hXMIxqfj7id+1RqZiWfdHPi9e7w0708jUyWRUcp9AR5S/qAnF9xM+CfbN5VntXVXPpxlcp4/qIWS/uNesIfxzRi9hlprnSB+Eug95/V/6if+XrBkWwl+r5GwmxAAHNVKrkY+sJog= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363189; c=relaxed/simple; bh=1ZJTXrPZn6vLFzJGF1fV1gJqddlvwbfkmv7zOk/8R78=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=opcr8i6kczZK21H/5XBZKFOkjIx5X3IfMwKaJxLOWECN6GYwamFbSwx0VZM3QVSZBtiqRCA2o6FXFf9vU8++yWlzhkDh/yW9dd53VIRDBZYym4WDBcdoQkvClKX7CxhTG1XC0WaP7S8nWeKj0pYNvwEOy51hMzs7C7GcDlCgNPQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AxwjClCx; arc=none smtp.client-ip=209.85.218.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AxwjClCx" Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-ac25520a289so154799166b.3 for ; Fri, 07 Mar 2025 07:59:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363185; x=1741967985; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OHR5hQJDkdouKch6tQfcQ2eHsBRq8rM+srmvQwbKcbs=; b=AxwjClCxwmPpiQlldVnIgLCHTph9wWIdm9CE7iZ2Uzaw+V2PlJkE559/zYG7QyrqWR cdEih6u92AAZpWKq3ozAbPyw1WBcN8XiqcqT6V3tMxamrYvC7nSk+/lKVpWL6/x/H8bW rcGYmXjHk0gD19Ak+QVNdsSOmvvw23AwmP2rkyhxN2Z5KKfWilLsXmKCIKIknv5TQlZE +0YyeZLjzXdoOPRY0KoIcnOXhzcmF5fmQxXXzYAXha0kWCrU+0GjlS99RVMUmaUTgCyj XwkkdUyrhEA1AQ3G4S2MH1RU3acK7esQGEZERLyYtys9ccku1C7Uv18bzA6gJVHQr0wf kMBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363185; x=1741967985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OHR5hQJDkdouKch6tQfcQ2eHsBRq8rM+srmvQwbKcbs=; b=e39AJE95PhiYJZQBDuHOsB+xNoi0v118h/o67NTmCcwQ9q9pevbixefOxFrBhSc/UJ KRR11I2DCAkQPx9IHeFgt1sJreVtyZ/7F7fMi3/1694Bwej/uEW2uIOk9f06UYfXx+Wv Z/1eHbEHSHuI1inLA1CPX26wAf5enflTQm4Id/6oQaqIX3qrA6/UJdtZp1YXrSl342+R G4g3V6030pzTXvsoaWKLS6uBStsdbbKc2jfNQm5P9iAG1ZI0AgunJ2s7mlVscIGiktIF CrruaJKGxIXdal87zL5UjtbYPOWk+8XoIvJCycQ0Ls1Qiqluj0guWpHLvo5X7h63FtkG cTNA== X-Gm-Message-State: AOJu0YxgOTZQtfXokwxAg2QVEQdhdkSxI3IB+3xi9NXdtQlqYq71cnjK GnPgi6oygPZlFP6IoiozygSBunscNptnFjd2K22uPWXCWZfrEYF5oss0RQ== X-Gm-Gg: ASbGncvG2I9WGQHPEztjP1L9BAvqj7EWq5JyBslBv+kwQoH5w8LnDAtPnTopSBQBseI DtTXLxfXiS4rPt8xHUbrvPc6DFU4KlGYl0GpaHu8WKmIIPb3KHwuhwWMdK9kBooQ4cevMvnPZiM ULPnaOyz6bWYIl11QPP0/pNa2JAbGHNmXNsbzfGQPUG+kQcARkion7IReD40mwt5co6RP0+GCxX lk9oyiaVEPb0ZLFjrw9pv6EswisIaDpxBhSabdTWCjRMpTC4lHwLR3pw43UrwVRQwvxZWXjhOVU 3D3/FTvbAqrYDErokh/dt9qCvXwk X-Google-Smtp-Source: AGHT+IF2MHboKdA9+T38PGP3XzJnMxx60yoUGgR38nYPWNcxHR2BqRlHUK0DPID8YN/AvjSMpFvxEQ== X-Received: by 2002:a17:907:3204:b0:abf:38a:6498 with SMTP id a640c23a62f3a-ac2530295d1mr410388766b.55.1741363185042; Fri, 07 Mar 2025 07:59:45 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:44 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 1/9] io_uring: introduce struct iou_vec Date: Fri, 7 Mar 2025 16:00:29 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I need a convenient way to pass around and work with iovec+size pair, put them into a structure and makes use of it in rw.c Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +++++ io_uring/rsrc.c | 9 +++++++++ io_uring/rsrc.h | 16 ++++++++++++++++ io_uring/rw.c | 17 +++++++---------- io_uring/rw.h | 4 ++-- 5 files changed, 39 insertions(+), 12 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 35fc241c4672..9101f12d21ef 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -110,6 +110,11 @@ struct io_uring_task { } ____cacheline_aligned_in_smp; }; +struct iou_vec { + struct iovec *iovec; + unsigned nr; +}; + struct io_uring { u32 head; u32 tail; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 28783f1dde00..bac509f85c80 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1260,3 +1260,12 @@ int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg) fput(file); return ret; } + +void io_vec_free(struct iou_vec *iv) +{ + if (!iv->iovec) + return; + kfree(iv->iovec); + iv->iovec = NULL; + iv->nr = 0; +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 284e300e63fb..f35e1a07619a 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -145,4 +145,20 @@ static inline void __io_unaccount_mem(struct user_struct *user, atomic_long_sub(nr_pages, &user->locked_vm); } +void io_vec_free(struct iou_vec *iv); + +static inline void io_vec_reset_iovec(struct iou_vec *iv, + struct iovec *iovec, unsigned nr) +{ + io_vec_free(iv); + iv->iovec = iovec; + iv->nr = nr; +} + +static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv) +{ + if (IS_ENABLED(CONFIG_KASAN)) + io_vec_free(iv); +} + #endif diff --git a/io_uring/rw.c b/io_uring/rw.c index 5ee9f8949e8b..ad7f647d48e9 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -87,9 +87,9 @@ static int io_import_vec(int ddir, struct io_kiocb *req, int ret, nr_segs; struct iovec *iov; - if (io->free_iovec) { - nr_segs = io->free_iov_nr; - iov = io->free_iovec; + if (io->vec.iovec) { + nr_segs = io->vec.nr; + iov = io->vec.iovec; } else { nr_segs = 1; iov = &io->fast_iov; @@ -101,9 +101,7 @@ static int io_import_vec(int ddir, struct io_kiocb *req, return ret; if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - io->free_iov_nr = io->iter.nr_segs; - kfree(io->free_iovec); - io->free_iovec = iov; + io_vec_reset_iovec(&io->vec, iov, io->iter.nr_segs); } return 0; } @@ -151,7 +149,7 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) return; - io_alloc_cache_kasan(&rw->free_iovec, &rw->free_iov_nr); + io_alloc_cache_vec_kasan(&rw->vec); if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -201,7 +199,7 @@ static int io_rw_alloc_async(struct io_kiocb *req) rw = io_uring_alloc_async_data(&ctx->rw_cache, req); if (!rw) return -ENOMEM; - if (rw->free_iovec) + if (rw->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; rw->bytes_done = 0; return 0; @@ -1327,7 +1325,6 @@ void io_rw_cache_free(const void *entry) { struct io_async_rw *rw = (struct io_async_rw *) entry; - if (rw->free_iovec) - kfree(rw->free_iovec); + io_vec_free(&rw->vec); kfree(rw); } diff --git a/io_uring/rw.h b/io_uring/rw.h index bf121b81ebe8..529fd2f96a7f 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -9,13 +9,13 @@ struct io_meta_state { }; struct io_async_rw { + struct iou_vec vec; size_t bytes_done; - struct iovec *free_iovec; + struct_group(clear, struct iov_iter iter; struct iov_iter_state iter_state; struct iovec fast_iov; - int free_iov_nr; /* * wpq is for buffered io, while meta fields are used with * direct io From patchwork Fri Mar 7 16:00:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006693 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D990922156E for ; Fri, 7 Mar 2025 15:59:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363190; cv=none; b=mDYCE3xMCSpkqbKzGOmdNB+0is3+pneDPjlvrIOGIGCIdZNAVKXnbM2abP2hZTW9EQpTl0bSiJp3X3AqPv6OiGUDcAY74vUCAqH/sRu0xkI6nH0pLY79Mo5YJl8raEdym1MgxZbp378WrNwGVhsJP5lsUnf/YQcOjcu0Jvd2k4Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363190; c=relaxed/simple; bh=U0p9sJ78MOBEr0imh6uZmRYspNOFe/F09i9T6gAkJRc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Q8BPvNX+n721LMa16Pfe5Q5uvtVU7qRV+juG6mnglS/yz+Xkv6smu+wGl0e+FIQgvDs31wKfshDTkoeSggngTxMxDjMnHXvYO9ELDIRboJq7nxYy4jJp9qHzZVVJhXdAtKYW8W6eS/JzGpuIgyLy64Y1KJluscpaw9oKcU2zJ3M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eMtPEhCL; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eMtPEhCL" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5dca468c5e4so3647368a12.1 for ; Fri, 07 Mar 2025 07:59:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363186; x=1741967986; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2v5a/ERJ2E7F2f+gymnI/P8uV1lIMDW+iluW2JhkoQ4=; b=eMtPEhCL9UTtaQ/OAhZDPe7kG/cQPSceRxFgrsU0uZ4AMTDRtQYd/gwAB2pSVW+doH 2RBBex9jmsohcJiRvkjNmzNj5tRTkfTUoFId2Sdqh/loXEZs7qh7gqkReKk0cYYwdD5V td4JtYsj1rjv4XvxTzBuNLmmQp8O8iZD8DZAbzmTiejPyXv/geTNb/r/ae/jsLCO3RgA kffzyWf4LGW/hfqYg2zK/VPhjWDYJaeAvweFTrintVGlB5ysI9PpJ4IoYekqb9fe3Lz+ uDNxFZh+d4e9Z9+IFkVkw2djkz5/78sGW7A+guaQIsqflH3GfZRb1Trr586gVzZWauKW Zvkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363186; x=1741967986; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2v5a/ERJ2E7F2f+gymnI/P8uV1lIMDW+iluW2JhkoQ4=; b=cYRMyK7XCblxVzHJSOl/3ZILlJD0iFvVNF8wqjyJyNJ0Eh92EZ+pGKWoIvvmjv3rEM o0j+ri6Apj0Hk2sXqXyCfIELxWUiz5jnRFmuFP31xX49ao/Fz6ISNqHXA+E5vxPYy82B WTNb8L517pS6IekvMBr7FdN+bx3Hxpc3lcpPstF53bbZNQOt7EL8kz3ZE56Pp6sQ1nI7 rYQ421L2JOSmtDv54Nh3zdwmPKwi4Bv9keTOTczczpzaY9JpUfyTQrdAk0Z631BEC09S w5wYH/mNo65q23BytUHmm3ayY2R17TFzoal0cVpYr92PAbO+RDhGQTMVRBKs3VebbtM8 LfhA== X-Gm-Message-State: AOJu0Yx6/NtlBdN7MufuUZ8afQtiR9a+JaFbTnifGPMRQrBlQto4OmZQ +K4gdPWdjH6tY5X6PsGvYP9RXK0yr88zD4yvYxQZ+Zcc+NvWr/FZb0ZkWA== X-Gm-Gg: ASbGncvIGiG8EXezglM+3keYvZ8a3te5WvCxcbbXRxeNWfg9mlBN66KeQrczw8/5iaB 16ctAJ2QpLUgfHM8g6K431Bcjz0ljOVY9xlWlBnMFQuyd3vLHHWqNJPWs5/0Z0K6NaIwjU+msRy XL5YCPvf7QQMJYcCY2pztZbkobNdB2OnXdwfC95p8iJlOu3McE2+ZAGEkk8mJf/oid5YjVzKMBs oMZlxOiYeDnZvnq/FVkdLhh2uM1pIyRGGYl68fLDH/sRpswIxpGFTZqTulTK8bv9PAdInfS4i+g +v2bgZ0M72STR5vMO4fLwmEwqKY2 X-Google-Smtp-Source: AGHT+IFyBk1iPEsq948u7GMSZJTJp5zsg/C/dEwEzsQhr2fbFgt9wUKMWEBMUzcPKpDz5i9dcBxWUQ== X-Received: by 2002:a17:907:97c6:b0:abf:4c82:22b1 with SMTP id a640c23a62f3a-ac252738126mr428833566b.32.1741363186309; Fri, 07 Mar 2025 07:59:46 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:45 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 2/9] io_uring: add infra for importing vectored reg buffers Date: Fri, 7 Mar 2025 16:00:30 +0000 Message-ID: <60bd246b1249476a6996407c1dbc38ef6febad14.1741362889.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add io_import_reg_vec(), which will be responsible for importing vectored registered buffers. The function might reallocate the vector, but it'd try to do the conversion in place first, which is why it's required of the user to pad the iovec to the right border of the cache. Overlapping also depends on struct iovec being larger than bvec, which is not the case on e.g. 32 bit architectures. Don't try to complicate this case and make sure vectors never overlap, it'll be improved later. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 7 +- io_uring/rsrc.c | 128 +++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 ++ 3 files changed, 138 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9101f12d21ef..cc84f6e5a64c 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -111,8 +111,11 @@ struct io_uring_task { }; struct iou_vec { - struct iovec *iovec; - unsigned nr; + union { + struct iovec *iovec; + struct bio_vec *bvec; + }; + unsigned nr; /* number of struct iovec it can hold */ }; struct io_uring { diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index bac509f85c80..71fe47facd4c 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1269,3 +1269,131 @@ void io_vec_free(struct iou_vec *iv) iv->iovec = NULL; iv->nr = 0; } + +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries) +{ + gfp_t gfp = GFP_KERNEL | __GFP_NOWARN; + struct iovec *iov; + + iov = kmalloc_array(nr_entries, sizeof(iov[0]), gfp); + if (!iov) + return -ENOMEM; + + io_vec_free(iv); + iv->iovec = iov; + iv->nr = nr_entries; + return 0; +} + +static int io_vec_fill_bvec(int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + struct iovec *iovec, unsigned nr_iovs, + struct iou_vec *vec) +{ + unsigned long folio_size = 1 << imu->folio_shift; + unsigned long folio_mask = folio_size - 1; + u64 folio_addr = imu->ubuf & ~folio_mask; + struct bio_vec *res_bvec = vec->bvec; + size_t total_len = 0; + unsigned bvec_idx = 0; + unsigned iov_idx; + + for (iov_idx = 0; iov_idx < nr_iovs; iov_idx++) { + size_t iov_len = iovec[iov_idx].iov_len; + u64 buf_addr = (u64)(uintptr_t)iovec[iov_idx].iov_base; + struct bio_vec *src_bvec; + size_t offset; + u64 buf_end; + + if (unlikely(check_add_overflow(buf_addr, (u64)iov_len, &buf_end))) + return -EFAULT; + if (unlikely(buf_addr < imu->ubuf || buf_end > (imu->ubuf + imu->len))) + return -EFAULT; + if (unlikely(!iov_len)) + return -EFAULT; + if (unlikely(check_add_overflow(total_len, iov_len, &total_len))) + return -EOVERFLOW; + + /* by using folio address it also accounts for bvec offset */ + offset = buf_addr - folio_addr; + src_bvec = imu->bvec + (offset >> imu->folio_shift); + offset &= folio_mask; + + for (; iov_len; offset = 0, bvec_idx++, src_bvec++) { + size_t seg_size = min_t(size_t, iov_len, + folio_size - offset); + + bvec_set_page(&res_bvec[bvec_idx], + src_bvec->bv_page, seg_size, offset); + iov_len -= seg_size; + } + } + if (total_len > MAX_RW_COUNT) + return -EINVAL; + + iov_iter_bvec(iter, ddir, res_bvec, bvec_idx, total_len); + return 0; +} + +static int io_estimate_bvec_size(struct iovec *iov, unsigned nr_iovs, + struct io_mapped_ubuf *imu) +{ + unsigned shift = imu->folio_shift; + size_t max_segs = 0; + unsigned i; + + for (i = 0; i < nr_iovs; i++) + max_segs += (iov[i].iov_len >> shift) + 2; + return max_segs; +} + +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned iovec_off, + unsigned issue_flags) +{ + struct io_rsrc_node *node; + struct io_mapped_ubuf *imu; + struct iovec *iov; + unsigned nr_segs; + + node = io_find_buf_node(req, issue_flags); + if (!node) + return -EFAULT; + imu = node->buf; + if (imu->is_kbuf) + return -EOPNOTSUPP; + if (!(imu->dir & (1 << ddir))) + return -EFAULT; + + iov = vec->iovec + iovec_off; + nr_segs = io_estimate_bvec_size(iov, nr_iovs, imu); + + if (sizeof(struct bio_vec) > sizeof(struct iovec)) { + size_t bvec_bytes; + + bvec_bytes = nr_segs * sizeof(struct bio_vec); + nr_segs = (bvec_bytes + sizeof(*iov) - 1) / sizeof(*iov); + nr_segs += nr_iovs; + } + + if (WARN_ON_ONCE(iovec_off + nr_iovs != vec->nr) || + nr_segs > vec->nr) { + struct iou_vec tmp_vec = {}; + int ret; + + ret = io_vec_realloc(&tmp_vec, nr_segs); + if (ret) + return ret; + + iovec_off = tmp_vec.nr - nr_iovs; + memcpy(tmp_vec.iovec + iovec_off, iov, sizeof(*iov) * nr_iovs); + io_vec_free(vec); + + *vec = tmp_vec; + iov = vec->iovec + iovec_off; + req->flags |= REQ_F_NEED_CLEANUP; + } + + return io_vec_fill_bvec(ddir, iter, imu, iov, nr_iovs, vec); +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index f35e1a07619a..0d5c18296130 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -61,6 +61,10 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req, int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, unsigned issue_flags); +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned iovec_off, + unsigned issue_flags); int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg); int io_sqe_buffers_unregister(struct io_ring_ctx *ctx); @@ -146,6 +150,7 @@ static inline void __io_unaccount_mem(struct user_struct *user, } void io_vec_free(struct iou_vec *iv); +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries); static inline void io_vec_reset_iovec(struct iou_vec *iv, struct iovec *iovec, unsigned nr) From patchwork Fri Mar 7 16:00:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006694 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C75E222156A for ; Fri, 7 Mar 2025 15:59:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363191; cv=none; b=OtclzY7y82hndqdQC0U9uQVDf7MXZ+PY2GTEMwa0HuGVMYTO7//HB3FxLFPKzbKMU0wOR+iCrQpuPM7exV6hfm4j0GJyYM96VwYLXW0suHdWhVaU4oNN4I4XfT3jl9M7hcPma/kMABrpJWkEEUekXAkhU7crxdBcXEiRgM6i2SI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363191; c=relaxed/simple; bh=CTDOfIKLiLS36vq6sL2sgn5lKY5hbvt0oCtkq6B9W9M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VrrMcIVMt55dmdjqiKazMxHA/MHhPkFQ3iy76qRqcKsPdBLHwROKeRbUPmGRyt/a9pVB/ngAXMKgLRC2DU+CBNwatDLiaa2vuFzpdgyPfZB/GpMJX7GVFwrxrbOXwsjkUEVblwsv31EftzgfwU0uqtZD2fODVZw/vmVbAHIUroM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LnBRAuU8; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LnBRAuU8" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-abf45d8db04so309858566b.1 for ; Fri, 07 Mar 2025 07:59:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363187; x=1741967987; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=REKT0J5DTDfZxhTt80Rxhy0qO38B5z+7gPhcbN7SaII=; b=LnBRAuU8vQqN9zq0VQIzVq5tgjo698/8Nf8J3R3u1jjsfl+tg55k51NFqk2m/qi68D 76oWw0BgKgJCVTU1Q5X3rb5+OAN6NvPoLn5fqy/Wb46iQ7pRI/Ua4cOSDbeRqqhTojcD sSUe27Lg0v1IWWe8yLIXwwqfVV4CLy+w7rdj0FmxJx//y03Zmbh65A02cPySU+nkSnV6 ViR3VPs3sogKupsmH9ZieBTZ6IVqeot3rNuT9lY0Z3mZ5qGIxQZsn+R/B5qWnwmPCj6R WQPaEISDUmKw/whLVi6Y0RU4zRkzoLrnc/pOcMohG3GBcntuD8jvVmxEUXUktaOdn36D qHrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363187; x=1741967987; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=REKT0J5DTDfZxhTt80Rxhy0qO38B5z+7gPhcbN7SaII=; b=DbOXT4RBlBqD86zsOfHlqUeVTq+0B8DDfXAkZiN2vYaL+5rVegTuk1nP/XC4EXOke1 v6RZ/O8F5XQA3cbwbkriGJawGno91hAKln4mXq6K335zvGMUhnen/sp0hl3G98/Z1vEY gXQygy9nHCe05AnuG7xtoDE7arbIDI4dxnKII27KSbaqNkTzueg8rmaKrlIx+2rDVLJS S5qcr9tU2hZJdh13xdQnDZ1eMMQpGhgU+IdCinpQGDbXeUvmYKhnhpLHiy2PvQGLnDpI V53fRSbMFYTsfVjALewiMihHZHkYRFys9vwlSX8TKaay6nVbZ1+GwkVJtn9zxxS5wEm2 Fkyw== X-Gm-Message-State: AOJu0YzbCBgrKktE3jWPCu2V4G77+6xf8y4+PfjFhJq9SOLgOcVF6JPp 5aVfzR/ZNZj8ujv4o6k4onWBDnWuxGIrCNfJwUlERL8kT+RIb5XI5p4QrQ== X-Gm-Gg: ASbGncs7VEq8hbKFY/OQQazN7snLLWNJMJOp6oauqLXv022qOij006FZ5+oy6v5+cgq HGYp32CVgRcFlt5buv2E1IJt6KM4H9kmqK2hY6y47JhzNbymZkKGOcTqwu/Q5vLe8Tk4mwg8uRR BC0avHaIJWm+BduQ5HmZtOp5pcnnhIn5n5HvfBmvdKWceXzF5G5kRrmiLsfWzOmeqAyg/SJ12VX eov2ek/FJlbMJVEZwxalgYLO4nFotkoSNWo7W0Ki/8L1cWjLL2EE0NUUyoA56scqqirnpBw8xk7 Mr5tk+7YwlXTMBrA3t6DskkcsDRC X-Google-Smtp-Source: AGHT+IH99pHifUa/pfyYPQcoE2fd5F1+3SNbhJmNu/sgS6YPnCTS4AAkCm40ViUENemOL7XIPmXFwQ== X-Received: by 2002:a17:907:d25:b0:abf:4f72:538e with SMTP id a640c23a62f3a-ac252f42b35mr379317266b.55.1741363187284; Fri, 07 Mar 2025 07:59:47 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:46 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 3/9] io_uring/rw: implement vectored registered rw Date: Fri, 7 Mar 2025 16:00:31 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Implement registered buffer vectored reads with new opcodes IORING_OP_WRITEV_FIXED and IORING_OP_READV_FIXED. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 2 ++ io_uring/opdef.c | 39 +++++++++++++++++++++++++++ io_uring/rw.c | 51 +++++++++++++++++++++++++++++++++++ io_uring/rw.h | 2 ++ 4 files changed, 94 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 3d99bf9bbf61..9e5eec7490bb 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -281,6 +281,8 @@ enum io_uring_op { IORING_OP_LISTEN, IORING_OP_RECV_ZC, IORING_OP_EPOLL_WAIT, + IORING_OP_READV_FIXED, + IORING_OP_WRITEV_FIXED, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/opdef.c b/io_uring/opdef.c index db77df513d55..7fd173197b1e 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -540,6 +540,35 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_READV_FIXED] = { + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_readv_fixed, + .issue = io_read, + }, + [IORING_OP_WRITEV_FIXED] = { + .needs_file = 1, + .hash_reg_file = 1, + .unbound_nonreg_file = 1, + .pollout = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_writev_fixed, + .issue = io_write, + }, }; const struct io_cold_def io_cold_defs[] = { @@ -775,6 +804,16 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_EPOLL_WAIT] = { .name = "EPOLL_WAIT", }, + [IORING_OP_READV_FIXED] = { + .name = "READV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, + [IORING_OP_WRITEV_FIXED] = { + .name = "WRITEV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, }; const char *io_uring_get_opcode(u8 opcode) diff --git a/io_uring/rw.c b/io_uring/rw.c index ad7f647d48e9..4c4229f41aaa 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,6 +381,57 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } +static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + struct io_async_rw *io = req->async_data; + const struct iovec __user *uvec; + size_t uvec_segs = rw->len; + struct iovec *iov; + int iovec_off, ret; + void *res; + + if (uvec_segs > io->vec.nr) { + ret = io_vec_realloc(&io->vec, uvec_segs); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + /* pad iovec to the right */ + iovec_off = io->vec.nr - uvec_segs; + iov = io->vec.iovec + iovec_off; + uvec = u64_to_user_ptr(rw->addr); + res = iovec_from_user(uvec, uvec_segs, uvec_segs, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, 0); + iov_iter_save_state(&io->iter, &io->iter_state); + return ret; +} + +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_DEST); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_DEST); +} + +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_SOURCE); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_SOURCE); +} + /* * Multishot read is prepared just like a normal read/write request, only * difference is that we set the MULTISHOT flag. diff --git a/io_uring/rw.h b/io_uring/rw.h index 529fd2f96a7f..81d6d9a8cf69 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -32,6 +32,8 @@ struct io_async_rw { int io_prep_read_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_readv(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_writev(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_read(struct io_kiocb *req, const struct io_uring_sqe *sqe); From patchwork Fri Mar 7 16:00:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006695 Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8670E194C78 for ; Fri, 7 Mar 2025 15:59:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363192; cv=none; b=insEzv5OFoWDFPAVlxzE6XYEOJ1vAmJyRwo54AqzwrDZXD8FO7tggfm6WvfY1eqCUquGrLDruCDy9bKq+haBcYpS7eQdAvbjZEnB7GQozoquwzahMv0QKu00x0o0yYAajXBcXAXD0gImjGBXCpvwzlPVrASM6mUSmYbA0bWCFCA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363192; c=relaxed/simple; bh=gNKF/L5J2pAWM9hL6RvdhhPWEP4Yq0qat+0Q7PL2fuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oFjYftPfFm/30KjLUkPOwYPGYO5gtD9UGKyf2zEEes5NIrJYnZb46w8eh6hwV6ZU/Zn3CsmNhe/fgMN3W+2rMrDZUdXw3T2sFdV1Oq+Ooc+CDs+x0WF4VHYPxaMmafzc/PddyhaDzPdYn8ogR/ABIYpAgZHe2gFfbmQ2FSz9pK8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FJ0VZAY+; arc=none smtp.client-ip=209.85.218.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FJ0VZAY+" Received: by mail-ej1-f53.google.com with SMTP id a640c23a62f3a-ac2400d1c01so326436566b.1 for ; Fri, 07 Mar 2025 07:59:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363188; x=1741967988; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=93tvkzOu0v7DF9VuZKcubARhjargqjujprNt6uu0n0M=; b=FJ0VZAY+yvxL5mE1UhWRYElrcCEg91v/gq9szfnPdT3bw0PHVhJjPB5Fo/OaR2oVCo O0dMG2AZFjRHdyqWcXVtw/SiUz0eaq//jI0vYnRdyl/YmGbXBAzhMHtNlx4/TLcuQ4Zy ADmCiK5IfwjyymToUrOO5BoNJZgBViGjV2dzQWnDvc+aJM2SL1cQRi6h1f7f+j/AG3X8 zO0Noi1sjWc6P5lkLJh7SsboBniyRux3nq9/3JsW5Ilp0aIRzysDgykPcxGGlcMMEPCi jw4WEvoiHcz1UoaN+y3f+j1lyEOoM4IHLXeXhxxjsEGYXf0quMk7EkHic+IwYjD+cvp6 3+gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363188; x=1741967988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=93tvkzOu0v7DF9VuZKcubARhjargqjujprNt6uu0n0M=; b=cVeJQVCOS7xZG3fYcIoMqNyndm37vzEeChTpXyB0gyqG5BwBo8UVInjnVxQhNJIWap IlFcbTeGLpPQrB9R3xpd9UiT0Ho99YEXME5sOoJe9hKbu7iUk9ZJkg092YxUcT/T9t9/ u0krsc5kVbMfnIAwtK1e46pIwm8i8mxOefJBvnr0C0hOLe1d8/B/1BQ2iVvna0UpAGqe 2X9sc6GZA+DLhwsC/b844dls5qtYL1fwk61k6tDCmPKh43FgHx4FpdFApPINXr7IUvHh PdOWRUiujaEaEIkqbsqkfhP62MRF4q+LbvyWcSM6vUNpMd+o5ibwzs5QmAsXzwrJPvSG AYQw== X-Gm-Message-State: AOJu0YwLCL22TEJ1BEgn4dh/kpBSHv/2rE1Vw8CBjWENPPkdV5vMMBwA Q1zwoHz5ujGtCMjbJ3vhMTKnmeRwG3jkOSVjk/Y1ghJ5B8v1MOVvkeBgYQ== X-Gm-Gg: ASbGncsXiVJN/ujtiIpJ53rm91Un+hA6yrbUOOVHAlBaZ9olB4F+0AU9hUS2/TJBvbO FOEo3SJ96CYuQFOd/U6QDDTae1pnbfUQznlvlKi7rut9i/ffTzy9QLTu+xYUA1jrPOsc7hsar1f 7PFQH7DufAjtsg9F89c6t3+TRn6E0eNQg3pEXwB4PIgvxxrnGQU+hoZO0rAya7bOyYj7K/lPfms 5mi/ofufQAbmhyM/PSS3c08gePD26T0fC1azAaJizbue7c0NfvqWR06SMxxHnVy7fX+vrJL86V7 n7FUGY3kgryIC+fNJW+rv4oWkaHK X-Google-Smtp-Source: AGHT+IGV9eH1SpNxkOV4jv5qlfSbSVvRiVruu0fnQeARy7WuDeD+rNa1ovI/PzduZEvIgsPB523Eag== X-Received: by 2002:a17:907:84c1:b0:abf:68b5:f78b with SMTP id a640c23a62f3a-ac26cc4b0bamr2294166b.24.1741363188173; Fri, 07 Mar 2025 07:59:48 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:47 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 4/9] io_uring/rw: defer reg buf vec import Date: Fri, 7 Mar 2025 16:00:32 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Import registered buffers for vectored reads and writes later at issue time as we now do for other fixed ops. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +++ io_uring/rw.c | 42 +++++++++++++++++++++++++++------- 2 files changed, 37 insertions(+), 8 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index cc84f6e5a64c..0e87e292bfb5 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -502,6 +502,7 @@ enum { REQ_F_BUFFERS_COMMIT_BIT, REQ_F_BUF_NODE_BIT, REQ_F_HAS_METADATA_BIT, + REQ_F_IMPORT_BUFFER_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -584,6 +585,8 @@ enum { REQ_F_BUF_NODE = IO_REQ_FLAG(REQ_F_BUF_NODE_BIT), /* request has read/write metadata assigned */ REQ_F_HAS_METADATA = IO_REQ_FLAG(REQ_F_HAS_METADATA_BIT), + /* resolve padded iovec to registered buffers */ + REQ_F_IMPORT_BUFFER = IO_REQ_FLAG(REQ_F_IMPORT_BUFFER_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, io_tw_token_t tw); diff --git a/io_uring/rw.c b/io_uring/rw.c index 4c4229f41aaa..e62f4ce34171 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,7 +381,25 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } -static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) +static int io_rw_import_reg_vec(struct io_kiocb *req, + struct io_async_rw *io, + int ddir, unsigned int issue_flags) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + unsigned uvec_segs = rw->len; + unsigned iovec_off = io->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, issue_flags); + if (unlikely(ret)) + return ret; + iov_iter_save_state(&io->iter, &io->iter_state); + req->flags &= ~REQ_F_IMPORT_BUFFER; + return 0; +} + +static int io_rw_prep_reg_vec(struct io_kiocb *req) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); struct io_async_rw *io = req->async_data; @@ -406,10 +424,8 @@ static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) if (IS_ERR(res)) return PTR_ERR(res); - ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, - uvec_segs, iovec_off, 0); - iov_iter_save_state(&io->iter, &io->iter_state); - return ret; + req->flags |= REQ_F_IMPORT_BUFFER; + return 0; } int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -419,7 +435,7 @@ int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) ret = __io_prep_rw(req, sqe, ITER_DEST); if (unlikely(ret)) return ret; - return io_rw_prep_reg_vec(req, ITER_DEST); + return io_rw_prep_reg_vec(req); } int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -429,7 +445,7 @@ int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) ret = __io_prep_rw(req, sqe, ITER_SOURCE); if (unlikely(ret)) return ret; - return io_rw_prep_reg_vec(req, ITER_SOURCE); + return io_rw_prep_reg_vec(req); } /* @@ -906,7 +922,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret; loff_t *ppos; - if (io_do_buffer_select(req)) { + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_DEST, issue_flags); + if (unlikely(ret)) + return ret; + } else if (io_do_buffer_select(req)) { ret = io_import_rw_buffer(ITER_DEST, req, io, issue_flags); if (unlikely(ret < 0)) return ret; @@ -1117,6 +1137,12 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret, ret2; loff_t *ppos; + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_SOURCE, issue_flags); + if (unlikely(ret)) + return ret; + } + ret = io_rw_init_file(req, FMODE_WRITE, WRITE); if (unlikely(ret)) return ret; From patchwork Fri Mar 7 16:00:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006696 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0111E221700 for ; Fri, 7 Mar 2025 15:59:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363193; cv=none; b=XsWTj/kJPEuk0GSLOQnlb1BqrXuMCaachqLuC29zpE8f2rsqSqiu6Quy/EULXU/V3NylVym5QKGLJ1np/CIsSbp4SwrLhwt1vcSI3a3fB/8Yy6Xr+aiCl1LG5rd8D6RyvNVaUCWfngyCFSEbauIhU0p81RJQv2CR9ooO4B80rAs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363193; c=relaxed/simple; bh=q/FZLY6NX7KPIL9q2sRYyOUjPSfSZOW4huImrRHQGAc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NKxa1cJFI2EM6/A6KVMO2iBBKzHKU5rubxeG6CsmKjsG+pumefOpFS+ULZi1+Cguw947hMlr8w24gkZmyla373fvvTD8bRr3WVAHmjV4GTs+dzfVPFSqyB2NjlO+2t+Khe8qyehiCmDqr8/phaXfwx8gGhF0C2uqrvDOEiHJt1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bufB9Tsj; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bufB9Tsj" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5e4ebc78da5so3783903a12.2 for ; Fri, 07 Mar 2025 07:59:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363190; x=1741967990; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=bufB9TsjEUcNygk9ED7RNbWJTPvkuRC80AIVWXAVX9P47HgnTd4HVS61GWfudUvUjW 95+i3Ea5Tljdth0NJdi2hR2882EKRWr8ohPfkiYs877VBI+piyD2cqqlVy8rfq/EkREW nZcU78SeS0ZwAdrMMponUCAmG+t93W8IW3Pv9Bfd3qq+TQOSQkvRjqceMzrshv4RRjyF MBiTivr073aPWEmGsO6zmGqWnAUWsF9qyOBXkerp7GQbLhEoCqCfuJ0qGs828BnMmeYg kUGeLFB/onoWOTZmrbTB/md/LNy5KHvoBZfoSeyZEMUnhuOmNTJ2ZXLBLDlpZ0YZItSS K9cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363190; x=1741967990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=NAMGwdpjVFUGJEhKZ7ZCpJ5AQhS9d/krw2PKrwOpSJOPJnheovczmeiIUSB5U+Q3Js uYPH6gbc2enhqKceBLTc+IJ6LpZIrLmcg9smw5bq8mJvRCpsGPUtymafAeloaf0bPklT P+kcDqhfvySfjltZEsvo15oJT/PvAz7Xb2TdqnmEvGO016BUg/O3kMr8z2h6mxa1gce3 Ltuxc5tN832BZ2RnXVweexiwiOq2FlOaQMJFUmbhuIjKZ8EtQk8AmEzK/twR/dg3EGyg eTpK5qTGhk3x/X1Xcs9VREiS5FW4oFGsYkAOsHDLGEO4WcRqrLJ8pWl/9LCaX69XezwC CU9w== X-Gm-Message-State: AOJu0YwkIvHe0VMyjBa3R96r7bdRBhuk71vZuIzpT4ye4ziEi33MPNHY 9zUta2OmCoM+SJcb+BIv28SC+RqgdjIznoAfXkNng3ADsB3z5OjPvz23XQ== X-Gm-Gg: ASbGncun5efxW4i+vyglAI/Jh9XKOSYgrUSrxQClptDzN5ve0fMz7LErbL26jSSo2G2 Xc0+Xa4By8cvYMDkdvPPBnU8ziJ34Dq/Ffd8KwK72Qmf6CbKdt/9UA7xTxBdaLNrroJwSLvSyZH xJY7XWyj+D1icENBobc7MI52KL6YCUpM04FAoRh/ZBn7O6D+ef5czayomlUCEO06sNRmGpEXsTn voX/2CAZf1JHVi6eRy/Pc7jeLfMtuo3Bao6uCKF2RxqDLchdiImrGsVTLySbAVVL7/+5ntiFu/L J6T6cMGGCVMMh8VBYGUqECAX80Ht X-Google-Smtp-Source: AGHT+IESAV9FwjXI0vCPxaqBaKD+zN4bGui3c5TxUArk14HeoW1pYdOWE40jQPjX3kFQViZzelhinQ== X-Received: by 2002:a17:907:1c1f:b0:abf:72c1:6e6c with SMTP id a640c23a62f3a-ac252ba1906mr475468066b.45.1741363189458; Fri, 07 Mar 2025 07:59:49 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:48 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 5/9] io_uring/net: combine msghdr copy Date: Fri, 7 Mar 2025 16:00:33 +0000 Message-ID: <25795660f7b31f9273911c99f495d9c2b169ecda.1741362889.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Call the compat version from inside of io_msg_copy_hdr() and don't duplicate it in callers. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 905d1ee01201..33076bd22c16 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -287,6 +287,24 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct user_msghdr __user *umsg = sr->umsg; int ret; + iomsg->msg.msg_name = &iomsg->addr; + iomsg->msg.msg_iter.nr_segs = 0; + + if (io_is_compat(req->ctx)) { + struct compat_msghdr cmsg; + + ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ddir, save_addr); + if (ret) + return ret; + + memset(&msg, 0, sizeof(msg)); + msg->msg_namelen = cmsg.msg_namelen; + msg->msg_controllen = cmsg.msg_controllen; + msg->msg_iov = compat_ptr(cmsg.msg_iov); + msg->msg_iovlen = cmsg.msg_iovlen; + return 0; + } + ret = io_copy_msghdr_from_user(msg, umsg); if (unlikely(ret)) return ret; @@ -323,18 +341,6 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_SOURCE, - NULL); - sr->msg_control = iomsg->msg.msg_control_user; - return ret; - } - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; @@ -710,21 +716,7 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_DEST, - &iomsg->uaddr); - memset(&msg, 0, sizeof(msg)); - msg.msg_namelen = cmsg.msg_namelen; - msg.msg_controllen = cmsg.msg_controllen; - } else { - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); - } - + ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, From patchwork Fri Mar 7 16:00:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006697 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9F95221708 for ; Fri, 7 Mar 2025 15:59:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363194; cv=none; b=AjZaCpf30t4xJjP6ltMOjiJFB/HUzfEQSWiiLhVK+yWhUSnSOlX2MuFwiirAB2FYBaRfL/USM4aqvRmm3RFvXmCyRTOPdThgJBjTGY8Y1XSnxYaVem8VKCtH1IaDgDeu0O8kU+YxuaG5qKvC5TOr8tEexf7yn209ajihOUTHGn0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363194; c=relaxed/simple; bh=dFLQOj2ZLIbINKKwcv5tlrqtAVZRjLUjvTJ5sPCh7Go=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sCf4XcaFjXSzBXwHdrV1fr6rD6rhPPLwRo93AJ0DarDuiF0IuKSjwrLjwXIi3UtX5pL9QqWSZd376B5OrQqfTAhD3jVItR9lcTECswq9BTXRZA+CxPAMEDHVnATbXstHOs5K/DfB00RXSwLHnyAFL19OMqeqkrtcU12OOZ2haMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nYyzi0oB; arc=none smtp.client-ip=209.85.218.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nYyzi0oB" Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-ac2400d1c01so326444966b.1 for ; Fri, 07 Mar 2025 07:59:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363190; x=1741967990; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=nYyzi0oBTaqufiDXhi29AitTWF9bzojZ7i7dyKRGURaQ29PhTQlbQORHAObZTJoac6 b7JYKW4U/F/Jgy55ukKMy6A+Rr4eAHwG513IDeULR4MktbNFK/Sn8A4zD3obuIQ8l9za WTZQrH4HzUdF4/rJLewNajjht+7PyzQc47G0NhK/INUlis4jrNHrqdrMzxxWl8xwiDJB rJrSa7QHzTTrAd5v+M4QnMBN9dfjhtPEHnCsZf/jww6bHZw7x0u9KFfGyeOYn02zHlcH fAUWKDmVfn2ElUSsiXlkUW4Ui9h+BHenDXqLd99ADSNIC151Iy/69bm9wh3kTH4Z5QJ+ nfmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363190; x=1741967990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=I69J+e1nP7JRHbSivUg+CgC0SBjVSxKl39IMsUiqy40mjqfEiWgFK06A03mxiOd4Wr tWUBw2yCSaWWaZqRR4NBGpTf8XPZjDBXADQ7fExvSWV3chw7IyXiRa2KSsjAXBAdjU6w ERm+jYzHQHiayH3tnao12Be0Gfd2r+doCbF6stId5+kpL1v059Kf7UVHPNDrIZlIx0j5 j97mvGhGVZDr6JmxZYGBY8Lyw1DCqNKG2QT0Tl6aoxA8JondO7TjCbXi/0Ei3SmgM91y QU3WwHSZ8cepOKEOAdxETQF7SVQlcGQZrqkZjL9rlrV0sxtL09giZzisKTjGIbYO2dWi Irgg== X-Gm-Message-State: AOJu0YyGvWlftp61RJvvxFmvWQIauURULAhAU3drFeD21y+g87nTzSSW njGA8FknRwiihb1Kk9liKaiU6Zq03ZH99U7lMbOpBkF9s1VhJvYo3n2/yg== X-Gm-Gg: ASbGncuFF0UwtwaPZnOc6DQ5kWOWasnCl9iI8hb/rj20FL9F3u8dc0xHwVhTW2gQ9r5 2FvFOfJ9+E5Va5Hcp+ib1PvTdlE9pjs9Kx8Jc1YZqS2pqMCcD5bsbcnTAx7oKmuPF649Utretrj miGtMvCY7I06DIUyC7JytToXPqSGWNxyo/3adVR6X+38YIZZzwbDDGOSu/K5r6Cp9hu+wS5wk46 jT2rPeImq7CMCxAX1IeX0tIbEqNX60XFpUIbag5oHsAXdDg2ZWLD1thERhFbfvhn0B83qH3sGi5 2Je8nTB1/LwuUe8Au/IG8dEF/O0U X-Google-Smtp-Source: AGHT+IHREUlUgY1xD3cb8snnrqKyFaNBjxZiyrxPDgvRG5l9cYyvNjCdFc+b4lp058/Dmd9FMIu8Xw== X-Received: by 2002:a17:907:7f23:b0:abf:4a3b:454d with SMTP id a640c23a62f3a-ac26ca31a38mr6506766b.7.1741363190353; Fri, 07 Mar 2025 07:59:50 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:49 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 6/9] io_uring/net: pull vec alloc out of msghdr import Date: Fri, 7 Mar 2025 16:00:34 +0000 Message-ID: <9600ea6300f620e65d39da481c22605ddc898850.1741362889.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I'll need more control over iovec management, move io_net_import_vec() out of io_msg_copy_hdr(). Signed-off-by: Pavel Begunkov --- io_uring/net.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 33076bd22c16..cbb889b85cfc 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -253,12 +253,8 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, return -EFAULT; sr->len = tmp_iov.iov_len; } - - return 0; } - - return io_net_import_vec(req, iomsg, (struct iovec __user *)uiov, - msg->msg_iovlen, ddir); + return 0; } static int io_copy_msghdr_from_user(struct user_msghdr *msg, @@ -328,10 +324,8 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, return -EFAULT; sr->len = tmp_iov.iov_len; } - return 0; } - - return io_net_import_vec(req, iomsg, msg->msg_iov, msg->msg_iovlen, ddir); + return 0; } static int io_sendmsg_copy_hdr(struct io_kiocb *req, @@ -342,6 +336,12 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, int ret; ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_SOURCE); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; return ret; @@ -719,6 +719,13 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) { + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_DEST); + if (unlikely(ret)) + return ret; + } return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, msg.msg_controllen); } From patchwork Fri Mar 7 16:00:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006698 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAE42221713 for ; Fri, 7 Mar 2025 15:59:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363195; cv=none; b=rSceNQL4NqrqyaNb2LTyzbQ02bsgskASWYgF+F/HbsSHv1vxjT62TerCYVe3NUwZptGmMcSSfbTlxGFz1Kpml6PfXa7vLV/qzbzjdu7KQJ1/o/vJ0cp1HQqdI/rZNIfNPyzmA/8WlJzj1F7stttwp4Gn5Eq/Dl0rDUPL8qCLY0Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363195; c=relaxed/simple; bh=hdM/yveeqx/cwJF39LxfDEQxflleuBmQE+zj5qDwnIg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N6/+tnfrRrVzxm52o2H4Wct015TzeIKBIxCLtTsC5FhAJ6VJjhGYuSueTJg5LtLXT2eWnCCymMC2nX5vGsq2UEPB5Jy4n5nQLPq/OyC206OXEts3bIpfPy/6ksxAeMPRHGt3IRRPHDd0Zjyl8vdsn5Uac4BwiruzsgHcqFNfIJE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eS87PlY+; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eS87PlY+" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-aaec61d0f65so405653266b.1 for ; Fri, 07 Mar 2025 07:59:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363192; x=1741967992; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Zac3vga/E698ojFWUHp/OWmE620rH3affaJufXu525s=; b=eS87PlY+V4O6hXRw3kLvC6XBw0IeQl+xPcVGZMorXTV3JdwElyChd1yD1MlieGZ7om z6bn1pYM/IGYjgFzUVkUcNIHErpflPMH4fq2uwVj9YUTRR2sXwLqp4t/LeSnzweqW3aJ dehflWq1SUVTaAnJQ1a9ymsxI9joZPBCXCRz23p9nbdK9EfB1x+BpqvrlsF08iU2BqTB ILN+KHY4j+ZDyTVuBT3cN0ThirbFE/SEqjkKqMh2zC1yfLdVsDJ6YtIvn+ABbTwmAPbJ Wk+J5t/VUoa5xWuwTTpd0RcU+e48vVM0DreWCE5DQ0JcnF+onvX0BhPGEKekEk3sm1Lp wX2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363192; x=1741967992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Zac3vga/E698ojFWUHp/OWmE620rH3affaJufXu525s=; b=s76mSIYB4E6Z89gYuQxeLiXBbP1uB0EGzWe3YpZOpsgMXKsw12ZbdCZiC25yihAXsI n59azRTbFC91+KCXkQX1hRZegAUYQOtwOzImNqE6yGwIeupA3jQpvYN+3abz8lYQqEks s5fkNhdayrM8uTOp/BElFWfVKZ0jh3GQUARv/owoBYnULyorx/iReplCyO8428nlqDvz fgbaXbnZxEnLbCMcR6a3kxhCWyleqAW6sPzFGf1bSOrRKoYYZ77BegYxmgI62rnNWOaV 1FD/4bWXs6eInKGUCyE0yI2kLzC6C4OEK0b9JlViP1BSg6+eSwaP8f9aM3XF2cFcWqC+ 3hAA== X-Gm-Message-State: AOJu0YzXwLEqAkiKLHI641YRsNxB7UeDKf2jxj7U6zouXHKUtTAPBtPL fOwOBUEAKLVEzSTfv6GoE7qKODsmEVWVXj5LEk22v/4NbZBb13waxII+6Q== X-Gm-Gg: ASbGncuFiubr0lVVXlZOznhMODbPvPnqgMyY/d0pqr6gUgv0L7VxLkobt5np3M+zLYu cU2kvNAWM0d5hYkvaB+DJqupzKy5xOiioFGtI62ekQaHCyvchHFUN+5o0qrnh/PiFa3gsEFlvOi IJmUQblbONObdfg5iAtBZh3fLL2hHvRh61M5EPWEKZgx53Ec7OjFqMKBLUGN8Q+hTrmc+NBp7P5 T8dayIXJ/ZyW9pNrPkt5KVnOA5uW79r47alXpSAQdLLX4keqbaGv8vTDUgevazJlVYeSjz+dESa ECo6UeqV6WkCPj0ynM6itCO4beaX X-Google-Smtp-Source: AGHT+IGBb9pG0RCCgyU60lVLo5dZwJoyA2mHQGei0kcapQYDJgg6adu3zgeTHCyTto1FGB2O6TDiMg== X-Received: by 2002:a17:907:3604:b0:ac1:dc0f:e03e with SMTP id a640c23a62f3a-ac252628a87mr488727266b.13.1741363191529; Fri, 07 Mar 2025 07:59:51 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:50 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 7/9] io_uring/net: convert to struct iou_vec Date: Fri, 7 Mar 2025 16:00:35 +0000 Message-ID: <6437b57dabed44eca708c02e390529c7ed211c78.1741362889.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert net.c to use struct iou_vec. Signed-off-by: Pavel Begunkov --- io_uring/alloc_cache.h | 9 -------- io_uring/net.c | 51 ++++++++++++++++++------------------------ io_uring/net.h | 6 ++--- 3 files changed, 25 insertions(+), 41 deletions(-) diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 7f68eff2e7f3..d33ce159ef33 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -16,15 +16,6 @@ bool io_alloc_cache_init(struct io_alloc_cache *cache, void *io_cache_alloc_new(struct io_alloc_cache *cache, gfp_t gfp); -static inline void io_alloc_cache_kasan(struct iovec **iov, int *nr) -{ - if (IS_ENABLED(CONFIG_KASAN)) { - kfree(*iov); - *iov = NULL; - *nr = 0; - } -} - static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, void *entry) { diff --git a/io_uring/net.c b/io_uring/net.c index cbb889b85cfc..a4b39343f345 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -136,11 +136,8 @@ static bool io_net_retry(struct socket *sock, int flags) static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) { - if (kmsg->free_iov) { - kfree(kmsg->free_iov); - kmsg->free_iov_nr = 0; - kmsg->free_iov = NULL; - } + if (kmsg->vec.iovec) + io_vec_free(&kmsg->vec); } static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) @@ -154,7 +151,7 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } /* Let normal cleanup path reap it if we fail adding to the cache */ - io_alloc_cache_kasan(&hdr->free_iov, &hdr->free_iov_nr); + io_alloc_cache_vec_kasan(&hdr->vec); if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -171,7 +168,7 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) return NULL; /* If the async data was cached, we might have an iov cached inside. */ - if (hdr->free_iov) + if (hdr->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; return hdr; } @@ -182,10 +179,7 @@ static void io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg { if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - kmsg->free_iov_nr = kmsg->msg.msg_iter.nr_segs; - if (kmsg->free_iov) - kfree(kmsg->free_iov); - kmsg->free_iov = iov; + io_vec_reset_iovec(&kmsg->vec, iov, kmsg->msg.msg_iter.nr_segs); } } @@ -208,9 +202,9 @@ static int io_net_import_vec(struct io_kiocb *req, struct io_async_msghdr *iomsg struct iovec *iov; int ret, nr_segs; - if (iomsg->free_iov) { - nr_segs = iomsg->free_iov_nr; - iov = iomsg->free_iov; + if (iomsg->vec.iovec) { + nr_segs = iomsg->vec.nr; + iov = iomsg->vec.iovec; } else { nr_segs = 1; iov = &iomsg->fast_iov; @@ -468,7 +462,7 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret) if (iter_is_ubuf(&kmsg->msg.msg_iter)) return 1; - iov = kmsg->free_iov; + iov = kmsg->vec.iovec; if (!iov) iov = &kmsg->fast_iov; @@ -584,9 +578,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, .nr_iovs = 1, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode = KBUF_MODE_FREE; } @@ -599,9 +593,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, if (unlikely(ret < 0)) return ret; - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } sr->len = arg.out_len; @@ -1085,9 +1079,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg .mode = KBUF_MODE_EXPAND, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode |= KBUF_MODE_FREE; } @@ -1106,9 +1100,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg } iov_iter_init(&kmsg->msg.msg_iter, ITER_DEST, arg.iovs, ret, arg.out_len); - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } } else { @@ -1874,8 +1868,7 @@ void io_netmsg_cache_free(const void *entry) { struct io_async_msghdr *kmsg = (struct io_async_msghdr *) entry; - if (kmsg->free_iov) - io_netmsg_iovec_free(kmsg); + io_vec_free(&kmsg->vec); kfree(kmsg); } #endif diff --git a/io_uring/net.h b/io_uring/net.h index b804c2b36e60..43e5ce5416b7 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -2,12 +2,12 @@ #include #include +#include struct io_async_msghdr { #if defined(CONFIG_NET) - struct iovec *free_iov; - /* points to an allocated iov, if NULL we use fast_iov instead */ - int free_iov_nr; + struct iou_vec vec; + struct_group(clear, int namelen; struct iovec fast_iov; From patchwork Fri Mar 7 16:00:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006699 Received: from mail-ej1-f47.google.com (mail-ej1-f47.google.com [209.85.218.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7924F22156B for ; Fri, 7 Mar 2025 15:59:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363197; cv=none; b=BzPhmyEmz0CXy8rEyZQ0sD16auvMODCRPGpPfa6TxxVS63bE4L8Z9uQJvml3x65Im8f+P+eEPGdKNbfbL/b9tH/4Yi38h9vWe/9G9aypVCU3pPmoayuutjk0qurLPk7f+P2mP6jDN8FoHGkbqHK+ul0HQW6A/ewTdllWayxTcdI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363197; c=relaxed/simple; bh=+ko6qVLwFbY8p2fCkuMHNRIwwTZQjN0AL9uVp4qcr08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KaxMpjg/pOw64o3yiMJD6lYPDPZKhft2wRESGiSLl2OUVxXaldzZ83tJKj8lJqjm7ElRlYc6SMw9sa6uxHhpTBuiKa2LJ5OVngrB3EKyNF8OOtDqsSJdokNGLZhhmgFT+RX7DFnT/+BFCpdbXTZMbSPWjQiOwkliUwpxtF3z8uA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=c7ffU+wz; arc=none smtp.client-ip=209.85.218.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c7ffU+wz" Received: by mail-ej1-f47.google.com with SMTP id a640c23a62f3a-abf615d5f31so380925266b.2 for ; Fri, 07 Mar 2025 07:59:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363193; x=1741967993; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=c7ffU+wzkCUeF/8p726acNqKHgYa3BsIz8uzk1YG6SB54kXJ6ISpKm3a3JwGQbEP2u p3SffSHc331P3Qsil1ZiEe5glsjrRrAKjcB/6iW1s9PGytPQgnfv3R75ZZlxHQtK1iTh GJRwIHkWkayqjN69N+Kun+T7/7Ex4b5P9Jiw2aQ4g+8sl56rDrqXG+/8U+kPSGvdlo7x 4eLBTPfemuRdJ+RAKY8GY+md+MCkcNomCqYDSl9wolS9ZLZj/eGjeSqQZ+6ztaftKRMP cwpBUH2t5QtR8TVfIlxj9tzruKbkThGvuF0ksU2kckhTYgRhZIzGmRkeNGkBzBvdVyyr YWZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363193; x=1741967993; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=bQ8F3yi+6pRp1x9nO0fzvqIR6pCLlJ+W4CgcLZ8Acq1AB8MsEfzGkPb0HOuKbJ24wA pWf8os42rDvtXsI8A4Pyl+/oiafXH/bZYhD8aHG/4LqJPNSYLlCVrxzh8PrY9f94e4za PUA0XdmF2sEHpQH8rOAAJR8hh7MlGpfO6JvDm3/cI02hKg8lXKMiRSPS+L+yCwOIjYXU 7l3xw3dIWBPS8FLHSGi+OHimV+w666tlR3QgbwbWW5eDMddJ2NR+ho1IhYX6v3995RSi /30hGqeYSJ1v+ybQ+dl2NZD9wTQtr/nrd5m9EGeZfXEAYE4zzwmnoxzgZJD6hGpSoESh 4/FA== X-Gm-Message-State: AOJu0YzFN0v92RwNlQ+K3F2k1QpKuGIQD+9taUtNBA/tYKkSt10Dgvfd Q78Ejqn85lZRqwtkmonTkrlh7Qs5C3DV6I7caNo8LO2+qSUrMVeFTNMdCQ== X-Gm-Gg: ASbGncsLyJ2k63y2f7AE5OJ+H7Ddg00CCmbwgTiCperu9XnOGWLPGIOTfhrUmF4d5M2 FDbl2y7priSIE/bNw3fiVKBTnWiB0QOFz0vn2upJ+oBl8iS02f5SJkbP/eMug1yapY3ed1LNf8V 2s93OdwcTVocPlwdlt2KcZle+fD6ZbuFoF2yLxltatrB23YO3xOfSAmssKYt7ZsTR2n91861H/y RJszpRCXlE0XMtHmPlMaeuBYSGVlxfzYfPLCTYt1c6jZccO/ZE4IonFZ08WcrnivYhXWjbCCTyC Ud5Vgvxv8uPSAs3ba6I55Yh7yWFS X-Google-Smtp-Source: AGHT+IEJwFn4cbFZCXEtmv6uj8zTX03Aq/LaWQ0s1B2IxViGBe8PpdOB/EtXCA8vQqsWu9YOSR+9RA== X-Received: by 2002:a17:907:158a:b0:ac1:e0fd:a81e with SMTP id a640c23a62f3a-ac252630d02mr317185166b.21.1741363192770; Fri, 07 Mar 2025 07:59:52 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:51 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 8/9] io_uring/net: implement vectored reg bufs for zctx Date: Fri, 7 Mar 2025 16:00:36 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add support for vectored registered buffers for send zc. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index a4b39343f345..5e27c22e1d58 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -395,6 +395,44 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe return io_sendmsg_copy_hdr(req, kmsg); } +static int io_sendmsg_zc_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg = req->async_data; + struct user_msghdr msg; + int ret, iovec_off; + struct iovec *iov; + void *res; + + if (!(sr->flags & IORING_RECVSEND_FIXED_BUF)) + return io_sendmsg_setup(req, sqe); + + sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); + + ret = io_msg_copy_hdr(req, kmsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + sr->msg_control = kmsg->msg.msg_control_user; + + if (msg.msg_iovlen > kmsg->vec.nr || WARN_ON_ONCE(!kmsg->vec.iovec)) { + ret = io_vec_realloc(&kmsg->vec, msg.msg_iovlen); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + iovec_off = kmsg->vec.nr - msg.msg_iovlen; + iov = kmsg->vec.iovec + iovec_off; + + res = iovec_from_user(msg.msg_iov, msg.msg_iovlen, kmsg->vec.nr, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + kmsg->msg.msg_iter.nr_segs = msg.msg_iovlen; + req->flags |= REQ_F_IMPORT_BUFFER; + return ret; +} + #define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE) int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -1333,8 +1371,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->opcode != IORING_OP_SEND_ZC) { if (unlikely(sqe->addr2 || sqe->file_index)) return -EINVAL; - if (unlikely(zc->flags & IORING_RECVSEND_FIXED_BUF)) - return -EINVAL; } zc->len = READ_ONCE(sqe->len); @@ -1350,7 +1386,7 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return -ENOMEM; if (req->opcode != IORING_OP_SENDMSG_ZC) return io_send_setup(req, sqe); - return io_sendmsg_setup(req, sqe); + return io_sendmsg_zc_setup(req, sqe); } static int io_sg_from_iter_iovec(struct sk_buff *skb, @@ -1506,6 +1542,22 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) unsigned flags; int ret, min_ret = 0; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; + + if (req->flags & REQ_F_IMPORT_BUFFER) { + unsigned uvec_segs = kmsg->msg.msg_iter.nr_segs; + unsigned iovec_off = kmsg->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ITER_SOURCE, &kmsg->msg.msg_iter, req, + &kmsg->vec, uvec_segs, iovec_off, + issue_flags); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; + req->flags &= ~REQ_F_IMPORT_BUFFER; + } + sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; @@ -1524,7 +1576,6 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_control_user = sr->msg_control; kmsg->msg.msg_ubuf = &io_notif_to_data(sr->notif)->uarg; - kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); if (unlikely(ret < min_ret)) { From patchwork Fri Mar 7 16:00:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006700 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 921AF221712 for ; Fri, 7 Mar 2025 15:59:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363198; cv=none; b=RtH2lsiz817aaYJhLTOiXA/PUE1mGhSxWwtPGi6DerMNXdqclV+59+PntSUD3jNmqgSJh+4ZjX3/v0dTBWOqT1SQVI+CEkHCEJRFrvfuGruBEj/nYc99W9900oktWX2WxEXoz3eNFR1kNlmfecjgrBZiledUIgTOy/X0faNBLOM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741363198; c=relaxed/simple; bh=c43Svu1iU7sMmAeZKrfwQmrejzQqcdDB09RJDv98Y9w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BYRkL18FL7f7MF0M70dB3HX8ULYRA5BMsmXiBLCzN4W3WrAO9OcRnomCOJ1JYMBMV8Cioxq5Lhx2biay0H0sOkg+fMRaIuVwjrhes8UV8K4E9TaSv1RvbblvQrYLyHcwstG0xrHMDZ2a7OmWZXa2HNBBTm5ZsUfJ8/gDOzyVjjk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mceVADc+; arc=none smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mceVADc+" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5e5e1a38c1aso1142671a12.2 for ; Fri, 07 Mar 2025 07:59:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741363194; x=1741967994; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jl4Ox8E80bgKYyEccBIF1Yq4r8LoalAtGCisRYqp/Z8=; b=mceVADc+9y+dohvo6/nqQ1/ekZiQGiEZtfn0e/YnKZQZWEYPCLdvBh69gOkRbPz4Vb uciIVKNk4GwDhHP0soj7I3SK4EsHVRqmhk0DT+5KP8kIfytKXe1aH1zUc1dymrUIOufZ s08/UKsaZQoXFwPis6cIHES7VTFUDyIMUl7eidthtS+QHqLaTOlho+kiQfb7ms5awOKC cbwZg8la82WEOYoWrqfjKUo3o3FCUPIROw1xaVj+zYpNVRtpd11gnrf95Wrq91+lmwc6 e2NBtbsKK2uppeUn+cpK4EUXeN10C9kEljQTS4uhFXLoDWVY8lgtailJPrFOb/KaBydI Zc8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741363194; x=1741967994; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jl4Ox8E80bgKYyEccBIF1Yq4r8LoalAtGCisRYqp/Z8=; b=HuKo6ro93g4T0ZcLnGG6yk8KNsndFVLStI/vV3CtVPN52sCu8c01t9p2fyBDqy3wlp RSFCOTOGtJrSYHGLTNO5ZAPGx0o+owSWR9cboZrrxosu4n5v8vLlhCYMyo4RgGWXYqkM VwLXWWD9TW4UbTAEl7d6d6urq0OGhMoydaC6v3d+6oMKu2ftdedGSKfH1iTl84cI7s9s Edl/hAotmNcBNMsTWA9L43fAuZWfDx0xL66684y+fkZatC3+Yf6ANDTZCx7v+AQDunlj Ne3R88IE1qoCjjvaRC3ZyJzx/JdDmY8vN9Jnjx1Q6q7Tix5D6Td1iYXgRwya00lX52Ff yK2A== X-Gm-Message-State: AOJu0YyeA7UUDHoTowb06CcY3WKeD19ZmfNE+Ekt7sIbLYDl2DPOw+0p r3zTEALOWxF3F8yTpMENNagmhB+OZwq8RNME1PlyjKDNv7vFOuXbKu/8rQ== X-Gm-Gg: ASbGncuyVK+Vj+gkGHdC1wkYaTFrEEWWj1s0Y8AGq0qVQ+eDCrY1oc1tZdCvVA45fIb L0CIt0W24H5HvWLohNCgrPh6fjbAg2UvDGns6CGjdhGcd4uzTihRDXytqLdpYU44kxzS81O2F+k Ih+h2G5zGJ0NEp54JB9L8VR6QFOMfmttm6QPKRaVaCkaWaBP0p5RoU0Zs9FwIB/4QWgIiZhsTGX Ap6serBcuxT3TpCC1H6qtxpQriPgYSE8HEvoZD2uDYXNDno/Di8EAB1NHQWHGyMhKf6xyBxe5Uu sHPySrEFHogUfpLkCoTAb0KSg1b3 X-Google-Smtp-Source: AGHT+IFg3ZLvzQYYHAjOpQH6c8T7mBzy21H2tATg0eYXJ7i9jqQkFzhiFBifKAmRZpYkj3UK84Vgpw== X-Received: by 2002:a05:6402:90c:b0:5dc:caab:9447 with SMTP id 4fb4d7f45d1cf-5e5e22da292mr10298092a12.18.1741363194318; Fri, 07 Mar 2025 07:59:54 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:a068]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac2399d7a17sm297369166b.179.2025.03.07.07.59.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:59:53 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v4 9/9] io_uring: cap cached iovec/bvec size Date: Fri, 7 Mar 2025 16:00:37 +0000 Message-ID: <823055fa6628daa24bbc9cd77c2da87e9a1e1e32.1741362889.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Bvecs can be large, put an arbitrary limit on the max vector size it can cache. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 3 +++ io_uring/rsrc.h | 2 ++ io_uring/rw.c | 3 +++ 3 files changed, 8 insertions(+) diff --git a/io_uring/net.c b/io_uring/net.c index 5e27c22e1d58..ce104d04b1e4 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -152,6 +152,9 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) /* Let normal cleanup path reap it if we fail adding to the cache */ io_alloc_cache_vec_kasan(&hdr->vec); + if (hdr->vec.nr > IO_VEC_CACHE_SOFT_CAP) + io_vec_free(&hdr->vec); + if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 0d5c18296130..b0097c06b577 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -5,6 +5,8 @@ #include #include +#define IO_VEC_CACHE_SOFT_CAP 256 + enum { IORING_RSRC_FILE = 0, IORING_RSRC_BUFFER = 1, diff --git a/io_uring/rw.c b/io_uring/rw.c index e62f4ce34171..bf35599d1078 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -150,6 +150,9 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) return; io_alloc_cache_vec_kasan(&rw->vec); + if (rw->vec.nr > IO_VEC_CACHE_SOFT_CAP) + io_vec_free(&rw->vec); + if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA;