From patchwork Tue Mar 4 15:40:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000972 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24DED2BF3EE for ; Tue, 4 Mar 2025 15:39:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102776; cv=none; b=mHV2/i7ZUF7wdzTWQ6KCWpOOr4OFHm2OUplbd1J6wmWojvfmwyffnl6gZhA6LbcDtd8k5cKK++hz5RSEj6IPhSLiA5dDkIBA1ZoqLFQ/QSzmqqUDK/wJStq+dEzwFBkgt2w7HV4BqJKcvxMrO9X+GiOGfr2q2htk8ow09Otm0lQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102776; c=relaxed/simple; bh=1d6ejyiPzFsCP/aRWcF09WW+bogzzbEaHRwv2dQmFps=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l1Ubq68t2G1OuyojDxt+/vMamIF6myVKBHfL3cFg5IsFmh5J5YAFfOyYSSVaqqoIT1+dIEB50ATfllWpTjBmwxSBHbFYCjvibtg1JMOC6NRPaNtqKkxcxrzd7Wuv4Nl1wN0txwQrQXPiZtyPysnMDNdnFj7SRC2ELFGothJqosg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hJe2IuyF; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hJe2IuyF" Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-aaee2c5ee6eso839141166b.1 for ; Tue, 04 Mar 2025 07:39:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102773; x=1741707573; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t91HrX4L7T7hAfZHnOYG8rdRL8w+TWHVfgfNwFTUYR0=; b=hJe2IuyFDaDSLeEW6KqnPVlU7WYKsMaRreM8Xbr47dnmMr7vlYl/yJE3mvDbryFhLG eerXJeq7lGzVTwZBipAsKOP0Fyksk0mcNuZD1s86fK72FwtA+IBDO3tFUmL9HJyyUSmZ YMZuuoCsbTDJ4UnmSlBPGfINVqnEMwLBukDIaUdmDDiLyswOh4+pEtQo5q5oOYqZi2GA vGtCyaelIhsiZY3aKBrp/p5NYecj8sO2xXPoUALz6zyNPs3I121rQIgr/LQ+tNW2qbKs o3oZ0Aps0xj1oKRM6Z9NQgEyiwi0ehD6E9uwuAoGYQt6YKk68zyT51DyUi9zR/t6d2Xw NwKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102773; x=1741707573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t91HrX4L7T7hAfZHnOYG8rdRL8w+TWHVfgfNwFTUYR0=; b=XouXJqK8bBa8Cf8qD9OYq64Ai9GQLw7aCv2KbWWvaptSUMyYfDJKGOdDKd3TpCGP/+ sasDH42mK9I92/LySHAB03ynq9O6kpxMyMgVNevOvQcYGhVZ0D+9Q8KQquiMQzbhGXX9 onWbEOHjCnMVsK/fuFY8EOYmdrTqzE9QR21jjQvBJMaZmWT0E8Nd/gU7MwotbIKO2HHf k8sdP4HuvT+vXbC72QgsjzPh3BRIQPDuGFCBRbbWLqHaB8JmwUfOy00i/hKMmhno4bo4 mNaCEVuPsWeCUAjwMGfw4DXKZit6uHbnUG0QhHQFg76jYdKupFPzkZR5/OlBg785y7M2 YrCA== X-Gm-Message-State: AOJu0Yx0/TaZePxQzF26gSUKtjAgud5jpvfBFogsun7SdQlbFhXU2M18 v76eJivAdCM2Wnu2HKPaDx+pkQntdJiTqyB5B5v+3IpoKwkCO3e/z84SWg== X-Gm-Gg: ASbGnctiiARZgARFcSmgQa+R4Li436hIcrm+qmqOdy7SlI0vM6Shrp+R75fZ6UHgX7w JicV/DyVi9ohWH3IhkRCtZUqF/HzeUvGKkc3tCCUoAlv4MpFtVekUt6k8ansjWVWthmxPBF1fUB TJX1bciH3DUAQBUOJZVNhhn1jhfdYOuI124ucAuURpHNTW0a7GLC1yBPRz5qoknCkxXHIVdHiPL 8+5VMxF2Mnjm6KFALVAxiwfs+tIeuzYElC7jva3BZ3lFheMKvUrdtWXzFJeGG7jnNDbCm8FDdRn Q5gyL5e87OAJNu+17XMVHRP9w2dg X-Google-Smtp-Source: AGHT+IF+gX1CuwYUt5SAiOpKhQHMz0CxBZWBVNKjgKuy/Laqtd/JRSNMG0aJsMwYqsDi8FpDN+yLdw== X-Received: by 2002:a17:907:2d0b:b0:abf:77f3:d1d8 with SMTP id a640c23a62f3a-abf77f3d653mr972926366b.19.1741102772620; Tue, 04 Mar 2025 07:39:32 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:31 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 1/9] io_uring: introduce struct iou_vec Date: Tue, 4 Mar 2025 15:40:22 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I need a convenient way to pass around and work with iovec+size pair, put them into a structure and makes use of it in rw.c Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +++++ io_uring/rsrc.c | 9 +++++++++ io_uring/rsrc.h | 17 +++++++++++++++++ io_uring/rw.c | 17 +++++++---------- io_uring/rw.h | 4 ++-- 5 files changed, 40 insertions(+), 12 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 35fc241c4672..9101f12d21ef 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -110,6 +110,11 @@ struct io_uring_task { } ____cacheline_aligned_in_smp; }; +struct iou_vec { + struct iovec *iovec; + unsigned nr; +}; + struct io_uring { u32 head; u32 tail; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d6ac41840900..9b05e614819e 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1264,3 +1264,12 @@ int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg) fput(file); return ret; } + +void io_vec_free(struct iou_vec *iv) +{ + if (!iv->iovec) + return; + kfree(iv->iovec); + iv->iovec = NULL; + iv->nr = 0; +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 284e300e63fb..ff78ead6bc75 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -4,6 +4,7 @@ #include #include +#include enum { IORING_RSRC_FILE = 0, @@ -145,4 +146,20 @@ static inline void __io_unaccount_mem(struct user_struct *user, atomic_long_sub(nr_pages, &user->locked_vm); } +void io_vec_free(struct iou_vec *iv); + +static inline void io_vec_reset_iovec(struct iou_vec *iv, + struct iovec *iovec, unsigned nr) +{ + io_vec_free(iv); + iv->iovec = iovec; + iv->nr = nr; +} + +static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv) +{ + if (IS_ENABLED(CONFIG_KASAN)) + io_vec_free(iv); +} + #endif diff --git a/io_uring/rw.c b/io_uring/rw.c index 5ee9f8949e8b..ad7f647d48e9 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -87,9 +87,9 @@ static int io_import_vec(int ddir, struct io_kiocb *req, int ret, nr_segs; struct iovec *iov; - if (io->free_iovec) { - nr_segs = io->free_iov_nr; - iov = io->free_iovec; + if (io->vec.iovec) { + nr_segs = io->vec.nr; + iov = io->vec.iovec; } else { nr_segs = 1; iov = &io->fast_iov; @@ -101,9 +101,7 @@ static int io_import_vec(int ddir, struct io_kiocb *req, return ret; if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - io->free_iov_nr = io->iter.nr_segs; - kfree(io->free_iovec); - io->free_iovec = iov; + io_vec_reset_iovec(&io->vec, iov, io->iter.nr_segs); } return 0; } @@ -151,7 +149,7 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) return; - io_alloc_cache_kasan(&rw->free_iovec, &rw->free_iov_nr); + io_alloc_cache_vec_kasan(&rw->vec); if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -201,7 +199,7 @@ static int io_rw_alloc_async(struct io_kiocb *req) rw = io_uring_alloc_async_data(&ctx->rw_cache, req); if (!rw) return -ENOMEM; - if (rw->free_iovec) + if (rw->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; rw->bytes_done = 0; return 0; @@ -1327,7 +1325,6 @@ void io_rw_cache_free(const void *entry) { struct io_async_rw *rw = (struct io_async_rw *) entry; - if (rw->free_iovec) - kfree(rw->free_iovec); + io_vec_free(&rw->vec); kfree(rw); } diff --git a/io_uring/rw.h b/io_uring/rw.h index bf121b81ebe8..529fd2f96a7f 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -9,13 +9,13 @@ struct io_meta_state { }; struct io_async_rw { + struct iou_vec vec; size_t bytes_done; - struct iovec *free_iovec; + struct_group(clear, struct iov_iter iter; struct iov_iter_state iter_state; struct iovec fast_iov; - int free_iov_nr; /* * wpq is for buffered io, while meta fields are used with * direct io From patchwork Tue Mar 4 15:40:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000973 Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49623255220 for ; Tue, 4 Mar 2025 15:39:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102778; cv=none; b=foBS3XpyiQbaoY4NmDq4jG7r63sHCXwkVl7WzGKAz3Lx5zbEqMB49sBOhNzHcIJfckZEV9YXzMUEqp+mDQ8EJ+NOIrr9h0hYCm6MoYVExJzmbZjS1F9y6ntwalEiknh6VrxbH9bX+F2ZOPjs856BSBu5/r61R+sUUvGNKhvGjmE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102778; c=relaxed/simple; bh=BvSYcKZjJx2mcN6MisvhcNvhByGtN1LJqFZE5JCFyus=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LoiF41TIMIvwxudKgNCiHRXOKBfkMnMqrWtD/0cMgWsKA2ZGcclZOeEPAR4Zfn7wFDqvo3nlQYw1Rz1LldgiUnswUuvEQfq+dNLobg+dalb3V5djSDiBmjGTTDl+r36FZdG/tepjzOrFrWFW9q/3mjJvsh46DIg0mnT8owBSxk8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=S9THVGdl; arc=none smtp.client-ip=209.85.218.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="S9THVGdl" Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-ab744d5e567so980242366b.1 for ; Tue, 04 Mar 2025 07:39:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102774; x=1741707574; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/qLKGOdBl3cLNSyd8QE3mj2clAx9J3/j7tRp1rR5kGA=; b=S9THVGdlCOXkZwCuEVo5FskGBUqFBN1Tcgpk0siIptGlRdgA67O5nQN3yPg9C07C5p kggV5aHav5AjfD8Ga04bPkGTFO/vyQmZvcLDI+fgGK0tifA8reNCg9/wUKx/Hp+IWoHs kfYPsE3Xp2k7txCialtPp95jwpIt6vGYtHCu3xPxdqgQfN43huP7T3JgTygjiROcIpXl zLM/9w1HGiwoLqmy5ZT6FRclglGc0KQgOoBlsLzDhFShMqGk6J0nnHtm4DdCwlB7qpVQ GqvIQBpFsFJkYqAIDFSgjS+SUeSfPfNbdBA97M7G0S391+ZB2V8/jRnNq/h7wY2FKOWm AKUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102774; x=1741707574; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/qLKGOdBl3cLNSyd8QE3mj2clAx9J3/j7tRp1rR5kGA=; b=QuyoAed2O+E5cgtZ//tcHkwxcRJVeaIjwy1tT3kKZl9umPv9X4W9ErHk/FyxnPesyc Mqe44ly3PXOOcYCL7hJ8ZoyG24Ml38rCv7Q5jfaUQ9DkOaao4EkrCGL18Dkj/jfMDDE7 ueirTZoQEn1Wbi89TComNaYo20fRLWwT4W0HJINs7LlK2MnpyxhkUQyzGKK0+Lv+NW5A ln0QQVIQINX1Cm4Ta41XwRA2ILahUgJIq1IncR6WNbTlhhmtGtgPiUQaSLZyDxP/ukwb jeoLlF+HY/oiduPjKSuGTwzegmHICjaLS5/E8ktVou+6GSH/Wlz9VpYtSVpfL+473Qqw lecQ== X-Gm-Message-State: AOJu0YyuPFuAtlx65TfbUNKoZfFEVPtLqQKKKQWNHIqh5aHm+s/0b88J HDfRGoGv/Jsp7b+gT5KLgghjw6UqH05/R+4MmX5E72lH7N+NXXKNqnhAEw== X-Gm-Gg: ASbGncteidr7XIjXu0QYqtc6+aNgXOlVPyXIxFZRC5rDt5HJ/gaYc0NXsvm8Nm6H65z KQ4e0diTkD2S5f+KRkoNmr3bOlPja2wmSE+lU1x2KeRRm+12Ph/9BqBL9L2KOAl/Xn2PKgsCBY1 6fR+pApR6idSe92R/NwWGc8YrbPSteOLiX+z7UNnVMIH85Q3N+ZHaeFMJC3VbfxlOzYM8F450zv iPNvKgpgjMoGTLhdmW1kD7akXmcgS9GuHWznvHxaDn2kY3BRarkwEPiFSoQBRvPpxGFSrE4ugOp WpHGmDHZKYxwHwn2ug7SJBlLr+u0 X-Google-Smtp-Source: AGHT+IFnSvS5AA6q3trOnwz01vkMEsxJkG3SHhEHr6Is9agUdriICyt+uPbb+V13o8bhS2hswzCbAA== X-Received: by 2002:a17:907:d24:b0:ac0:4364:407e with SMTP id a640c23a62f3a-ac1f0eb69d1mr396984366b.4.1741102774029; Tue, 04 Mar 2025 07:39:34 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:33 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 2/9] io_uring: add infra for importing vectored reg buffers Date: Tue, 4 Mar 2025 15:40:23 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add io_import_reg_vec(), which will be responsible for importing vectored registered buffers. iovecs are overlapped with the resulting bvec in memory, which is why the iovec is expected to be padded in iou_vec. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +- io_uring/rsrc.c | 122 +++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 ++ 3 files changed, 131 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9101f12d21ef..b770a2b12da6 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -111,7 +111,10 @@ struct io_uring_task { }; struct iou_vec { - struct iovec *iovec; + union { + struct iovec *iovec; + struct bio_vec *bvec; + }; unsigned nr; }; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 9b05e614819e..38743886bbf4 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1267,9 +1267,131 @@ int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg) void io_vec_free(struct iou_vec *iv) { + BUILD_BUG_ON(sizeof(struct bio_vec) > sizeof(struct iovec)); + if (!iv->iovec) return; kfree(iv->iovec); iv->iovec = NULL; iv->nr = 0; } + +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries) +{ + gfp_t gfp = GFP_KERNEL | __GFP_NOWARN; + struct iovec *iov; + + iov = kmalloc_array(nr_entries, sizeof(iov[0]), gfp); + if (!iov) + return -ENOMEM; + + io_vec_free(iv); + iv->iovec = iov; + iv->nr = nr_entries; + return 0; +} + +static int io_vec_fill_bvec(int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + struct iovec *iovec, unsigned nr_iovs, + struct iou_vec *vec) +{ + unsigned long folio_size = 1 << imu->folio_shift; + unsigned long folio_mask = folio_size - 1; + u64 folio_addr = imu->ubuf & ~folio_mask; + struct bio_vec *res_bvec = vec->bvec; + size_t total_len = 0; + unsigned bvec_idx = 0; + unsigned iov_idx; + + for (iov_idx = 0; iov_idx < nr_iovs; iov_idx++) { + size_t iov_len = iovec[iov_idx].iov_len; + u64 buf_addr = (u64)iovec[iov_idx].iov_base; + struct bio_vec *src_bvec; + size_t offset; + u64 buf_end; + + if (unlikely(check_add_overflow(buf_addr, (u64)iov_len, &buf_end))) + return -EFAULT; + if (unlikely(buf_addr < imu->ubuf || buf_end > (imu->ubuf + imu->len))) + return -EFAULT; + if (unlikely(!iov_len)) + return -EFAULT; + if (unlikely(check_add_overflow(total_len, iov_len, &total_len))) + return -EOVERFLOW; + + /* by using folio address it also accounts for bvec offset */ + offset = buf_addr - folio_addr; + src_bvec = imu->bvec + (offset >> imu->folio_shift); + offset &= folio_mask; + + for (; iov_len; offset = 0, bvec_idx++, src_bvec++) { + size_t seg_size = min_t(size_t, iov_len, + folio_size - offset); + + bvec_set_page(&res_bvec[bvec_idx], + src_bvec->bv_page, seg_size, offset); + iov_len -= seg_size; + } + } + if (total_len > MAX_RW_COUNT) + return -EINVAL; + + iov_iter_bvec(iter, ddir, res_bvec, bvec_idx, total_len); + return 0; +} + +static int io_estimate_bvec_size(struct iovec *iov, unsigned nr_iovs, + struct io_mapped_ubuf *imu) +{ + unsigned shift = imu->folio_shift; + size_t max_segs = 0; + unsigned i; + + for (i = 0; i < nr_iovs; i++) + max_segs += (iov[i].iov_len >> shift) + 2; + return max_segs; +} + +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned iovec_off, + unsigned issue_flags) +{ + struct io_rsrc_node *node; + struct io_mapped_ubuf *imu; + struct iovec *iov; + unsigned nr_segs; + + node = io_find_buf_node(req, issue_flags); + if (!node) + return -EFAULT; + imu = node->buf; + if (imu->is_kbuf) + return -EOPNOTSUPP; + if (!(imu->dir & (1 << ddir))) + return -EFAULT; + + iov = vec->iovec + iovec_off; + nr_segs = io_estimate_bvec_size(iov, nr_iovs, imu); + + if (WARN_ON_ONCE(iovec_off + nr_iovs != vec->nr) || + nr_segs > vec->nr) { + struct iou_vec tmp_vec = {}; + int ret; + + ret = io_vec_realloc(&tmp_vec, nr_segs); + if (ret) + return ret; + + iovec_off = tmp_vec.nr - nr_iovs; + memcpy(tmp_vec.iovec + iovec_off, iov, sizeof(*iov) * nr_iovs); + io_vec_free(vec); + + *vec = tmp_vec; + iov = vec->iovec + iovec_off; + req->flags |= REQ_F_NEED_CLEANUP; + } + + return io_vec_fill_bvec(ddir, iter, imu, iov, nr_iovs, vec); +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index ff78ead6bc75..f1496f7d844f 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -62,6 +62,10 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req, int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, unsigned issue_flags); +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned iovec_off, + unsigned issue_flags); int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg); int io_sqe_buffers_unregister(struct io_ring_ctx *ctx); @@ -147,6 +151,7 @@ static inline void __io_unaccount_mem(struct user_struct *user, } void io_vec_free(struct iou_vec *iv); +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries); static inline void io_vec_reset_iovec(struct iou_vec *iv, struct iovec *iovec, unsigned nr) From patchwork Tue Mar 4 15:40:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000974 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9512F280CC4 for ; Tue, 4 Mar 2025 15:39:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102779; cv=none; b=c85mdoJovAd9X/FLyfN+sLZ7bPfZqgAjlu7Pajmci37HBZMyyqEOnuIi3CF8jZdSrZ7pjh5j1XMFnIJ7OGExmY9pK/YOK13Nv5Hi77Cf0JyEYMUa/28UypAoAVdUPJON6Hz4J1pR6D51g8IqZGIDm1DDCDt0SRgAu6dvAF3kdpM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102779; c=relaxed/simple; bh=z68zN+i+UX2eMqGbGdS9FjdQEKbOuwEhXBJrRHSKZrQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ebIarzQkqnVuKC8M5H2CacNiUA5/nY8DCin1S9QC4PLmY271wBTHcmvTCRGeDi0FLFh5jFEDAh5abxY6bvnqd4j5IpzUS9aZ2eH2qXWlGn6/ykN5BkeqLy8W/CPtQn4bGWcdC8IfEIT6Xuwe7Hohou55dQns0UyhVoaM9woEMb0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=R92IPqWJ; arc=none smtp.client-ip=209.85.218.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="R92IPqWJ" Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-aaee2c5ee6eso839148866b.1 for ; Tue, 04 Mar 2025 07:39:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102775; x=1741707575; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1OuBKjwGr0UgawHOumxXWMKAPfGEiLxbUH2ylFtbJ/Q=; b=R92IPqWJlQNx6u2uw5oWES7JkoBeHnrfwTlxKJu2KbadgybaYhhrrqJce1nSyBi5tt PxPUTBNwyXZx4WRWQlbvQN0itD+BTL4Y8xxjxeWVBD7+ZDT8WtRQ1FKi6RtkuILpRRTj NLS2Wi8KjORZGTqEcA4dD1TT0copmPnRwuknk6QA3iKt8kQjENbdLoovbqkOPiMjmUCg yFv5qhroS/5ugEkksLQyPFAnlEDYtrZ8F6CTVBQsMcDOlbnPY/pammBbaL/7DntpbV/0 1a+ku6sL+fQoDCB1/HaF9vxVUiWg2nlcJBt0VckMFDrI/qezZI4sFak9yIJ5oUBHIcJ6 68ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102775; x=1741707575; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1OuBKjwGr0UgawHOumxXWMKAPfGEiLxbUH2ylFtbJ/Q=; b=lWejVClj+kikuiq6ivj8OWVrephNYMYBB1T8GABjEWYCgWpOJHHrrKJUPH6vXnmyks vWL28hGmrvMk7CVohyUz9HdaKrg11ev9GqQmg5kmD6YbxDlYLigdxa6S1XeIaHScJvmw Wo8EJXnm7x3ekwHqE/7b6eGMiJ/tviUH5a6+cW/79YfeDKmayYp3Av0JDbtJHvqTj9pQ hp+3/8M8QY+uJo9Q64dA0p35+mYNR/1U9mB8hQMwgeBn2QN0IK7uxmjPjf84aWloKSZL n+Lx7m/Dn+RjbF7DNEMTAiQwzcEmPNy7SuLKADzqRQikIsRtnJELvyGhskQMMCrsd4+c WLyQ== X-Gm-Message-State: AOJu0YxqzsO8SPT29fZazthU3F0jlFSu80ixUUd8PjfpmXHUpEmWUgVz dF5x5V2+J1KQWpmZGl7dM2sFfRKKHPhkY+Ma4kp4zGNcYB6gPF+V4Tl4tw== X-Gm-Gg: ASbGncsJtUMMIoi9JXrI7gZMto3onU+lkA0kOQFTVefo4mXF7xVwr2Fe2jeaDWb7IDR nVNCFc2IXhBq4u6JZfvLryKf/gspPZXAuH5fDTptnLamdIaoXDaNBQOYPs2lei28KT1ePS5FNm8 hS/v54SkwA45eL6hMwTJRAgau/iQpC7UinN6Sk5XujlLi1u3J7QrnItO41Ls7FmJoaKY9wuEIZc u8iMt+oY7RMieu1CAcqETXeyIGqC2JzjnB3IU8Ay/ZCiwCA23xk90unb0MjR0EVPndH9X+InI6W mr9fyUE/Us73zdJlB1F8g1ZMpXhQ X-Google-Smtp-Source: AGHT+IGluVtNgHr+YrNc/9AEjetNP299tL1fJT+OT5WwItcxeCX2SVjperWjdb34jkhMNddCCboQ2g== X-Received: by 2002:a17:907:7a8e:b0:abf:6d24:10bb with SMTP id a640c23a62f3a-abf6d241c9emr1034399966b.44.1741102775331; Tue, 04 Mar 2025 07:39:35 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:34 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 3/9] io_uring/rw: implement vectored registered rw Date: Tue, 4 Mar 2025 15:40:24 +0000 Message-ID: <7d37e52d36b645fb7c70f41a85fc704d8bca5f4a.1741102644.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Implement registered buffer vectored reads with new opcodes IORING_OP_WRITEV_FIXED and IORING_OP_READV_FIXED. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 2 ++ io_uring/opdef.c | 39 +++++++++++++++++++++++++++ io_uring/rw.c | 51 +++++++++++++++++++++++++++++++++++ io_uring/rw.h | 2 ++ 4 files changed, 94 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 1e02e94bc26d..9dd384b369ee 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -280,6 +280,8 @@ enum io_uring_op { IORING_OP_BIND, IORING_OP_LISTEN, IORING_OP_RECV_ZC, + IORING_OP_READV_FIXED, + IORING_OP_WRITEV_FIXED, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 9511262c513e..6655d2cbf74d 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -529,6 +529,35 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_READV_FIXED] = { + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_readv_fixed, + .issue = io_read, + }, + [IORING_OP_WRITEV_FIXED] = { + .needs_file = 1, + .hash_reg_file = 1, + .unbound_nonreg_file = 1, + .pollout = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_writev_fixed, + .issue = io_write, + }, }; const struct io_cold_def io_cold_defs[] = { @@ -761,6 +790,16 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_RECV_ZC] = { .name = "RECV_ZC", }, + [IORING_OP_READV_FIXED] = { + .name = "READV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, + [IORING_OP_WRITEV_FIXED] = { + .name = "WRITEV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, }; const char *io_uring_get_opcode(u8 opcode) diff --git a/io_uring/rw.c b/io_uring/rw.c index ad7f647d48e9..4c4229f41aaa 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,6 +381,57 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } +static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + struct io_async_rw *io = req->async_data; + const struct iovec __user *uvec; + size_t uvec_segs = rw->len; + struct iovec *iov; + int iovec_off, ret; + void *res; + + if (uvec_segs > io->vec.nr) { + ret = io_vec_realloc(&io->vec, uvec_segs); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + /* pad iovec to the right */ + iovec_off = io->vec.nr - uvec_segs; + iov = io->vec.iovec + iovec_off; + uvec = u64_to_user_ptr(rw->addr); + res = iovec_from_user(uvec, uvec_segs, uvec_segs, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, 0); + iov_iter_save_state(&io->iter, &io->iter_state); + return ret; +} + +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_DEST); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_DEST); +} + +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_SOURCE); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_SOURCE); +} + /* * Multishot read is prepared just like a normal read/write request, only * difference is that we set the MULTISHOT flag. diff --git a/io_uring/rw.h b/io_uring/rw.h index 529fd2f96a7f..81d6d9a8cf69 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -32,6 +32,8 @@ struct io_async_rw { int io_prep_read_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_readv(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_writev(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_read(struct io_kiocb *req, const struct io_uring_sqe *sqe); From patchwork Tue Mar 4 15:40:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000975 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2170298CA0 for ; Tue, 4 Mar 2025 15:39:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102780; cv=none; b=NQjQPO/h2lra+3IDFQvx4w2RzzFJxYAWGpvtQlU8Com8kAx4AKGc57jSkt/+3nguAOPoVDlUbV+0sD9P74GfQBssjzBoOCDFwBZ+f/AMZBuCa+D7GzD6hY4zUzBpLCle0v4JtMas9u2cj/4VhsujhXs/VsM4UTJCVt5ik0kJBRc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102780; c=relaxed/simple; bh=Dfyl5JUYMQ4WXsedoYmAVv71GmQLPCka+J2WuBab3mg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oornH+p3vBSzbO8Jo4x4minVjCWR8Z+UlD+4BDFigdF4qZoLOjndPCFDh1LbQy4Y8z+kpBTyG0z6XtqizjhCT/caA/f06U1hYPKFHZ2zqZfP1wNm2qDVfUY8G1V+b7JwSL07yPXa5gBovyqvBRnqE94NY7QmhrOfIoib8eyt3R8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hP7NOZdM; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hP7NOZdM" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5e549af4927so4100701a12.2 for ; Tue, 04 Mar 2025 07:39:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102777; x=1741707577; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=S++4h6XEEInGoer4Mhouq45inPSSKuv2sG0OsLdnyFA=; b=hP7NOZdMjHo8OvwHYf3KX8dtDoFW9LYFCK+0xytwjRTqns5zBemUnDT02kDTj/pIOO 4ejCMlYrdDhef8bQeYp7s2KYNyO90z1RPcP6oksLPOHjYPYAiQZJfq9rQ5wf9eQ3dxbx 5AupLGyhr9DRGKwmNOw28iNRiHrzltj4vPuk72PsZ6J7pPYZxkZYUDG5tVwj0bk9zBFs 9qSCVprlo17zRsS96m8b1bEFOY3CdJk9eDKZS29Pv+tJXjAcdRgswKCZzhvhK/3FFnrP Y18mrlyUj9MzQKytBBJIOxIIpaszUql1pGQLUKuBqzpgFhPrmYGS5FMUUUGWNBRx5CuN 6JpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102777; x=1741707577; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S++4h6XEEInGoer4Mhouq45inPSSKuv2sG0OsLdnyFA=; b=h56VjDhvlDGDuxD1yrC6dnH6kJivNW8NfNlGta8huyCcWpR0UaMUg7tPtaDrIQKJv4 oGJPaGTTlRSSA82kwU3Ewd7PP0k8SWdKapB2r2FzyNJ0tXnEUI6J+UZ/ca7umBdGG1hC 8Qlm7Ecz3lviZG4eSOq2DFy8zO349ZvULlpZIDWD8cM6mtne6A2JtW1+gUkjtpmjH9iQ CmFEUq9xZd9AkWoDs30jMdU+ITGU0g4XtFUkjkFJHNuJ8vPw+QzmnzEV4fDNTD5QVC+o XIjZfuzVgho3vGAmcuhePmnl10hrm9GLn3A2KK3OZWb39x0oGauC8zqNPcrYgx4wqSYj bAoQ== X-Gm-Message-State: AOJu0YzATS4hGJ0HonGLP0mFspqPoPCWRN3Mc85X2UcqUwFwREb1fA3C 9fChSW2gJ+WSN/z9Dm+Pb7gWOAtz7EvEUxW7FA5RRdsSYqaMaQFHusllgA== X-Gm-Gg: ASbGncuVVfNGG2o6Hy+cvFmCJF8Ytlb575GCAxB5Kk5AGYMfIihRctopc+sjSgygVTE XoCyPVDCGAugtZ99ZjuBClQA5waXtjD3bR92+Ir6YKpBaLOm+RvZJKrVuJKy4tDV36B9LZdRYjb mA6QosKKDqbucxIU3LJfcyVII0pzuP5CEJgAowD2zHQTtnHNtJVPTGaZSb+Fep7BCZS4UsPW54B 5WDlIO4Vgs7/J1VZ199A5H9vG/N961XnPdhfbua97pjFJsBaefmyYXZCzwNxXcshybDRBbqFo2J ASebruzQz0u14uWBDVPSMW/kvKNo X-Google-Smtp-Source: AGHT+IES3BAXsq1p5J+bL1Pg8YxKjD7sqkv+HHL3zFzay07Bpa0wOI2LAZ5Y/d6gDZU5DBvWiLHZOw== X-Received: by 2002:a17:907:9815:b0:abf:641a:5727 with SMTP id a640c23a62f3a-abf641a5b44mr1054379566b.7.1741102776680; Tue, 04 Mar 2025 07:39:36 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:35 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 4/9] io_uring/rw: defer reg buf vec import Date: Tue, 4 Mar 2025 15:40:25 +0000 Message-ID: <5b1f8b6ce1b39c5bc0c7f42119f941fdf0621505.1741102644.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Import registered buffers for vectored reads and writes later at issue time as we now do for other fixed ops. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +++ io_uring/rw.c | 42 +++++++++++++++++++++++++++------- 2 files changed, 37 insertions(+), 8 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index b770a2b12da6..d36fccda754b 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -502,6 +502,7 @@ enum { REQ_F_BUFFERS_COMMIT_BIT, REQ_F_BUF_NODE_BIT, REQ_F_HAS_METADATA_BIT, + REQ_F_IMPORT_BUFFER_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -584,6 +585,8 @@ enum { REQ_F_BUF_NODE = IO_REQ_FLAG(REQ_F_BUF_NODE_BIT), /* request has read/write metadata assigned */ REQ_F_HAS_METADATA = IO_REQ_FLAG(REQ_F_HAS_METADATA_BIT), + /* resolve padded iovec to registered buffers */ + REQ_F_IMPORT_BUFFER = IO_REQ_FLAG(REQ_F_IMPORT_BUFFER_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, io_tw_token_t tw); diff --git a/io_uring/rw.c b/io_uring/rw.c index 4c4229f41aaa..e62f4ce34171 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,7 +381,25 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } -static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) +static int io_rw_import_reg_vec(struct io_kiocb *req, + struct io_async_rw *io, + int ddir, unsigned int issue_flags) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + unsigned uvec_segs = rw->len; + unsigned iovec_off = io->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, issue_flags); + if (unlikely(ret)) + return ret; + iov_iter_save_state(&io->iter, &io->iter_state); + req->flags &= ~REQ_F_IMPORT_BUFFER; + return 0; +} + +static int io_rw_prep_reg_vec(struct io_kiocb *req) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); struct io_async_rw *io = req->async_data; @@ -406,10 +424,8 @@ static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) if (IS_ERR(res)) return PTR_ERR(res); - ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, - uvec_segs, iovec_off, 0); - iov_iter_save_state(&io->iter, &io->iter_state); - return ret; + req->flags |= REQ_F_IMPORT_BUFFER; + return 0; } int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -419,7 +435,7 @@ int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) ret = __io_prep_rw(req, sqe, ITER_DEST); if (unlikely(ret)) return ret; - return io_rw_prep_reg_vec(req, ITER_DEST); + return io_rw_prep_reg_vec(req); } int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -429,7 +445,7 @@ int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) ret = __io_prep_rw(req, sqe, ITER_SOURCE); if (unlikely(ret)) return ret; - return io_rw_prep_reg_vec(req, ITER_SOURCE); + return io_rw_prep_reg_vec(req); } /* @@ -906,7 +922,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret; loff_t *ppos; - if (io_do_buffer_select(req)) { + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_DEST, issue_flags); + if (unlikely(ret)) + return ret; + } else if (io_do_buffer_select(req)) { ret = io_import_rw_buffer(ITER_DEST, req, io, issue_flags); if (unlikely(ret < 0)) return ret; @@ -1117,6 +1137,12 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret, ret2; loff_t *ppos; + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_SOURCE, issue_flags); + if (unlikely(ret)) + return ret; + } + ret = io_rw_init_file(req, FMODE_WRITE, WRITE); if (unlikely(ret)) return ret; From patchwork Tue Mar 4 15:40:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000976 Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45EC0298CB1 for ; Tue, 4 Mar 2025 15:39:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102781; cv=none; b=jDhzLOXbzore0P7KwCNm43VvyqagJyXGRiZK77VL/DJM6eQtyhL7LsoyYOldF2mXinfdkM1IV7DHozE3tFa7HAL/MYnFeWL+Fk2oN9HaVOrbIgZPynw9iXRFH4kB43jAFhoNNHGMaHnnj0j29T4gnHjn0plT2J6EMyTEbspTK20= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102781; c=relaxed/simple; bh=q/FZLY6NX7KPIL9q2sRYyOUjPSfSZOW4huImrRHQGAc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K9CSTeebjzENmXZL5Un4fk4ZtuBWnyqXERJYF5+AeCwfrvHKshKdxf2ZUaN0o/fs8pTsNODLAPTlAnHywibWGckMdtgKqqDAsKV0g8xbm6yperkRo5fuNaMa8bo2DaocwV4Ds0oP047RJOp6OrFoeUs3p3g4ujzQkmqGW7idj3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ukz/llAq; arc=none smtp.client-ip=209.85.218.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ukz/llAq" Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-abf538f7be0so530306766b.3 for ; Tue, 04 Mar 2025 07:39:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102778; x=1741707578; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=Ukz/llAqzz6nnEYUXolgAhE2DfJkyrC3eWtiRPhJ2gNv4teohr07Ckqf36d+cy5rd8 aeIlWTdxV5AD06/OWFdACSPw2ZwR/XA2CSZgBP6Fr/td10j60q1IzLsTO+Zu/ph727dp igkr1C5CmPciWqrOjy6yIE+E6iJSDk9Fz4E3rdXKR1NNO1s6hZ8Qn3cwB7WZ/k7pdvwv 7IBpcu0rkPzoe4Wk79SIU4DKgvMkW3GzkCKMDOhq0om2TUaWhQn2ALCL4RawVYKifn+z zLelnPB4cxoP0TeuwX7mlHviPCx1GEc3RNnZTaj/+LgjtLEVaHhvlBRnntWmQSMtUcA3 nKAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102778; x=1741707578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=Sqs4C4Q7LX+TPgyEccfJbfJd2s9Y+fGqTcuqOalDy0qK55CVMgHrHixKqq8nK8xUTs FUKi1QLzOIrdgSiEKy96+oO6DYEjirwd3XLTOYQbySFFUrMCePjbvb8RtWvG8h+Q78Ay CSw754Qh7j3LiTHzkuRmk+9XRtzY2N7AUim0D0yH0I+/OoK3msDHbz7I/oKNBcOkKFC9 WGZpkB7tWN87Mmk7E9Qjn/mmqc8xtCIfD4h+1LH8IBevYTKiRYfeEnhdjPai8HS6Z4oJ 3Jy0mzM086F3y5IFIgiJQRBVTfOc7CagoeB1SknHqkleXHMC9HXkVP8NW8FFaSGl+Q7V UgCQ== X-Gm-Message-State: AOJu0YyJypv+bbN0Q2K4EGzTRGVHimYkGL76j6bwZI9tHcYcICLYDx8f ITF6xPI/SYWFnum0dUAKdlXajLnaXHtoi5JABRNsAhfbKLk8SyJQGBINlA== X-Gm-Gg: ASbGncu2KbHG2eYkoHWpBeDjW30yOoRfdFYjnKfSKxOEEAeGHDFM3f3SR+FbMu/XAxm 9qnRz+QR5BYeNwN4rY518fqmVt+y8sNwupHNXYi7gupGnwYb+9fTEmOINKCd+vxem5Mt62ojlrN oP7veWd1Y2aSZGvkEsNokLkfqaNSFB2JthrWZcZ7mN3ipULGBj5wrGQD8tOxOEnYFWRhdMa8SXH 2c2XcXFpt9rPVdX6t9VkpWM9HOjF0kWQsVLGlJr3kJnzGB4nfuBUoRkIX8aQjUpLyzRp71McreS jvZHq9tniBg7MNQJKNAVpVw0yx3f X-Google-Smtp-Source: AGHT+IEtrgiSzJbLJs07GbvYBSwaLhqcbcC8rTo/06QM7KJC5+aYR1gtYi1diGno/MnOfnHrgjNFPA== X-Received: by 2002:a17:906:d54f:b0:ac1:ffde:7706 with SMTP id a640c23a62f3a-ac1ffde7a3emr156100066b.25.1741102778207; Tue, 04 Mar 2025 07:39:38 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:37 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 5/9] io_uring/net: combine msghdr copy Date: Tue, 4 Mar 2025 15:40:26 +0000 Message-ID: <71be0fd66b80847a948855d825f6db0bb24e049f.1741102644.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Call the compat version from inside of io_msg_copy_hdr() and don't duplicate it in callers. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 905d1ee01201..33076bd22c16 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -287,6 +287,24 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct user_msghdr __user *umsg = sr->umsg; int ret; + iomsg->msg.msg_name = &iomsg->addr; + iomsg->msg.msg_iter.nr_segs = 0; + + if (io_is_compat(req->ctx)) { + struct compat_msghdr cmsg; + + ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ddir, save_addr); + if (ret) + return ret; + + memset(&msg, 0, sizeof(msg)); + msg->msg_namelen = cmsg.msg_namelen; + msg->msg_controllen = cmsg.msg_controllen; + msg->msg_iov = compat_ptr(cmsg.msg_iov); + msg->msg_iovlen = cmsg.msg_iovlen; + return 0; + } + ret = io_copy_msghdr_from_user(msg, umsg); if (unlikely(ret)) return ret; @@ -323,18 +341,6 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_SOURCE, - NULL); - sr->msg_control = iomsg->msg.msg_control_user; - return ret; - } - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; @@ -710,21 +716,7 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_DEST, - &iomsg->uaddr); - memset(&msg, 0, sizeof(msg)); - msg.msg_namelen = cmsg.msg_namelen; - msg.msg_controllen = cmsg.msg_controllen; - } else { - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); - } - + ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, From patchwork Tue Mar 4 15:40:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000977 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D93FC298CC0 for ; Tue, 4 Mar 2025 15:39:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102783; cv=none; b=BSx2au0cD+/vZ7PsDkHhZEOA/fjjLvgbuHoczf3MVCX2MSuMBSTyTTVcWBibsRJet5LvCeQLTOU0KmUsiHYr/H5zeXzJQ2+YSRk4FPLahUqyZqCChTwr3jPWIbTb+/kBPvB9ipN4HRRxWjpEr6SS5ygqmE0ufjIgN3bmHpIbnnM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102783; c=relaxed/simple; bh=dFLQOj2ZLIbINKKwcv5tlrqtAVZRjLUjvTJ5sPCh7Go=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SKrnpYZeA4DjSV8hjrGdLX6805MeA7Yzrf0BZpObIjNu0eQCEVE+njO2f/x1z/8MytI+c6gpApBX3xvoN55OMfuwhlYho50rs6b6DcTYwW2+flrgjqhVWnnts6LAp4QON2I5vcgEd5/LWxOdAigesgoNv9Oe+/FRC3ipf0mWmhg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=e2/Ix5ii; arc=none smtp.client-ip=209.85.218.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="e2/Ix5ii" Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-abfe7b5fbe8so317715766b.0 for ; Tue, 04 Mar 2025 07:39:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102780; x=1741707580; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=e2/Ix5ii5OQnGxMJ3vq1Cl9PTF89ydKooRd/cnf8QPbwZtAXg3bSZ4PdJAilRrcOMr 1XKONrX2ZU/w6rI04KK0EuCyfhZ7O5B0E45mC10J0egye3EkUFQm05OubNMLFoWzRud5 OW3xpm8q6l62lqlBMy1lcXfV6FATZWSwfjWi2g5bSC5yPZV2rJrCQwQjRGD/o7tujSAP n/Q11WTnDmADvEEkpuxA9qOmaajrVELPSW6Hap89AFdYnwbPTumwviP6X21rAIE3Mxsb VYK5MAAA0IYn3bvoVobtK4cxdkCF4QZrWYodPLYSdzApt87JWqq2dZ23NINYJd9riPuh yKgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102780; x=1741707580; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=XX0lx7hXVv6G10/M0h5aNOUcmPisRpn5zYzxA/BBWhO9E+ahPnNLG3LIVxmHkFnoD+ a77f0O0t3BH/YwLNuBLw0q4ApJGmIBoGwb3bWGayxD08mMAuftPH7zUZrMq9jbrHlL2H RklxKNgUUwhNV3Oxbkuj5UbSaknXVbGBJn28JAa5IkMQzsr6s9YKRY9ZFwunkDhqfBIk csu9HtpjIA1pmPpCJdtxyYwOiv4kd05FS12pcU3S4BLsSW/N15WxGzvp160+KJWRQeR7 dQhqZoYHjvo03X9YpKphyOSe20WNG3+MhB7Jyo8V5VfEpOvWROAHayjy478xGDM+rgwm PkkQ== X-Gm-Message-State: AOJu0YwofqqkiAa/Zy99yqLuFX0pouffLjqNFT5w1WxQOIcp/123gD0b Kr+7Qrwt8+ZO7Crlrm+jj5cIMmL1csAMNSQkzPi1RPD8nnncO3AGBYRzOw== X-Gm-Gg: ASbGncsDXhuWzNAOhRiQE4s3dii7QFnikeE05JrUqyzUkRY3fbO2GbNA5CKpTGr0rBg 1duqubVqc5D3Fzy0ShN+wc9gDf3SzMzrdx2+p9qfpepYtZnRaAciT0tdN3L4y+wKCr2z8yjoec3 Sb57x1MJ0izQq4PVCQu0TOmCaSj4n1BBn40QIpyY/FYVDz8QMulK00D7DUqLcMl84lPWn+cVHpV 1nUEH/l1MmzCpl7xfDiBxbuizei+QRymqKQ9NEiOr3+WPNfj5i4Xj6ecLoO5vZMYBhF/obF9glS z7fpt2FY2WKKv/dKzQAEIWbfpUtP X-Google-Smtp-Source: AGHT+IFZriAoSOp8xxvRw4vAE9UxOIJvovmxB3/Tv8kdwinW9oWTcfgPKQ1Sz6jIeCu2HzEn1cBHvQ== X-Received: by 2002:a05:6402:34c6:b0:5de:4a8b:4c9c with SMTP id 4fb4d7f45d1cf-5e4d6b62f6emr45746979a12.32.1741102779457; Tue, 04 Mar 2025 07:39:39 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:38 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 6/9] io_uring/net: pull vec alloc out of msghdr import Date: Tue, 4 Mar 2025 15:40:27 +0000 Message-ID: <3e9edc3be022d9dc2ada8577c73b3532f836aaad.1741102644.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I'll need more control over iovec management, move io_net_import_vec() out of io_msg_copy_hdr(). Signed-off-by: Pavel Begunkov --- io_uring/net.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 33076bd22c16..cbb889b85cfc 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -253,12 +253,8 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, return -EFAULT; sr->len = tmp_iov.iov_len; } - - return 0; } - - return io_net_import_vec(req, iomsg, (struct iovec __user *)uiov, - msg->msg_iovlen, ddir); + return 0; } static int io_copy_msghdr_from_user(struct user_msghdr *msg, @@ -328,10 +324,8 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, return -EFAULT; sr->len = tmp_iov.iov_len; } - return 0; } - - return io_net_import_vec(req, iomsg, msg->msg_iov, msg->msg_iovlen, ddir); + return 0; } static int io_sendmsg_copy_hdr(struct io_kiocb *req, @@ -342,6 +336,12 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, int ret; ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_SOURCE); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; return ret; @@ -719,6 +719,13 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) { + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_DEST); + if (unlikely(ret)) + return ret; + } return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, msg.msg_controllen); } From patchwork Tue Mar 4 15:40:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000978 Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C6B9298CB1 for ; Tue, 4 Mar 2025 15:39:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102785; cv=none; b=pmjaO4VLbOWCd7qdT12mm3+yf9d95eyh1q6SIDmKo1T2UMMEynnGtvCN1I0b/NfKtYdHge13Lk6tCX5WvecFt3T0XmuhwT3mzHHc0L63UTUcCm/BnbOQ0AJaqcLmo3IdJV5jQUg2XhPt5XusQbq1MNXiHzSejHi+VrDSRuGeERs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102785; c=relaxed/simple; bh=zWZarhNyJi53T++SjywLAAUqP8VMK2X1hgd352VZQsk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VxbsLEOZTIWZJWZ6xKvmKWT1JUlnEF6OHrFY9IKyh8wxggUWN2LOq1D9vq+h+5DHDQievq3UQKhbSMPa7Rc1Wh/3Nyj5C7XhcwtSCrFkmVQNKCkrirv8eBNms8iC7al75IH/t/D9PvzCsNFbeHZRoxeeN6PlYhpQt34WprpVlto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Yl2erDxx; arc=none smtp.client-ip=209.85.218.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Yl2erDxx" Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-abf45d8db04so547435666b.1 for ; Tue, 04 Mar 2025 07:39:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102781; x=1741707581; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7RWfRi6x1mSPiivij2EBvrzcTuGy50QoeAUbZiSRBp0=; b=Yl2erDxxvKlBSXiKMgGGxAORzJfvwr8Hn+nICktGhyaV3lBYF3khpSUBeE979v14WX oEDuRZVfzeMGtidc2KQW5gJEWB2NKKj81QIdrSliOoFW00fU6MJTk6gBnsiQW4cKEqbN xV0DxUIr+FoYt8YPBqpNp7JeOEggTfOv9qRldUwzbXBIkXFSaJfq2E54TdcUlckltqKM UjkfXU383PhTAPRgdW+OGDxyemp3LvnJH9roSaoKwTJRpuNvWX8uXiWaKn30QlFmjebc OlimAS1HlSvCbACLEJ8ixKXZE0k1mMpl27k7eK3J2RWNTSaJoXKzXFR4u/mIMdOqytZk cmEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102781; x=1741707581; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7RWfRi6x1mSPiivij2EBvrzcTuGy50QoeAUbZiSRBp0=; b=tKZTclnt/2pkzf1wFrFYhhIgUuAl7NCCV8V5XLANA6+hGiu0wX6zGk3w4Ov8Ev29SQ XevU1m/ONvRR+AEQ78UaML5ga6SXQjahaaugQxXGRScNCG1TMU7mx+u0lkSS6/B0Ik2I y6qO4H5hSXI8EBGuKNiC4W2OZLDXHS0Xh1QxIxB70MFlzVYBP5DB3eE3C4aTXie+EEVb QT3pClFNdsZdAAE8NwCXwOWGso6paryDEopXy76EdXL1h42gxU3za38k1HOtrRu8LJqQ VV8LXVl5ZL8ZvsaXJ+BiPQwnmp9W/lLSRRPP2zC1LrdN0QmpdnVl/ySiTEQH4ZKFnr+o ZFRQ== X-Gm-Message-State: AOJu0YzoiBPr3+FRHiV29690GuZaH+xpYZUKhz7EhYpXR3HnXEZt5ibI 70BhxX3V8i8BpoFBdtdaoqWAi3Odbr+nyWvo9bVEX89I5xm7TXpvXG7vog== X-Gm-Gg: ASbGnctoWygQUDfpIWvImdc8rMmEqkssktR4hGMC+KE6sB8OIubh0TGfJjmQN90pWqk hFIjQATfDtwF5WbOfSjWyRJGwzYGUlE1kMCmBjAgMOEWFwz5Q2sZ6toiz2BUcs7EhM7mKzFtXxF gKTC3CNl+6XD3aPmXfn+9lm+695+rpVpnx64DZ9Q7LwFt5gB0OilWHtjAyElvrFea8Zktgw/uDb eJtndV4r2y6AHeWUs/PnfpIVYVYfrAyDI/WjuDAKCg0cmK/6CQ89aRqR14hzGUd5t1HRyz4ChGK OzibfcO5BAXlVg/1Sjp37GI5gL6E X-Google-Smtp-Source: AGHT+IHMUHrwzkgKWkhbjtHORjGFuk9rE/fkXaSlVEt/16ru+vPKkCRSKX8XcrCVrXY5mPsp7YcOIg== X-Received: by 2002:a17:907:9622:b0:abf:4f72:538e with SMTP id a640c23a62f3a-abf4f728fd5mr1581937366b.55.1741102780780; Tue, 04 Mar 2025 07:39:40 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:39 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 7/9] io_uring/net: convert to struct iou_vec Date: Tue, 4 Mar 2025 15:40:28 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert net.c to use struct iou_vec. Signed-off-by: Pavel Begunkov --- io_uring/alloc_cache.h | 9 -------- io_uring/net.c | 51 ++++++++++++++++++------------------------ io_uring/net.h | 6 ++--- 3 files changed, 25 insertions(+), 41 deletions(-) diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 0dd17d8ba93a..7094d9d0bd29 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -16,15 +16,6 @@ bool io_alloc_cache_init(struct io_alloc_cache *cache, void *io_cache_alloc_new(struct io_alloc_cache *cache, gfp_t gfp); -static inline void io_alloc_cache_kasan(struct iovec **iov, int *nr) -{ - if (IS_ENABLED(CONFIG_KASAN)) { - kfree(*iov); - *iov = NULL; - *nr = 0; - } -} - static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, void *entry) { diff --git a/io_uring/net.c b/io_uring/net.c index cbb889b85cfc..a4b39343f345 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -136,11 +136,8 @@ static bool io_net_retry(struct socket *sock, int flags) static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) { - if (kmsg->free_iov) { - kfree(kmsg->free_iov); - kmsg->free_iov_nr = 0; - kmsg->free_iov = NULL; - } + if (kmsg->vec.iovec) + io_vec_free(&kmsg->vec); } static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) @@ -154,7 +151,7 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } /* Let normal cleanup path reap it if we fail adding to the cache */ - io_alloc_cache_kasan(&hdr->free_iov, &hdr->free_iov_nr); + io_alloc_cache_vec_kasan(&hdr->vec); if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -171,7 +168,7 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) return NULL; /* If the async data was cached, we might have an iov cached inside. */ - if (hdr->free_iov) + if (hdr->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; return hdr; } @@ -182,10 +179,7 @@ static void io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg { if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - kmsg->free_iov_nr = kmsg->msg.msg_iter.nr_segs; - if (kmsg->free_iov) - kfree(kmsg->free_iov); - kmsg->free_iov = iov; + io_vec_reset_iovec(&kmsg->vec, iov, kmsg->msg.msg_iter.nr_segs); } } @@ -208,9 +202,9 @@ static int io_net_import_vec(struct io_kiocb *req, struct io_async_msghdr *iomsg struct iovec *iov; int ret, nr_segs; - if (iomsg->free_iov) { - nr_segs = iomsg->free_iov_nr; - iov = iomsg->free_iov; + if (iomsg->vec.iovec) { + nr_segs = iomsg->vec.nr; + iov = iomsg->vec.iovec; } else { nr_segs = 1; iov = &iomsg->fast_iov; @@ -468,7 +462,7 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret) if (iter_is_ubuf(&kmsg->msg.msg_iter)) return 1; - iov = kmsg->free_iov; + iov = kmsg->vec.iovec; if (!iov) iov = &kmsg->fast_iov; @@ -584,9 +578,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, .nr_iovs = 1, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode = KBUF_MODE_FREE; } @@ -599,9 +593,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, if (unlikely(ret < 0)) return ret; - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } sr->len = arg.out_len; @@ -1085,9 +1079,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg .mode = KBUF_MODE_EXPAND, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode |= KBUF_MODE_FREE; } @@ -1106,9 +1100,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg } iov_iter_init(&kmsg->msg.msg_iter, ITER_DEST, arg.iovs, ret, arg.out_len); - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } } else { @@ -1874,8 +1868,7 @@ void io_netmsg_cache_free(const void *entry) { struct io_async_msghdr *kmsg = (struct io_async_msghdr *) entry; - if (kmsg->free_iov) - io_netmsg_iovec_free(kmsg); + io_vec_free(&kmsg->vec); kfree(kmsg); } #endif diff --git a/io_uring/net.h b/io_uring/net.h index b804c2b36e60..43e5ce5416b7 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -2,12 +2,12 @@ #include #include +#include struct io_async_msghdr { #if defined(CONFIG_NET) - struct iovec *free_iov; - /* points to an allocated iov, if NULL we use fast_iov instead */ - int free_iov_nr; + struct iou_vec vec; + struct_group(clear, int namelen; struct iovec fast_iov; From patchwork Tue Mar 4 15:40:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000979 Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94267298CC2 for ; Tue, 4 Mar 2025 15:39:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102788; cv=none; b=jQNYb74eQ5TiNHxSL/3WkaPm0LEBE8WDWYh9Oe5K1Qcn1HLsivFhouZQ8dltqfm7R4VyBsLSZFL1K9VnMIuD0h/BDI5XSVg+kJyaQPcPhJQoE2x039ePYqqXnXCoJRpG9IjuLIIpa7rgt4f43ZwP6C0ZmzjuP1RMfbnW6fAfkEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102788; c=relaxed/simple; bh=+ko6qVLwFbY8p2fCkuMHNRIwwTZQjN0AL9uVp4qcr08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=skXr+j+Bl1DS9yL27K4lIg/+dTahY7fvPBBlJ4qxZgRww74v6nMktbG4C5t25NrbzbMGy/t7X//zwlbcBlHFImEEhwwzfpq0/WX8Wy3mhpXH79Jq1A/T3k341gLeh68WzeeVLw8UeHfAsTHdUwk477TVyxV7cNZLeh1oFAoiXBI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=e2s502pV; arc=none smtp.client-ip=209.85.208.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="e2s502pV" Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-5dee07e51aaso10808691a12.3 for ; Tue, 04 Mar 2025 07:39:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102782; x=1741707582; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=e2s502pVCkXgkM8zMK1hVwPJ43lbTnwaPYIivTt11o7rvd+Ok+zI6yeNyRpG47CBhM ihr0VBwbSEVRGxykbXwNZ0CFJ5CJZAkeIImO7KFqdsyK7gDMmjBcruFLj0ReHnRMUKuE GDFbtCZOelY7wWFY4JPSO5PKTcu2M+PUmc0fycEMj/VAaGr4xjUH2RJCu/T7h0m6UkAA sQqzpoxY8BgGq2fLQ4PQJBNs0MenfPPKG5MGgd0wl1d1yX8HmgRALUW3rstOrJI0gDa7 VLZ4IaTcXX6nVlhp1TlWiplysp7cIKjoLb1b7GFDeWilRdxZP6WbapAWoFDx5Zkfxj9S ovYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102782; x=1741707582; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=YffEtfwDCfUD8wnqykZudJNM//au58xvqcL+qxuX+cwee7Ki/3btCcAEHREM/E38b6 pfYMeFXRUbYYuue6OHMtQP/S1u9/UGbjEecflh5YwuWEU41a6qdVKkvA+aVW7iHBFn2C knfliLZZqYPZHB0PyIxbgzd1JOg+jlbqtvUXfg4txnsQj4UC8j20ZlAlsfpN1d2TyArY yRSZjwfdcydfsBx888wXnZ/oJqtLdVj2h6Zglb10iX3nFKEq3BumH1f1ROE3WLecmfNq 5LH7Sl5aqqjsyUZLZRPv7GDpIA29Wcbghewr3ycm0VLdrFGjZGMvkkgYggimZ1yED8My RODw== X-Gm-Message-State: AOJu0YxudYMX2dqwP+RcR3ZQ3pHy7hRaJULoLgivJM2dzbSs2SKFzvMz pWWTpBVCeJmzRsMiGm56fAqMcCuou4qSTZ+3OuUZvp3eXuwRe4PpfxidOQ== X-Gm-Gg: ASbGncvaw1LLyqgsNFSE8v/poY8HM2PvqhBa2yhOV4bfPpU3djsaY6IwvomcrvBi/le 50X73KJZTfP3amfo1L/EcQY5G/SqLCWpejZpdY7pIL7SWdTQS2NCuoARKh/ngbqNIkd/C00FUZe D7MGKEesyAv89p07wB8at6BJxQ9c2VJw0a7635HeNJMLq6THmiuJUnrQMeZMOKldex9doejO9vp tQWPOzoqeUXGY1NC016OAyKaNwAO7J5fvNHIXgGG6Q6Ff68gtKYMVzaT5tGc4LvhGWm+LeSBSZA LwVB3VhchNylHY1PFhOG2hMK2r8f X-Google-Smtp-Source: AGHT+IF34EN7IqNH71tO9SUK+H1SrNCen0EOMQbCOMPvTq2DkNDEQy16A1SCj3loxqXH/zaL2MimOw== X-Received: by 2002:a17:907:96a7:b0:abf:5b43:5f27 with SMTP id a640c23a62f3a-abf5b4360damr1113282366b.19.1741102782401; Tue, 04 Mar 2025 07:39:42 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:41 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 8/9] io_uring/net: implement vectored reg bufs for zctx Date: Tue, 4 Mar 2025 15:40:29 +0000 Message-ID: <78e94466329f9c2814174219be2b72c03995f329.1741102644.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add support for vectored registered buffers for send zc. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index a4b39343f345..5e27c22e1d58 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -395,6 +395,44 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe return io_sendmsg_copy_hdr(req, kmsg); } +static int io_sendmsg_zc_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg = req->async_data; + struct user_msghdr msg; + int ret, iovec_off; + struct iovec *iov; + void *res; + + if (!(sr->flags & IORING_RECVSEND_FIXED_BUF)) + return io_sendmsg_setup(req, sqe); + + sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); + + ret = io_msg_copy_hdr(req, kmsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + sr->msg_control = kmsg->msg.msg_control_user; + + if (msg.msg_iovlen > kmsg->vec.nr || WARN_ON_ONCE(!kmsg->vec.iovec)) { + ret = io_vec_realloc(&kmsg->vec, msg.msg_iovlen); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + iovec_off = kmsg->vec.nr - msg.msg_iovlen; + iov = kmsg->vec.iovec + iovec_off; + + res = iovec_from_user(msg.msg_iov, msg.msg_iovlen, kmsg->vec.nr, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + kmsg->msg.msg_iter.nr_segs = msg.msg_iovlen; + req->flags |= REQ_F_IMPORT_BUFFER; + return ret; +} + #define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE) int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -1333,8 +1371,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->opcode != IORING_OP_SEND_ZC) { if (unlikely(sqe->addr2 || sqe->file_index)) return -EINVAL; - if (unlikely(zc->flags & IORING_RECVSEND_FIXED_BUF)) - return -EINVAL; } zc->len = READ_ONCE(sqe->len); @@ -1350,7 +1386,7 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return -ENOMEM; if (req->opcode != IORING_OP_SENDMSG_ZC) return io_send_setup(req, sqe); - return io_sendmsg_setup(req, sqe); + return io_sendmsg_zc_setup(req, sqe); } static int io_sg_from_iter_iovec(struct sk_buff *skb, @@ -1506,6 +1542,22 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) unsigned flags; int ret, min_ret = 0; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; + + if (req->flags & REQ_F_IMPORT_BUFFER) { + unsigned uvec_segs = kmsg->msg.msg_iter.nr_segs; + unsigned iovec_off = kmsg->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ITER_SOURCE, &kmsg->msg.msg_iter, req, + &kmsg->vec, uvec_segs, iovec_off, + issue_flags); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; + req->flags &= ~REQ_F_IMPORT_BUFFER; + } + sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; @@ -1524,7 +1576,6 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_control_user = sr->msg_control; kmsg->msg.msg_ubuf = &io_notif_to_data(sr->notif)->uarg; - kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); if (unlikely(ret < min_ret)) { From patchwork Tue Mar 4 15:40:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14000980 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73EAB298CD4 for ; Tue, 4 Mar 2025 15:39:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102788; cv=none; b=Vdk/+lOqn9ig3mO/89BhuRV/93byq+E+Iryz17NLNE3ar4Zy2SXOC6dQk6mSXtYVOGmS20vRtAdmCt0UzjND2kU+PRX285Hbxu3orhF7QYzbW+EDtwa/yX0/vofLIciGDbhp+Ztu+uh6Qftl3YawlIffAZy7dWfaWK0gad3fLoA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741102788; c=relaxed/simple; bh=4sK4VkwDijMad+gj86h50HxY2xUtZTdkNgwm9T1N7Fw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tvRmDzcK/q/CE+9EY5IjqrgYaXcWbWz2wq80fFBg+acOykybuXvcWPP1B7P7XwWKNia8T/xnt92cYGXMr07j+DCRKkA5fc6XFyln7D9awWUB+K04tx9I1WX2SzI81LsHQrfdQCmW6Yjq7HZxvjDgXWloL/gu9o8Tlo/PCGqwAYI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JtwB7r8J; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JtwB7r8J" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-abec8b750ebso1006558366b.0 for ; Tue, 04 Mar 2025 07:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741102784; x=1741707584; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o/9RNEb6QyHbVKv47iIof+34KhXOtpAV60Dp/7krGt8=; b=JtwB7r8JDYXwN3nAIAdaUNEYZD4O83F8dsI/+r60a7HCqNuSqB9DBrbwD4kAsRbi60 aK1lt0VJaIKgBdpmBFt49O6WW8NUks77TucrqqtezPjraQsr6Wv+mERoWW2h2/IM+tJq GExyr31a/mXnDFv0YDGC7oyRI8wSb2oDn8vA9K7t2dbSbv7FoOgcy95njTUwjIGBkxGn 6QYJNFRuI4z5iKkJJB/OfGEeHRYta5OlzsbbanqSRIlp7+GiFO6zEPP+gdH9DwZvrgl1 XvABKPk3grVThHpOsXPfcWfDJQ6+7dzPxqSL0kGgJ9SLP1WO7icTYgmTAzxv2Yv78Afe FYHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741102784; x=1741707584; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o/9RNEb6QyHbVKv47iIof+34KhXOtpAV60Dp/7krGt8=; b=MYbzZ2O12ot+D3t9RSOxmputf1CVlYm+4vcDGjdeYh3d+4kA1aUOZUmjiIK1Ah4YLB SrfXo8jPk2l35MyGlJ87azpp/CAnwOvQghvoIsP/ypg1DbPQVZ3KDr/WZYOrTbK7G28n 70tyfBI+bKlN+Ael0KZYOEr9wgKXQvBJmj8fzUUVL4OCKwttwKGabezJtfiIEng2QgwL ky/RiE433aa/f3iTP0F683zhfV7jHbzU4TJ9MUymAAEHQz55tdW1wTu3bAkIMys3RI7P AgJopyzxr6vmnaEBiYOJIxHcCUXbLP0g97OynrKUMd/mLrjdNhJeDeRa8cWTWmra/4rs Epwg== X-Gm-Message-State: AOJu0YxcbjwDXLlww7fGxE9cxRFnZPwepjAzX/8PVhD9OuwAaYu0DJL8 Ba4uMfW/KqDy+xkZsYlsGZyKinXMb7BjzZE/BUT2/EHCIUCbeyWba7M4Sg== X-Gm-Gg: ASbGncsK96/6IhQB/GN9Sb5EAih6z4tAD7KGzzwp5FCvZq7qPv/guqotg7Nff38G/Sd OuuhtHiJLBEhGcBNRFBWIRss2vYfh19zCnC59RKXiNpCe8Y8fS22cy+BRs0n0LwEVZciNcN31v9 lB9xDZuxreD+GkQjTySsI4UxOeaDgmyx2szeSmCjvLklogqqO8+UlyIWOXvblBSDI3SddIQIiDy K58NbAvI6cWGWdPwKoNV+2njSwTtz0rwH5MkzR2SiBdRqVjKGq6VHsna0X53CuzvsUMkeT+S81h /zKR3PqeCD4osc+qES5858XdqiSC X-Google-Smtp-Source: AGHT+IE4Wu79UPtv3RFcvZpLKQktzC0n27BtsI2xx0Ldv57Nrue9ofSuxEKYxV39rLcjISZiogT4Hg== X-Received: by 2002:a17:907:6a0e:b0:abf:4ca9:55ff with SMTP id a640c23a62f3a-abf4ca9583dmr1527233466b.32.1741102784064; Tue, 04 Mar 2025 07:39:44 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:3bd7]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ac1ecafa17fsm168420966b.162.2025.03.04.07.39.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Mar 2025 07:39:42 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH v2 9/9] io_uring: cap cached iovec/bvec size Date: Tue, 4 Mar 2025 15:40:30 +0000 Message-ID: <50ffecb2f87a3b80300d5c0d8482e20ed897ccd9.1741102644.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Bvecs can be large, put an arbitrary limit on the max vector size it can cache. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 3 +++ io_uring/rsrc.h | 2 ++ io_uring/rw.c | 3 +++ 3 files changed, 8 insertions(+) diff --git a/io_uring/net.c b/io_uring/net.c index 5e27c22e1d58..ce104d04b1e4 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -152,6 +152,9 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) /* Let normal cleanup path reap it if we fail adding to the cache */ io_alloc_cache_vec_kasan(&hdr->vec); + if (hdr->vec.nr > IO_VEC_CACHE_SOFT_CAP) + io_vec_free(&hdr->vec); + if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index f1496f7d844f..0bfcdba12617 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -6,6 +6,8 @@ #include #include +#define IO_VEC_CACHE_SOFT_CAP 256 + enum { IORING_RSRC_FILE = 0, IORING_RSRC_BUFFER = 1, diff --git a/io_uring/rw.c b/io_uring/rw.c index e62f4ce34171..bf35599d1078 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -150,6 +150,9 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) return; io_alloc_cache_vec_kasan(&rw->vec); + if (rw->vec.nr > IO_VEC_CACHE_SOFT_CAP) + io_vec_free(&rw->vec); + if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA;