From patchwork Fri Mar 7 15:49:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006673 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 284DC21D3C0 for ; Fri, 7 Mar 2025 15:48:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362494; cv=none; b=hBwBmZvP61EDarIgmu0ECqYSkftSJ0u3+/W0EUhJE0I/ddCGBzDgxO9VAsXOyOjaGxk73a4rUtDQ+1invTMU745U1F948fxdO2xeGaIK0Rc4AaRK+fxQx/V9/FFS7uP+aKtA1iiPB8HFAewJp/KbYZw0q+Wcu2LcsFbYKbcG7wg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362494; c=relaxed/simple; bh=Q8PdENXAiLSA678FCapF070tkBMsP1ce5Gykksi+Hcc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KUlBje9DnuvRzp8VStNbS+tVeuzT5wB6+QZB1osUoSnTCBRqxkH8YusJ8kZXdWZM7ibK+5GEZGgf1guilyRDeg5Ijuy+0k6DEklqRbLHSWxGe7JruH0/iRbnsE8hiGf2xkp3YiGcGR4K+tXlRDADsz1fv3NE6HzaxnlhpqrCqGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=K2t0nmGB; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="K2t0nmGB" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5e0b70fb1daso3683873a12.1 for ; Fri, 07 Mar 2025 07:48:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362491; x=1741967291; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oGewhYsabKASpjkFax+fSFYao6hHzoa78HnivAvFalA=; b=K2t0nmGB/hheIbzQodTNIxD26WRp6axmjKHOiF2WN6ts2DWmZNx93Jp8pb3a7a/gLw te7Fwsp2mfr4a6olb8bSd+ORjtmtozvmgj4GoMRTpeijvQNO6sjVuVIlohA8+3quYdFg PU12NbX2xo568tORS/9mgflFGfG+MawX78cSIYYf+r9JHlHThe/zzDXQ5nePRnAhNJqw RlIYcAiV7bcPRe1vLCUBfQSekJR7zsBbjssRK4H/RivPqDyDsTfy/Fhxf2iT4mKR9R4m z7jUCgxisBtHjIE2lmIkPfKlgLayT8XqvxTD6UTy4MqSVvLEG2YgdOyRUSYMhg0G/cT6 rLWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362491; x=1741967291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oGewhYsabKASpjkFax+fSFYao6hHzoa78HnivAvFalA=; b=m6ypTyClkVVdIPqDyhMN3euLs1UTzYWd8cZHZJ/zGKoxqHb1BOPb7HW0GMpctOJ+ix Hs540N/scfjLk0pbaWYQj9ewZRRnbE4dnGBVGwH/xlqRM31jbfFaHIkdQOJLR90hPx9L sVNQKE7EQnY/un7ZHmRyoWmBdxc/X1B3V9iPMgrxMD5tJSCW7ESoYLzsPMPuW9MLwN4O dQFmiwetp5fYnNJWFGR++3iHG716f4Itrr7JZIoa8wHIJjGzCDHxE9YCdYPGv5hK/FkW jcmycB4bofxlasAwLpWQOWBoe2HZeHH/mCuEzi9ClP35vSmOFMzffYngtYzCsWCSDamh IhAA== X-Gm-Message-State: AOJu0YzYaSakAgT5+2Ifrbzhi3MlmR85W18b/W/MUyVjaex71Q+YaNig nqm1xFdSKBmLbmpsuDZ3UbQJyGuzjxcA88iSq1GJchRhAcq0f+dyWUcaNQ== X-Gm-Gg: ASbGncsvdksUdJWKKoY6Kx1BCB5PQCfqEykQLuXz3yCGOdZSSkSQW3JicUR6tZ+T3AI ZnFn79SzFWAzAjtvfTjUAedCqbkVDq5yl5UOICuLDumMqXVbzMdZxuoHcgoS6tdJX9zG7X8/IHP CXb10s4JjDgVJeQOvoLvQvFYAU+5BFG5DpwJWSbKopywN0V333oypqc3KRjrd3Lg86A+4EzKf3K TQK0iRSc2M20S+yyoVWFyT0Gi7pinJRBOc/npbqiKgRcZzlVO6uA+tL6Nat7m4gDPw7o9DoH5F6 g6WgPp0CFsu35o1yl0Wkl+gUpHFr X-Google-Smtp-Source: AGHT+IF9rUDg+UlQM7x7vGlNMJkJgvGM91lCW0qlgToELVBvIkY5ymi3nTOLLoMQa43wgi3OFpcGbA== X-Received: by 2002:a05:6402:518a:b0:5d9:82bc:ad06 with SMTP id 4fb4d7f45d1cf-5e5e22bf16amr4766917a12.3.1741362490583; Fri, 07 Mar 2025 07:48:10 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:09 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 1/9] io_uring: introduce struct iou_vec Date: Fri, 7 Mar 2025 15:49:02 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I need a convenient way to pass around and work with iovec+size pair, put them into a structure and makes use of it in rw.c Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +++++ io_uring/rsrc.c | 9 +++++++++ io_uring/rsrc.h | 17 +++++++++++++++++ io_uring/rw.c | 17 +++++++---------- io_uring/rw.h | 4 ++-- 5 files changed, 40 insertions(+), 12 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 35fc241c4672..9101f12d21ef 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -110,6 +110,11 @@ struct io_uring_task { } ____cacheline_aligned_in_smp; }; +struct iou_vec { + struct iovec *iovec; + unsigned nr; +}; + struct io_uring { u32 head; u32 tail; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 28783f1dde00..bac509f85c80 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1260,3 +1260,12 @@ int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg) fput(file); return ret; } + +void io_vec_free(struct iou_vec *iv) +{ + if (!iv->iovec) + return; + kfree(iv->iovec); + iv->iovec = NULL; + iv->nr = 0; +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 284e300e63fb..ff78ead6bc75 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -4,6 +4,7 @@ #include #include +#include enum { IORING_RSRC_FILE = 0, @@ -145,4 +146,20 @@ static inline void __io_unaccount_mem(struct user_struct *user, atomic_long_sub(nr_pages, &user->locked_vm); } +void io_vec_free(struct iou_vec *iv); + +static inline void io_vec_reset_iovec(struct iou_vec *iv, + struct iovec *iovec, unsigned nr) +{ + io_vec_free(iv); + iv->iovec = iovec; + iv->nr = nr; +} + +static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv) +{ + if (IS_ENABLED(CONFIG_KASAN)) + io_vec_free(iv); +} + #endif diff --git a/io_uring/rw.c b/io_uring/rw.c index 5ee9f8949e8b..ad7f647d48e9 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -87,9 +87,9 @@ static int io_import_vec(int ddir, struct io_kiocb *req, int ret, nr_segs; struct iovec *iov; - if (io->free_iovec) { - nr_segs = io->free_iov_nr; - iov = io->free_iovec; + if (io->vec.iovec) { + nr_segs = io->vec.nr; + iov = io->vec.iovec; } else { nr_segs = 1; iov = &io->fast_iov; @@ -101,9 +101,7 @@ static int io_import_vec(int ddir, struct io_kiocb *req, return ret; if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - io->free_iov_nr = io->iter.nr_segs; - kfree(io->free_iovec); - io->free_iovec = iov; + io_vec_reset_iovec(&io->vec, iov, io->iter.nr_segs); } return 0; } @@ -151,7 +149,7 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) return; - io_alloc_cache_kasan(&rw->free_iovec, &rw->free_iov_nr); + io_alloc_cache_vec_kasan(&rw->vec); if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -201,7 +199,7 @@ static int io_rw_alloc_async(struct io_kiocb *req) rw = io_uring_alloc_async_data(&ctx->rw_cache, req); if (!rw) return -ENOMEM; - if (rw->free_iovec) + if (rw->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; rw->bytes_done = 0; return 0; @@ -1327,7 +1325,6 @@ void io_rw_cache_free(const void *entry) { struct io_async_rw *rw = (struct io_async_rw *) entry; - if (rw->free_iovec) - kfree(rw->free_iovec); + io_vec_free(&rw->vec); kfree(rw); } diff --git a/io_uring/rw.h b/io_uring/rw.h index bf121b81ebe8..529fd2f96a7f 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -9,13 +9,13 @@ struct io_meta_state { }; struct io_async_rw { + struct iou_vec vec; size_t bytes_done; - struct iovec *free_iovec; + struct_group(clear, struct iov_iter iter; struct iov_iter_state iter_state; struct iovec fast_iov; - int free_iov_nr; /* * wpq is for buffered io, while meta fields are used with * direct io From patchwork Fri Mar 7 15:49:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006674 Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 916F0DF71 for ; Fri, 7 Mar 2025 15:48:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362496; cv=none; b=VYy1uUm1c3muAchS4qelkClMmmjtmZTKsHmg063UCedD8PSKEaajcgH/wHX2OcmXjveCbDUHdWMlTGYCxnn33ZZkNtHyEatn3e+9X6EvB6m93ImcmO+L7WpaaRBmvFsN7q0ODL2LlqqYVXuMoBG2hpd4EgbrRvIq1omZftmP8uA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362496; c=relaxed/simple; bh=+R+is5ZIjZWSMpaQwhdZechILV8Q6KDGIXuR4qpgbSM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CWOsmNC21//IWC3vcNDTEOqfGm5uJ/ckakF84kb2aoXY33OyStEB6nr1XR/I9HmdbXWaIh1qz0oqgZ4GkduQ1f8rH2KNiQ964KjIBkMN18RGV7uE34VWStv3Vt+auPMx6f7hbiyDlMqzwrxlKcZcE4VbEGa7v4Z/SpJIKQNHepA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JBSWF2ww; arc=none smtp.client-ip=209.85.208.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JBSWF2ww" Received: by mail-ed1-f46.google.com with SMTP id 4fb4d7f45d1cf-5e5e22e6ed2so1772167a12.3 for ; Fri, 07 Mar 2025 07:48:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362492; x=1741967292; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ERi6eo1GFgBzJUmgrKyeNKQEukSFbVEJOWBSsBRX7ZQ=; b=JBSWF2wwsYXVWzwH0gTF5tCiN4Uyu+hvSQuAiW0IA8sYG17eOzk4eL0+8taTn1InVm lnqKWd33JCWBxnaaTXZGyyMqiZ66qUMu/8+MZKS1bzBLbo99FuvtgP8pgTga/yGrgMVO b5CeB2anKEqXHGo/XsBwkHFLiIrrUmLl+1ZJRDucpcfyQuJ5DChjOkqm8iMfek/+VnyH 4aioWWOLlosnQ7JRV8EhyK02zpIon3DKIiCFyvAG6+nM/clu881mSsI3nvpotFpZcLM6 MPRkvsaRq34zRNGGga4ZqqZ/GxGwQ2pwnAFdrv6MpzNOCcwfG7K0jZRmXpz4txmzfDtD q4DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362492; x=1741967292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ERi6eo1GFgBzJUmgrKyeNKQEukSFbVEJOWBSsBRX7ZQ=; b=RPQqWzss/Pd4bVUYIpKfVr2FVaMhFeM44XHc7yNNNErebBUShxJxfDKrIEWxFJUsQ2 J+3vB412vtrd1zLf6+Fu6rlpyaSHEPXTYvj2TXF7/az93jNAr9rd4jg0VOda9nGBsWaG 2P5rXxyMXDI/DJ1pSUjThIBNFt0mEBZRZzGl4ImbE+69Ca8MV2w9biCroWzIGhicJlhF voix77vATZip+N9Jlz9/cK76TFPFVdoC2OyFOFWlaU5wE+JTQDvO4ty7yL3gxhJNliAV P8xc3ftC7WSnUdC7oNYX208yJQyrwOjAbnr7B4ZjgVDB4PtLvyG1dWInFMVjvmZACkI9 u00w== X-Gm-Message-State: AOJu0YybZdRY3QCbrgIuAUCitgmUmxSN+B/6nokCOr+sf9L6Tvm/4ZeS yTa89l1QmteZkG1qJn7fPAdvxd3Knt42gqD1eFtoNK/gVlNRIcAntavXag== X-Gm-Gg: ASbGncthIMPA3O5RsYNOxy4/WiftIffEuQ4Js1cZ8v5EJVsn+14wohOQRnj9iS2LZy8 5THeHLtIOBpgEfHH2BRUiXgV4ZKyHfcoqne8eOHQPU3KrxzIxjyEYgv9mmwCGJ+0KQ0w2WEATXe Mk7ttYCZXG+3B/0jlW42UY6qv07nj0Nf1JIGefFd45yHUJZ2CJExMnDg9zfKyK0ebNciLvre9VD ttbKD6njndqOYAHkhVZ1IRZuF1gi4SO5eyD5a9MnMU26UKx1CtDmKQ72WbWzKrMKERJbaWBj2Lu LWLX4tCCmYpYJxpiO12vhLVN41mi X-Google-Smtp-Source: AGHT+IHyrhyxhw4WMheV+3vPBRqx9UznAD3HsIojBS+FTli+iTKLT84MG263QT2J/W7foOAHDyANrA== X-Received: by 2002:a05:6402:3553:b0:5e4:d4d3:569d with SMTP id 4fb4d7f45d1cf-5e5e24bbaa6mr3671901a12.22.1741362492079; Fri, 07 Mar 2025 07:48:12 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:10 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 2/9] io_uring: add infra for importing vectored reg buffers Date: Fri, 7 Mar 2025 15:49:03 +0000 Message-ID: <67b7ff6530ae2b44a8e69704302601d40a694da4.1741361926.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add io_import_reg_vec(), which will be responsible for importing vectored registered buffers. The function might reallocate the vector, but it'd try to do the conversion in place first, which is why it's required of the user to pad the iovec to the right border of the cache. Overlapping also depends on struct iovec being larger than bvec, which is not the case on e.g. 32 bit architectures. Don't try to complicate this case and make sure vectors never overlap, it'll be improved later. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 7 +- io_uring/rsrc.c | 128 +++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 ++ 3 files changed, 138 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9101f12d21ef..cc84f6e5a64c 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -111,8 +111,11 @@ struct io_uring_task { }; struct iou_vec { - struct iovec *iovec; - unsigned nr; + union { + struct iovec *iovec; + struct bio_vec *bvec; + }; + unsigned nr; /* number of struct iovec it can hold */ }; struct io_uring { diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index bac509f85c80..8327c0ffca68 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1269,3 +1269,131 @@ void io_vec_free(struct iou_vec *iv) iv->iovec = NULL; iv->nr = 0; } + +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries) +{ + gfp_t gfp = GFP_KERNEL | __GFP_NOWARN; + struct iovec *iov; + + iov = kmalloc_array(nr_entries, sizeof(iov[0]), gfp); + if (!iov) + return -ENOMEM; + + io_vec_free(iv); + iv->iovec = iov; + iv->nr = nr_entries; + return 0; +} + +static int io_vec_fill_bvec(int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + struct iovec *iovec, unsigned nr_iovs, + struct iou_vec *vec) +{ + unsigned long folio_size = 1 << imu->folio_shift; + unsigned long folio_mask = folio_size - 1; + u64 folio_addr = imu->ubuf & ~folio_mask; + struct bio_vec *res_bvec = vec->bvec; + size_t total_len = 0; + unsigned bvec_idx = 0; + unsigned iov_idx; + + for (iov_idx = 0; iov_idx < nr_iovs; iov_idx++) { + size_t iov_len = iovec[iov_idx].iov_len; + u64 buf_addr = (u64)iovec[iov_idx].iov_base; + struct bio_vec *src_bvec; + size_t offset; + u64 buf_end; + + if (unlikely(check_add_overflow(buf_addr, (u64)iov_len, &buf_end))) + return -EFAULT; + if (unlikely(buf_addr < imu->ubuf || buf_end > (imu->ubuf + imu->len))) + return -EFAULT; + if (unlikely(!iov_len)) + return -EFAULT; + if (unlikely(check_add_overflow(total_len, iov_len, &total_len))) + return -EOVERFLOW; + + /* by using folio address it also accounts for bvec offset */ + offset = buf_addr - folio_addr; + src_bvec = imu->bvec + (offset >> imu->folio_shift); + offset &= folio_mask; + + for (; iov_len; offset = 0, bvec_idx++, src_bvec++) { + size_t seg_size = min_t(size_t, iov_len, + folio_size - offset); + + bvec_set_page(&res_bvec[bvec_idx], + src_bvec->bv_page, seg_size, offset); + iov_len -= seg_size; + } + } + if (total_len > MAX_RW_COUNT) + return -EINVAL; + + iov_iter_bvec(iter, ddir, res_bvec, bvec_idx, total_len); + return 0; +} + +static int io_estimate_bvec_size(struct iovec *iov, unsigned nr_iovs, + struct io_mapped_ubuf *imu) +{ + unsigned shift = imu->folio_shift; + size_t max_segs = 0; + unsigned i; + + for (i = 0; i < nr_iovs; i++) + max_segs += (iov[i].iov_len >> shift) + 2; + return max_segs; +} + +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned iovec_off, + unsigned issue_flags) +{ + struct io_rsrc_node *node; + struct io_mapped_ubuf *imu; + struct iovec *iov; + unsigned nr_segs; + + node = io_find_buf_node(req, issue_flags); + if (!node) + return -EFAULT; + imu = node->buf; + if (imu->is_kbuf) + return -EOPNOTSUPP; + if (!(imu->dir & (1 << ddir))) + return -EFAULT; + + iov = vec->iovec + iovec_off; + nr_segs = io_estimate_bvec_size(iov, nr_iovs, imu); + + if (sizeof(struct bio_vec) > sizeof(struct iovec)) { + size_t bvec_bytes; + + bvec_bytes = nr_segs * sizeof(struct bio_vec); + nr_segs = (bvec_bytes + sizeof(*iov) - 1) / sizeof(*iov); + nr_segs += nr_iovs; + } + + if (WARN_ON_ONCE(iovec_off + nr_iovs != vec->nr) || + nr_segs > vec->nr) { + struct iou_vec tmp_vec = {}; + int ret; + + ret = io_vec_realloc(&tmp_vec, nr_segs); + if (ret) + return ret; + + iovec_off = tmp_vec.nr - nr_iovs; + memcpy(tmp_vec.iovec + iovec_off, iov, sizeof(*iov) * nr_iovs); + io_vec_free(vec); + + *vec = tmp_vec; + iov = vec->iovec + iovec_off; + req->flags |= REQ_F_NEED_CLEANUP; + } + + return io_vec_fill_bvec(ddir, iter, imu, iov, nr_iovs, vec); +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index ff78ead6bc75..f1496f7d844f 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -62,6 +62,10 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req, int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, unsigned issue_flags); +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned iovec_off, + unsigned issue_flags); int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg); int io_sqe_buffers_unregister(struct io_ring_ctx *ctx); @@ -147,6 +151,7 @@ static inline void __io_unaccount_mem(struct user_struct *user, } void io_vec_free(struct iou_vec *iv); +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries); static inline void io_vec_reset_iovec(struct iou_vec *iv, struct iovec *iovec, unsigned nr) From patchwork Fri Mar 7 15:49:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006675 Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1FA9194C78 for ; Fri, 7 Mar 2025 15:48:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362497; cv=none; b=EP7EZlWGpzdseHdpbdVD/qP05iYa/Z0aEVQ5JPgOpPJQ47jN/rizQYtXyQIKzeiLyrWoDH8pSLg/QZsO8H5bxbuWZCOzIV5cR4od4robi+7D9ouIzguTTP5ybX/BI5l45eIiqEzDKY2mp3N8bi0bP/ojtV/05ket0SB/iiP8RQ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362497; c=relaxed/simple; bh=CTDOfIKLiLS36vq6sL2sgn5lKY5hbvt0oCtkq6B9W9M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SmSetNzUManLWJ55LRQksHLbIRtHGnMBPsly3Zn74uCuzQaV3RULJeYHyzF9N6CyhsqqAfspgsjfBj4v6XZqooBasYi4dcfQIdmkpsJ2rRsB49UvhrdBN1s7srUZTDRrI6wAjty/ASPViRxYMlbc2CE/2IImOHTidiwJiAiwvgo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XTJA2GJX; arc=none smtp.client-ip=209.85.208.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XTJA2GJX" Received: by mail-ed1-f51.google.com with SMTP id 4fb4d7f45d1cf-5e5cded3e2eso2739825a12.0 for ; Fri, 07 Mar 2025 07:48:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362494; x=1741967294; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=REKT0J5DTDfZxhTt80Rxhy0qO38B5z+7gPhcbN7SaII=; b=XTJA2GJXKKEsHSGTouSB+QvviV0w4GIZyz+RGGa6E/8IdhByxhS1Okr7Gttr60Omiq kxi+CVarsbzaDuCPeB70JDPtcpERLBTRlWGDy2myv8+wBQf3pnpT0XGzNQmJpFKeJ3lS ubCfuBdfmiZ0AZVlkSBG8WHBApkn+6VyjDlwT83J1wmBMCs15QIIrTEVZOxE68dH98b/ JzVRw4KrBMtj59qWBrw00qNf9APE4sVaXXe+GmglxxGBDapXORoD/Vqk59dDp5djt0I6 o/1KBV6HXiPF5hLNNn+ataympqpQ7YO11drRxA+Qvcn3hU7yxrR93a6w2m4HMg1E5VKm PCEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362494; x=1741967294; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=REKT0J5DTDfZxhTt80Rxhy0qO38B5z+7gPhcbN7SaII=; b=HvvhHkKpwBuyIjNBHQsZHTsJKrSOn/jYe7V4Q5BxWLkPgMicfXU1p0M4UUAKnDoRg4 jb63xh1RBvEoNREuGXgO8F4TZk69NCeIv8MJnbd++tqQmq5LbNL+SuOVX8ANxaQ+r6lT jEaqjjA+di3BV2h4HgW90h7we7WNzl40b8QeKqZ5LWhtlb3yWlwPXNbzO8mywhOwHlYI KuqLTz8XEEmknouzGADxQSBUXw/dqDaGHEXb/QgoP2oWiVsBiuSr6YrAWFqp83j+44zY 16SiKGcQ1wSZiGDpOJfBcywjyNb8tHlN7Wjx8a6QtI0jGqJKhWxwIXttQTw3NKn6lKEs 9E5g== X-Gm-Message-State: AOJu0YxygNqCZLy8PkBKtYJG4Mbqk951DdM8qxVgCuGNWM8pESwCG5JB vpM7KINigSfyI+Kb36u6nB8+3HDQ3Rfji6e030FpQFc1Sp5Ge8qvuWQM1Q== X-Gm-Gg: ASbGncuRI9uPcIa2CrnxcrOiKADK2Ruz7FKmt4a6cLbz/9k6a/l6Ryp/2n/U+DIPEKc 8yeaAVvq4L4CT40ja/Oz6QdT8oUbfc/V0IKFJAraRKm+03pnFOfo8XMNl09Mslfk+Aat/KZyIRH a2hv8GamGxMe2QxFBUNomDUBhZKt5j5gZaY1H3/2YHwSQBgMLiJIfnfWh0a7qkeQSa6k0+VtG+y OpDxqqLLrFOupmAmyhpg9haXWNEcqwpv1zrplRXrfy8YPqgXIwwO5PllsSKdl+/zk1dNWTqbDt9 IDLAkJrK8D8wrB/JSrS59e9AaPFx X-Google-Smtp-Source: AGHT+IFFy4WaJE3ajLQ1SRwAJzN7hQgbSbqi4RDF9DWFDuDMdSHp5dm2Wc9bvgfrKYcHuL2m1f1xlA== X-Received: by 2002:a05:6402:2793:b0:5dc:cf9b:b048 with SMTP id 4fb4d7f45d1cf-5e5e22a96bbmr3705563a12.1.1741362493511; Fri, 07 Mar 2025 07:48:13 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:12 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 3/9] io_uring/rw: implement vectored registered rw Date: Fri, 7 Mar 2025 15:49:04 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Implement registered buffer vectored reads with new opcodes IORING_OP_WRITEV_FIXED and IORING_OP_READV_FIXED. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 2 ++ io_uring/opdef.c | 39 +++++++++++++++++++++++++++ io_uring/rw.c | 51 +++++++++++++++++++++++++++++++++++ io_uring/rw.h | 2 ++ 4 files changed, 94 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 3d99bf9bbf61..9e5eec7490bb 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -281,6 +281,8 @@ enum io_uring_op { IORING_OP_LISTEN, IORING_OP_RECV_ZC, IORING_OP_EPOLL_WAIT, + IORING_OP_READV_FIXED, + IORING_OP_WRITEV_FIXED, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/opdef.c b/io_uring/opdef.c index db77df513d55..7fd173197b1e 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -540,6 +540,35 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_READV_FIXED] = { + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_readv_fixed, + .issue = io_read, + }, + [IORING_OP_WRITEV_FIXED] = { + .needs_file = 1, + .hash_reg_file = 1, + .unbound_nonreg_file = 1, + .pollout = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_writev_fixed, + .issue = io_write, + }, }; const struct io_cold_def io_cold_defs[] = { @@ -775,6 +804,16 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_EPOLL_WAIT] = { .name = "EPOLL_WAIT", }, + [IORING_OP_READV_FIXED] = { + .name = "READV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, + [IORING_OP_WRITEV_FIXED] = { + .name = "WRITEV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, }; const char *io_uring_get_opcode(u8 opcode) diff --git a/io_uring/rw.c b/io_uring/rw.c index ad7f647d48e9..4c4229f41aaa 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,6 +381,57 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } +static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + struct io_async_rw *io = req->async_data; + const struct iovec __user *uvec; + size_t uvec_segs = rw->len; + struct iovec *iov; + int iovec_off, ret; + void *res; + + if (uvec_segs > io->vec.nr) { + ret = io_vec_realloc(&io->vec, uvec_segs); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + /* pad iovec to the right */ + iovec_off = io->vec.nr - uvec_segs; + iov = io->vec.iovec + iovec_off; + uvec = u64_to_user_ptr(rw->addr); + res = iovec_from_user(uvec, uvec_segs, uvec_segs, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, 0); + iov_iter_save_state(&io->iter, &io->iter_state); + return ret; +} + +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_DEST); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_DEST); +} + +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_SOURCE); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_SOURCE); +} + /* * Multishot read is prepared just like a normal read/write request, only * difference is that we set the MULTISHOT flag. diff --git a/io_uring/rw.h b/io_uring/rw.h index 529fd2f96a7f..81d6d9a8cf69 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -32,6 +32,8 @@ struct io_async_rw { int io_prep_read_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_readv(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_writev(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_read(struct io_kiocb *req, const struct io_uring_sqe *sqe); From patchwork Fri Mar 7 15:49:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006676 Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01BCDDF71 for ; Fri, 7 Mar 2025 15:48:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362498; cv=none; b=Y+1zJRgZLfHZxVoanyfcOYt9PakEgv90/b9kfAUMLSFgwFAojzkc6AyBzOKppCgxpSXSWUkjEO0XWBUJx+W24agoHARBH27bkReMgxHxls0sZny7cHEHX1lbk+rRPl3u6zt/A+pUpfNF5WMKMStL/8v3tnEz6fC5EzAzivhdPdo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362498; c=relaxed/simple; bh=gNKF/L5J2pAWM9hL6RvdhhPWEP4Yq0qat+0Q7PL2fuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=vBSAa+kK3wL4R286SwHuslr240MiRH1JNLCclZEARsYl7LNssGjwNE6K+tyXZdJXBEImSzEagaecbK1cOgcLnhQKZl2g9vk8K7jJs70KPuOW9+It32emW8XmSSs3kbxOXuyaBkUslTbOeIoLL6pQkYFp4CXNav4Sz+kDmvfKtj0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MkdgrSB8; arc=none smtp.client-ip=209.85.208.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MkdgrSB8" Received: by mail-ed1-f43.google.com with SMTP id 4fb4d7f45d1cf-5e5e7fd051bso1012066a12.0 for ; Fri, 07 Mar 2025 07:48:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362495; x=1741967295; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=93tvkzOu0v7DF9VuZKcubARhjargqjujprNt6uu0n0M=; b=MkdgrSB8/H7r9EC+eLNdtSGmwgIsxuOfWg2MZt7yEDe/i0fAagm0/vrNWMEto+dbqX Q1lITNDz4tA8VGjQx68e/by1QA+rBpMFWNQ13zC5wjwYkG0BbsnOkJHRi/25wHeRTldp 1uvjYwCwcNDUa5mrqyi2oyO8gyQKl0wdqyx9TAcPwPneSkdth/pgM6s2sDd/IBf7EMLy qw7SaCfXXWH2wYshCcdbga6wSx3sN7u89tWTa5lS2aFQ5BlZb+zUAW1ram3Byii1pIzM kS2+93IeplEeL+fzIIm0PRLLreKvVfVsrT+tYu6wKJUr2lGlpKmmp6eqBBKqFEzPEHgD 80sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362495; x=1741967295; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=93tvkzOu0v7DF9VuZKcubARhjargqjujprNt6uu0n0M=; b=NZfuczU6dUH6HJ84s4T/JwAcF8S+dW1VA3tZDfvfKLq6tY5JAu/9FxOXzYcPsN7cPM yGj2jmmC7JprJxxJniPG6iZ0BLDfMTqWjny+rLYgh3YBzIne0v8LKc4u1Njc+N14wXVg UsBp4Pz6SsyNX1JgKHjQR5gvDzSVxhKr6o6oQRxswQYZeFY7iSpsK8FgqzJvH6v7+BeN ED2qW5b4Tb1d9N5hzJI2eXH3wmit1smJnmomYsKmnmPpybLcWuise0003hmWOBMHkQzP RzthoReh6yUklZa0/XeJGMESfmX2cC/5Ucnw4fZ5mG0zBB40ITjPzfAeRm8lZEWoM2Oa lACg== X-Gm-Message-State: AOJu0Yx5BT84KGWhRWnnzdl/81rh4juW80PP/vJYSxjISqwFmvCPqpnU hDQ6FhxAnurtvhjrbInjQB9nNU6JjG4kMmAFY8ydGbB6q5kILzSAdcPdeQ== X-Gm-Gg: ASbGncuwaZ6HtyrSyoRjHemSyWm9aKTAF9wFcue2kcQZU/zLAIVrvX2DFysFHGqbqpX bwBHtRK2SXf8G6rn2MOpCfwKteuJOQNtjLmXZ2JK3UNGP1Z4aSYZaQrdylMrcHG0Rp9OppBnAO3 xaHu+5a6gCa0Oc+sYAAtAytGD4N0bSnftO660ii0bRxicPPWJCpAmWGVbjE9svYplbNlclV4i+c ndmK/KnwN0OwlZ1DSGyMtMnN0noHFotFuuhKvLpY+0W7H7qo9Yq/7n1a4lmoU5fMADQLfXv/rjO D50GccKJjxDdDeeiza/tqp5d+HZs X-Google-Smtp-Source: AGHT+IEtQCqShBzxU3wM1ns//mwtr6og3quzCi8dHnjyWACbmbhVQQ90qPJ+6sCe+n8wqLc9yac1Ig== X-Received: by 2002:a05:6402:27d3:b0:5e4:d2d4:b4f3 with SMTP id 4fb4d7f45d1cf-5e5e22cb5a3mr4748811a12.14.1741362494745; Fri, 07 Mar 2025 07:48:14 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:13 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 4/9] io_uring/rw: defer reg buf vec import Date: Fri, 7 Mar 2025 15:49:05 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Import registered buffers for vectored reads and writes later at issue time as we now do for other fixed ops. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +++ io_uring/rw.c | 42 +++++++++++++++++++++++++++------- 2 files changed, 37 insertions(+), 8 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index cc84f6e5a64c..0e87e292bfb5 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -502,6 +502,7 @@ enum { REQ_F_BUFFERS_COMMIT_BIT, REQ_F_BUF_NODE_BIT, REQ_F_HAS_METADATA_BIT, + REQ_F_IMPORT_BUFFER_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -584,6 +585,8 @@ enum { REQ_F_BUF_NODE = IO_REQ_FLAG(REQ_F_BUF_NODE_BIT), /* request has read/write metadata assigned */ REQ_F_HAS_METADATA = IO_REQ_FLAG(REQ_F_HAS_METADATA_BIT), + /* resolve padded iovec to registered buffers */ + REQ_F_IMPORT_BUFFER = IO_REQ_FLAG(REQ_F_IMPORT_BUFFER_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, io_tw_token_t tw); diff --git a/io_uring/rw.c b/io_uring/rw.c index 4c4229f41aaa..e62f4ce34171 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,7 +381,25 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } -static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) +static int io_rw_import_reg_vec(struct io_kiocb *req, + struct io_async_rw *io, + int ddir, unsigned int issue_flags) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + unsigned uvec_segs = rw->len; + unsigned iovec_off = io->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, issue_flags); + if (unlikely(ret)) + return ret; + iov_iter_save_state(&io->iter, &io->iter_state); + req->flags &= ~REQ_F_IMPORT_BUFFER; + return 0; +} + +static int io_rw_prep_reg_vec(struct io_kiocb *req) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); struct io_async_rw *io = req->async_data; @@ -406,10 +424,8 @@ static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) if (IS_ERR(res)) return PTR_ERR(res); - ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, - uvec_segs, iovec_off, 0); - iov_iter_save_state(&io->iter, &io->iter_state); - return ret; + req->flags |= REQ_F_IMPORT_BUFFER; + return 0; } int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -419,7 +435,7 @@ int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) ret = __io_prep_rw(req, sqe, ITER_DEST); if (unlikely(ret)) return ret; - return io_rw_prep_reg_vec(req, ITER_DEST); + return io_rw_prep_reg_vec(req); } int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -429,7 +445,7 @@ int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) ret = __io_prep_rw(req, sqe, ITER_SOURCE); if (unlikely(ret)) return ret; - return io_rw_prep_reg_vec(req, ITER_SOURCE); + return io_rw_prep_reg_vec(req); } /* @@ -906,7 +922,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret; loff_t *ppos; - if (io_do_buffer_select(req)) { + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_DEST, issue_flags); + if (unlikely(ret)) + return ret; + } else if (io_do_buffer_select(req)) { ret = io_import_rw_buffer(ITER_DEST, req, io, issue_flags); if (unlikely(ret < 0)) return ret; @@ -1117,6 +1137,12 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret, ret2; loff_t *ppos; + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_SOURCE, issue_flags); + if (unlikely(ret)) + return ret; + } + ret = io_rw_init_file(req, FMODE_WRITE, WRITE); if (unlikely(ret)) return ret; From patchwork Fri Mar 7 15:49:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006677 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F24A521D3D5 for ; Fri, 7 Mar 2025 15:48:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362499; cv=none; b=oV1rlhzUkryttj/YqXQpUUOTdOazVKdo1sAyw39BRu0E9KeSycExrZ3AwOkZVCncRoy7XO3TB50wkl3b3EsnlR63Kk2MFIyUmrs/nXcB6HWxAwzsKEuA9UATPl1FELvmxIKc6YROIXbrMy4mUj6b+uQDmNX/FuANv2CnFZaj584= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362499; c=relaxed/simple; bh=q/FZLY6NX7KPIL9q2sRYyOUjPSfSZOW4huImrRHQGAc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V8qgN28nA73YqnCC5xAaA0moeVq4eeNyy122yo57hY1yNR8IzAP5lI6ydysbiYbA2LNR2VvP1QUECLVFZ7JI+wJI6Bjbz3Klbvkxc1y5dnhzOPFzvQlZbsJKZ4qlD6K0Nzp/zw+jyr94TXoDDKti/271r0XwYfa5phNDUdZiHXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VRPB4vXm; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VRPB4vXm" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5e04064af07so3771453a12.0 for ; Fri, 07 Mar 2025 07:48:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362496; x=1741967296; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=VRPB4vXmxLdFycOWknF4RJA6jk/DfKOF8+sP7my/XUsqSlZ/jYyWP0qc+mmAzbZoIK KcDm5wIDoOMR9dIADXDtcqCWGLtAuxz1pKp8Agy388zwXH74GaOAHPdQlgPXRabBelBA t0BRk96D3DxGQJVkJGA4GOK2s4S8CZSnyQhc7J9vlFuiBO5K58WDdQ7mCJ0T2S+0qwpl EtvByEq4kjMj4a58xOq4fjRIY8Tx0sHkS9QwVTerzDjMqNiJieb5P5QloR9qkyofBy3j 8czbF23jyfvY0UPMJ6xicSIKmNPKBtbCCaTcOSzGj+sQow8bbVtwxrPYpesbyNXcztwK XS4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362496; x=1741967296; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=P9lXc4nL9yBiPLGy7AB11DSoGpO37gtv4EDoF+rozUYu+QDKPXxDf1aNJKy26HCF82 8wYEorNJp+fFmpyYnr9nqnJmsSyha/DLw0K7zPr3uhaFGyYRcUCHodD3l80yDu4ol+MI kGfQPlv+1/fQrfrZtLTpatuKTkwewDNZVHHIWK4FjLM9Tj7h2w1Ba+pCnLUgZkgyAwGb 8pAPxqOPYiuB/uDV+9Jmbc3IduA8Yuo6FtD8FvVvXhYxHr9/3HzcsCVQrK3Lbq1H6ASQ wF8b9X6VXbdfXXpPiPZFneLyMFl+hrziwP6kJ7C0trnh/ocZUmx6dgwDr7Nwvs/Vtmb3 uV4A== X-Gm-Message-State: AOJu0YxmSpvRGCC4mRPZXKnQ3AJjJSsouB+67NFBojoUqOY9PwCV3Clh 2jn+kI2hVtHAABjfyhOTjkkn4xXAscaqdCwuG7a8dz1BSdbSC3ieje0Ftg== X-Gm-Gg: ASbGnctFF/hQ3fLLRFBLLoj5TfLsse8kdKRWTsi6N0uBVR8bELnuri2ZEieBf4PRQt2 HI6cz7ZkHfW999bDy8rkmiddSz08kz+beR3dr1X6ltlWn6SoJvzuwvPSVkORf4bhH3Vblu3H6IH zQyagxVzAnv3yburE6HVqLK7+SScFOrQDDNvPE+JNrWS1DndLFvtuzzR1raCRzNFGYVVw5UvILA wdT1Cuq57TEuZZVGL+LMRkMvx7d8tZs+wvWKX3qMXLNh05GkwZyLsZvaft0xbsHfwWgoyRpdAxQ hYOnQ/EQZ1/A7M7gh7DIoHmUsXed X-Google-Smtp-Source: AGHT+IGFSV9ZSbOz3IWTyLUIM7qtuI6iq6wAEVZ8fN/ZSNPOSM3gyokHk8UPblCgR2yVlUUbJRMg6A== X-Received: by 2002:a05:6402:84a:b0:5e0:36c0:7b00 with SMTP id 4fb4d7f45d1cf-5e5e24d402cmr4225999a12.31.1741362495836; Fri, 07 Mar 2025 07:48:15 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:15 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 5/9] io_uring/net: combine msghdr copy Date: Fri, 7 Mar 2025 15:49:06 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Call the compat version from inside of io_msg_copy_hdr() and don't duplicate it in callers. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 905d1ee01201..33076bd22c16 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -287,6 +287,24 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct user_msghdr __user *umsg = sr->umsg; int ret; + iomsg->msg.msg_name = &iomsg->addr; + iomsg->msg.msg_iter.nr_segs = 0; + + if (io_is_compat(req->ctx)) { + struct compat_msghdr cmsg; + + ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ddir, save_addr); + if (ret) + return ret; + + memset(&msg, 0, sizeof(msg)); + msg->msg_namelen = cmsg.msg_namelen; + msg->msg_controllen = cmsg.msg_controllen; + msg->msg_iov = compat_ptr(cmsg.msg_iov); + msg->msg_iovlen = cmsg.msg_iovlen; + return 0; + } + ret = io_copy_msghdr_from_user(msg, umsg); if (unlikely(ret)) return ret; @@ -323,18 +341,6 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_SOURCE, - NULL); - sr->msg_control = iomsg->msg.msg_control_user; - return ret; - } - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; @@ -710,21 +716,7 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_DEST, - &iomsg->uaddr); - memset(&msg, 0, sizeof(msg)); - msg.msg_namelen = cmsg.msg_namelen; - msg.msg_controllen = cmsg.msg_controllen; - } else { - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); - } - + ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, From patchwork Fri Mar 7 15:49:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006678 Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DABEDF71 for ; Fri, 7 Mar 2025 15:48:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362501; cv=none; b=qWlVoykWx6Qqe2ifygwbOZ5/PhrhLRIoH9FENEY56qdhzVDCbsSsxV5+k+oLk/9C11xfnTEoGD9dvVSa3PH8maTc1wS/R/Y/4am6XZmuLm3znlRs2cLNeyvw31eR34Hup967yzgVSKC9r0A7TOTz02M7i8VVd9/4aZP6mBZZxrs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362501; c=relaxed/simple; bh=dFLQOj2ZLIbINKKwcv5tlrqtAVZRjLUjvTJ5sPCh7Go=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=m60cndu9ooRQXSftstmgZN67NajNqaO6Nh01LraMe+sHC8AqefiaE1TDv4F9ZT07FUOpjEsmx8Alq9MWZ0Q4h9+qa+SbRX9DMKmoZN6F87D5wEVZLLo8/Av3Q9CtyqPCe72bvbKxtjXZUAg3UMFQWvTBnovEmG07IQNvgfq07SQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dSm90Dz/; arc=none smtp.client-ip=209.85.218.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dSm90Dz/" Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-ab78e6edb99so310725266b.2 for ; Fri, 07 Mar 2025 07:48:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362497; x=1741967297; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=dSm90Dz/MgaUYub7LDuoPz2+fnNYRh6qvl5EYGqgIZIecOIlMcI/vias095IgYOCrq 2jamepk6lnf4XHMfa1puiq5apD3okZZBib1S6RujKVgRbXezfE1gc72yR5NoZGWuQlL+ Nh5LyQBpR62S/zavVxf9zMN63056W+i1YFEdMQMg91JARWfvYLcUMvSkN6AxzLBRJV3B dQ40B3cxnrLsjaocEH9sH7uRfiGyMtlPzoNYc1agxPsLrkVnMiDsOA5hvQgN6ckIgCpo csvc7AcmMovosN4Ydsa/3Apm7uOhDe/WELy1xQIJHpyNCvYEKy916vqHKdgWqZhebOVz sxzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362497; x=1741967297; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=UxVuJQD5cPnxBG5/k2Oq3MbPTcqfCwbwVGqBJLhJTkEzmHXZsyq7e5VsMt/4mVGlhO 8zUOxpfYMi71b1Q5O6nm9MefnqwTWoDMeI0JV7X4FjsSCt+HeEj+OG7CML6am7ekg3gb MRvdLgEPokeLK533+Ybkf7xAkrJ+yIBxc0gNnf2GpCjNMJqWDO4IJWRtKSznYWUUBpIX idL9RVP7pXTcEJSJT8bP3Z7I5ft1qbtGscVoKeIBAsyko3+Kwl6MxN8d5JM6oaqa5DID 8G8bm53pP28ELp0e17rs3VeRFeF36f95XcG3hlAF84Nnqz/wT7f+NpczmJV+q0HrDj2K P/fQ== X-Gm-Message-State: AOJu0YzMaguag+IDDlncMougbwTXzEO9uPDeJiLdjptG/xOE7oLDP7rq OyX0ZRX9NLU+HTQdlBfnaoahYQAxmMabAHE8ogRdJzF896t8fgmGPSHSTQ== X-Gm-Gg: ASbGnct1sFFFErCTBWRvCi/IPE4PkMgSxukvH8AvJwYbqMmUTYawu0K/JQQqxr7CoEe baoDNsff3hD+r9qlOtPB586Uq82zztd9iMnMo5wpksZTo2vC7VjKhmsH6+LuQIukG1OnCScdCnM 489ObGkzU1NpqjCnFCwLzY8mlnB4egvUGyoE+ZBcz8E/1InCh9DGqBpdZdMSLXUqXfqpKk5NxVe 7hvF6C8G17uog2AJwWGpbtLkuoeWWidfA8rcFga/3S4QnklXk2TVxtxqYcXJveiE9WR7Zz877Rt blgyfcGs4yMyY3OcPx0GBXi0JHDi X-Google-Smtp-Source: AGHT+IFtiesCHuNrORx96JsQDdJG2OxoxpxMdqYEE5kiMfy9VSZYOWXnHl0Ur1pkEz5EXiHdShECJg== X-Received: by 2002:a17:907:2d8d:b0:abf:6db5:c9a9 with SMTP id a640c23a62f3a-ac252f9f653mr479879066b.39.1741362497021; Fri, 07 Mar 2025 07:48:17 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:16 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 6/9] io_uring/net: pull vec alloc out of msghdr import Date: Fri, 7 Mar 2025 15:49:07 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I'll need more control over iovec management, move io_net_import_vec() out of io_msg_copy_hdr(). Signed-off-by: Pavel Begunkov --- io_uring/net.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 33076bd22c16..cbb889b85cfc 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -253,12 +253,8 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, return -EFAULT; sr->len = tmp_iov.iov_len; } - - return 0; } - - return io_net_import_vec(req, iomsg, (struct iovec __user *)uiov, - msg->msg_iovlen, ddir); + return 0; } static int io_copy_msghdr_from_user(struct user_msghdr *msg, @@ -328,10 +324,8 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, return -EFAULT; sr->len = tmp_iov.iov_len; } - return 0; } - - return io_net_import_vec(req, iomsg, msg->msg_iov, msg->msg_iovlen, ddir); + return 0; } static int io_sendmsg_copy_hdr(struct io_kiocb *req, @@ -342,6 +336,12 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, int ret; ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_SOURCE); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; return ret; @@ -719,6 +719,13 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) { + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_DEST); + if (unlikely(ret)) + return ret; + } return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, msg.msg_controllen); } From patchwork Fri Mar 7 15:49:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006680 Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0CC65194C78 for ; Fri, 7 Mar 2025 15:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362503; cv=none; b=e7jz/rDnPFDJEUzGykXWMRWSUQ1n8e2f4ddLzqWwL7RQXxT9dmsJxcIkAsyEyBKgG3cGJOsXTf8HHMd2nWs7z+TndzbgxXWDog2Z8qFaQoSYkahbURRFqU0jx+fdp4CxhGYkGfvwq6rLPrZvzxh28Y6gxGCoj/QjykYIyumsdJ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362503; c=relaxed/simple; bh=hdM/yveeqx/cwJF39LxfDEQxflleuBmQE+zj5qDwnIg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jchXRgZP3Am7sHSSC0O0IAtKnk/mV3NUFHaMQ6eYpIU2IWQwi0ZBAbJ8EZWN4Yc1oc3XPXr/fJG1s31fy1g2fG6Cnf2O9U5olnp+VDSMAhDTkirtvyEJokUSe55M2asPd7ooWFK+6Ei9BpKf+Em56OIcobj0MomQ85mtwnkPAaw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WwajzKdN; arc=none smtp.client-ip=209.85.208.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WwajzKdN" Received: by mail-ed1-f53.google.com with SMTP id 4fb4d7f45d1cf-5e549af4927so3571832a12.2 for ; Fri, 07 Mar 2025 07:48:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362498; x=1741967298; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Zac3vga/E698ojFWUHp/OWmE620rH3affaJufXu525s=; b=WwajzKdNaIRQaaoIy5wH5ALwnBVA9B7EMvnX9TO9zAHCi11CvIAWt4Sy7TmYm1DaVd N/YA2SSH9fopP7iLlWHrQjxzUm8eSAjarJipcKtSmRncdiTkis5c/9UGjjtyaLwdaFN+ Py9z+Fo+7ZEqe0lS+9ZWuchvWo1oy2FpKDQ2O1WIxg17Zj/wDIy0HjcEmwWUh07bK4VM NfOhblcus2zcmnvO8Z+s+v1pSbL6B/ckmNh/+E6BGuf8S+jz0rBC+TEk9Wp3jomy+k6y lziN8S27nBoeg6vxvCOtGmFHkPeMYY47klsCIx3KY2osdgYmMs47rxjKYk0S5u/oZGM0 hcEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362498; x=1741967298; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Zac3vga/E698ojFWUHp/OWmE620rH3affaJufXu525s=; b=pMjH/7F/ty92WQ27zfevF96mxstj6VYt8DSTl+EEYrkfsYlIc6uGC01d/S4ufDIG56 PRXaqC0zdLfphh26VGCW04MkkbmtPd0STZBfFk0DV4HInK32LcUWttbZ8aiABxRbL3x5 M/RdTtLoAIG4u8PH5z4hfyxVv3k44BTIpyILV3xMvXT/v2ttfDUmJvwkS2WJJH/2VCok 5B+lNt0QH5cGhxz3OcZe9GJr0GVYRdDbWIhnRUbfQ4fxHjOgyEgK2qSkDpPZIkQ+fzce 9TMBY1L+GdGFADG95hkrsXc4QNHXFnEA+ZZ8Zj7JHXSlQJi8jhIO3Aqdw1lZHj95eV5e HQuw== X-Gm-Message-State: AOJu0Yz1joD39lI0TPwUVfRbKrtQW0iFkE9DwlMYTU+1cRn3bR2xtl9S hcE20Kd3+givASg8hTAjwvuDFD9w4C4dWqR+s39KYBuVx5+JldeRApb2nw== X-Gm-Gg: ASbGncu2NCwsrUB0BO/6dVhQ7piUyDWnjXkVBLFm7diUUuA0UMxED03keqtdwMZN4vl yKAqtBNaNttTDhHWPA92f+UzG+RH+V9UAwogLFfTnJAOzaqmOKh0KJqayKZchta7ZOcDHn4qwCY Sj+gK8k7VoYjbvuOohN5u/lU3GgvxMP2tGNzM5nBZlvqo7WjRoRD0rDhsAm/HGj1irIOKFBM33G 2unraXIsTjlk/3VdrR0I3Avf6Ti+/hASbRrUDLMXPa9BsooI3kDYvb3X/yfZ6krE13m8dhiuao3 Of0ESufrRl3ZxoCQf9a4/cedEpr+ X-Google-Smtp-Source: AGHT+IHyo8PN39OruwYo6/8uZoc5QntCXAl1kiNVmtVbsj/VhQG+lt0dllXa562T8Ip8hYyKkK23ig== X-Received: by 2002:a05:6402:5106:b0:5e4:c8fd:3ba0 with SMTP id 4fb4d7f45d1cf-5e5e22c05acmr4365401a12.6.1741362498176; Fri, 07 Mar 2025 07:48:18 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:17 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 7/9] io_uring/net: convert to struct iou_vec Date: Fri, 7 Mar 2025 15:49:08 +0000 Message-ID: <4527d9152e7ebc75e3481879e724518e6a990778.1741361926.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert net.c to use struct iou_vec. Signed-off-by: Pavel Begunkov --- io_uring/alloc_cache.h | 9 -------- io_uring/net.c | 51 ++++++++++++++++++------------------------ io_uring/net.h | 6 ++--- 3 files changed, 25 insertions(+), 41 deletions(-) diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 7f68eff2e7f3..d33ce159ef33 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -16,15 +16,6 @@ bool io_alloc_cache_init(struct io_alloc_cache *cache, void *io_cache_alloc_new(struct io_alloc_cache *cache, gfp_t gfp); -static inline void io_alloc_cache_kasan(struct iovec **iov, int *nr) -{ - if (IS_ENABLED(CONFIG_KASAN)) { - kfree(*iov); - *iov = NULL; - *nr = 0; - } -} - static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, void *entry) { diff --git a/io_uring/net.c b/io_uring/net.c index cbb889b85cfc..a4b39343f345 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -136,11 +136,8 @@ static bool io_net_retry(struct socket *sock, int flags) static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) { - if (kmsg->free_iov) { - kfree(kmsg->free_iov); - kmsg->free_iov_nr = 0; - kmsg->free_iov = NULL; - } + if (kmsg->vec.iovec) + io_vec_free(&kmsg->vec); } static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) @@ -154,7 +151,7 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } /* Let normal cleanup path reap it if we fail adding to the cache */ - io_alloc_cache_kasan(&hdr->free_iov, &hdr->free_iov_nr); + io_alloc_cache_vec_kasan(&hdr->vec); if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -171,7 +168,7 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) return NULL; /* If the async data was cached, we might have an iov cached inside. */ - if (hdr->free_iov) + if (hdr->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; return hdr; } @@ -182,10 +179,7 @@ static void io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg { if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - kmsg->free_iov_nr = kmsg->msg.msg_iter.nr_segs; - if (kmsg->free_iov) - kfree(kmsg->free_iov); - kmsg->free_iov = iov; + io_vec_reset_iovec(&kmsg->vec, iov, kmsg->msg.msg_iter.nr_segs); } } @@ -208,9 +202,9 @@ static int io_net_import_vec(struct io_kiocb *req, struct io_async_msghdr *iomsg struct iovec *iov; int ret, nr_segs; - if (iomsg->free_iov) { - nr_segs = iomsg->free_iov_nr; - iov = iomsg->free_iov; + if (iomsg->vec.iovec) { + nr_segs = iomsg->vec.nr; + iov = iomsg->vec.iovec; } else { nr_segs = 1; iov = &iomsg->fast_iov; @@ -468,7 +462,7 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret) if (iter_is_ubuf(&kmsg->msg.msg_iter)) return 1; - iov = kmsg->free_iov; + iov = kmsg->vec.iovec; if (!iov) iov = &kmsg->fast_iov; @@ -584,9 +578,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, .nr_iovs = 1, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode = KBUF_MODE_FREE; } @@ -599,9 +593,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, if (unlikely(ret < 0)) return ret; - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } sr->len = arg.out_len; @@ -1085,9 +1079,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg .mode = KBUF_MODE_EXPAND, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode |= KBUF_MODE_FREE; } @@ -1106,9 +1100,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg } iov_iter_init(&kmsg->msg.msg_iter, ITER_DEST, arg.iovs, ret, arg.out_len); - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } } else { @@ -1874,8 +1868,7 @@ void io_netmsg_cache_free(const void *entry) { struct io_async_msghdr *kmsg = (struct io_async_msghdr *) entry; - if (kmsg->free_iov) - io_netmsg_iovec_free(kmsg); + io_vec_free(&kmsg->vec); kfree(kmsg); } #endif diff --git a/io_uring/net.h b/io_uring/net.h index b804c2b36e60..43e5ce5416b7 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -2,12 +2,12 @@ #include #include +#include struct io_async_msghdr { #if defined(CONFIG_NET) - struct iovec *free_iov; - /* points to an allocated iov, if NULL we use fast_iov instead */ - int free_iov_nr; + struct iou_vec vec; + struct_group(clear, int namelen; struct iovec fast_iov; From patchwork Fri Mar 7 15:49:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006679 Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D281DF71 for ; Fri, 7 Mar 2025 15:48:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362503; cv=none; b=ScVn4GkA7sybvVAzMJRXqq1KqOiYvgJFxGcSRJsaKzTnvb4AWs7XYSUml0Z2kseKQlUHkvrDBcjI+3Vrd8wO25fdqDHdQTgntVD5hehTrLOfes44gaOhk8VVYl39sTEMHMkYOBRO6SDX9gaCCM9xmU/NOGtnJi6xcfnNNxd8bZE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362503; c=relaxed/simple; bh=+ko6qVLwFbY8p2fCkuMHNRIwwTZQjN0AL9uVp4qcr08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZKTTn9N8UbioiO4jBY8vCeIrjfJP4Jqxpb79VqS7AVbV2I0XKb+nGan+798DpQv4ySXrUCLqAZ3AzqoJuD47HcbamlgN0rmiZ/SH7n2Y3ZknAiJgqdgyPwZR7/3yzqJuz9ZvkioPuPqZ8NZpC3aZVPQRgz5aZhQ7IWgB6H/xgTM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VdydvhaU; arc=none smtp.client-ip=209.85.208.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VdydvhaU" Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-5e5c7d6b96fso3185434a12.3 for ; Fri, 07 Mar 2025 07:48:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362499; x=1741967299; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=VdydvhaUg5Mdvo+6pey7ci+gjz/lg0GStXgyQNiacYVHAYDukHqUtUhWez5gBENs1x GcGHWl8q2Q9SXWVX0fczYM2d2sOoC4zmihjGGaBx236EGEORf6sWxppF8pLB3Xpxt+sE S/vSBHRR3wS5YE2a1gnu4GvlkvghcHWIdvsZHjXhY5AbOrDZ0we2gvyPaWfZH2ojK0TD quPXutSDo92rh12NwRn/m8oMZlCDg9RotY426GJcw86BWXXqCKPmdlkFL/0y9W8iTmRS zJZW+OR18Kv3uvRdJ3a+EP8OlR8QE0S9zkhBIgwMylJ1BLEDRrAYfzSOBGJImmVaPUcO VGRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362499; x=1741967299; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=wlVrP+qK7MbdbT4PmaFBCGNnd038fcBLhK+w8flxMQGxmL5y4H5A9iQP42QTRcI2Yh VC6pN2k1dmn/dcMf7NrRfC4MnXUGgf6uNLLPemG6UKOhk+cWYcxuXzl4RGCjocNjCE2D fP1KUfL+4rpflU6CE3+zb+SNrQIySwiAl/TZ7zwggNfzKeFiY1VhhpKsQ+yuxxY/lFrv Os6WZwRFPKdLRoxp9aiiqtlNIzLuNw/vbEDFW7BZHiI8qXdtbh2eCiZhdsZk7VzNhO5d 1eDW0bW30ckPdIUC5er2tkc9p6H8ztSDdAEMRTIAqIsKuKx1W8RN2mKxQ5YKZyLSDBIq aTmg== X-Gm-Message-State: AOJu0YzMIILr1PUh7Sr7r9bQUzccUw3xeJgzw7bJ92n050hS7LKLxw8N Z/OjVTQra4qxydsxOehMTdWeAXHW7rysvsJbsSAYTQPv0lwy61/06ygTTw== X-Gm-Gg: ASbGncuOcjAn9ax5ckzp3LyMAONzXmqUn47YzOYk+vAvkMUEDSfAVTCDr2+5HNASvgF pKO04T3ZFmOpVr1ytZ/mArcglie5Da81K1bmIuMFyyrMnVi9aSD8iPkGxOoOrpacNGShrOhQOIQ NJsVBsVSeyF0NIPuXk7c1IyYumxSpTeRd14KQ6cDY6/VCNwIEJku5psquOULnuuVLwNOQpk/Swb JlMl3W25l2I89QU+KspwTbHM07pAZrecPYiXesoxKXdw9uZz2WHxM+MdHShJKdzyy7YIDDzJFjw O+55rXvTHYU1/hhpeGY23g8UHpgO X-Google-Smtp-Source: AGHT+IF2tLXtQyqecrT83VDAwCbmRXARWHJkEL39+aFQZQpoqpfFSFC04GaIDnabETmi07kSUDn/LQ== X-Received: by 2002:a50:8dca:0:b0:5e5:bd8d:edb9 with SMTP id 4fb4d7f45d1cf-5e5e22db548mr3514793a12.10.1741362499192; Fri, 07 Mar 2025 07:48:19 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:18 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 8/9] io_uring/net: implement vectored reg bufs for zctx Date: Fri, 7 Mar 2025 15:49:09 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add support for vectored registered buffers for send zc. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index a4b39343f345..5e27c22e1d58 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -395,6 +395,44 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe return io_sendmsg_copy_hdr(req, kmsg); } +static int io_sendmsg_zc_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg = req->async_data; + struct user_msghdr msg; + int ret, iovec_off; + struct iovec *iov; + void *res; + + if (!(sr->flags & IORING_RECVSEND_FIXED_BUF)) + return io_sendmsg_setup(req, sqe); + + sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); + + ret = io_msg_copy_hdr(req, kmsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + sr->msg_control = kmsg->msg.msg_control_user; + + if (msg.msg_iovlen > kmsg->vec.nr || WARN_ON_ONCE(!kmsg->vec.iovec)) { + ret = io_vec_realloc(&kmsg->vec, msg.msg_iovlen); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + iovec_off = kmsg->vec.nr - msg.msg_iovlen; + iov = kmsg->vec.iovec + iovec_off; + + res = iovec_from_user(msg.msg_iov, msg.msg_iovlen, kmsg->vec.nr, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + kmsg->msg.msg_iter.nr_segs = msg.msg_iovlen; + req->flags |= REQ_F_IMPORT_BUFFER; + return ret; +} + #define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE) int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -1333,8 +1371,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->opcode != IORING_OP_SEND_ZC) { if (unlikely(sqe->addr2 || sqe->file_index)) return -EINVAL; - if (unlikely(zc->flags & IORING_RECVSEND_FIXED_BUF)) - return -EINVAL; } zc->len = READ_ONCE(sqe->len); @@ -1350,7 +1386,7 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return -ENOMEM; if (req->opcode != IORING_OP_SENDMSG_ZC) return io_send_setup(req, sqe); - return io_sendmsg_setup(req, sqe); + return io_sendmsg_zc_setup(req, sqe); } static int io_sg_from_iter_iovec(struct sk_buff *skb, @@ -1506,6 +1542,22 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) unsigned flags; int ret, min_ret = 0; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; + + if (req->flags & REQ_F_IMPORT_BUFFER) { + unsigned uvec_segs = kmsg->msg.msg_iter.nr_segs; + unsigned iovec_off = kmsg->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ITER_SOURCE, &kmsg->msg.msg_iter, req, + &kmsg->vec, uvec_segs, iovec_off, + issue_flags); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; + req->flags &= ~REQ_F_IMPORT_BUFFER; + } + sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; @@ -1524,7 +1576,6 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_control_user = sr->msg_control; kmsg->msg.msg_ubuf = &io_notif_to_data(sr->notif)->uarg; - kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); if (unlikely(ret < min_ret)) { From patchwork Fri Mar 7 15:49:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 14006681 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21C9221CFFA for ; Fri, 7 Mar 2025 15:48:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362504; cv=none; b=i+ywdgf2OZSGO5+SnWxgJKhEwVQA3nwbF9JY4bltTPEmPoCc5NQkjLgWZ2L7e5bGCq2vYgJpo+GjDam2/a44VrSkuFvyINaecEc5HJ8SmCL88swg4efWrRFRqVcJBZF5YMzl4A2vXM6PkFSJy3vlnn0mk5pq1OvTjueI9mrEqBE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741362504; c=relaxed/simple; bh=4sK4VkwDijMad+gj86h50HxY2xUtZTdkNgwm9T1N7Fw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gRHJYHIrdGamNbGBDPxaow+RuNW7YeU1og/xdTf/MlQHIWhYj4fL/LT/3dZf7TkITvELENn9UZDhbmubEl25cpBDGZmzRZPSpAWVzvamwCHV1LE9NYHZ+BqUJ4RRePitSmIRUcGtfw1wYbqlkNFF7NK08wlJ1hydmZ1KW153wIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jfHIaye8; arc=none smtp.client-ip=209.85.208.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jfHIaye8" Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-5e5cd420781so3316298a12.2 for ; Fri, 07 Mar 2025 07:48:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741362501; x=1741967301; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o/9RNEb6QyHbVKv47iIof+34KhXOtpAV60Dp/7krGt8=; b=jfHIaye88tdXeJFJmDuQveGfse3Z2im53OUPGAYXgtADVN3EqM/NfIc1J11633lI0r R2ZkjzQEx416PITj3jzNn40jht/AjnYa7Gjf0ysDuh/TKQTxV3e9hHmUlQ/jOVALkKJ3 3j+PydbXuQZbLrM2tP/DPKAMfhp3Y8o/pkrdF4GFOJGKevsJS7fx4YRxzjHGyW4M3jiH YaGRm/zBP+rPC2aXqcPnbg5PdoHAznzYqezaTTJ3Zb4XUb1W6+jte4QhNsCrSwFIruN8 9AyQNDLthhQPBzaMiT02LurVreAtwIes5M6qdX/0fbW4d9U5i2C9UqzWmr4hwbIjCaL+ IlyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741362501; x=1741967301; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o/9RNEb6QyHbVKv47iIof+34KhXOtpAV60Dp/7krGt8=; b=lPmgk1gpTA0hMRz0NOPBQ4YoIVrHwLqVOOjbZnsZpU1uqHbtKZOmOgURf85+jkt4QT CbJXxK37Iy1S9VezTo467MNqn1HFzx0BKzp1iYrPZemvpseAi4+o/hJbWGB3DxB5PJV6 5vr9nkteWkAjyF2EgEi8lO0Sx0YyjGDJGiNhsuoW7CEspVZL9FyYVmcJqveWEqyJWrd9 gFW7JlLmw5JPTA45vHEDrCfkF7ExpBfAea55nJx8z1Np2lNuJI9+1gIrqGr6GMseISAy BFLBweG0LtoiG+TRJFhaDJgTmQAI/5qT78/qURqQ9/TTXObEqIFK9bp03XAQiR8Gsto+ i+Gg== X-Gm-Message-State: AOJu0Yxa1Jzz60GLflYQZ2kBTcVv5gr+ME8zPklP+MS0f0RBEwHsu4IH 35+/53k/S4Nk6UJR0KTWeUuToxYdH2uia1cj5TmeZjrVfTGadxX7cIMEjg== X-Gm-Gg: ASbGncuY7B/chsYoiUUY6mllsSpGhH99otbt+Tyc8hYvbezYzItbt1nnOxHHh8qqrMh DP73rvQeiaddiRTllBPDgANjlxg8NfMzXr6eZVaT4QEA7VleNvZusxE/8Jhspxmhi3knmCOIiW/ vjaI+aI6eCvng4PelCYojI/jnYJmtQjZ88k0cvr+iItZR0uzIE6lUSfx583Q/VD9gA7FMt7rhaD h7K/Yr4X++yNpV7lFoa/kYLx7iilOS+yMwxmnbKqMU2+f2AgV7C6xEUNhOH6vISR9m/CTCs/aUM wh49lOwOeo63/iCuJZk3MVv+zpKz X-Google-Smtp-Source: AGHT+IGQdtbNt3VBFgp28EyvTz9nw3WDbp4yK2Mltnhd6abQAF61FPdAFCIAt1sHBjkYadrIabzxiA== X-Received: by 2002:a05:6402:2346:b0:5e0:60ed:f355 with SMTP id 4fb4d7f45d1cf-5e5e22d98dbmr3955446a12.18.1741362500788; Fri, 07 Mar 2025 07:48:20 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:1422]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5e5c766a033sm2665591a12.56.2025.03.07.07.48.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Mar 2025 07:48:19 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH v3 9/9] io_uring: cap cached iovec/bvec size Date: Fri, 7 Mar 2025 15:49:10 +0000 Message-ID: <097f9237221826b1eebf5b03336be176fd2ee0ca.1741361926.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Bvecs can be large, put an arbitrary limit on the max vector size it can cache. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 3 +++ io_uring/rsrc.h | 2 ++ io_uring/rw.c | 3 +++ 3 files changed, 8 insertions(+) diff --git a/io_uring/net.c b/io_uring/net.c index 5e27c22e1d58..ce104d04b1e4 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -152,6 +152,9 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) /* Let normal cleanup path reap it if we fail adding to the cache */ io_alloc_cache_vec_kasan(&hdr->vec); + if (hdr->vec.nr > IO_VEC_CACHE_SOFT_CAP) + io_vec_free(&hdr->vec); + if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index f1496f7d844f..0bfcdba12617 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -6,6 +6,8 @@ #include #include +#define IO_VEC_CACHE_SOFT_CAP 256 + enum { IORING_RSRC_FILE = 0, IORING_RSRC_BUFFER = 1, diff --git a/io_uring/rw.c b/io_uring/rw.c index e62f4ce34171..bf35599d1078 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -150,6 +150,9 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) return; io_alloc_cache_vec_kasan(&rw->vec); + if (rw->vec.nr > IO_VEC_CACHE_SOFT_CAP) + io_vec_free(&rw->vec); + if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA;