From patchwork Mon Mar 3 15:50:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999067 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E3F0213E67 for ; Mon, 3 Mar 2025 15:50:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017005; cv=none; b=C+Lh6RF/cL0UbeiqwLnvD7t1+1o3jRjFMeiVhih+Nu8syNQhuYogIxJIijLzkorHUJWstHHRUKJEhJTUQ/iCxczBYxAU7jNXR4EeZO1QEDEBClY2vtw0aeYt6lHj6V9nnnF7XTzLq2VU4K9lr0+MJSW9UyYyd46MZcRxp2BhVJE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017005; c=relaxed/simple; bh=eb7zhO5hrORnDMVtuvQJNO7Eek1fNmob5BJW+IsWq7Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nAQekfM/0lS5Oyk9L5s8jCoeX2GL2C6CmkXTNrveDODAPQTb/SYzfAPVCqcqvlqpKqiDPVaWjd30BTN32wibHKEjIh3lzzdjb5Ju+MVBw1NRzYDFpA6b2zj0WC1VYXEWP/MK7e/euGR0cisG5f4THm3JB6qDyB3eVW+9Pu2hPr0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=J1dYdYvB; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J1dYdYvB" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-abf615d5f31so330221866b.2 for ; Mon, 03 Mar 2025 07:50:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017002; x=1741621802; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g5aJ+LTP0iCupC2CddhzwQvv8bbrm69RkycI+0Z/um4=; b=J1dYdYvBvQN04hITLW89jxiYQUVPpbUiRspq2RyfBgzhL85nnG9JG1zJiZeCLZjYxh oYSlzzAGkaB1/o/mLD2VYEsi5yEBBHAmDJ2OGIWgi4/MEs9BlvL8zyc41ZbiFuL4IBi4 EKjKBfrZN0DEhgqHIdYSNxc4tH/KGyGJUY9kRkT+80KjJMAdJkaSEJm6X9Zqu5FQ8v3H 0DFfoG5ZTQgO6SPKesxecQQSSKRRC5fmP9u+Z8eZx+j/3KH75crvJ7VKkLoFs9Q/tk/P zzmogKAqFTLKRrGgD+Ym2xGJa4zQFo23b4RxSg/yOlWWbXC5t/ofsf/G+DF6YkMocgsJ ImPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017002; x=1741621802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g5aJ+LTP0iCupC2CddhzwQvv8bbrm69RkycI+0Z/um4=; b=XbT0abyAcNylEdmQxVJmZZnR/HfbnVJLtCqIcuzYgvmHw1oRzh3Gd8UFUmt5mcCEMW y336Iq7r0BHFE1ohO7f3HQ4IpuQ++NqojPm1Pm1oz5P7a5cbiPfK2m2/HVKrEo6MDkpB Pppvbalqz3ar6WK9mIW74nagVMxzjKj80Jqu3FlUGQ/gqQNcldvUOmtSHETM71RvSHoD ee6x84RcJQ5XFPNlAU4ssR4Nw1b58BvX33Da+n0w8nQf0jFcWL6l5DChwAWVm/rmIljO BFQKOKvwXLgAYSZu53aDt9o06pGA8fIUav3FQ67BpJ64hw2PEq5QQsZ19Mt+0gEsRtSM x6Tg== X-Gm-Message-State: AOJu0Yzj0Ma6gNURPJWM0r7w9WXygEI8gvpDV+eUUPaiHk/Wvbz1CUS7 /JslHceM6dNRm4x8Bv2f1B+/+RyJlBSJ2RhdFunHeXeE1YB6vU+PQWFwQA== X-Gm-Gg: ASbGncsAaxLcvWY0DCVu6b5J/lHfO5dmQoxAfVL2QLssNxtzh0YZWntHEzvjgwGIw/5 snXz/cXyTdisjov7ENShmJRHL88aErdUVGZuq/yiy9MfBKx86XciIqKrU+nJvfTePnBPRHaBvf7 oUtu8/H5IbRoV/e1B9n0yA8PIidiOgErSAP2Xn2Jk2iKvb55Q9h5CpGdYwZq5GO11RSkB4raFiP hmc5gxDyd8CP9ZE2LEAR31SBE1/BqGkqd0Sxbydn68j98P+GCToRyLaqFEo5kOTHmllfcU9tlLH phAH/JyNanznqfbw0HJYwQ/Kl6Ze X-Google-Smtp-Source: AGHT+IFLSSgRqADlfOBo5Y3PcN8Gtoj/a1bP99JLnQppEzvqPfXAFuzp1PEJgNNGwrki8ZvH52mEwA== X-Received: by 2002:a17:907:3f90:b0:ac1:db49:99b8 with SMTP id a640c23a62f3a-ac1db49aa62mr293249766b.20.1741017001746; Mon, 03 Mar 2025 07:50:01 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:00 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 1/8] io_uring: introduce struct iou_vec Date: Mon, 3 Mar 2025 15:50:56 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I need a convenient way to pass around and work with iovec+size pair, put them into a structure and makes use of it in rw.c Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +++++ io_uring/rsrc.c | 9 +++++++++ io_uring/rsrc.h | 17 +++++++++++++++++ io_uring/rw.c | 17 +++++++---------- io_uring/rw.h | 6 ++++-- 5 files changed, 42 insertions(+), 12 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 35fc241c4672..9101f12d21ef 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -110,6 +110,11 @@ struct io_uring_task { } ____cacheline_aligned_in_smp; }; +struct iou_vec { + struct iovec *iovec; + unsigned nr; +}; + struct io_uring { u32 head; u32 tail; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d6ac41840900..9b05e614819e 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1264,3 +1264,12 @@ int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg) fput(file); return ret; } + +void io_vec_free(struct iou_vec *iv) +{ + if (!iv->iovec) + return; + kfree(iv->iovec); + iv->iovec = NULL; + iv->nr = 0; +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 662244282b2c..e3f1cfb2ff7b 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -3,6 +3,7 @@ #define IOU_RSRC_H #include +#include enum { IORING_RSRC_FILE = 0, @@ -144,4 +145,20 @@ static inline void __io_unaccount_mem(struct user_struct *user, atomic_long_sub(nr_pages, &user->locked_vm); } +void io_vec_free(struct iou_vec *iv); + +static inline void io_vec_reset_iovec(struct iou_vec *iv, + struct iovec *iovec, unsigned nr) +{ + io_vec_free(iv); + iv->iovec = iovec; + iv->nr = nr; +} + +static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv) +{ + if (IS_ENABLED(CONFIG_KASAN)) + io_vec_free(iv); +} + #endif diff --git a/io_uring/rw.c b/io_uring/rw.c index 5ee9f8949e8b..ad7f647d48e9 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -87,9 +87,9 @@ static int io_import_vec(int ddir, struct io_kiocb *req, int ret, nr_segs; struct iovec *iov; - if (io->free_iovec) { - nr_segs = io->free_iov_nr; - iov = io->free_iovec; + if (io->vec.iovec) { + nr_segs = io->vec.nr; + iov = io->vec.iovec; } else { nr_segs = 1; iov = &io->fast_iov; @@ -101,9 +101,7 @@ static int io_import_vec(int ddir, struct io_kiocb *req, return ret; if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - io->free_iov_nr = io->iter.nr_segs; - kfree(io->free_iovec); - io->free_iovec = iov; + io_vec_reset_iovec(&io->vec, iov, io->iter.nr_segs); } return 0; } @@ -151,7 +149,7 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) return; - io_alloc_cache_kasan(&rw->free_iovec, &rw->free_iov_nr); + io_alloc_cache_vec_kasan(&rw->vec); if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -201,7 +199,7 @@ static int io_rw_alloc_async(struct io_kiocb *req) rw = io_uring_alloc_async_data(&ctx->rw_cache, req); if (!rw) return -ENOMEM; - if (rw->free_iovec) + if (rw->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; rw->bytes_done = 0; return 0; @@ -1327,7 +1325,6 @@ void io_rw_cache_free(const void *entry) { struct io_async_rw *rw = (struct io_async_rw *) entry; - if (rw->free_iovec) - kfree(rw->free_iovec); + io_vec_free(&rw->vec); kfree(rw); } diff --git a/io_uring/rw.h b/io_uring/rw.h index bf121b81ebe8..e86a3858f48b 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -3,19 +3,21 @@ #include #include +#include "rsrc.h" + struct io_meta_state { u32 seed; struct iov_iter_state iter_meta; }; struct io_async_rw { + struct iou_vec vec; size_t bytes_done; - struct iovec *free_iovec; + struct_group(clear, struct iov_iter iter; struct iov_iter_state iter_state; struct iovec fast_iov; - int free_iov_nr; /* * wpq is for buffered io, while meta fields are used with * direct io From patchwork Mon Mar 3 15:50:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999068 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9565214A6C for ; Mon, 3 Mar 2025 15:50:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017007; cv=none; b=Ej7e5jT3U/D5hwjDxKp2S7LmGypb3ljdW/5Y1jiiNnArWnJMz1YV9cjG3X+NyWsggQZkIGaRxzRkPx3gJeiAh+SYB9ESGndEnAL8tvtAb7Y+SIpiCMS4ASqSzlbcUQE3hNKQcgA+8HOsAHjj3xYxmw2V1CugQKmbrrhCHjSDlaQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017007; c=relaxed/simple; bh=b/+M6EYHNbSHsVgKpBR+If1LLwtdyB8f0ukymqfRDmI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XtQYFd+ldtzG/Gf/n52VimO8Dqf4mPeaF1r49u8PAR0eTN9y0PMEsqt5rx/YX4DKmY38e/EbbgDURkcOjQQPR5PMgUDZQYXcR0kj6SEg6NzWxiKwEYsuMch6w3lZsVmfJ1ckLToLpcWGppg5zVj/JQYAxMC4/3Oz3I2qj6hvoO4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LFhjakTT; arc=none smtp.client-ip=209.85.208.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LFhjakTT" Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-5e4ad1d67bdso7236149a12.2 for ; Mon, 03 Mar 2025 07:50:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017003; x=1741621803; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IlMfiIq/U2rZ1pw2Rwc3/yXfiAFA292UD6WiVz5J0yI=; b=LFhjakTTs0tWaUd80GATqFvCgDIx54PR2wDY86jabBdUWIUV1TIYuby8ZVQErK2OMI Z8ytrAcs3tIHO2daLtnTYJGWIx73jqR9NN68ziVqIKsZjLUD1++b5t71lE+iM1bO1e6G oCUVSRyWwj3jhdHUJW1p3RUUrDFnYxBTwsVz+n9f6qZmLC6bdEjWj4I+K5xAjJvDT3Dd bjv2T7a9LFC16RmGJZgpFKOChtVrnlDXG4OZTwFVjlUV9y5KBGMnE0BMNp71X9CpD6Dj 3qtT35rqatUkq6v5eX1niRaVpIBelohu/SgqWhreVus+W1FGmGM4sevs2iP6bvsx/2io 0lZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017003; x=1741621803; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IlMfiIq/U2rZ1pw2Rwc3/yXfiAFA292UD6WiVz5J0yI=; b=T9gpj6Y2r5cpGqMYliv2oz67bV2/p5Ao5R9ky94CH/g4MZ+vEePx6eUB4l7EmJDX80 X9ezi0/43pz2qHJ0oBmH68SF7Kb9x7sdenDGG56q9sIQx566wQL4uxkxwPVwGFHr+TwX qNiFfTK9hWkNalXmPrYg8UGLUjjoQH3rffrgdNeEWz4tf7ylTTzbo6sv1D1jDVXR36fZ Mp4md/t/Ay5SDrf7veZt6PFSxNSsqb/SycqTe1ypIDU4DNczWu89ysLxH3g8FewOIgqJ pcTM3W5bWtzawiTyElUTDCt/W6zxffq2dPinvA2Mr1JYjsZboBgmuCK0KO2SM9WDqoPi ECng== X-Gm-Message-State: AOJu0Yw8AeUNAeqADrkorQR5NI1uo4NkR9plvqTSiaEvbbCoNebnNU5o Er6atKOHu87dEwa1s6CQ4p639+NSQT30CT/FrUxlENRx9PRPYOfPCFRdpw== X-Gm-Gg: ASbGncvVKpsGV7wvWzqyUJ9re9z+o7CXm/KQ9/nNE0Mzkdjz35hpB339F6m5nVNDwI5 UsZhO5IqQQCtsWz3wGYD4MvY6z/cZ7NViH0nr35rBv39tep4t7NR6ai5ogr5DaIS3VuZVp8PWwj lNc24y3TLS4GgWQyqb6OMBJ8D+IhvcPMSZEf7VGmgp7SN3K9/NE4EZA7w2AlaqWCnpxWsPN3mSC OwZW8f/FxoyiQhmr2ogej2lVoULLYf8PG70jLbvY4eChSY+aNL4WI/Oj5wK4krWLWLD06czJ9J5 kx2+yuyKebiKINui/ID62BSidg++ X-Google-Smtp-Source: AGHT+IE43N9jJYr6j3HuDd77e3s7s/590LS4lAxa/rElsYSAakBDkpg3e9EcN3thTDs2HWzbesLULg== X-Received: by 2002:a17:906:da88:b0:ab7:851d:4718 with SMTP id a640c23a62f3a-abf265a306emr1857174166b.36.1741017003255; Mon, 03 Mar 2025 07:50:03 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:02 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 2/8] io_uring: add infra for importing vectored reg buffers Date: Mon, 3 Mar 2025 15:50:57 +0000 Message-ID: <841b4d5b039b9db84aa1e1415a6d249ea57646f6.1741014186.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add io_import_reg_vec(), which will be responsible for importing vectored registered buffers. iovecs are overlapped with the resulting bvec in memory, which is why the iovec is expected to be padded in iou_vec. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +- io_uring/rsrc.c | 124 +++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 5 ++ 3 files changed, 133 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9101f12d21ef..b770a2b12da6 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -111,7 +111,10 @@ struct io_uring_task { }; struct iou_vec { - struct iovec *iovec; + union { + struct iovec *iovec; + struct bio_vec *bvec; + }; unsigned nr; }; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 9b05e614819e..1ec1f5b3e385 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1267,9 +1267,133 @@ int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg) void io_vec_free(struct iou_vec *iv) { + BUILD_BUG_ON(sizeof(struct bio_vec) > sizeof(struct iovec)); + if (!iv->iovec) return; kfree(iv->iovec); iv->iovec = NULL; iv->nr = 0; } + +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries) +{ + struct iovec *iov; + + WARN_ON_ONCE(nr_entries <= 0); + + iov = kmalloc_array(nr_entries, sizeof(iov[0]), GFP_KERNEL); + if (!iov) + return -ENOMEM; + + io_vec_free(iv); + iv->iovec = iov; + iv->nr = nr_entries; + return 0; +} + +static int io_vec_fill_bvec(int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + struct iovec *iovec, int nr_iovs, + struct iou_vec *vec) +{ + unsigned long folio_size = (1 << imu->folio_shift); + unsigned long folio_mask = folio_size - 1; + struct bio_vec *res_bvec = vec->bvec; + size_t total_len = 0; + int bvec_idx = 0; + int iov_idx; + + for (iov_idx = 0; iov_idx < nr_iovs; iov_idx++) { + size_t iov_len = iovec[iov_idx].iov_len; + u64 buf_addr = (u64)iovec[iov_idx].iov_base; + u64 folio_addr = imu->ubuf & ~folio_mask; + struct bio_vec *src_bvec; + size_t offset; + u64 buf_end; + + if (unlikely(check_add_overflow(buf_addr, (u64)iov_len, &buf_end))) + return -EFAULT; + if (unlikely(buf_addr < imu->ubuf || buf_end > (imu->ubuf + imu->len))) + return -EFAULT; + + total_len += iov_len; + /* by using folio address it also accounts for bvec offset */ + offset = buf_addr - folio_addr; + src_bvec = imu->bvec + (offset >> imu->folio_shift); + offset &= folio_mask; + + for (; iov_len; offset = 0, bvec_idx++, src_bvec++) { + size_t seg_size = min_t(size_t, iov_len, + folio_size - offset); + + res_bvec[bvec_idx].bv_page = src_bvec->bv_page; + res_bvec[bvec_idx].bv_offset = offset; + res_bvec[bvec_idx].bv_len = seg_size; + iov_len -= seg_size; + } + } + if (total_len > MAX_RW_COUNT) + return -EINVAL; + + iov_iter_bvec(iter, ddir, res_bvec, bvec_idx, total_len); + return 0; +} + +static int io_estimate_bvec_size(struct iovec *iov, unsigned nr_iovs, + struct io_mapped_ubuf *imu) +{ + unsigned shift = imu->folio_shift; + size_t max_segs = 0; + unsigned i; + + for (i = 0; i < nr_iovs; i++) + max_segs += (iov[i].iov_len >> shift) + 2; + return max_segs; +} + +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + int nr_iovs, unsigned iovec_off, + unsigned issue_flags) +{ + struct io_rsrc_node *node; + struct io_mapped_ubuf *imu; + unsigned cache_nr; + struct iovec *iov; + unsigned nr_segs; + int ret; + + node = io_find_buf_node(req, issue_flags); + if (!node) + return -EFAULT; + imu = node->buf; + if (imu->is_kbuf) + return -EOPNOTSUPP; + + iov = vec->iovec + iovec_off; + ret = io_estimate_bvec_size(iov, nr_iovs, imu); + if (ret < 0) + return ret; + nr_segs = ret; + cache_nr = vec->nr; + + if (WARN_ON_ONCE(iovec_off + nr_iovs != cache_nr) || + nr_segs > cache_nr) { + struct iou_vec tmp_vec = {}; + + ret = io_vec_realloc(&tmp_vec, nr_segs); + if (ret) + return ret; + + iovec_off = tmp_vec.nr - nr_iovs; + memcpy(tmp_vec.iovec + iovec_off, iov, sizeof(*iov) * nr_iovs); + io_vec_free(vec); + + *vec = tmp_vec; + iov = vec->iovec + iovec_off; + req->flags |= REQ_F_NEED_CLEANUP; + } + + return io_vec_fill_bvec(ddir, iter, imu, iov, nr_iovs, vec); +} diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index e3f1cfb2ff7b..769ef5d76a4b 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -61,6 +61,10 @@ struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req, int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, unsigned issue_flags); +int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + int nr_iovs, unsigned iovec_off, + unsigned issue_flags); int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg); int io_sqe_buffers_unregister(struct io_ring_ctx *ctx); @@ -146,6 +150,7 @@ static inline void __io_unaccount_mem(struct user_struct *user, } void io_vec_free(struct iou_vec *iv); +int io_vec_realloc(struct iou_vec *iv, unsigned nr_entries); static inline void io_vec_reset_iovec(struct iou_vec *iv, struct iovec *iovec, unsigned nr) From patchwork Mon Mar 3 15:50:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999069 Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED5A921481D for ; Mon, 3 Mar 2025 15:50:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017008; cv=none; b=W3bVxymouiA8oDXw+6dmuTbwfzIgjzUaoCWj5uQ2sZ+sNGWMMbcSjlltWEu4EnTS0ysXt8peec580I9S0G1vZlbblfl6rjUJ3RTjs+dfNNF9La29HNNMP0u82pLOq6YOPctmrimXHZh5NvBQYPfAO8u/gFTwT131NDfh9i+/CbA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017008; c=relaxed/simple; bh=/k+C4WrIcyZoM7DP6xOSy0mBPHla12k1sfjHglsWHKg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JBwWkNcTr6s4Oa8ZC+RCcIHdta2Ld+WfzRLePv33BB6u3nLEmiqmmo51+whzAOgtIbw01jP+of9KsgaYBhzlG6EgSynVuHFh7z2JaPvZt3vRytdvBsJD4y/r0z8z+DP6yzlyyRxqsDMMVmDUsLRi5L4AuzVwY2+zADlw/quNzv8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lVNg8c/I; arc=none smtp.client-ip=209.85.208.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lVNg8c/I" Received: by mail-ed1-f53.google.com with SMTP id 4fb4d7f45d1cf-5e4b410e48bso6983169a12.0 for ; Mon, 03 Mar 2025 07:50:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017005; x=1741621805; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MXOQyWnlq5tzOu2N0NICEtzunR0KjmiKmSNM3Jri7zM=; b=lVNg8c/IsBpaWIb8Wxt0ZllF8Co0ekaj0ujaqROdpM0raj9DVdDRh77K7t9rL7kH4+ +lvq3IEKqRY7IxNBz8IuIrMzKgViOMJEHkH7zL0Y9W4Pjf+DKGo2rXG3KZOyvUf44vyq PzLbbH2r3h5LsoSfxeeqlTHDWsTQovsvDJGaxUQuCqmekzxBWmfjOXxUZjgF6maqv+fh 7USLofssZ82O9Z+NToF28lhSYPJROYU/Eud+58UQf+o4afMs/MxVz2V6ZTPS27K4opPy oMuDTJ8YjzJUcxQ7YT9FuzgPJx9BXA+D0CmVG4jYtoDzogNFCsN0R5C66qZSUmuXJPWw TBjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017005; x=1741621805; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MXOQyWnlq5tzOu2N0NICEtzunR0KjmiKmSNM3Jri7zM=; b=LjTLORv8siQHKA8gPsF5xiwvw+vrdVFEl67hebg5q5MEB8kurC0qau9yc8bWJLYWf0 O9ZGeq0Cz36l9zkjsvZpc38kusZq/vNi/C8ZEmcLKblnzrSdyqKQ6pu1zyO90aMD+fF0 chrcBWRmLetLz8wjK0uSHVna9FpMg/JdAzvpxqm+GFtdu4UTpluZY2jGTWcOtd/P7Q+N N5G2cK3HqPMjnSXxXXXV6tCwRwgn/5E/FSGXdBYJc0/rkTuyTMvLuiSyNqvjrAU+6iQ6 CGBSl0dDBNI7ONJf4/2RCaXVSV2Ag1n3t0UwOhNMpKpuc3WHVzDFUMTC4wTJBDuk93Rb O4Vg== X-Gm-Message-State: AOJu0YxV7xOUnE2TH76yaMoAW+jE58Wp0AS1C4H0yn44ooTj1G+sRCUX Yp4Gg1vPoY/EsNv9JPS+cbTYKZ9d7TI2/FT5A6X8T+bYLtEDkr2HKmyGWg== X-Gm-Gg: ASbGnctPzlQyxeL7VnOm/1gVaYy85jN1LKaYBi0AGfPYERokHfi7L6+ctggdGq7akz6 42ljwHHDJ5E7AX41nx6Yr9mP0E0HD5q1A96JWMvshHnkzcIMfdlJOqqg4e/UBrkVBCaPuFigCzs IAmoTFOop7bFxL7da24NMZPIVQNynCV5LbW7j3dw3Y5B06N71MtkcsRQE5Qu2ylPI58fb2um9o2 uH+sfeBAjSV/1Nu2r3olIbX1A0WgOHyZBLoBmVqRhC5kkbfO+kuQ07rLXA/3D1FEyCDcdcQjPEp h8eZ75ZB2wTolc0fW3sv+IQgcpj6 X-Google-Smtp-Source: AGHT+IHuOrAZfretqhG4WseQbBXsQ/v93bwmCEqyNVRhm+hhSR/Z0iPuHqYAd6aMOFslnY/R4v0fEw== X-Received: by 2002:a17:907:3f29:b0:ac1:eebf:9d0 with SMTP id a640c23a62f3a-ac1eebf0a41mr13495566b.31.1741017004663; Mon, 03 Mar 2025 07:50:04 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:03 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 3/8] io_uring/rw: implement vectored registered rw Date: Mon, 3 Mar 2025 15:50:58 +0000 Message-ID: <444a0ab0c3dc6320c5604a06284923a7abefa145.1741014186.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Implement registered buffer vectored reads with new opcodes IORING_OP_WRITEV_FIXED and IORING_OP_READV_FIXED. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 2 ++ io_uring/opdef.c | 39 +++++++++++++++++++++++++++ io_uring/rw.c | 51 +++++++++++++++++++++++++++++++++++ io_uring/rw.h | 2 ++ 4 files changed, 94 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 1e02e94bc26d..9dd384b369ee 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -280,6 +280,8 @@ enum io_uring_op { IORING_OP_BIND, IORING_OP_LISTEN, IORING_OP_RECV_ZC, + IORING_OP_READV_FIXED, + IORING_OP_WRITEV_FIXED, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 9511262c513e..6655d2cbf74d 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -529,6 +529,35 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_READV_FIXED] = { + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_readv_fixed, + .issue = io_read, + }, + [IORING_OP_WRITEV_FIXED] = { + .needs_file = 1, + .hash_reg_file = 1, + .unbound_nonreg_file = 1, + .pollout = 1, + .plug = 1, + .audit_skip = 1, + .ioprio = 1, + .iopoll = 1, + .iopoll_queue = 1, + .vectored = 1, + .async_size = sizeof(struct io_async_rw), + .prep = io_prep_writev_fixed, + .issue = io_write, + }, }; const struct io_cold_def io_cold_defs[] = { @@ -761,6 +790,16 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_RECV_ZC] = { .name = "RECV_ZC", }, + [IORING_OP_READV_FIXED] = { + .name = "READV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, + [IORING_OP_WRITEV_FIXED] = { + .name = "WRITEV_FIXED", + .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, + }, }; const char *io_uring_get_opcode(u8 opcode) diff --git a/io_uring/rw.c b/io_uring/rw.c index ad7f647d48e9..4c4229f41aaa 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,6 +381,57 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } +static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + struct io_async_rw *io = req->async_data; + const struct iovec __user *uvec; + size_t uvec_segs = rw->len; + struct iovec *iov; + int iovec_off, ret; + void *res; + + if (uvec_segs > io->vec.nr) { + ret = io_vec_realloc(&io->vec, uvec_segs); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + /* pad iovec to the right */ + iovec_off = io->vec.nr - uvec_segs; + iov = io->vec.iovec + iovec_off; + uvec = u64_to_user_ptr(rw->addr); + res = iovec_from_user(uvec, uvec_segs, uvec_segs, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, 0); + iov_iter_save_state(&io->iter, &io->iter_state); + return ret; +} + +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_DEST); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_DEST); +} + +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = __io_prep_rw(req, sqe, ITER_SOURCE); + if (unlikely(ret)) + return ret; + return io_rw_prep_reg_vec(req, ITER_SOURCE); +} + /* * Multishot read is prepared just like a normal read/write request, only * difference is that we set the MULTISHOT flag. diff --git a/io_uring/rw.h b/io_uring/rw.h index e86a3858f48b..475b6306a316 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -34,6 +34,8 @@ struct io_async_rw { int io_prep_read_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_writev_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_readv(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_writev(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_prep_read(struct io_kiocb *req, const struct io_uring_sqe *sqe); From patchwork Mon Mar 3 15:50:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999070 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57395214A78 for ; Mon, 3 Mar 2025 15:50:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017010; cv=none; b=grcKiHf4rtqpOIqxlqYRMEot7xKlK7NDty0gFaWXIdvXAb9o4+UwlnIym378P8Z6oD75jgwtC8zUrJItXR8qoINLOIxw2G5T/RWhb8F5BaTBdBmtwynfiCy4LKriLM7mMR5eJeZBud8iXckqh8cftUn/FHr7efpWzt8UovJm5CQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017010; c=relaxed/simple; bh=lHmXTgZLZPrFWQA2uBEMU/GbCtmIu6SrxYKZmNwzLF0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YnKnUeW21Ld9Ns+FSdMbb3BPLF1aVPhn5MfC6Ce/3BchMjESSxRN7mKmYKkmHA63jRHWdFhdXAnDX3yHh8JUfghi5KE2DCYv6YQ4TXSKxeuQcu37i8R/HQlHLZJ3U0/DUVjHMog2VBupzbu6FnHlgEfJUjFuaBnIRk3bEwlTHNI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fJ1rm8me; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fJ1rm8me" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-ab771575040so1011586066b.1 for ; Mon, 03 Mar 2025 07:50:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017006; x=1741621806; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qOI8bdVbcoPe8sCdbKgJ/CnnnozyzBKVPBUSTN1RX5w=; b=fJ1rm8meZi1p09ide7nSdLZQJXtmPKpQiF95LKKTCVnHNmRsxwk4I7aSXC+JshgkrZ eS91AJC1Slfvxa6HOdxOSOMOGTpKZmvyEGJRnFHJ694Eig9S35rlj0sajFVyboSvbTup 1VhowIjo+EBx8zikgaHj9EkG1dZheCF2QtyX2mwDzr/MrRb8OZvK9MnRQ1NejTnwZl3P +ogMc9jyTHyFR8mtrb6OGTnB01MNTtrarJ+YEO8J9eMaWky2oEDc4/HcaS2cL5hiM3Ch bD8z0Mkvbbjipp8X/WCEGmY2y0LkXjJbjOR4g5otfH3ggoAmnTeliCqr6fCieji1v3BD DZBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017006; x=1741621806; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qOI8bdVbcoPe8sCdbKgJ/CnnnozyzBKVPBUSTN1RX5w=; b=fiFF2zL4alqowONOq118IcRJsvBe9+ba6/Sfsx6+lTJtTvnqRtphTSSpT/4thLMjkW F6oieItbu+GJYIZK+uAYSJiegF450HfQT3HcabnIpU4j+OuvUA8DRc9Y0ZtvOEx4pECZ Ip7g/Ctu4mFgl04iI+pIOqSwJROcNlf67aeP5xevvayVvb7WJbWSEtjbZMExFh88QhHX tGh53ZeXgEPue2c9IJ3LFvRKok7w6Q+p6/EL+asKf1Au7xA00jkkIc7IMzL0t3Tqrb3a TOJoI4OQAOeuQmoBb1I5HnT/K9M07Nv4511Ba5b5RWtUTrtIpddrvaT8kNQe1NGp3Fro fahA== X-Gm-Message-State: AOJu0Yze5U4l8LZsj1w/7mLWhPkIPhrgVikyP981hB/n12yjiAnZpSwL nSnaCdgZOY/JYTtndfRpJKJDa29VfF1LMvHIycj6qlkAtbYAO+7FTpvPxw== X-Gm-Gg: ASbGncvRzBTJtIOmw1dVwq4D/86AtmygsyGZ/cxKRzuIO5P4Ctba6+qznqzPAhUZocz YR79HXb6d8/MRnHwfVoI8PgoFP15dAku2JzHazX7UR+Llbno6GI7AkpE3QetiO0ixsSyW8GUNDO fo1ABdsDAlFbxlGHVtnrsPrBQQHEeVKI59Yc3OBrTx8qoHvc9AYh4sIAGZAL2TRbd55Q0OKA5QP ppLxaHv5b0oC49lOWy4KAcjAOjjdWKqMgOsXprOvbh1TvnIrAjfISaP5WBQXjRW3bXC+Th+lOb/ 6xY8hpTqfKYnzx4tVL1nvtNY5cnX X-Google-Smtp-Source: AGHT+IG8VnigtDDJepQuYFkEFzgfRIXO008KSy3xpbq04HuITU2qGMbMU16oNEehgKwwZWzUeS+7+g== X-Received: by 2002:a17:907:7f92:b0:ac1:edc5:d73b with SMTP id a640c23a62f3a-ac1edc5d856mr28321566b.8.1741017005894; Mon, 03 Mar 2025 07:50:05 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:05 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 4/8] io_uring/rw: defer reg buf vec import Date: Mon, 3 Mar 2025 15:50:59 +0000 Message-ID: X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Import registered buffers for vectored reads and writes later at issue time as we now do for other fixed ops. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +++ io_uring/rw.c | 36 +++++++++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index b770a2b12da6..d36fccda754b 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -502,6 +502,7 @@ enum { REQ_F_BUFFERS_COMMIT_BIT, REQ_F_BUF_NODE_BIT, REQ_F_HAS_METADATA_BIT, + REQ_F_IMPORT_BUFFER_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -584,6 +585,8 @@ enum { REQ_F_BUF_NODE = IO_REQ_FLAG(REQ_F_BUF_NODE_BIT), /* request has read/write metadata assigned */ REQ_F_HAS_METADATA = IO_REQ_FLAG(REQ_F_HAS_METADATA_BIT), + /* resolve padded iovec to registered buffers */ + REQ_F_IMPORT_BUFFER = IO_REQ_FLAG(REQ_F_IMPORT_BUFFER_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, io_tw_token_t tw); diff --git a/io_uring/rw.c b/io_uring/rw.c index 4c4229f41aaa..33a7ab2a8664 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -381,6 +381,24 @@ int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_prep_rw(req, sqe, ITER_SOURCE); } +static int io_rw_import_reg_vec(struct io_kiocb *req, + struct io_async_rw *io, + int ddir, unsigned int issue_flags) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + unsigned uvec_segs = rw->len; + unsigned iovec_off = io->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, + uvec_segs, iovec_off, issue_flags); + if (unlikely(ret)) + return ret; + iov_iter_save_state(&io->iter, &io->iter_state); + req->flags &= ~REQ_F_IMPORT_BUFFER; + return 0; +} + static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); @@ -406,10 +424,8 @@ static int io_rw_prep_reg_vec(struct io_kiocb *req, int ddir) if (IS_ERR(res)) return PTR_ERR(res); - ret = io_import_reg_vec(ddir, &io->iter, req, &io->vec, - uvec_segs, iovec_off, 0); - iov_iter_save_state(&io->iter, &io->iter_state); - return ret; + req->flags |= REQ_F_IMPORT_BUFFER; + return 0; } int io_prep_readv_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -906,7 +922,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret; loff_t *ppos; - if (io_do_buffer_select(req)) { + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_DEST, issue_flags); + if (unlikely(ret)) + return ret; + } else if (io_do_buffer_select(req)) { ret = io_import_rw_buffer(ITER_DEST, req, io, issue_flags); if (unlikely(ret < 0)) return ret; @@ -1117,6 +1137,12 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ssize_t ret, ret2; loff_t *ppos; + if (req->flags & REQ_F_IMPORT_BUFFER) { + ret = io_rw_import_reg_vec(req, io, ITER_SOURCE, issue_flags); + if (unlikely(ret)) + return ret; + } + ret = io_rw_init_file(req, FMODE_WRITE, WRITE); if (unlikely(ret)) return ret; From patchwork Mon Mar 3 15:51:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999071 Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D4C3214A73 for ; Mon, 3 Mar 2025 15:50:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017011; cv=none; b=UnuTfxy7NWzBJMZi5oQiDsiCgKhTEJdUtiINwzVGUyZN+bqYIQ0QMjzh38niWU6Ar0FerUighy8g7iLypU8NH5vmXkqPhZy5g7jrVBb0G308/M10gXZG8TO5iBgSD9SD9k44oU8KipBabX1SfvqyDfFpNZ1LEM3ZjbsCGTmw51o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017011; c=relaxed/simple; bh=q/FZLY6NX7KPIL9q2sRYyOUjPSfSZOW4huImrRHQGAc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z/jWYobc2SLopYk8fLPgiob7fiD5kOJPZk9nZ60/n/8UWcrX2RQtpMpv2Mu6zOlH3J0ftsmgmGqE+Tv3UXkVyquw80fWA9O5o5dc0J6/UUnzPmbPR1NE3jkvqHkLfslPt5+0Umnk9UJCAw01T3pOAM6haKovc9TK5ftwznng124= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=G35oYLmb; arc=none smtp.client-ip=209.85.218.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="G35oYLmb" Received: by mail-ej1-f49.google.com with SMTP id a640c23a62f3a-abf3cf3d142so358064166b.2 for ; Mon, 03 Mar 2025 07:50:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017007; x=1741621807; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=G35oYLmbAWuADJQvHRHmOHcTz3QXsjyHxcxyqWmP9EVuLRK2Uc13o7rBQB+s1U8G1D fyAQPz0PXTdFhI4xoOr1/RQjQubQDab2LArdo3fYJn5SGyb3bnZNeqcFMG31kZSvqZ9S 2XpT91VKAFlLz7FOk+TBDJQQLFxPrWbmMeFY1jb7B/IJ0u5crfwRJzZa2S6tTHNekIP3 M4k3ONnHu4RCs8/AWjo0U1sOw9i9Qb9AuvGI8SQFU8MApw5rVMOe0BHmsEutmIG1t4i0 FPTvZsRwUGCDbOT4pxW4HSDadObbt2PFR5Dhl7QHObGvAdmn383z3uvCPBKebs2K7JpT HDSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017007; x=1741621807; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VMFfmh910ijk2bF3gt2GcETiZRuBH9+UmQ8at+7H2ws=; b=cdY0nINAc53EST3wnfF1AzX7nBtc8/5GiK/dRvRs+x4IYL3w7HaPE7nDkaFlj5gUM2 IpmE4itJTYS6hPwiF0tUb3hC2Qq4vqxwwkQkH9yAIHeRCm+3t2J5/uTWRTGBzZWE7nZ1 NyeLACmuXzTf3N03w7Cpsx6RnO5z1B8tVpMHzi5puUYFQnTAOH30l2ySQZlb14RKRfBz l2WxG/dReY9JTZtYrLsjpWTDrlRScRnman2zsa+RIC65GS286dWY8Zb9XjB7bs2KFN8M 2KWFIgiZ59oSim2UZtWVBYUy+EcWJ/LIk2msdTe3ZGCEf+0j1O0Y4f1UAxcVV2fqXcqY FuQA== X-Gm-Message-State: AOJu0YwGHG2/bzr2cTa6AEwEPCddzjXwCZA7FAY+f7LGnPAihu6R14bi KrrjYaLjCEK+11YhHLk9vQYhvT1ceXXloEmZhnunrs15EAur0j8fe0a2RQ== X-Gm-Gg: ASbGncuaszHm/hdnV2/ZPeC6D+Zdigj6vlwm/s4hZYXG0mWZR+rouQy9aH951mthJlH ZPoxxgyo0YHIqxWFbyf/Fi6R1dXHm2jNuBF6BG4on3kOuLmmVXx0VP5SmalqiHVgqv6Wi+8DQyf JXaISzSCeE0/hoQLWQUnIWxmJI0LvBqW5JpYrD2yDtm/1RaJNhnIw3AYkSdVVVleMA/jRTfWSlL niSzQ/y0rb+Lu9A9j4Zd1mXiUbVfMltNckdXbJyqkGhP2IuXC8pbBoVr7mMtMGLswR1eGfPGwmQ LFVYx4G8dL8rcrFrUwu0+m+c2vf2 X-Google-Smtp-Source: AGHT+IEsKn4k5lnP374tskqgJJg/1CcMvM9epqoq78NM4lyJNlpbcHIH2nnr2krLQDpXbZ7/AhOBmA== X-Received: by 2002:a05:6402:4316:b0:5dc:9589:9f64 with SMTP id 4fb4d7f45d1cf-5e4d6ad8c0dmr37532194a12.13.1741017007219; Mon, 03 Mar 2025 07:50:07 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:06 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 5/8] io_uring/net: combine msghdr copy Date: Mon, 3 Mar 2025 15:51:00 +0000 Message-ID: <569c679f6eb96a976021db2618aa0a3566b1ebef.1741014186.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Call the compat version from inside of io_msg_copy_hdr() and don't duplicate it in callers. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 905d1ee01201..33076bd22c16 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -287,6 +287,24 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct user_msghdr __user *umsg = sr->umsg; int ret; + iomsg->msg.msg_name = &iomsg->addr; + iomsg->msg.msg_iter.nr_segs = 0; + + if (io_is_compat(req->ctx)) { + struct compat_msghdr cmsg; + + ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ddir, save_addr); + if (ret) + return ret; + + memset(&msg, 0, sizeof(msg)); + msg->msg_namelen = cmsg.msg_namelen; + msg->msg_controllen = cmsg.msg_controllen; + msg->msg_iov = compat_ptr(cmsg.msg_iov); + msg->msg_iovlen = cmsg.msg_iovlen; + return 0; + } + ret = io_copy_msghdr_from_user(msg, umsg); if (unlikely(ret)) return ret; @@ -323,18 +341,6 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_SOURCE, - NULL); - sr->msg_control = iomsg->msg.msg_control_user; - return ret; - } - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; @@ -710,21 +716,7 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, struct user_msghdr msg; int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - - if (io_is_compat(req->ctx)) { - struct compat_msghdr cmsg; - - ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_DEST, - &iomsg->uaddr); - memset(&msg, 0, sizeof(msg)); - msg.msg_namelen = cmsg.msg_namelen; - msg.msg_controllen = cmsg.msg_controllen; - } else { - ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); - } - + ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, From patchwork Mon Mar 3 15:51:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999072 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF48C214A68 for ; Mon, 3 Mar 2025 15:50:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017012; cv=none; b=Ru1b1fT2LFg+3iRWCEhvkGAjCK7EDFuQWlC+4bEm7sLonqhp1WsH8khLiBoeZjy25I4MHdK0TYwaPW67sMkie5LelVeso2jeeWpAQ2EwHKrfIgDFzE7rY7CJ3XIFtU4GkpdaOTA/Umx6eMR5NoyiEQHqEAxN7Q6C2Y53VnsLzmo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017012; c=relaxed/simple; bh=dFLQOj2ZLIbINKKwcv5tlrqtAVZRjLUjvTJ5sPCh7Go=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u8senyEWkPauWPffNxCFhaAdjwALRxmCTrbI3+T6qqnAQfuT1ZPHmMWHpUv03b41RvIKO6Q9qrRNzqADOjM42qxCVLTsquokw9SgaHTISanl4dUqtaJtdYkL+hQJYc3aTqgUOoR8jzOT7oZf/q0u5ED7vE53tPX97NG+Z0c6WZA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dqeCb729; arc=none smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dqeCb729" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5e033c2f106so5222851a12.3 for ; Mon, 03 Mar 2025 07:50:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017009; x=1741621809; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=dqeCb729v11PK/dvlvvXCk8+NwiFdAh+BpzROWdPRcNfNQPbDfdu1DV9IvOXUOgpxu Twuh/B2K62/RpDbn3hbFsUYo6v8X7jN23+5Ne3nYpO/PrWHN0COc7kZu+MmuR6G2oLV0 SQP1wGc+JJu1LOmrIdX9WKHQ6cg1zR47z0t12+q2q7/AifOolJniq8+AAZcILqab2Hmx sDYHsZuhW/+h1afV3n6TDnrAATC1vCaAbHeAOExMIUGjMyEqFWI+JoEo0eoLobBotMr/ Zrz27CAb9AW6hfsmzFm4RGS1cz4CpE6jjntLN0gvpcjevATpaFGKAI5AggJhWyXOGkbw GV8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017009; x=1741621809; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xq9H3dhjnKdFz5nEV9/T6Wusp8ITCyq4E/MzF/cO3lc=; b=wSguBSBfLdEBT/nOA1Um6Yrb8QU5ouJT0+GLjqiUoJl3g/ng+mk8DN4DbPmddrNTOv WqYU5O9k9SS9gfbHhGKvrN81EDYkJDEpFG34wpPN5DvriCL2jVr190F89FJMbTeGtGtb /NgtG2sUeEK4FFOLxX7MWdXZcu1V6EuXv59k9gquNUgBPvhCjKvT5rC8/d6bbwzbHdnP mCmb1L2NxK3fgvcLk43mY6DbYMEYg0XwLMe5CIAIFrZNmAv3phcceXvyNP4SX2s26cA8 Mr8Cr6hAWqpT8vS37rjH7WVon4z0hoMD1K9Y3dh6nmHTFYF+deOvBgYJEDcQKtzLH6Ry gOCA== X-Gm-Message-State: AOJu0YxGImfatCISFKqDgv3dUqgw5A0si/t1UYOCzZC5ujHmPAXgM7vL nHsai8+UKwhBeNWTwAdS2rxw9Wapwqk5JDY6qwS8n7iFRgeGf4AIF6CP/A== X-Gm-Gg: ASbGncvks/vfE6Xe3VdzaZHYkLwDmv4StDUpCpHcAcj70D6AcDw9dn/oqi61GaaVysX NNVWzYrJDylzm/KSK7Z0ALbigFaeKbjPXo/piniRrsz5l1S2F7M/Q/aySupVoIHwV2+4I4wThgQ 7k7An5iOZEoBrLkoM0vLHTaNHnjZ8gPq0Qot7ZrRv9ippd6uXl1uFcBXmzLG4bCbf4MIhmYIkT/ 3SLh9bL9MaLcm26jYJjGSMPxq4/mguFrnVFe3ZQ8xPK2iSDQa50316USXLqCUswreKdUK4vck6N U8DvhPX34BdX0r+cyXyC1QjdyQe3 X-Google-Smtp-Source: AGHT+IE+iN/p5Bsx4Qg6XD+UpUhQNchjUI5vLe8Fy6Cod7Xli4OgnfeJCqByb1OdGNiRRlsT0c0DFQ== X-Received: by 2002:a05:6402:1d4d:b0:5dc:796f:fc86 with SMTP id 4fb4d7f45d1cf-5e4d6af436amr37347679a12.16.1741017008478; Mon, 03 Mar 2025 07:50:08 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:07 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 6/8] io_uring/net: pull vec alloc out of msghdr import Date: Mon, 3 Mar 2025 15:51:01 +0000 Message-ID: <7ddd011c69f9cc12229c43696655fc0d36a54916.1741014186.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 I'll need more control over iovec management, move io_net_import_vec() out of io_msg_copy_hdr(). Signed-off-by: Pavel Begunkov --- io_uring/net.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 33076bd22c16..cbb889b85cfc 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -253,12 +253,8 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, return -EFAULT; sr->len = tmp_iov.iov_len; } - - return 0; } - - return io_net_import_vec(req, iomsg, (struct iovec __user *)uiov, - msg->msg_iovlen, ddir); + return 0; } static int io_copy_msghdr_from_user(struct user_msghdr *msg, @@ -328,10 +324,8 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, return -EFAULT; sr->len = tmp_iov.iov_len; } - return 0; } - - return io_net_import_vec(req, iomsg, msg->msg_iov, msg->msg_iovlen, ddir); + return 0; } static int io_sendmsg_copy_hdr(struct io_kiocb *req, @@ -342,6 +336,12 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, int ret; ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_SOURCE); /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; return ret; @@ -719,6 +719,13 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST, &iomsg->uaddr); if (unlikely(ret)) return ret; + + if (!(req->flags & REQ_F_BUFFER_SELECT)) { + ret = io_net_import_vec(req, iomsg, msg.msg_iov, msg.msg_iovlen, + ITER_DEST); + if (unlikely(ret)) + return ret; + } return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, msg.msg_controllen); } From patchwork Mon Mar 3 15:51:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999073 Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49A0D214A6E for ; Mon, 3 Mar 2025 15:50:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017014; cv=none; b=LtY8/vOcsn30k30KWTNHv85+zyOua1X9IdPX2ckFNLPVsOCL8YNwnqocNY38hsxlEFXrX+z5lEm7sffT2ulvhcmQzqpzfjBvkP/8IeKLS4GeJT41XeJnu9KEmrEDFZG1iuUL5XKMxg30hsWJBVijaBYnIDWxk+1R1EIp51eEEfg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017014; c=relaxed/simple; bh=zWZarhNyJi53T++SjywLAAUqP8VMK2X1hgd352VZQsk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VKUas4vqmXqJC+yLxMhlPEVNYE9sb283dehu1GJAwsTodOx53bmX1HrHRs3VO45Hu9raETf2jI/jQ1PhRb/odGZPyPn7V1fm0U0GOevRCFRlCVPCtHmjoo60YS4m+ws9FtG1PQ+qZ3KMXxEZKAE+zb2F+winiDTLmh0A3iWV3Ss= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IIJUBar8; arc=none smtp.client-ip=209.85.218.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IIJUBar8" Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-abf4cebb04dso413378966b.0 for ; Mon, 03 Mar 2025 07:50:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017010; x=1741621810; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7RWfRi6x1mSPiivij2EBvrzcTuGy50QoeAUbZiSRBp0=; b=IIJUBar8/k+bxtmvHurlEzB5Cv2jK35MaecjFRlkM2PmK8l7ixxSNN7UPzWjC6PjUv L36iymqPYLV2vv+IPzJRSxSZcJdtgRxFkxYRXUtAhA7nYxwnl/HLg3wC+o5SXvTWOuNQ QIasMwTJmZqi9gLTHf32BPpTUYQyeUu0QyYTKXvO4Vf7O6MahN6O5x+FieSLt6j3Ozrg MfuW0bcV+Sjc0wAeiKw6gSJ51Es2KYdz/mwwJU7DiTxwP86R3Y8QzGQ4zWMwg44GZaEe /t3wEeEB8wlzVQxHW983pwElsqr3WBSIOKMzfq1mQuS3GBaQLfmzx4FAcF2SGsy+94px 6Nmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017010; x=1741621810; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7RWfRi6x1mSPiivij2EBvrzcTuGy50QoeAUbZiSRBp0=; b=ufnHa0h7J0JejG2LjaHKBwnFyibdGStttvTxjPSa0aNJ3lfrPvyLgzp7UFbn6j8b26 +IHjTROGPK4ysFy3S44AL6ab1WIr+Hffit0hKuA5EczOo8IBJ4Tr49qomC3iu/ofubyD EBk64Ausuakdpp2eQYxTBdRogIXhHS8+T9hKnH9X8ibt2+XCQAzDRvjGpV46R5RAbJ3F ozjAFZCeXyHishTDAIvevoMUPiD/yESDlGNpSMlvICMQI7i9UAWro0NKqSYSFjy/Z7IA Nfa6yD1TUcQCt46oVHJhxXf74c5F4WRwOG3K+Rly5/ApxlDqL6+P9e9qVVoQg9Hc+ipl pVPA== X-Gm-Message-State: AOJu0YwZ2a983PqL95IeLhiCog9bme098OSTTT+vUyPUPU/xANK5kPJl 7TQki/EvJbyJ1jX2QLzEsypDO7ok0ZRX5tHLDqTh8JrvMFK+m9TVTnAplg== X-Gm-Gg: ASbGncu8ckkqawTbPePo6cpRmbV3ezIClv0e50NdXdehYN5x6icgb1Zq/3WZaKsDwKT jCRQhBsgr2nvTsjJ/F+Sl+MTd0rIkxalo2toaZ+6u5mZr/t+GY/teq2lOLORBVviyyBHv4A8Qie 6EadB1KgKadzPwxodBHmOFdbZ9LUUZi16W+58VQP0kLl4vK9LZ+0kIcWOQBVsFeE+xwJtp/6I6K M0giJ6GZhGf+e6gINiCTe16V+ktmb8cGIWgumYuwePfkWzfzF+sfcKEr7tJ++qXJ5aUNgU5rXFA bYtSreC6dZIOFMgOvbLs1xmP8Lz9 X-Google-Smtp-Source: AGHT+IE7eDNYUt6NKBIYxCRHut2Ooazzn759df/qNoG+3i/G1s06Av6sl2D0z0JB1DSAgoI8e6C/mQ== X-Received: by 2002:a17:906:acc:b0:abf:4a3b:454d with SMTP id a640c23a62f3a-abf4a3b62demr1076997466b.7.1741017009882; Mon, 03 Mar 2025 07:50:09 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:09 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 7/8] io_uring/net: convert to struct iou_vec Date: Mon, 3 Mar 2025 15:51:02 +0000 Message-ID: <52aad74e49bfff5d7751ddd232cd4e2733dabae9.1741014186.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert net.c to use struct iou_vec. Signed-off-by: Pavel Begunkov --- io_uring/alloc_cache.h | 9 -------- io_uring/net.c | 51 ++++++++++++++++++------------------------ io_uring/net.h | 6 ++--- 3 files changed, 25 insertions(+), 41 deletions(-) diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 0dd17d8ba93a..7094d9d0bd29 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -16,15 +16,6 @@ bool io_alloc_cache_init(struct io_alloc_cache *cache, void *io_cache_alloc_new(struct io_alloc_cache *cache, gfp_t gfp); -static inline void io_alloc_cache_kasan(struct iovec **iov, int *nr) -{ - if (IS_ENABLED(CONFIG_KASAN)) { - kfree(*iov); - *iov = NULL; - *nr = 0; - } -} - static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, void *entry) { diff --git a/io_uring/net.c b/io_uring/net.c index cbb889b85cfc..a4b39343f345 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -136,11 +136,8 @@ static bool io_net_retry(struct socket *sock, int flags) static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) { - if (kmsg->free_iov) { - kfree(kmsg->free_iov); - kmsg->free_iov_nr = 0; - kmsg->free_iov = NULL; - } + if (kmsg->vec.iovec) + io_vec_free(&kmsg->vec); } static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) @@ -154,7 +151,7 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } /* Let normal cleanup path reap it if we fail adding to the cache */ - io_alloc_cache_kasan(&hdr->free_iov, &hdr->free_iov_nr); + io_alloc_cache_vec_kasan(&hdr->vec); if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -171,7 +168,7 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) return NULL; /* If the async data was cached, we might have an iov cached inside. */ - if (hdr->free_iov) + if (hdr->vec.iovec) req->flags |= REQ_F_NEED_CLEANUP; return hdr; } @@ -182,10 +179,7 @@ static void io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg { if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - kmsg->free_iov_nr = kmsg->msg.msg_iter.nr_segs; - if (kmsg->free_iov) - kfree(kmsg->free_iov); - kmsg->free_iov = iov; + io_vec_reset_iovec(&kmsg->vec, iov, kmsg->msg.msg_iter.nr_segs); } } @@ -208,9 +202,9 @@ static int io_net_import_vec(struct io_kiocb *req, struct io_async_msghdr *iomsg struct iovec *iov; int ret, nr_segs; - if (iomsg->free_iov) { - nr_segs = iomsg->free_iov_nr; - iov = iomsg->free_iov; + if (iomsg->vec.iovec) { + nr_segs = iomsg->vec.nr; + iov = iomsg->vec.iovec; } else { nr_segs = 1; iov = &iomsg->fast_iov; @@ -468,7 +462,7 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret) if (iter_is_ubuf(&kmsg->msg.msg_iter)) return 1; - iov = kmsg->free_iov; + iov = kmsg->vec.iovec; if (!iov) iov = &kmsg->fast_iov; @@ -584,9 +578,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, .nr_iovs = 1, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode = KBUF_MODE_FREE; } @@ -599,9 +593,9 @@ static int io_send_select_buffer(struct io_kiocb *req, unsigned int issue_flags, if (unlikely(ret < 0)) return ret; - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } sr->len = arg.out_len; @@ -1085,9 +1079,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg .mode = KBUF_MODE_EXPAND, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->vec.iovec) { + arg.nr_iovs = kmsg->vec.nr; + arg.iovs = kmsg->vec.iovec; arg.mode |= KBUF_MODE_FREE; } @@ -1106,9 +1100,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg } iov_iter_init(&kmsg->msg.msg_iter, ITER_DEST, arg.iovs, ret, arg.out_len); - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->vec.iovec) { + kmsg->vec.nr = ret; + kmsg->vec.iovec = arg.iovs; req->flags |= REQ_F_NEED_CLEANUP; } } else { @@ -1874,8 +1868,7 @@ void io_netmsg_cache_free(const void *entry) { struct io_async_msghdr *kmsg = (struct io_async_msghdr *) entry; - if (kmsg->free_iov) - io_netmsg_iovec_free(kmsg); + io_vec_free(&kmsg->vec); kfree(kmsg); } #endif diff --git a/io_uring/net.h b/io_uring/net.h index b804c2b36e60..43e5ce5416b7 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -2,12 +2,12 @@ #include #include +#include struct io_async_msghdr { #if defined(CONFIG_NET) - struct iovec *free_iov; - /* points to an allocated iov, if NULL we use fast_iov instead */ - int free_iov_nr; + struct iou_vec vec; + struct_group(clear, int namelen; struct iovec fast_iov; From patchwork Mon Mar 3 15:51:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13999074 Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8981D214A68 for ; Mon, 3 Mar 2025 15:50:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017015; cv=none; b=gEvqW32Uy5by8wxTTNyG6c0eefkx5yMN0fAh6s8LUlnhr2gQt0P40iXoa/pAkGX8em5pG6gTaaZb1Z//by+MPoRf9bVeoWc8d0KaZIvXjdkyihdbNYapim+w++BRvI38wIpKXZSJkgCcSsoOv/NH8kRfbOULusc7WQjdF6PdgRc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741017015; c=relaxed/simple; bh=+ko6qVLwFbY8p2fCkuMHNRIwwTZQjN0AL9uVp4qcr08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aGXHDdk7GNeUKOylL1G0+QNjOIPG8UvDq67bkn6UJjA73S2Wxuw93rpJDXmq6Zk5L3z4ieieTpzXoHyDEwu/KEi6yWvkqkmD04+q3qKJPNrIs7LavPyEB+Q+7mvG4Or3aIwZTmN2VJvNTH7SqpaMokH0D4TpzMF66daefgyrcJE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FTVkC/uw; arc=none smtp.client-ip=209.85.218.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FTVkC/uw" Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-abf42913e95so415519866b.2 for ; Mon, 03 Mar 2025 07:50:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741017011; x=1741621811; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=FTVkC/uw+lzubZgQ6Ti/aOUB0dfHCYveL0kZ9e/DS9N1MHjwCaLdx/WE2qkQj/Szjr 75r7UE10AFwm2VJJ9FUk5YM1yHC6eCcvl7mDkV+NFRw2peZNNO/UWz9bxISl6SjqaSJZ 3BmQN2eCG1MIRdwTgsBVsTR9z93c+Vz7e3GFGEHG/qXaW2g9tMCfjf/T+Jkko/AQgxM0 Ks57UOxdxiwrLO2EwheycyDW+n0+j1C6QMsWDNE+oFhjB0ZsT5xAJ5hIYPzSQ7KEefEd EPwixWuDXPcDAAwerzW38tyIG149QfwikKJJypv4o0n874g1nV/KzPBn8iIzmhrMqnvk VzSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741017011; x=1741621811; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yljV5A1erT6qwC3xVxpqpVfrN9r/MNCu0zylOYIEozc=; b=bM0+D6T6YsD67zFCil4YGSY9k7D9Zc8gLPwci7PGcCg3HUu0WpViMUa9bD/2F41mjd nowenqQ7DWk/5RI0Zrn8vQdlXxIlP145hlqC8vYEKbXdG2ezQUAy8sFObDhI4eb8li0N rnWdd3tzYFjWWRL+vZcetUZmHuZsr99/oDUyHcRrIMS2ZRZohyYyjJ8Gij4sO2plDup/ /n1+y+xW63jyN8GvVisMqkXjILifUm+PkhNVDrDEQ355emuJ3sML6jCdxeJ5SCluZ0Rc jAkGsYe0RqiCZUtnIdD4xI1zzxVXJmCi9Z3ReiYq3hnS5muWZtvUZvwqG9guToohGEu0 3mSw== X-Gm-Message-State: AOJu0YxJZJ8C5lGIkhVljPJiw6sVUWsqooJvoK3TMKGkCqbj5R6A5+Fd 2Jxm5LzGRtCZryIW2xTNFfj5ko7o4GtHYRDx3sljcFBz828cbjYwjw0wcA== X-Gm-Gg: ASbGncvgCkx0wj73qRujRYy+l0I8k/63Pv9N09pzWMMnEGSnXVJ5APvDqFSdI3oM7g+ Xj+R/c2/XI5CNAzoytLiBzqGyxuXSzyLqOOio2MLfSlXgS4sMGkhp7A0NzJ8xgCCTB5Z3CdaY6T 95sQS4pn4r722XXOGWtTsyBtAF/YxB7USg6ft9Ko31oaCo+VI831o6v3nH74MKN4Yf8C2K1GLgB 1Him7dhp0w18LBeKvRTokL1DzpXmXIEv69RTo3mclJ+FRmvxJDv/gmoot8XxwV9b9uxeOvBRRoT 7Xj4Czu4ukwaMcvdOKHGeJE9dtQD X-Google-Smtp-Source: AGHT+IHcUsv5eBQVod+isN0ix1b/5ICTR+hGIlxbIcGYZhT61r9aA/Cvxef6pGkt4U6fK+rTb2OiGg== X-Received: by 2002:a17:907:97cd:b0:abb:33ff:c5f5 with SMTP id a640c23a62f3a-abf269b4357mr1584049666b.48.1741017011188; Mon, 03 Mar 2025 07:50:11 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::1:299a]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-abf4e50c80esm492335266b.61.2025.03.03.07.50.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 07:50:10 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Andres Freund Subject: [PATCH 8/8] io_uring/net: implement vectored reg bufs for zctx Date: Mon, 3 Mar 2025 15:51:03 +0000 Message-ID: <0b47cabcec226a2630fabf0491856ec78748ef04.1741014186.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add support for vectored registered buffers for send zc. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index a4b39343f345..5e27c22e1d58 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -395,6 +395,44 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe return io_sendmsg_copy_hdr(req, kmsg); } +static int io_sendmsg_zc_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg = req->async_data; + struct user_msghdr msg; + int ret, iovec_off; + struct iovec *iov; + void *res; + + if (!(sr->flags & IORING_RECVSEND_FIXED_BUF)) + return io_sendmsg_setup(req, sqe); + + sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); + + ret = io_msg_copy_hdr(req, kmsg, &msg, ITER_SOURCE, NULL); + if (unlikely(ret)) + return ret; + sr->msg_control = kmsg->msg.msg_control_user; + + if (msg.msg_iovlen > kmsg->vec.nr || WARN_ON_ONCE(!kmsg->vec.iovec)) { + ret = io_vec_realloc(&kmsg->vec, msg.msg_iovlen); + if (ret) + return ret; + req->flags |= REQ_F_NEED_CLEANUP; + } + iovec_off = kmsg->vec.nr - msg.msg_iovlen; + iov = kmsg->vec.iovec + iovec_off; + + res = iovec_from_user(msg.msg_iov, msg.msg_iovlen, kmsg->vec.nr, iov, + io_is_compat(req->ctx)); + if (IS_ERR(res)) + return PTR_ERR(res); + + kmsg->msg.msg_iter.nr_segs = msg.msg_iovlen; + req->flags |= REQ_F_IMPORT_BUFFER; + return ret; +} + #define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE) int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -1333,8 +1371,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->opcode != IORING_OP_SEND_ZC) { if (unlikely(sqe->addr2 || sqe->file_index)) return -EINVAL; - if (unlikely(zc->flags & IORING_RECVSEND_FIXED_BUF)) - return -EINVAL; } zc->len = READ_ONCE(sqe->len); @@ -1350,7 +1386,7 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return -ENOMEM; if (req->opcode != IORING_OP_SENDMSG_ZC) return io_send_setup(req, sqe); - return io_sendmsg_setup(req, sqe); + return io_sendmsg_zc_setup(req, sqe); } static int io_sg_from_iter_iovec(struct sk_buff *skb, @@ -1506,6 +1542,22 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) unsigned flags; int ret, min_ret = 0; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; + + if (req->flags & REQ_F_IMPORT_BUFFER) { + unsigned uvec_segs = kmsg->msg.msg_iter.nr_segs; + unsigned iovec_off = kmsg->vec.nr - uvec_segs; + int ret; + + ret = io_import_reg_vec(ITER_SOURCE, &kmsg->msg.msg_iter, req, + &kmsg->vec, uvec_segs, iovec_off, + issue_flags); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; + req->flags &= ~REQ_F_IMPORT_BUFFER; + } + sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; @@ -1524,7 +1576,6 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_control_user = sr->msg_control; kmsg->msg.msg_ubuf = &io_notif_to_data(sr->notif)->uarg; - kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); if (unlikely(ret < min_ret)) {