From patchwork Wed Oct 23 02:38:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13846329 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E6183E47B for ; Wed, 23 Oct 2024 02:37:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651067; cv=none; b=hLOuXiZP7glsbkuF0FyzYNy7RTQGW1E6pIsH1OKIlgJc8CBCoGlZSiZbsHotTCAoVJsKafpDjrAEBWe5nU1i3X3u1Tc4821k7O8ipBIAoxEhI8TpBCC3kqtv0xYvaqNnPu409IavimHHrq7pEE/p1jtIUNZ5esVONt1Pc5XT2aQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651067; c=relaxed/simple; bh=EfHPem0do0GYFx0J09ST4Cctmb18cdKIyDDQs68dbFk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ewDOwFVxsJoH55p/Kok1hg+biRjEc8i6QUlskDeu0vYiR8hpkfLBy+GhSrWpLLoQiGxxq3scHHucR7pJ0CiN3mKEWayKo7BIKoK+VEzB4+K2i4rN7K00aQYdQSMtRBi70aGvpY2QwfXVhClsLCJXx0IHd2jZ2SY7pTVI3MgfL9g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SId/y5rG; arc=none smtp.client-ip=209.85.128.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SId/y5rG" Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-4314f38d274so85721095e9.1 for ; Tue, 22 Oct 2024 19:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729651064; x=1730255864; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cMUVtpGdejGLLGPI3KH9jr++Ui3+MXActRRl7F9MiG0=; b=SId/y5rG0HYRoF6WID9+cYzUPgxKRBkaKOaFzNwqV+zWsR2UdeHwRvbHNegX+64ypa OHw3ih6CpfQ1T9alljbW3yLPLDTumb6KWOjr8yu2T6K0u7afQonXhDRlMhH3XRI/71/l pFiEvsIxY7+ye/D6r2DhJg3BSAbCdbgbDucw9dyAyBs+gmCmGBGI0CdD+Gcrvlar8aCt U8nwtvaLrzcEWf6DPtay4PYCLm7Se90u8P7fW77i7XbmZmAOaHDkMy8mna/njv/BtuwW aqdXQ8R72cUz+IffI6F8UICFLYGt80v/TQ8FpO3R0Yaz9ZTIOTNSaEDkK3wRSwHVsOr6 DKXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729651064; x=1730255864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cMUVtpGdejGLLGPI3KH9jr++Ui3+MXActRRl7F9MiG0=; b=TfM2j6dn5zuJ6NUgYhk8IWTMTfahVQ9jBz10Qx6D2q/a6ZSFw4eIGRoDg0ARLm+XCL ANyQ4FFHpgYzxDnhlVPpCkK4Daqpzn3jdHktlGvSE/swaBpMhFmlpKTF1nqcLw6xZmFl lwYNMPSTKUWUlE1/qsgr2JrMOB2YW2mT8+nL1ohUa/VdDnitjKG7PvvHVzmDD1ghiJOw qa5xKhBAVDrSjA+dTKSLPq23Fh2lHpljF30mIZHV2JEusy0iNzu7A6aJXbl4rvEPS0ww ALef8miJRm1xLRcu0ne3T2G9j8eu0bXntTnDOn2dzqstBtz4Vn36l8KCXdVbFNPeMl7k E3Jw== X-Gm-Message-State: AOJu0Yxz231G5iTRI66JQ1TwWaueMJOgnuY4Ll5deMDxBN/g5vCY4FJN dAT/fsbTV84SKzPsYZsM/kJtL17RTYvnJAMCIgk+WgIvgDhHvKmQM/PnZw== X-Google-Smtp-Source: AGHT+IGJmXL397yq8TyJXKBKMg+ObMyezQt41230F9t/wCVT1PJgGlEL04FSpQWUyQe/9se0IEn5NQ== X-Received: by 2002:adf:e952:0:b0:37d:3705:84e7 with SMTP id ffacd0b85a97d-37efcf05c67mr750166f8f.17.1729651063420; Tue, 22 Oct 2024 19:37:43 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.112]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a9a91371046sm410418766b.139.2024.10.22.19.37.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2024 19:37:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 1/4] io_uring/net: introduce io_kmsg_set_iovec Date: Wed, 23 Oct 2024 03:38:18 +0100 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A prep patch, add a helper function taking an allocated iovec and assigning it to the kmsg cache. It'll be expanded upon in the following patch. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 0a0b148a153c..bd24290fa646 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -125,12 +125,18 @@ static bool io_net_retry(struct socket *sock, int flags) return sock->type == SOCK_STREAM || sock->type == SOCK_SEQPACKET; } +static inline void io_kmsg_set_iovec(struct io_async_msghdr *kmsg, + struct iovec *iov, int nr) +{ + kmsg->free_iov_nr = nr; + kmsg->free_iov = iov; +} + static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) { if (kmsg->free_iov) { kfree(kmsg->free_iov); - kmsg->free_iov_nr = 0; - kmsg->free_iov = NULL; + io_kmsg_set_iovec(kmsg, NULL, 0); } } @@ -174,8 +180,7 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) if (!io_alloc_async_data(req)) { hdr = req->async_data; - hdr->free_iov_nr = 0; - hdr->free_iov = NULL; + io_kmsg_set_iovec(hdr, NULL, 0); return hdr; } return NULL; @@ -187,10 +192,9 @@ static int io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg, { if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - kmsg->free_iov_nr = kmsg->msg.msg_iter.nr_segs; if (kmsg->free_iov) kfree(kmsg->free_iov); - kmsg->free_iov = iov; + io_kmsg_set_iovec(kmsg, iov, kmsg->msg.msg_iter.nr_segs); } return 0; } @@ -623,8 +627,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) return ret; if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + io_kmsg_set_iovec(kmsg, arg.iovs, ret); req->flags |= REQ_F_NEED_CLEANUP; } sr->len = arg.out_len; @@ -1107,8 +1110,7 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg iov_iter_init(&kmsg->msg.msg_iter, ITER_DEST, arg.iovs, ret, arg.out_len); if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { - kmsg->free_iov_nr = ret; - kmsg->free_iov = arg.iovs; + io_kmsg_set_iovec(kmsg, arg.iovs, ret); req->flags |= REQ_F_NEED_CLEANUP; } } else { From patchwork Wed Oct 23 02:38:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13846330 Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F5A649659 for ; Wed, 23 Oct 2024 02:37:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651068; cv=none; b=s1e6dv3c97XbA/zjmdTjZRE8abpfEOUyptbRDL+Nt0utUVgnz/ggLhmobVXRXSecP8mzTik0rq5GpcrEeFbB7LNCWyR1YfEDue/ezVr0ou05aQ+D4V07MtZDNZ5o0dO+Th+WPrPxQhl3PvzhpS1Wy0BxkM0ziWZdtEld1McV7as= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651068; c=relaxed/simple; bh=DGJiRNm1uxrVPT+D0PYHhHPXANKsKAeXgHzLGUqZzS4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u/tRG3T/+7teKTP6Nq0/lMrLAk8cjBLxx9id436J53EWTCZ5NKGk2VMqe6DALnOz8SADzduOc2g+R1xtEbuHTYHZWzhC45Kx8OjbssTaAcwyDUzYEKkIdqIX+j2l3fIHVyiG/AlB9lD2U21R2DeyZbg0vCaEDKmrFtPmwYTyQWo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YEzZS6Cn; arc=none smtp.client-ip=209.85.208.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YEzZS6Cn" Received: by mail-ed1-f51.google.com with SMTP id 4fb4d7f45d1cf-5c9c28c1ecbso8095359a12.0 for ; Tue, 22 Oct 2024 19:37:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729651065; x=1730255865; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JLns/gCvXAoxA+K3r+ie+TtYMtfuDR/vLKoP6gq63Z4=; b=YEzZS6CneZAauYin/JoRauF57bl+IexFs0qPkK5mhu64HndK3oA0UHXs3lUFTYY20z t/OCPPl675uXszPHFfeFo4pXDqg2vrP6Xuj5IvYL/KYmojOUPN/p7cblSaZD43ya73En TO7Qk4m828B0izE4ab7V+yMj/e/5nhGMzAg9oLZpRGS8o0ISOg2SVeXHya3wLQ77ctZF PJWjXBEX++17Gvk7sLu4UN4hl8LLPVI3AcOjbbyBLCeeZ+djbTi92NhlJ10uKlkEJQik vQ6HoFuiutNzt5781+omHsgFrgWMK/uGlTqxqkM3fTmY2TbPyaZsGPaYl3qmm4EnHkqh v3eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729651065; x=1730255865; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JLns/gCvXAoxA+K3r+ie+TtYMtfuDR/vLKoP6gq63Z4=; b=fRCi1vL9o3tRWkhmhmGGRqALqGd9OykZVXbbt6tJAzAR8I/JcLcRAgVFECYh+W0pBn 41tXRVr0Pr9U2cOJums/2VCv9JiKaN4+iH7rky/l+EUT/izPz6oLuXY/a9Z9Inm4B+ot GRxM8FM0RKsjdl43VHvog22uQomuqBV1iqquhgNDnK6yGySLgpH0KJbLazmObKMkOaZk bV3NH6BORsPjDnaRvendwzpEYv+8kbXk9KofnWFbgJVZXmzozEuJ6Rs62Uqf1FYfVG79 9AEh5Z/ekNL0ykxX0OCrnQzTvfaZlmbFbnZWjX9GfBJNrwgdWLJ1rCdSnRtm9oeqP3HU 9aCw== X-Gm-Message-State: AOJu0YwfgUm3JQUKAnDQzlb7/Y/KkpT+wOJPGaidBeRNsR2/+05R0lFu qA/sCCO0fExGTf1rllvlbczd2aeVSFqq6BLnDFnzDY1b2AGGboA2+8moEg== X-Google-Smtp-Source: AGHT+IH5Ge6B8XYenhlOfEyQqfK2mpVWjk9mxcZVnsIbi20vN91j7QBagWFJr50KxeqPlWO+4RWjbQ== X-Received: by 2002:a17:907:9726:b0:a99:523d:2108 with SMTP id a640c23a62f3a-a9abf866556mr100410366b.16.1729651064927; Tue, 22 Oct 2024 19:37:44 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.112]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a9a91371046sm410418766b.139.2024.10.22.19.37.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2024 19:37:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/4] io_uring/net: allow mixed bvec/iovec caching Date: Wed, 23 Oct 2024 03:38:19 +0100 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We're going to have allocated bvecs shortly, we need a place to store them and intra releasing it. Reuse the struct io_async_msghdr iovec caching for that. Get rid of typing and switch to bytes instead of keeping the number of iov elements the cached array can store. Performance wise it should be just fine as divisions will be compiled into binary shifts. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 67 +++++++++++++++++++++++++++----------------------- io_uring/net.h | 4 +-- 2 files changed, 38 insertions(+), 33 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index bd24290fa646..bc062b5a7a55 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -128,14 +128,19 @@ static bool io_net_retry(struct socket *sock, int flags) static inline void io_kmsg_set_iovec(struct io_async_msghdr *kmsg, struct iovec *iov, int nr) { - kmsg->free_iov_nr = nr; - kmsg->free_iov = iov; + kmsg->free_vec_bytes = nr * sizeof(*iov); + kmsg->free_vec = iov; +} + +static int io_kmsg_nr_free_iov(struct io_async_msghdr *kmsg) +{ + return kmsg->free_vec_bytes / sizeof(struct iovec); } static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) { - if (kmsg->free_iov) { - kfree(kmsg->free_iov); + if (kmsg->free_vec) { + kfree(kmsg->free_vec); io_kmsg_set_iovec(kmsg, NULL, 0); } } @@ -143,7 +148,7 @@ static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) { struct io_async_msghdr *hdr = req->async_data; - struct iovec *iov; + void *vec; /* can't recycle, ensure we free the iovec if we have one */ if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) { @@ -152,10 +157,10 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } /* Let normal cleanup path reap it if we fail adding to the cache */ - iov = hdr->free_iov; + vec = hdr->free_vec; if (io_alloc_cache_put(&req->ctx->netmsg_cache, hdr)) { - if (iov) - kasan_mempool_poison_object(iov); + if (vec) + kasan_mempool_poison_object(vec); req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; } @@ -168,9 +173,9 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) hdr = io_alloc_cache_get(&ctx->netmsg_cache); if (hdr) { - if (hdr->free_iov) { - kasan_mempool_unpoison_object(hdr->free_iov, - hdr->free_iov_nr * sizeof(struct iovec)); + if (hdr->free_vec) { + kasan_mempool_unpoison_object(hdr->free_vec, + hdr->free_vec_bytes); req->flags |= REQ_F_NEED_CLEANUP; } req->flags |= REQ_F_ASYNC_DATA; @@ -192,8 +197,8 @@ static int io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg, { if (iov) { req->flags |= REQ_F_NEED_CLEANUP; - if (kmsg->free_iov) - kfree(kmsg->free_iov); + if (kmsg->free_vec) + kfree(kmsg->free_vec); io_kmsg_set_iovec(kmsg, iov, kmsg->msg.msg_iter.nr_segs); } return 0; @@ -220,9 +225,9 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, struct iovec *iov; int ret, nr_segs; - if (iomsg->free_iov) { - nr_segs = iomsg->free_iov_nr; - iov = iomsg->free_iov; + if (iomsg->free_vec) { + nr_segs = io_kmsg_nr_free_iov(iomsg); + iov = iomsg->free_vec; } else { iov = &iomsg->fast_iov; nr_segs = 1; @@ -270,9 +275,9 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct iovec *iov; int ret, nr_segs; - if (iomsg->free_iov) { - nr_segs = iomsg->free_iov_nr; - iov = iomsg->free_iov; + if (iomsg->free_vec) { + nr_segs = io_kmsg_nr_free_iov(iomsg); + iov = iomsg->free_vec; } else { iov = &iomsg->fast_iov; nr_segs = 1; @@ -478,7 +483,7 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret) if (iter_is_ubuf(&kmsg->msg.msg_iter)) return 1; - iov = kmsg->free_iov; + iov = kmsg->free_vec; if (!iov) iov = &kmsg->fast_iov; @@ -611,9 +616,9 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) .nr_iovs = 1, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->free_vec) { + arg.nr_iovs = io_kmsg_nr_free_iov(kmsg); + arg.iovs = kmsg->free_vec; arg.mode = KBUF_MODE_FREE; } @@ -626,7 +631,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret < 0)) return ret; - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_vec) { io_kmsg_set_iovec(kmsg, arg.iovs, ret); req->flags |= REQ_F_NEED_CLEANUP; } @@ -1088,9 +1093,9 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg .mode = KBUF_MODE_EXPAND, }; - if (kmsg->free_iov) { - arg.nr_iovs = kmsg->free_iov_nr; - arg.iovs = kmsg->free_iov; + if (kmsg->free_vec) { + arg.nr_iovs = io_kmsg_nr_free_iov(kmsg); + arg.iovs = kmsg->free_vec; arg.mode |= KBUF_MODE_FREE; } @@ -1109,7 +1114,7 @@ static int io_recv_buf_select(struct io_kiocb *req, struct io_async_msghdr *kmsg } iov_iter_init(&kmsg->msg.msg_iter, ITER_DEST, arg.iovs, ret, arg.out_len); - if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_iov) { + if (arg.iovs != &kmsg->fast_iov && arg.iovs != kmsg->free_vec) { io_kmsg_set_iovec(kmsg, arg.iovs, ret); req->flags |= REQ_F_NEED_CLEANUP; } @@ -1807,9 +1812,9 @@ void io_netmsg_cache_free(const void *entry) { struct io_async_msghdr *kmsg = (struct io_async_msghdr *) entry; - if (kmsg->free_iov) { - kasan_mempool_unpoison_object(kmsg->free_iov, - kmsg->free_iov_nr * sizeof(struct iovec)); + if (kmsg->free_vec) { + kasan_mempool_unpoison_object(kmsg->free_vec, + kmsg->free_vec_bytes); io_netmsg_iovec_free(kmsg); } kfree(kmsg); diff --git a/io_uring/net.h b/io_uring/net.h index 52bfee05f06a..65d497985572 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -7,8 +7,8 @@ struct io_async_msghdr { #if defined(CONFIG_NET) struct iovec fast_iov; /* points to an allocated iov, if NULL we use fast_iov instead */ - struct iovec *free_iov; - int free_iov_nr; + void *free_vec; + int free_vec_bytes; int namelen; __kernel_size_t controllen; __kernel_size_t payloadlen; From patchwork Wed Oct 23 02:38:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13846331 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF7AB3E47B for ; Wed, 23 Oct 2024 02:37:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651070; cv=none; b=uKNBthVx/SOYhn6vE7Sb6sGJve5HKy4ptveUCWqtRHPDiYN5JXSOtt4KxXzZJCgEaHTSOZpIIYlK6UGrszi3HX3kay0Z4gg7h249VLWV87p6+PalQe6/tC1m1YwMF3A6nmPWhWH9RBx0attOk2LP1YXkEs6OY2x8EezoF72GemI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651070; c=relaxed/simple; bh=wmy/0xUu8rdddj78sP03A17QHFp6S7LvaSQUEZU4HTU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=g3Kp3BErb5/M7SPdvjSTMtjzILVyRQCbR9TZ63xwLbnFSiP6P1N6abBhblQd/l8MxHaVoTG/oIDVs9QwDC/jWlm/v6h44YWOdvDUHEvuWPKfazZeN4ItjXuKptVel4El4HRyyLu7ABXqC2IPMaNgJlhz4qCxg2gOlX99kGVRDZ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mhZeWIhu; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mhZeWIhu" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a9a0f198d38so870843166b.1 for ; Tue, 22 Oct 2024 19:37:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729651067; x=1730255867; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BZmFUg4K3Gqw0+UKGPY7kgfm/3RIi1Y7zSVrNyfsSM8=; b=mhZeWIhumtgz0JYhAnnBFAUBsC7YSX5de1+1SLsKk5WG28BDAjDtlCtWDYdfVdlBT5 fQVQ/ePX8G47XzSAgCrIWg4bySLkChRlH3CDBq5P/EcZJIulEpmybLKd/RvHPcRjilPJ SjljfiQ8/rU/Btp+8kL+mUr5Z4SnK2CoZ4ArQVRlUDaI37uSFHWMQdUtM0gaBZY8+30e 4twWGQg13AQcVs40mLC5SFT2WNTGKNhaRl0HCf30+axFr1+8g11ggdHeZO+o8XXpjvzZ eonRvXpzyeBxX6xi96OBFV56Mjyr81qmo9IVOimq32y8eQWfGPwqzPIkltuUdQGAHqM1 bV3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729651067; x=1730255867; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BZmFUg4K3Gqw0+UKGPY7kgfm/3RIi1Y7zSVrNyfsSM8=; b=rGrBYeTTpODHSoL7RWZgyikWPPC2Ge4mSBXqEyn5F1Jkt21LPtRJCkrfuxo43uL7K/ 0HV5av3J4TVIa/m0IA1VWiOx4eL0wAOPLraCZ5rMiAm2C4P91B17U1OsiyH641KlJRd6 q5rV36vvadAXJBr7dHGLOHkgM1bbg+m00pGavssBMTCBN6lam5lKt3TILlLQkaLQFqB6 bANQg/z5Uj0l7kcwIIUQBos4KukVgtCPpWZmuisQwALrQaIDcs0+ZOG4ZjB0GniWBW2B cf5r7CcdFtm877dZNGvAR/laRHykQ09Mm77MNg0TXnO5tV+FIVeQtjGAvp1NoKuzsvUx FelQ== X-Gm-Message-State: AOJu0YxPpyAzyU0kKt6X74JYpeiKNRhVBbe80Bh8rB3kGR1BiqggSOIu ErramuW2CfvNTv7LglULz3Sh+UKMtN0RNtDBiG/lD5KeFWB/5YRV8rjWww== X-Google-Smtp-Source: AGHT+IFp/86cXf6G8J19MixL2hCyJGI2nPQizbSxWSSC9Y+ty4BD8aQo3VxSBYoTWWLcPY/Dj3ogWw== X-Received: by 2002:a17:907:72d4:b0:a99:2ab0:d973 with SMTP id a640c23a62f3a-a9abf96ce13mr86143266b.55.1729651066640; Tue, 22 Oct 2024 19:37:46 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.112]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a9a91371046sm410418766b.139.2024.10.22.19.37.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2024 19:37:46 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 3/4] io_uring: vectored registered buffer import Date: Wed, 23 Oct 2024 03:38:20 +0100 Message-ID: <2b49bea692514ef41d9e5cdd3a1516caf7eacd98.1729650350.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a helper that takes a registered buffer and an iovec with addresses pointing into that registered buffer, and return a new bvec corresponding to the given iovec. Essentially, each iov entry is resolved into a bvec array, which gives us an array of arrays of struct bio_vec, which the function flattens into a single long bvec. Note, max_segs is overestimated, that can be improved later. The allocation also can be optimised by doing it inline into the same array. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 3 +++ 2 files changed, 63 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index fa5f27496aef..6f9f3cb4a2ef 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1085,6 +1085,66 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, return ret; } +struct bio_vec *io_import_fixed_vec(int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + struct iovec *iovec, int nr_iovs) +{ + unsigned long folio_size = (1 << imu->folio_shift); + unsigned long folio_mask = folio_size - 1; + struct bio_vec *res_bvec; + size_t total_len = 0; + int max_segs = 0; + int bvec_idx = 0; + int iov_idx; + + if (WARN_ON_ONCE(!imu)) + return ERR_PTR(-EFAULT); + + for (iov_idx = 0; iov_idx < nr_iovs; iov_idx++) { + size_t iov_len = iovec[iov_idx].iov_len; + u64 buf_addr = (u64)iovec[iov_idx].iov_base; + u64 buf_end; + + if (unlikely(check_add_overflow(buf_addr, (u64)iov_len, &buf_end))) + return ERR_PTR(-EFAULT); + /* not inside the mapped region */ + if (unlikely(buf_addr < imu->ubuf || buf_end > (imu->ubuf + imu->len))) + return ERR_PTR(-EFAULT); + max_segs += (iov_len >> imu->folio_shift) + 2; + } + + res_bvec = kmalloc_array(max_segs, sizeof(*res_bvec), GFP_KERNEL); + if (!res_bvec) + return ERR_PTR(-ENOMEM); + + for (iov_idx = 0; iov_idx < nr_iovs; iov_idx++) { + size_t iov_len = iovec[iov_idx].iov_len; + u64 buf_addr = (u64)iovec[iov_idx].iov_base; + u64 folio_addr = imu->ubuf & ~folio_mask; + struct bio_vec *src_bvec; + size_t offset; + + total_len += iov_len; + /* by using folio address it also accounts for bvec offset */ + offset = buf_addr - folio_addr; + src_bvec = imu->bvec + (offset >> imu->folio_shift); + offset &= folio_mask; + + for (; iov_len; offset = 0, bvec_idx++, src_bvec++) { + size_t seg_size = min_t(size_t, iov_len, + folio_size - offset); + + res_bvec[bvec_idx].bv_page = src_bvec->bv_page; + res_bvec[bvec_idx].bv_offset = offset; + res_bvec[bvec_idx].bv_len = seg_size; + iov_len -= seg_size; + } + } + + iov_iter_bvec(iter, ddir, res_bvec, bvec_idx, total_len); + return res_bvec; +} + int io_import_fixed(int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, u64 buf_addr, size_t len) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 8ed588036210..675161cf8b92 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -66,6 +66,9 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc); int io_import_fixed(int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, u64 buf_addr, size_t len); +struct bio_vec *io_import_fixed_vec(int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + struct iovec *iovec, int nr_iovs); int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg); void __io_sqe_buffers_unregister(struct io_ring_ctx *ctx); From patchwork Wed Oct 23 02:38:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13846332 Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE291487A7 for ; Wed, 23 Oct 2024 02:37:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651072; cv=none; b=QaX5OcODgTM2Q7Ksub0fa4Xqw4r+3tbc6hSAA9BaWog8RdWSHzOEyMh+KKMTvq275SOp8I1dbsHB1J/P7Yy+SN3ly12e2VLfeBtY9W7HSn1x1QAbAeB4j9693dtqb0NjoHlxQYWM8A+w9ZR9ChhJMTCjsRujMyy1ef/41VXbDkk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729651072; c=relaxed/simple; bh=TxZJssuzzLpXjABpyyiaqo/BqMUvUtqrO4xYLcFoukw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ErA0ae9JqGeD17I3nWITaNY6usjPRJvMGfJmekwA9hsrAuDdsF0orMF3CrjjPZsrfnypdi3iodODfvO3sZzJ1ZUpMEcFT1ViLnNxQM909q3xJ1t60/sFrgbOEilnVvq+qLOaLyCpCdRpJBf5XyEJcDR6/Hm/6F0bAR5a99mso9s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MqqoQ6wO; arc=none smtp.client-ip=209.85.208.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MqqoQ6wO" Received: by mail-ed1-f47.google.com with SMTP id 4fb4d7f45d1cf-5c903f5bd0eso3366309a12.3 for ; Tue, 22 Oct 2024 19:37:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729651068; x=1730255868; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lC8EWOJ/M38xO5z8xunAoNX/+lyqVLN/bqiRMncMjMI=; b=MqqoQ6wOK35KJz5qdXTFwZMQeInGdjUoLhrdH0JzkCYtmWbwJlXdWNACCNVcWV8Y6T OKYYDPWFzKNv0j2yTdr63NFg5rlSvk1tgBA1zvcDzsC3tyr2TF6vAB0s2gO6B6OzEOzx fFIA8WyxjTP7hou8PxP5giWAfEL1iixnapPadNKD7LO4jPAX21v1/XWgQB+SR6ejVUju lQU2K9wDJptrWXmj0YYiSJYQtd3ejpCVve4FGmIJ2Kr9YGVgxkQynI9vqhxYQNKBNuLX GtFqE4G6a1dT3zvi6nd5ZPcC1UUN1sDwpAMB29Z7Hu+bZMjAs/8wOWPcRUopsIXDIcBw cpHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729651068; x=1730255868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lC8EWOJ/M38xO5z8xunAoNX/+lyqVLN/bqiRMncMjMI=; b=s0vIEXw+WKfGQhgov8fdAQq2hvUQFZ7FVn60FsId+HnNlUZUDlO70UYMhl++EjOm9f RqhLFVxh37Szng6vlYlzKxBEB5c0ikT/2kfS1AqdVdqIfbomhnJiVR3bAWngNoex0Yd3 P8paQUzE8ygY5WEmcZ/1H3AuAeJO7Ohmti66M+t5+Ht4Ly7aaPxjibbUU6LUwf4MlS1D x6gCiChXP1g6ojzTRTTed+osh08fPxkOFvhCTzmQcI3XjLRBqPXlwyjpe3gY9CPCyh3K ApBNhuNocGxPOd7OuceEmcCfNQCr9aDvyCwab5cFm36k0paN95T89NLui2bZnGZSkkwn XblQ== X-Gm-Message-State: AOJu0Yyb4DZ6Po4h/mXSV3y6sQDaPWUyJhom/HveSzH31Aee+uNCgFaO fUvZ0b1y3iuGNqtsauhN7rMdMGxf6K2nEuCL9gYfx8nCkkCxLAhkdwz7dQ== X-Google-Smtp-Source: AGHT+IELJDdeBoLDtN4SioPM//CuShks/3lPMxdQkVX/kOIe6DwgXZ5k9WN5ByE4zpUUwHg5fVyOiQ== X-Received: by 2002:a17:907:7290:b0:a9a:478:2ee3 with SMTP id a640c23a62f3a-a9abf92f3b4mr84937666b.40.1729651068523; Tue, 22 Oct 2024 19:37:48 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.112]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a9a91371046sm410418766b.139.2024.10.22.19.37.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2024 19:37:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 4/4] io_uring/net: sendzc with vectored fixed buffers Date: Wed, 23 Oct 2024 03:38:21 +0100 Message-ID: X-Mailer: git-send-email 2.46.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently, we can use registered buffers with send zerocopy but not sendmsg. However, users want to use it with zc sendmsg as well, and pass a scatter list into a registered buffer. Implement a vectored registered buffer support for sendmsg zerocopy. The ABI should be intuitive. The user should set sqe->buf_index to the desired registered buffer and also and pass IORING_RECVSEND_FIXED_BUF flag. msghdr should still point to an iovec with user addresses served to calculate offsets how it has always been with registered buffers. In other words, in most cases the user passes the same iovec it'd pass to the non-registered buffer version. It's the first step and requires some more work cleaning the infrastructure. It'll also need some imporvement on the bvec caching side. Note, we can easily enable it for non zc version, and even extend the feature to read/write requests. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 47 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 45 insertions(+), 2 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index bc062b5a7a55..6a19b6a7dc06 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -204,6 +204,18 @@ static int io_net_vec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg, return 0; } +static void io_net_bvec_assign(struct io_kiocb *req, struct io_async_msghdr *kmsg, + struct bio_vec *bvec, int max_segs) +{ + if (bvec) { + req->flags |= REQ_F_NEED_CLEANUP; + if (kmsg->free_vec) + kfree(kmsg->free_vec); + kmsg->free_vec_bytes = max_segs * sizeof(*bvec); + kmsg->free_vec = bvec; + } +} + static inline void io_mshot_prep_retry(struct io_kiocb *req, struct io_async_msghdr *kmsg) { @@ -267,6 +279,31 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, } #endif +static int io_send_setup_sg_fixed(struct io_kiocb *req, struct iovec *iovec, + int nr_iovs, int ddir) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *iomsg = req->async_data; + struct iov_iter *iter = &iomsg->msg.msg_iter; + struct io_ring_ctx *ctx = req->ctx; + struct io_mapped_ubuf *imu; + struct bio_vec *bvec; + int idx; + + if (unlikely(sr->buf_index >= ctx->nr_user_bufs)) + return -EFAULT; + idx = array_index_nospec(sr->buf_index, ctx->nr_user_bufs); + imu = READ_ONCE(ctx->user_bufs[idx]); + io_req_set_rsrc_node(sr->notif, ctx, 0); + + bvec = io_import_fixed_vec(ddir, iter, imu, iovec, nr_iovs); + if (unlikely(IS_ERR(bvec))) + return PTR_ERR(bvec); + + io_net_bvec_assign(req, iomsg, bvec, iter->nr_segs); + return 0; +} + static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct user_msghdr *msg, int ddir) { @@ -413,6 +450,14 @@ static int io_sendmsg_setup(struct io_kiocb *req, const struct io_uring_sqe *sqe ret = io_sendmsg_copy_hdr(req, kmsg); if (!ret) req->flags |= REQ_F_NEED_CLEANUP; + + if (sr->flags & IORING_RECVSEND_FIXED_BUF) { + struct iovec *iov; + + iov = kmsg->free_vec ? kmsg->free_vec : &kmsg->fast_iov; + return io_send_setup_sg_fixed(req, iov, + kmsg->msg.msg_iter.nr_segs, ITER_SOURCE); + } return ret; } @@ -1270,8 +1315,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->opcode != IORING_OP_SEND_ZC) { if (unlikely(sqe->addr2 || sqe->file_index)) return -EINVAL; - if (unlikely(zc->flags & IORING_RECVSEND_FIXED_BUF)) - return -EINVAL; } zc->len = READ_ONCE(sqe->len);