From patchwork Wed Mar 20 01:17:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597165 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 682797489 for ; Wed, 20 Mar 2024 01:22:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897781; cv=none; b=ma1NbhewZHA5bX+iSoVRUg1+mu4rkrGa+jzC4DBMSh8NEssTEMkyvw8lVa1lE7srRTP56/IWyOrXoKyiTo9Le1oTbWOoT5hFSZZv5MG2EUv9NoQoLDDbAEQ6lW0F14JGtlLxVaUlRFR3fwTZJGJxvCMw3zjZuev7OSQbXY/eWuc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897781; c=relaxed/simple; bh=7vMskN9crckJXLTbyOoQ5AdBSdpzGmEiV9oSGPK7AoQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AN1zT31rRpdMessxVzsMktXhZombpaD4E01dZYDthfnuQ3Qj1bmoLrWu5oHbGENl+ESLHCt5CfHOE3BOHnOd0q7UAs837iM6ao8TNK5wZiGFqAYTUZHWY/eUu3IP46UIxPKyAQBwht0+sus0KzmAEOPgwDI6ECnHaiiUndx+Ouk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=RxSc9OB/; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="RxSc9OB/" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-29c572d4b84so2010482a91.1 for ; Tue, 19 Mar 2024 18:22:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897777; x=1711502577; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UxL9SCcDmri6r5vo+Fn+IFDVaE2HKUZtNpPNcBwdPK4=; b=RxSc9OB/qwh6V6rNSWc+lSxa5w0WXJNgveWIVhfd4/pWyhhytmjsXVS9nH6BVuF7pS DJK8sEmoKSoC1+t4ra253lph9ny/XU8Uv63CHiB884q/hz0f4AZRF6J0afXFIVrzWsxG TI2zE1zmzRyvVz5BFstFW3psUFc++1DejMD7mGXPHEeuoeZxrwSRYmn89FIfsApL15J6 EGfbK0B6i88XZodXYW4C8cirLPBn2xPy+QdvLlIxlHLsRWYeZCMmNMHTLzDLih8tdQ02 LD5BBWANVqsnF+XLbR2vzrIKaaxgMWnPgKpXvmzi2agstU1xaCsDLLUWOXVAjaaFwo05 ylVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897777; x=1711502577; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UxL9SCcDmri6r5vo+Fn+IFDVaE2HKUZtNpPNcBwdPK4=; b=mpZbYcwk3GoCWuQ+uzIXAFmZyxIqaK9Og3E1hYlz7jZeetIPwu3e4Deo9n/ojDk175 zjxL/GNznFdmS+ZC31gRfzlJpTDpm6xEStJ1b2Vt5mS3CNn7qBSqinwcLzs5FlnIQDX0 8dO8bX5YOuXNgIhZ+mCZCzRljXxOkQx99O8GDE2flj71BgA5u7dpAuZ64CRUtV1/ohfC N0XaVmcSxbmebDCWsSLJtSUxoAr9uD7OWWN1/+AbXSCqrI97vudmVBcb0Ecw4+wJ7/X1 hCxw31OChn4pYvikNcYDrlYbXPu0yLkcRraljhGe9p9YUHwe1lsUlh15sLlI7vdLHpi+ QV1g== X-Gm-Message-State: AOJu0Yw9mzcCjWvmrdxwCzmLIuvxN7rvTz+fxMMgoLYV6Q4oqFAePKdG KsSP5cCJk4MOZurCuLfs/PgZbMOFRWbms1cWTBkTWgmoDtaMVxm40RyRaf6xSTcBftVaH6H6taM g X-Google-Smtp-Source: AGHT+IFk82h/TvNkThEhppFtuBH/+7EM97pBHvp/qGIB8GUHn5yUdbOwOqKD+I/6tBXTUyx+fsZHfg== X-Received: by 2002:a05:6a21:32a7:b0:1a3:622b:c405 with SMTP id yt39-20020a056a2132a700b001a3622bc405mr4611140pzb.3.1710897777157; Tue, 19 Mar 2024 18:22:57 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.22.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:22:55 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 01/15] io_uring/net: switch io_send() and io_send_zc() to using io_async_msghdr Date: Tue, 19 Mar 2024 19:17:29 -0600 Message-ID: <20240320012251.1120361-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 No functional changes in this patch, just in preparation for carrying more state then we have now, if necessary. While unifying some of this code, add a generic send setup prep handler that they can both use. This gets rid of some manual msghdr and sockaddr on the stack, and makes it look a bit more like the sendmsg/recvmsg variants. We can probably unify a bit more on top of this going forward. Signed-off-by: Jens Axboe --- io_uring/net.c | 202 ++++++++++++++++++++++++----------------------- io_uring/opdef.c | 1 + 2 files changed, 106 insertions(+), 97 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index ed798e185bbf..7912a4fb2d0b 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -322,36 +322,25 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, int io_send_prep_async(struct io_kiocb *req) { - struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr *io; int ret; if (req_has_async_data(req)) return 0; - zc->done_io = 0; - if (!zc->addr) + sr->done_io = 0; + if (!sr->addr) return 0; io = io_msg_alloc_async_prep(req); if (!io) return -ENOMEM; - ret = move_addr_to_kernel(zc->addr, zc->addr_len, &io->addr); - return ret; -} - -static int io_setup_async_addr(struct io_kiocb *req, - struct sockaddr_storage *addr_storage, - unsigned int issue_flags) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *io; - - if (!sr->addr || req_has_async_data(req)) - return -EAGAIN; - io = io_msg_alloc_async(req, issue_flags); - if (!io) - return -ENOMEM; - memcpy(&io->addr, addr_storage, sizeof(io->addr)); - return -EAGAIN; + memset(&io->msg, 0, sizeof(io->msg)); + ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &io->msg.msg_iter); + if (unlikely(ret)) + return ret; + if (sr->addr) + return move_addr_to_kernel(sr->addr, sr->addr_len, &io->addr); + return 0; } int io_sendmsg_prep_async(struct io_kiocb *req) @@ -475,45 +464,72 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -int io_send(struct io_kiocb *req, unsigned int issue_flags) +static struct io_async_msghdr *io_send_setup(struct io_kiocb *req, + struct io_async_msghdr *stack_msg, + unsigned int issue_flags) { - struct sockaddr_storage __address; struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct msghdr msg; - struct socket *sock; - unsigned flags; - int min_ret = 0; + struct io_async_msghdr *kmsg; int ret; - msg.msg_name = NULL; - msg.msg_control = NULL; - msg.msg_controllen = 0; - msg.msg_namelen = 0; - msg.msg_ubuf = NULL; - - if (sr->addr) { - if (req_has_async_data(req)) { - struct io_async_msghdr *io = req->async_data; + if (req_has_async_data(req)) { + kmsg = req->async_data; + } else { + kmsg = stack_msg; + kmsg->free_iov = NULL; - msg.msg_name = &io->addr; - } else { - ret = move_addr_to_kernel(sr->addr, sr->addr_len, &__address); + if (sr->addr) { + ret = move_addr_to_kernel(sr->addr, sr->addr_len, + &kmsg->addr); if (unlikely(ret < 0)) - return ret; - msg.msg_name = (struct sockaddr *)&__address; + return ERR_PTR(ret); + } + + if (!io_do_buffer_select(req)) { + ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, + &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ERR_PTR(ret); } - msg.msg_namelen = sr->addr_len; } + if (sr->addr) { + kmsg->msg.msg_name = &kmsg->addr; + kmsg->msg.msg_namelen = sr->addr_len; + } else { + kmsg->msg.msg_name = NULL; + kmsg->msg.msg_namelen = 0; + } + kmsg->msg.msg_control = NULL; + kmsg->msg.msg_controllen = 0; + kmsg->msg.msg_ubuf = NULL; + if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return io_setup_async_addr(req, &__address, issue_flags); + return ERR_PTR(io_setup_async_msg(req, kmsg, issue_flags)); + + return kmsg; +} + +int io_send(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr iomsg, *kmsg; + size_t len = sr->len; + struct socket *sock; + unsigned flags; + int min_ret = 0; + int ret; sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; - ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &msg.msg_iter); + kmsg = io_send_setup(req, &iomsg, issue_flags); + if (IS_ERR(kmsg)) + return PTR_ERR(kmsg); + + ret = import_ubuf(ITER_SOURCE, sr->buf, len, &kmsg->msg.msg_iter); if (unlikely(ret)) return ret; @@ -521,21 +537,19 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (issue_flags & IO_URING_F_NONBLOCK) flags |= MSG_DONTWAIT; if (flags & MSG_WAITALL) - min_ret = iov_iter_count(&msg.msg_iter); + min_ret = iov_iter_count(&kmsg->msg.msg_iter); flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; - msg.msg_flags = flags; - ret = sock_sendmsg(sock, &msg); + kmsg->msg.msg_flags = flags; + ret = sock_sendmsg(sock, &kmsg->msg); if (ret < min_ret) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_addr(req, &__address, issue_flags); + return io_setup_async_msg(req, kmsg, issue_flags); if (ret > 0 && io_net_retry(sock, flags)) { - sr->len -= ret; - sr->buf += ret; sr->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_addr(req, &__address, issue_flags); + return io_setup_async_msg(req, kmsg, issue_flags); } if (ret == -ERESTARTSYS) ret = -EINTR; @@ -545,6 +559,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; + io_req_msg_cleanup(req, kmsg, issue_flags); io_req_set_res(req, ret, 0); return IOU_OK; } @@ -1158,11 +1173,35 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, return ret; } +static int io_send_zc_import(struct io_kiocb *req, struct io_async_msghdr *kmsg) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + int ret; + + if (sr->flags & IORING_RECVSEND_FIXED_BUF) { + ret = io_import_fixed(ITER_SOURCE, &kmsg->msg.msg_iter, req->imu, + (u64)(uintptr_t)sr->buf, sr->len); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter; + } else { + io_notif_set_extended(sr->notif); + ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + ret = io_notif_account_mem(sr->notif, sr->len); + if (unlikely(ret)) + return ret; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; + } + + return ret; +} + int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) { - struct sockaddr_storage __address; struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); - struct msghdr msg; + struct io_async_msghdr iomsg, *kmsg; struct socket *sock; unsigned msg_flags; int ret, min_ret = 0; @@ -1173,67 +1212,35 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) if (!test_bit(SOCK_SUPPORT_ZC, &sock->flags)) return -EOPNOTSUPP; - msg.msg_name = NULL; - msg.msg_control = NULL; - msg.msg_controllen = 0; - msg.msg_namelen = 0; - - if (zc->addr) { - if (req_has_async_data(req)) { - struct io_async_msghdr *io = req->async_data; - - msg.msg_name = &io->addr; - } else { - ret = move_addr_to_kernel(zc->addr, zc->addr_len, &__address); - if (unlikely(ret < 0)) - return ret; - msg.msg_name = (struct sockaddr *)&__address; - } - msg.msg_namelen = zc->addr_len; - } - - if (!(req->flags & REQ_F_POLLED) && - (zc->flags & IORING_RECVSEND_POLL_FIRST)) - return io_setup_async_addr(req, &__address, issue_flags); + kmsg = io_send_setup(req, &iomsg, issue_flags); + if (IS_ERR(kmsg)) + return PTR_ERR(kmsg); - if (zc->flags & IORING_RECVSEND_FIXED_BUF) { - ret = io_import_fixed(ITER_SOURCE, &msg.msg_iter, req->imu, - (u64)(uintptr_t)zc->buf, zc->len); - if (unlikely(ret)) - return ret; - msg.sg_from_iter = io_sg_from_iter; - } else { - io_notif_set_extended(zc->notif); - ret = import_ubuf(ITER_SOURCE, zc->buf, zc->len, &msg.msg_iter); + if (!zc->done_io) { + ret = io_send_zc_import(req, kmsg); if (unlikely(ret)) return ret; - ret = io_notif_account_mem(zc->notif, zc->len); - if (unlikely(ret)) - return ret; - msg.sg_from_iter = io_sg_from_iter_iovec; } msg_flags = zc->msg_flags | MSG_ZEROCOPY; if (issue_flags & IO_URING_F_NONBLOCK) msg_flags |= MSG_DONTWAIT; if (msg_flags & MSG_WAITALL) - min_ret = iov_iter_count(&msg.msg_iter); + min_ret = iov_iter_count(&kmsg->msg.msg_iter); msg_flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; - msg.msg_flags = msg_flags; - msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; - ret = sock_sendmsg(sock, &msg); + kmsg->msg.msg_flags = msg_flags; + kmsg->msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; + ret = sock_sendmsg(sock, &kmsg->msg); if (unlikely(ret < min_ret)) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_addr(req, &__address, issue_flags); + return io_setup_async_msg(req, kmsg, issue_flags); - if (ret > 0 && io_net_retry(sock, msg.msg_flags)) { - zc->len -= ret; - zc->buf += ret; + if (ret > 0 && io_net_retry(sock, kmsg->msg.msg_flags)) { zc->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_addr(req, &__address, issue_flags); + return io_setup_async_msg(req, kmsg, issue_flags); } if (ret == -ERESTARTSYS) ret = -EINTR; @@ -1251,6 +1258,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) */ if (!(issue_flags & IO_URING_F_UNLOCKED)) { io_notif_flush(zc->notif); + io_netmsg_recycle(req, issue_flags); req->flags &= ~REQ_F_NEED_CLEANUP; } io_req_set_res(req, ret, IORING_CQE_F_MORE); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 9c080aadc5a6..b0a990c6bbff 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -602,6 +602,7 @@ const struct io_cold_def io_cold_defs[] = { .name = "SEND", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), + .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, .prep_async = io_send_prep_async, #endif From patchwork Wed Mar 20 01:17:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597166 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68881523D for ; Wed, 20 Mar 2024 01:23:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897782; cv=none; b=P5/S/Ks30nQ+bsGq9mNGTBaDzjqNRVmJ1Z3EfCqB89xexG7Ck72ZxtnTF/oos+WVljheZnrLhntGGrZy59KqWjgQUxkEJE9VI4S3noXT6IBN91CeVldixv8hNcbEYr6K4oKVE3DP4Cy6mc+MZDY3W2OmJbUkkD2+T1pUmrV/kdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897782; c=relaxed/simple; bh=+SQErtN0ghCoyRkAkCTpTxBxeXSRJLYOkbX9sdNg8Cg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=phpKG1niWysSpQ1HCPSveu8EqDKYBO1dGAHN0d1K0mkZNOqHXmZ9HTCef4ziIEd0Ng9PeiiZPx69GawE43q+hW3qMIBGlIwedWpLpLi/iBEomPpNXLbj0OGOD7OF7srssY4DDE/sedBfEbyPthBjGUuMWEjiX49hwOORSEQ3uxk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=DEKD0TP+; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="DEKD0TP+" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6e6c38be762so941715b3a.1 for ; Tue, 19 Mar 2024 18:23:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897779; x=1711502579; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m0ERWSADulJ+IP6ADJvW4gJHm8zTlL+F/+4A6drzXvo=; b=DEKD0TP+4k3PsyZXOxoWhaRvKacnLtDa3W/XArWf0ZSxfib2g6tRYUFII4AJrCpROY q+tINusqLOsWhLRkWo64mt9TT6Q5/JIEx3UCuoIfaA2XGOIq7V0642jS7nswl4Zz8P7w W5WAv/07eJtBfT43rsX10bUgbjKC5eeMOPESq0cYoMf6JUKdoFi8uoMUsk06/FPeEs4/ jn5allnv7vVJcbAnks6QVjTg3pcUAq4dtikFB1G7TnMk2CSj3kfe0U4D6L08tWx6DzBC XjtZuw2Op0y25fEn68CpiWaUBerPrG7asSULamv6lUgxIdqNq/Z5N7lnpktJgfKKXBBZ /LIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897779; x=1711502579; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m0ERWSADulJ+IP6ADJvW4gJHm8zTlL+F/+4A6drzXvo=; b=lgPNHM4ZiyWU6PpAgQwYFQq8CeL3DLCD3hUJWkiE9y8F0P4eeWDvCjeOdn0zwnVFq5 YRv/HjhQRlW/PoxGrNhB0aCfoGlDVFBChIDhcY4vPknYF8MTURLBEZJGvWJpVQYqTdAc fJWMRnoY+Y6DaSEHMOMPqY6MESpwTQJfkeiV/AAYwXIWrhwqEsyD9GAsWCdJtIEOtV5G 06jiAO3zgyF6Qxg44duTR0SEJGVWkilKn4LJ9fuotMq5PqHTZ86SejkM1rpTGzNvtkAl hluMOGYGy5EXRROsCMdGKAX1NtjltV10GTflcuVOuBt0hrGT5gNlRyIvKWkc/ciQFpIE d1oQ== X-Gm-Message-State: AOJu0Yw3N6uoKPsKqQCZuV/ajirHCsSGk3Q/t+B7dkwSxIFDPtO6jILE yFe3oEz9YXEdcjg/Ghfwr5L2RuwMQRMY3nkiVRWTq7YRpzcVDuGaoUm9GhQuNDaazV9VVk+3IFE Z X-Google-Smtp-Source: AGHT+IH0wxU1MeXp3H9GBKASvgbuec99hZxtOU+oymhAP/wWhBKBsjTyB2M5Se+XiBPlvhUHE+VGBQ== X-Received: by 2002:a05:6a00:9382:b0:6e6:864d:767 with SMTP id ka2-20020a056a00938200b006e6864d0767mr4278500pfb.3.1710897779040; Tue, 19 Mar 2024 18:22:59 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.22.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:22:57 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 02/15] io_uring/net: switch io_recv() to using io_async_msghdr Date: Tue, 19 Mar 2024 19:17:30 -0600 Message-ID: <20240320012251.1120361-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 No functional changes in this patch, just in preparation for carrying more state then we have now, if necessary. Signed-off-by: Jens Axboe --- io_uring/net.c | 77 +++++++++++++++++++++++++++++------------------- io_uring/net.h | 2 +- io_uring/opdef.c | 7 +++-- 3 files changed, 53 insertions(+), 33 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 7912a4fb2d0b..ff22f6cc859e 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -320,7 +320,7 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, return ret; } -int io_send_prep_async(struct io_kiocb *req) +int io_sendrecv_prep_async(struct io_kiocb *req) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr *io; @@ -707,13 +707,13 @@ static inline void io_recv_prep_retry(struct io_kiocb *req) * again (for multishot). */ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, - struct msghdr *msg, bool mshot_finished, - unsigned issue_flags) + struct io_async_msghdr *kmsg, + bool mshot_finished, unsigned issue_flags) { unsigned int cflags; cflags = io_put_kbuf(req, issue_flags); - if (msg->msg_inq > 0) + if (kmsg->msg.msg_inq > 0) cflags |= IORING_CQE_F_SOCK_NONEMPTY; /* @@ -727,7 +727,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, io_recv_prep_retry(req); /* Known not-empty or unknown state, retry */ - if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq < 0) { + if (cflags & IORING_CQE_F_SOCK_NONEMPTY || kmsg->msg.msg_inq < 0) { if (sr->nr_multishot_loops++ < MULTISHOT_MAX_RETRY) return false; /* mshot retries exceeded, force a requeue */ @@ -928,7 +928,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) else io_kbuf_recycle(req, issue_flags); - if (!io_recv_finish(req, &ret, &kmsg->msg, mshot_finished, issue_flags)) + if (!io_recv_finish(req, &ret, kmsg, mshot_finished, issue_flags)) goto retry_multishot; if (mshot_finished) @@ -942,29 +942,42 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) int io_recv(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct msghdr msg; + struct io_async_msghdr iomsg, *kmsg; struct socket *sock; unsigned flags; int ret, min_ret = 0; bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; size_t len = sr->len; + if (req_has_async_data(req)) { + kmsg = req->async_data; + } else { + kmsg = &iomsg; + kmsg->free_iov = NULL; + kmsg->msg.msg_name = NULL; + kmsg->msg.msg_namelen = 0; + kmsg->msg.msg_control = NULL; + kmsg->msg.msg_get_inq = 1; + kmsg->msg.msg_controllen = 0; + kmsg->msg.msg_iocb = NULL; + kmsg->msg.msg_ubuf = NULL; + + if (!io_do_buffer_select(req)) { + ret = import_ubuf(ITER_DEST, sr->buf, sr->len, + &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + } + } + if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return -EAGAIN; + return io_setup_async_msg(req, kmsg, issue_flags); sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; - msg.msg_name = NULL; - msg.msg_namelen = 0; - msg.msg_control = NULL; - msg.msg_get_inq = 1; - msg.msg_controllen = 0; - msg.msg_iocb = NULL; - msg.msg_ubuf = NULL; - flags = sr->msg_flags; if (force_nonblock) flags |= MSG_DONTWAIT; @@ -978,22 +991,23 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) return -ENOBUFS; sr->buf = buf; sr->len = len; + ret = import_ubuf(ITER_DEST, sr->buf, sr->len, + &kmsg->msg.msg_iter); + if (unlikely(ret)) + goto out_free; } - ret = import_ubuf(ITER_DEST, sr->buf, len, &msg.msg_iter); - if (unlikely(ret)) - goto out_free; - - msg.msg_inq = -1; - msg.msg_flags = 0; + kmsg->msg.msg_inq = -1; + kmsg->msg.msg_flags = 0; if (flags & MSG_WAITALL) - min_ret = iov_iter_count(&msg.msg_iter); + min_ret = iov_iter_count(&kmsg->msg.msg_iter); - ret = sock_recvmsg(sock, &msg, flags); + ret = sock_recvmsg(sock, &kmsg->msg, flags); if (ret < min_ret) { if (ret == -EAGAIN && force_nonblock) { - if (issue_flags & IO_URING_F_MULTISHOT) { + ret = io_setup_async_msg(req, kmsg, issue_flags); + if (ret == -EAGAIN && issue_flags & IO_URING_F_MULTISHOT) { io_kbuf_recycle(req, issue_flags); return IOU_ISSUE_SKIP_COMPLETE; } @@ -1001,16 +1015,14 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) return -EAGAIN; } if (ret > 0 && io_net_retry(sock, flags)) { - sr->len -= ret; - sr->buf += ret; sr->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return -EAGAIN; + return io_setup_async_msg(req, kmsg, issue_flags); } if (ret == -ERESTARTSYS) ret = -EINTR; req_set_fail(req); - } else if ((flags & MSG_WAITALL) && (msg.msg_flags & (MSG_TRUNC | MSG_CTRUNC))) { + } else if ((flags & MSG_WAITALL) && (kmsg->msg.msg_flags & (MSG_TRUNC | MSG_CTRUNC))) { out_free: req_set_fail(req); } @@ -1022,9 +1034,14 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) else io_kbuf_recycle(req, issue_flags); - if (!io_recv_finish(req, &ret, &msg, ret <= 0, issue_flags)) + if (!io_recv_finish(req, &ret, kmsg, ret <= 0, issue_flags)) goto retry_multishot; + if (ret == -EAGAIN) + return io_setup_async_msg(req, kmsg, issue_flags); + else if (ret != IOU_OK && ret != IOU_STOP_MULTISHOT) + io_req_msg_cleanup(req, kmsg, issue_flags); + return ret; } diff --git a/io_uring/net.h b/io_uring/net.h index 9d7962f65f26..281afef670a6 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -39,7 +39,7 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags); int io_send(struct io_kiocb *req, unsigned int issue_flags); -int io_send_prep_async(struct io_kiocb *req); +int io_sendrecv_prep_async(struct io_kiocb *req); int io_recvmsg_prep_async(struct io_kiocb *req); int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index b0a990c6bbff..77131826d603 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -604,13 +604,16 @@ const struct io_cold_def io_cold_defs[] = { .async_size = sizeof(struct io_async_msghdr), .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, - .prep_async = io_send_prep_async, + .prep_async = io_sendrecv_prep_async, #endif }, [IORING_OP_RECV] = { .name = "RECV", #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), + .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, + .prep_async = io_sendrecv_prep_async, #endif }, [IORING_OP_OPENAT2] = { @@ -687,7 +690,7 @@ const struct io_cold_def io_cold_defs[] = { .name = "SEND_ZC", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), - .prep_async = io_send_prep_async, + .prep_async = io_sendrecv_prep_async, .cleanup = io_send_zc_cleanup, .fail = io_sendrecv_fail, #endif From patchwork Wed Mar 20 01:17:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597167 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6D1E747F for ; Wed, 20 Mar 2024 01:23:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897784; cv=none; b=AkHKS3fCEFa54MIQDYAtXZ23F0FIpnpJ5lebJzXzEom9MTVmhuvg8ErlGDANiyfKpjR3u5p23NXh3so1LPTR7xLVvFrlRikrSspcbVBLpFIPFplrtxHogkMLB6BmW1+91KeVHmKRmDIBUEkX+mVeSAFUEpCqXauBCdu8qca4HGw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897784; c=relaxed/simple; bh=3t3JhBqwsUniiYPwfPjuNoMmoS8+rDgD/SutkzOPmQo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kkkZPUSJf2CAnnfetc+HLwp5IDF8H67IbxosZ3wtgaGFpnS/JjYXmJj1gyD4OIw15di1HDeUSoF+HXg2CdSymyE8ImfSyz5tMPhs/T9O4WWmNEQjrEct9AOh5U5+/VaxCsDn7qr0AhqS1ZVc8WgvafxIysUlSajSXeAYwPe1pHE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=meDzAHuj; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="meDzAHuj" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-29c572d4b84so2010495a91.1 for ; Tue, 19 Mar 2024 18:23:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897781; x=1711502581; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e10G5Hj/0YjRZ1wjbOIQLq1rOF/sLBwi2jZonL4N32Y=; b=meDzAHuj8rZkNqLjtKvMBMRBXJ7eAr2IuIKptRpoTHloGPqzEvQJiMonje09TV+veQ FYPi4kPuoabHXOPB4zBrzR3FHDjrv/ayGQzxhVwI7EOko2A5IS/bmCY1i0bs8KMgyRj2 VCJiWLo+29rUT84nQX8mpPTs4xfLzW12UBsYxFchx5aCX77EaHZsGC5cd+TNDipjjVYa ViFweTy6YlwLyota/SqgIIiYqVyHpaCfBGNGdYZUa04fCoxf0F5fbcDLVaRqXuuDYqjn 2O6qwIZ+AoMM7XyAjyRF63WAX6H8mxkJT5nNTvrBOFtoOiyXOberRjBk+QWr0dgzkZb7 WP0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897781; x=1711502581; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e10G5Hj/0YjRZ1wjbOIQLq1rOF/sLBwi2jZonL4N32Y=; b=u31v+VLUqGO2/z9O+8s90bMYOIIAtRk99+CUEGRSY1KM14OhTgqYfAK0oYiHiBrovQ pBQhZD8auwsETxorKWiupTX/mQsnhgdMTDU5C5IRRM9hj7HwXjlzcf70c7y062n7ROQE GFNTv5suh+s6AgMpZnGEjrNYdxWempPZrUfVvk1NEJrGmjgDCNriqcyi3ksh9UC3D/NX pvLoNOjE1NsZW+2SnIk976CIFQcF9F+Wmmccln8Z9vjerDR0ZiUBiLzfL44AfucP48+D 5xXesP0VxTXnoTwBkEUcyW/tI6dilMx6m50/S34UX35PHtqIvD4+7zliQJV6rauprRUm /jlg== X-Gm-Message-State: AOJu0YyNOqeQjCYgWWNwVLx+S9FaO+K9lcxs71mREIk738eoADLiA4ye /vnuFd83Prm31UrGe6jeCrg9BJfB25LoIotvsJCnJ80ahWQ70rFJnmdCrRoL/xMyhSi3uQtPMFT 4 X-Google-Smtp-Source: AGHT+IGAQi/8nmIjvauLwK271B0tDlm+Z89ao4GPvs2Ay+RXYaZ9dDScACS/5Yhziz997XdRNePcEA== X-Received: by 2002:a05:6a00:731c:b0:6e7:7e2a:839e with SMTP id lq28-20020a056a00731c00b006e77e2a839emr1929153pfb.2.1710897781592; Tue, 19 Mar 2024 18:23:01 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.22.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:22:59 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 03/15] io_uring/net: unify cleanup handling Date: Tue, 19 Mar 2024 19:17:31 -0600 Message-ID: <20240320012251.1120361-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Now that recv/recvmsg both do the same cleanup, put it in the retry and finish handlers. Signed-off-by: Jens Axboe --- io_uring/net.c | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index ff22f6cc859e..2389bb1cc050 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -690,10 +690,16 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } -static inline void io_recv_prep_retry(struct io_kiocb *req) +static inline void io_recv_prep_retry(struct io_kiocb *req, + struct io_async_msghdr *kmsg) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + if (kmsg->free_iov) { + kfree(kmsg->free_iov); + kmsg->free_iov = NULL; + } + req->flags &= ~REQ_F_BL_EMPTY; sr->done_io = 0; sr->len = 0; /* get from the provided buffer */ @@ -725,7 +731,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; - io_recv_prep_retry(req); + io_recv_prep_retry(req, kmsg); /* Known not-empty or unknown state, retry */ if (cflags & IORING_CQE_F_SOCK_NONEMPTY || kmsg->msg.msg_inq < 0) { if (sr->nr_multishot_loops++ < MULTISHOT_MAX_RETRY) @@ -734,10 +740,9 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, sr->nr_multishot_loops = 0; mshot_retry_ret = IOU_REQUEUE; } - if (issue_flags & IO_URING_F_MULTISHOT) + *ret = io_setup_async_msg(req, kmsg, issue_flags); + if (*ret == -EAGAIN && issue_flags & IO_URING_F_MULTISHOT) *ret = mshot_retry_ret; - else - *ret = -EAGAIN; return true; } @@ -748,6 +753,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, *ret = IOU_STOP_MULTISHOT; else *ret = IOU_OK; + io_req_msg_cleanup(req, kmsg, issue_flags); return true; } @@ -931,11 +937,6 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (!io_recv_finish(req, &ret, kmsg, mshot_finished, issue_flags)) goto retry_multishot; - if (mshot_finished) - io_req_msg_cleanup(req, kmsg, issue_flags); - else if (ret == -EAGAIN) - return io_setup_async_msg(req, kmsg, issue_flags); - return ret; } @@ -1037,11 +1038,6 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) if (!io_recv_finish(req, &ret, kmsg, ret <= 0, issue_flags)) goto retry_multishot; - if (ret == -EAGAIN) - return io_setup_async_msg(req, kmsg, issue_flags); - else if (ret != IOU_OK && ret != IOU_STOP_MULTISHOT) - io_req_msg_cleanup(req, kmsg, issue_flags); - return ret; } From patchwork Wed Mar 20 01:17:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597168 Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2911779CB for ; Wed, 20 Mar 2024 01:23:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897787; cv=none; b=Q8qnOSrEHqNHy1oRhqCzzYmsTv5dCrs1p0L1bA0U0Cqh7UJL4JQTC3ceJGB5WXw8SCsQSgokn9YKKavbuzMLHtSlB2zmDDYluFx/ucX1xKGfI6nQluol4n6eMsUyWEjk6MVLz4OXCF07tYYtaOWR2E3EOU+LfMbhzKMng7gBzW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897787; c=relaxed/simple; bh=tivHDBEL+WtYymt0cZGSjZSqi4yCY7Ied12bwk+0D2c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=k7bp8XMQtBLy6YlpmID+Zd0O4Bfns3wTbEConRbssQgfKFVYL8ZOk/ZmfrODETrSDTC4+69A/x8h+bMHq+C7gg9QfKAXGk3pTHGq234Y81u5bnjYuc+oda83Calz3DZYS6s9jcjKByC7keWpTtoZaIT5dHbiA92qYpmiV/ZBa6Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Ra5t1b3N; arc=none smtp.client-ip=209.85.210.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Ra5t1b3N" Received: by mail-ot1-f52.google.com with SMTP id 46e09a7af769-6e677ec9508so776624a34.1 for ; Tue, 19 Mar 2024 18:23:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897784; x=1711502584; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PVIzvL86EshOfsNRSGCowXS0RPvzEb/Uc/uFMq6Rfiw=; b=Ra5t1b3Ne4Qf8jQskrJZ4lrkoybwARc6uVxm4fAaZVjsiYr1vdy8cpRo2XUurvupnL XramJhvkPCJ6S3oE+yYtqbnppyMf+NQjyyH/GzyamkgNWkMXbkklZFyes1OQk7xZxASE OfwPZz/PQGa4X8Dg4ADXrdMglLBnrMGNdObL37z0oHfTbU1rkkesrGguGUSlUSsma8ZH 8KREe/YQCcAHyGfveJL0c6PLnRDu7/ctcvhrSWAg66qIZI30tC0gV/ETCXbYOBQhz7VV ukE1adw4Hmykru/ECJi9DdlHUgKPuQus2z5Jy/c0xX+Dy8b0sXWjlPuicbWlhWQ/mT1f RACw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897784; x=1711502584; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PVIzvL86EshOfsNRSGCowXS0RPvzEb/Uc/uFMq6Rfiw=; b=KoQ2pQxKTaN5OBVS3E0IWd5ce7RbTSTTMnd6a9+ahHQmJMS2gyXhfTHHMX8DyHTwnY dai8GKA82tDPUTm/w/kjGhn9IjsUEtHhpjFjtX3RsP3cXmSjPQBZv8U8CnED4ZriqfLX OvFLM/yODloNMeZY1fdaLf9wyMBiyMyWd95QZE/rwdZvcMNyuP1b1126S6NZyrFy6uti BSPtpRR9n3AqTYSzkGuJY63F8WUmZWPbmmqLI+tNRNb7dGl0t0hRciepMK+X0zWpkcXJ rBYEuKWaAWYm8H6/naxfoj8S3gmFl352xeMpJZNAHDbPk0QVf8/PVnwC/tc0wd3H0Ub/ CIXg== X-Gm-Message-State: AOJu0Yw6NAfn3iY2PAC6KU0G0/1kI0jf+K43RF+L0M3hp/HT/otcP5GS 4doVoKwtq576HWr+0p+4uZTV7c4fMHIC00wxJHjPZFWQV0+/xjP6IjQ4czrv0RauVSLS9DmTLe3 X X-Google-Smtp-Source: AGHT+IHPcTWabU47Un5Tvlb/fjYvkMt/T083TkXvvRFMcqIPY2rC8vAM966fg32UvHqTuuhDpFPixQ== X-Received: by 2002:a05:6358:7211:b0:17f:1d24:1432 with SMTP id h17-20020a056358721100b0017f1d241432mr843751rwa.3.1710897783573; Tue, 19 Mar 2024 18:23:03 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:02 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 04/15] io_uring/net: always setup an io_async_msghdr Date: Tue, 19 Mar 2024 19:17:32 -0600 Message-ID: <20240320012251.1120361-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than use an on-stack one and then need to allocate and copy if we have to go async, always grab one upfront. This should be very cheap, and potentially even have cache hotness benefits for back-to-back send/recv requests. For any recv type of request, this is probably a good choice in general, as it's expected that no data is available initially. For send this is not necessarily the case, as we expect space to be available. However, getting a cached io_async_msghdr is very cheap, and as it should be cache hot, probably the difference here is neglible, if any. A nice side benefit is that we can kill io_setup_async_msg completely, which has some nasty iovec manipulation code. Signed-off-by: Jens Axboe --- io_uring/net.c | 117 ++++++++++++++++++++----------------------------- 1 file changed, 47 insertions(+), 70 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 2389bb1cc050..776ebfea8742 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -161,36 +161,6 @@ static inline struct io_async_msghdr *io_msg_alloc_async_prep(struct io_kiocb *r return io_msg_alloc_async(req, 0); } -static int io_setup_async_msg(struct io_kiocb *req, - struct io_async_msghdr *kmsg, - unsigned int issue_flags) -{ - struct io_async_msghdr *async_msg; - - if (req_has_async_data(req)) - return -EAGAIN; - async_msg = io_msg_alloc_async(req, issue_flags); - if (!async_msg) { - kfree(kmsg->free_iov); - return -ENOMEM; - } - req->flags |= REQ_F_NEED_CLEANUP; - memcpy(async_msg, kmsg, sizeof(*kmsg)); - if (async_msg->msg.msg_name) - async_msg->msg.msg_name = &async_msg->addr; - - if ((req->flags & REQ_F_BUFFER_SELECT) && !async_msg->msg.msg_iter.nr_segs) - return -EAGAIN; - - /* if were using fast_iov, set it to the new one */ - if (iter_is_iovec(&kmsg->msg.msg_iter) && !kmsg->free_iov) { - size_t fast_idx = iter_iov(&kmsg->msg.msg_iter) - kmsg->fast_iov; - async_msg->msg.msg_iter.__iov = &async_msg->fast_iov[fast_idx]; - } - - return -EAGAIN; -} - #ifdef CONFIG_COMPAT static int io_compat_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, @@ -409,7 +379,7 @@ static void io_req_msg_cleanup(struct io_kiocb *req, int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr iomsg, *kmsg; + struct io_async_msghdr *kmsg; struct socket *sock; unsigned flags; int min_ret = 0; @@ -423,15 +393,17 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) kmsg = req->async_data; kmsg->msg.msg_control_user = sr->msg_control; } else { - ret = io_sendmsg_copy_hdr(req, &iomsg); + kmsg = io_msg_alloc_async(req, issue_flags); + if (unlikely(!kmsg)) + return -ENOMEM; + ret = io_sendmsg_copy_hdr(req, kmsg); if (ret) return ret; - kmsg = &iomsg; } if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; flags = sr->msg_flags; if (issue_flags & IO_URING_F_NONBLOCK) @@ -443,13 +415,13 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) if (ret < min_ret) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; if (ret > 0 && io_net_retry(sock, flags)) { kmsg->msg.msg_controllen = 0; kmsg->msg.msg_control = NULL; sr->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; } if (ret == -ERESTARTSYS) ret = -EINTR; @@ -465,7 +437,6 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) } static struct io_async_msghdr *io_send_setup(struct io_kiocb *req, - struct io_async_msghdr *stack_msg, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -475,8 +446,9 @@ static struct io_async_msghdr *io_send_setup(struct io_kiocb *req, if (req_has_async_data(req)) { kmsg = req->async_data; } else { - kmsg = stack_msg; - kmsg->free_iov = NULL; + kmsg = io_msg_alloc_async(req, issue_flags); + if (!kmsg) + return ERR_PTR(-ENOMEM); if (sr->addr) { ret = move_addr_to_kernel(sr->addr, sr->addr_len, @@ -506,7 +478,7 @@ static struct io_async_msghdr *io_send_setup(struct io_kiocb *req, if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return ERR_PTR(io_setup_async_msg(req, kmsg, issue_flags)); + return ERR_PTR(-EAGAIN); return kmsg; } @@ -514,7 +486,7 @@ static struct io_async_msghdr *io_send_setup(struct io_kiocb *req, int io_send(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr iomsg, *kmsg; + struct io_async_msghdr *kmsg; size_t len = sr->len; struct socket *sock; unsigned flags; @@ -525,7 +497,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!sock)) return -ENOTSOCK; - kmsg = io_send_setup(req, &iomsg, issue_flags); + kmsg = io_send_setup(req, issue_flags); if (IS_ERR(kmsg)) return PTR_ERR(kmsg); @@ -544,12 +516,12 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret = sock_sendmsg(sock, &kmsg->msg); if (ret < min_ret) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; } if (ret == -ERESTARTSYS) ret = -EINTR; @@ -740,9 +712,10 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, sr->nr_multishot_loops = 0; mshot_retry_ret = IOU_REQUEUE; } - *ret = io_setup_async_msg(req, kmsg, issue_flags); - if (*ret == -EAGAIN && issue_flags & IO_URING_F_MULTISHOT) + if (issue_flags & IO_URING_F_MULTISHOT) *ret = mshot_retry_ret; + else + *ret = -EAGAIN; return true; } @@ -844,7 +817,7 @@ static int io_recvmsg_multishot(struct socket *sock, struct io_sr_msg *io, int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr iomsg, *kmsg; + struct io_async_msghdr *kmsg; struct socket *sock; unsigned flags; int ret, min_ret = 0; @@ -858,15 +831,17 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (req_has_async_data(req)) { kmsg = req->async_data; } else { - ret = io_recvmsg_copy_hdr(req, &iomsg); + kmsg = io_msg_alloc_async(req, issue_flags); + if (unlikely(!kmsg)) + return -ENOMEM; + ret = io_recvmsg_copy_hdr(req, kmsg); if (ret) return ret; - kmsg = &iomsg; } if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; flags = sr->msg_flags; if (force_nonblock) @@ -908,17 +883,16 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (ret < min_ret) { if (ret == -EAGAIN && force_nonblock) { - ret = io_setup_async_msg(req, kmsg, issue_flags); - if (ret == -EAGAIN && (issue_flags & IO_URING_F_MULTISHOT)) { + if (issue_flags & IO_URING_F_MULTISHOT) { io_kbuf_recycle(req, issue_flags); return IOU_ISSUE_SKIP_COMPLETE; } - return ret; + return -EAGAIN; } if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; } if (ret == -ERESTARTSYS) ret = -EINTR; @@ -943,7 +917,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) int io_recv(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr iomsg, *kmsg; + struct io_async_msghdr *kmsg; struct socket *sock; unsigned flags; int ret, min_ret = 0; @@ -953,7 +927,9 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) if (req_has_async_data(req)) { kmsg = req->async_data; } else { - kmsg = &iomsg; + kmsg = io_msg_alloc_async(req, issue_flags); + if (unlikely(!kmsg)) + return -ENOMEM; kmsg->free_iov = NULL; kmsg->msg.msg_name = NULL; kmsg->msg.msg_namelen = 0; @@ -973,7 +949,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; sock = sock_from_file(req->file); if (unlikely(!sock)) @@ -1007,8 +983,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) ret = sock_recvmsg(sock, &kmsg->msg, flags); if (ret < min_ret) { if (ret == -EAGAIN && force_nonblock) { - ret = io_setup_async_msg(req, kmsg, issue_flags); - if (ret == -EAGAIN && issue_flags & IO_URING_F_MULTISHOT) { + if (issue_flags & IO_URING_F_MULTISHOT) { io_kbuf_recycle(req, issue_flags); return IOU_ISSUE_SKIP_COMPLETE; } @@ -1018,7 +993,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; } if (ret == -ERESTARTSYS) ret = -EINTR; @@ -1214,7 +1189,7 @@ static int io_send_zc_import(struct io_kiocb *req, struct io_async_msghdr *kmsg) int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr iomsg, *kmsg; + struct io_async_msghdr *kmsg; struct socket *sock; unsigned msg_flags; int ret, min_ret = 0; @@ -1225,7 +1200,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) if (!test_bit(SOCK_SUPPORT_ZC, &sock->flags)) return -EOPNOTSUPP; - kmsg = io_send_setup(req, &iomsg, issue_flags); + kmsg = io_send_setup(req, issue_flags); if (IS_ERR(kmsg)) return PTR_ERR(kmsg); @@ -1248,12 +1223,12 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret < min_ret)) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; if (ret > 0 && io_net_retry(sock, kmsg->msg.msg_flags)) { zc->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; } if (ret == -ERESTARTSYS) ret = -EINTR; @@ -1281,7 +1256,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr iomsg, *kmsg; + struct io_async_msghdr *kmsg; struct socket *sock; unsigned flags; int ret, min_ret = 0; @@ -1297,15 +1272,17 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) if (req_has_async_data(req)) { kmsg = req->async_data; } else { - ret = io_sendmsg_copy_hdr(req, &iomsg); + kmsg = io_msg_alloc_async(req, issue_flags); + if (unlikely(!kmsg)) + return -ENOMEM; + ret = io_sendmsg_copy_hdr(req, kmsg); if (ret) return ret; - kmsg = &iomsg; } if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; flags = sr->msg_flags | MSG_ZEROCOPY; if (issue_flags & IO_URING_F_NONBLOCK) @@ -1319,12 +1296,12 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret < min_ret)) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; req->flags |= REQ_F_BL_NO_RECYCLE; - return io_setup_async_msg(req, kmsg, issue_flags); + return -EAGAIN; } if (ret == -ERESTARTSYS) ret = -EINTR; From patchwork Wed Mar 20 01:17:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597169 Received: from mail-oo1-f51.google.com (mail-oo1-f51.google.com [209.85.161.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F27B96AB9 for ; Wed, 20 Mar 2024 01:23:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897789; cv=none; b=smhIoASikonefZe3tUwSeqwWMAf/RjpTUgZPHH4PZI/MMD4I5xz4bkZEQjnwcYOKHnmajmWjgfEzXZ+r0GjvA7ESZtEqOL5uacZLUA2xMzo3ZQUr86YLDPsJhzVnalXDPFBW/qR6hxevWoxONcuYbMdznL3jhOlI9Ey8eVTEcZo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897789; c=relaxed/simple; bh=kxp/f1iNAZs2D1AJZXzi1lJ0/2cNaq9CLp8NEOxtxIk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Azwmejf53uVcvaRqHYDJwgkcm4H0f7a3LaG9u5QRoiLRTZC/4KvRdNG5bSbq5qlLMuUUmMVXy3jTb7MAO9tfosn9tledQ/O+uPb/AUaQ9hvuvlXEZPxns1839mGIi5rp6KE9TZXo6wwka8raq3iCtpjdGC7SWSn1XR6WmFrnOvU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=wHRwsol8; arc=none smtp.client-ip=209.85.161.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="wHRwsol8" Received: by mail-oo1-f51.google.com with SMTP id 006d021491bc7-5a48320f0f3so250384eaf.1 for ; Tue, 19 Mar 2024 18:23:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897786; x=1711502586; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jh76wYFgOXm2vo5NRb3e0I5NUVtMiDvkSEJhtoXmii8=; b=wHRwsol8BkHk2eoLbR+SYvI6B6afo7YMRrPixZTOH9MLI9AthiAFHvq5/GksVzzcD4 v1k3BpxRvfOMkKgykdiLfhq74gvnuxREZBo2RRlrNbN0tYUbYLqn/3C/YuGKhdvxJhw6 ecSqBdUOP9H+KpRl5UeRtuM90NmCMOWdnr9sgqnd5IEkmPwZrULC/K56+Ne4gkhI3lnm pZgrA9astc/teV+PwHZvUHsG+HNh5/Z1a8aJEAnrKO9nekqpMj7BgYy6lPEbwEhIAZLX +IrWZcgAh6X9+aHrAdzTFgC7OeW5adF3lyFkjRh3EEkQJPLeZu8wOjbTsanuIVbKlOnQ nyBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897786; x=1711502586; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jh76wYFgOXm2vo5NRb3e0I5NUVtMiDvkSEJhtoXmii8=; b=fPMoAaM+eAf05usTMncQ1bleed6pySEMWBwjXgryazCM2IuSday88IXgLN1ddMSC0q syXA3PYkoSNsfsg9YBkAJnnQNzd7OWMBckSpBQ9k4FhnihqgsXfzAFhlFrDoPraYuvDr MGefXFk0R5o8c7EHTsPSPOcbDygheSKCCUQoEKGRWFHyf67CjvAjRD67tEZ/WXQQ79uw YgKHahEmOh8jdtVxIILwZ/uCm5tHQn17zWBN4cuDxhcNQQyqIV6VFg2TgFIqtVuo1kGT UOvTRhUCjJ13VfemfHlCObw4e+/tkzho/ld+OS/kmnhuOsZKtNiRoDmY2G3FtJBjZTy6 CBAA== X-Gm-Message-State: AOJu0YwHZ/moua1ZyJmB4d3nPU122wFoqDJ/SQ9Zu5IlhVbSdE7ziPKj UZpqa1ceEAdUJLh0JNysDFlmbPnMje1yWxny1yQs4FGsvCA/M+1mKuURl1/BvEhk3EOBDLw8l4G V X-Google-Smtp-Source: AGHT+IEycmslRdmFZQHmmq6a810nnmvW+wW3TKznqhy2X4QFTnKNoEXAGOWSd1U0C6KhtasbQOSk6A== X-Received: by 2002:a05:6358:5409:b0:17e:bbaf:4063 with SMTP id u9-20020a056358540900b0017ebbaf4063mr3740684rwe.2.1710897785679; Tue, 19 Mar 2024 18:23:05 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:04 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 05/15] io_uring/net: get rid of ->prep_async() for receive side Date: Tue, 19 Mar 2024 19:17:33 -0600 Message-ID: <20240320012251.1120361-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the io_async_msghdr out of the issue path and into prep handling, since it's now done unconditionally and hence does not need to be part of the issue path. This reduces the footprint of the multishot fast path of multiple invocations of ->issue() per prep, and also means that we can drop using ->prep_async() for recvmsg as we now do this setup on the prep side. Signed-off-by: Jens Axboe --- io_uring/net.c | 71 +++++++++++++++++++----------------------------- io_uring/net.h | 1 - io_uring/opdef.c | 2 -- 3 files changed, 28 insertions(+), 46 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 776ebfea8742..50758442188b 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -598,17 +598,36 @@ static int io_recvmsg_copy_hdr(struct io_kiocb *req, msg.msg_controllen); } -int io_recvmsg_prep_async(struct io_kiocb *req) +static int io_recvmsg_prep_setup(struct io_kiocb *req) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *iomsg; + struct io_async_msghdr *kmsg; int ret; - sr->done_io = 0; - if (!io_msg_alloc_async_prep(req)) + /* always locked for prep */ + kmsg = io_msg_alloc_async(req, 0); + if (unlikely(!kmsg)) return -ENOMEM; - iomsg = req->async_data; - ret = io_recvmsg_copy_hdr(req, iomsg); + + if (req->opcode == IORING_OP_RECV) { + kmsg->msg.msg_name = NULL; + kmsg->msg.msg_namelen = 0; + kmsg->msg.msg_control = NULL; + kmsg->msg.msg_get_inq = 1; + kmsg->msg.msg_controllen = 0; + kmsg->msg.msg_iocb = NULL; + kmsg->msg.msg_ubuf = NULL; + + if (!io_do_buffer_select(req)) { + ret = import_ubuf(ITER_DEST, sr->buf, sr->len, + &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + } + return 0; + } + + ret = io_recvmsg_copy_hdr(req, kmsg); if (!ret) req->flags |= REQ_F_NEED_CLEANUP; return ret; @@ -659,7 +678,7 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->msg_flags |= MSG_CMSG_COMPAT; #endif sr->nr_multishot_loops = 0; - return 0; + return io_recvmsg_prep_setup(req); } static inline void io_recv_prep_retry(struct io_kiocb *req, @@ -817,7 +836,7 @@ static int io_recvmsg_multishot(struct socket *sock, struct io_sr_msg *io, int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg; + struct io_async_msghdr *kmsg = req->async_data; struct socket *sock; unsigned flags; int ret, min_ret = 0; @@ -828,17 +847,6 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!sock)) return -ENOTSOCK; - if (req_has_async_data(req)) { - kmsg = req->async_data; - } else { - kmsg = io_msg_alloc_async(req, issue_flags); - if (unlikely(!kmsg)) - return -ENOMEM; - ret = io_recvmsg_copy_hdr(req, kmsg); - if (ret) - return ret; - } - if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) return -EAGAIN; @@ -917,36 +925,13 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) int io_recv(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg; + struct io_async_msghdr *kmsg = req->async_data; struct socket *sock; unsigned flags; int ret, min_ret = 0; bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; size_t len = sr->len; - if (req_has_async_data(req)) { - kmsg = req->async_data; - } else { - kmsg = io_msg_alloc_async(req, issue_flags); - if (unlikely(!kmsg)) - return -ENOMEM; - kmsg->free_iov = NULL; - kmsg->msg.msg_name = NULL; - kmsg->msg.msg_namelen = 0; - kmsg->msg.msg_control = NULL; - kmsg->msg.msg_get_inq = 1; - kmsg->msg.msg_controllen = 0; - kmsg->msg.msg_iocb = NULL; - kmsg->msg.msg_ubuf = NULL; - - if (!io_do_buffer_select(req)) { - ret = import_ubuf(ITER_DEST, sr->buf, sr->len, - &kmsg->msg.msg_iter); - if (unlikely(ret)) - return ret; - } - } - if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) return -EAGAIN; diff --git a/io_uring/net.h b/io_uring/net.h index 281afef670a6..654324739346 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -41,7 +41,6 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags); int io_send(struct io_kiocb *req, unsigned int issue_flags); int io_sendrecv_prep_async(struct io_kiocb *req); -int io_recvmsg_prep_async(struct io_kiocb *req); int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags); int io_recv(struct io_kiocb *req, unsigned int issue_flags); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 77131826d603..1368193edc57 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -536,7 +536,6 @@ const struct io_cold_def io_cold_defs[] = { .name = "RECVMSG", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), - .prep_async = io_recvmsg_prep_async, .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, #endif @@ -613,7 +612,6 @@ const struct io_cold_def io_cold_defs[] = { .async_size = sizeof(struct io_async_msghdr), .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, - .prep_async = io_sendrecv_prep_async, #endif }, [IORING_OP_OPENAT2] = { From patchwork Wed Mar 20 01:17:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597170 Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6BAF7464 for ; Wed, 20 Mar 2024 01:23:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897790; cv=none; b=A5SSkFp6AYNDiD+kWmQYOP1O2++X67un76+uJhNDZnmtyfA1OIwfyLpal/V2qy/1VZ92C30/apo8xP7517ltzKKzHNpnlD5ZmsY5LS+pu6SKUPyvAKcCP1m2jCVpicH/OPVApHoSJZykw1Ov0srFY89fx63E8ZSm+HfuJceERiY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897790; c=relaxed/simple; bh=TZQoeqSK3zMxlu7nCyBtUQb2tl9lGeNBurGesffu8kU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Yk9bDaRp1d9WWaJpKPl/FDaqBX4aDYAiK51S9pB8+ME8hLuXs/tykv/mHhGVx0qWiGd6P7niN4VagUekevuvL01iPCkEc/SDvxtz29CZl9RAftvE7wGfXmBLv0n2bUF62KGBm/QlWaRTlPzBxUX+vlLz2ho3cdirD3cnASSguSo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=hGZMddGl; arc=none smtp.client-ip=209.85.210.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="hGZMddGl" Received: by mail-ot1-f49.google.com with SMTP id 46e09a7af769-6e6a40a15c5so294995a34.0 for ; Tue, 19 Mar 2024 18:23:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897787; x=1711502587; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Klt+7YCKJK205MhC8Z3ij0luEyfDTMzh6WojoGKZ0xE=; b=hGZMddGlT4+q9sHeKzS4U3QXegBrPQaFDm/JdQ/hparC1uvKE2O22F8+TjvSlY6fu7 TFp6K2yfXjCaNvcn47A72UuejuNSZkyS2CYbagqa6//SLNjBbuEyEyePkC1XCA4ksMlC A4I85OWx2GitlIP1nIMORnQmxQOH4wuQGFvyoK1Cz6VMabGCBC578y/4OvFbFYhkaqNs AD0oJBWe4sUTIKKZHortLmwyn1VW9Se4sSRG6QbCysIyvvm47lSdgoerIDE/WzTyJnFB CelxwiOZFpGmxmAGh3kDDTF8OpNXu82Os4U+4R9lhOhPucrJ5CHq959VxyFdWtbDV+Yj IZtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897787; x=1711502587; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Klt+7YCKJK205MhC8Z3ij0luEyfDTMzh6WojoGKZ0xE=; b=B58TBKXeJvFEsDwD8KPwiLpBihwkO6hB2c8WT4XpvUIbhJFBvgsFNl129q2Y77AU+q TB5tPtn9q0xHMeEttmD6hPkEFvg9HlrRx9y/QK493VMmfj7gx8mnt2wD5jbGbfkz4Psh +jauwnhjGnoEoC7lBXYi+AaVilQgSKvtqL8GZlsnx+1WRx7NTDO48f5n8C1ZHpVxR1i7 48ech88ZNVxnMfehCfAME7Wg/P8zP4jgvfng7uM1DOL018oRLPbEAs7G5Ekhueh+IAxf 7By9l9OmdONG5qiXtlJOOXhjv6MKRg+kZdNTWYJIVnf9iKI1NQ2p3NnMhQWacHBAHoGA toqg== X-Gm-Message-State: AOJu0YwRJ0FR0LIiYiPyFavW/sTBGzdtT1KJwkiFZ886Y2F/CHaHNIFV 6pWWiJsNza3xijBHS0TcTePR/OcgzAr9IuLQ1Js1xM8oZfLMN9q7Zd1EYo+x/kz6MN5J9/fldBw v X-Google-Smtp-Source: AGHT+IHkBYMlCfLy27NEkOSNlI60wAg32L3VltnzedUDPhZoiYqU3OKPmmwqf1nKl9QJrl7fNypYJA== X-Received: by 2002:a05:6820:d0a:b0:5a4:6ac7:de6d with SMTP id ej10-20020a0568200d0a00b005a46ac7de6dmr11558565oob.1.1710897787333; Tue, 19 Mar 2024 18:23:07 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:06 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 06/15] io_uring/net: get rid of ->prep_async() for send side Date: Tue, 19 Mar 2024 19:17:34 -0600 Message-ID: <20240320012251.1120361-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the io_async_msghdr out of the issue path and into prep handling, e it's now done unconditionally and hence does not need to be part of the issue path. This means we can drop any usage of io_sendrecv_prep_async() and io_sendmsg_prep_async(), and hence the forced async setup path is now unified with the normal prep setup. Signed-off-by: Jens Axboe --- io_uring/net.c | 169 +++++++++++++++-------------------------------- io_uring/net.h | 2 - io_uring/opdef.c | 4 -- 3 files changed, 54 insertions(+), 121 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 50758442188b..94767d6c1946 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -290,50 +290,64 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, return ret; } -int io_sendrecv_prep_async(struct io_kiocb *req) +void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req) +{ + struct io_async_msghdr *io = req->async_data; + + kfree(io->free_iov); +} + +static int io_send_setup(struct io_kiocb *req) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *io; + struct io_async_msghdr *kmsg = req->async_data; int ret; - if (req_has_async_data(req)) - return 0; - sr->done_io = 0; - if (!sr->addr) - return 0; - io = io_msg_alloc_async_prep(req); - if (!io) - return -ENOMEM; - memset(&io->msg, 0, sizeof(io->msg)); - ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &io->msg.msg_iter); - if (unlikely(ret)) - return ret; - if (sr->addr) - return move_addr_to_kernel(sr->addr, sr->addr_len, &io->addr); + if (sr->addr) { + ret = move_addr_to_kernel(sr->addr, sr->addr_len, &kmsg->addr); + if (unlikely(ret < 0)) + return ret; + } + + if (!io_do_buffer_select(req)) { + ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, + &kmsg->msg.msg_iter); + if (unlikely(ret)) + return ret; + } + + if (sr->addr) { + kmsg->msg.msg_name = &kmsg->addr; + kmsg->msg.msg_namelen = sr->addr_len; + } else { + kmsg->msg.msg_name = NULL; + kmsg->msg.msg_namelen = 0; + } + kmsg->msg.msg_control = NULL; + kmsg->msg.msg_controllen = 0; + kmsg->msg.msg_ubuf = NULL; return 0; } -int io_sendmsg_prep_async(struct io_kiocb *req) +static int io_sendmsg_prep_setup(struct io_kiocb *req, int is_msg) { - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *kmsg; int ret; - sr->done_io = 0; - if (!io_msg_alloc_async_prep(req)) + /* always locked for prep */ + kmsg = io_msg_alloc_async(req, 0); + if (unlikely(!kmsg)) return -ENOMEM; - ret = io_sendmsg_copy_hdr(req, req->async_data); + + if (!is_msg) + return io_send_setup(req); + + ret = io_sendmsg_copy_hdr(req, kmsg); if (!ret) req->flags |= REQ_F_NEED_CLEANUP; return ret; } -void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req) -{ - struct io_async_msghdr *io = req->async_data; - - kfree(io->free_iov); -} - int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -362,7 +376,7 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->ctx->compat) sr->msg_flags |= MSG_CMSG_COMPAT; #endif - return 0; + return io_sendmsg_prep_setup(req, req->opcode == IORING_OP_SENDMSG); } static void io_req_msg_cleanup(struct io_kiocb *req, @@ -379,7 +393,7 @@ static void io_req_msg_cleanup(struct io_kiocb *req, int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg; + struct io_async_msghdr *kmsg = req->async_data; struct socket *sock; unsigned flags; int min_ret = 0; @@ -389,18 +403,6 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!sock)) return -ENOTSOCK; - if (req_has_async_data(req)) { - kmsg = req->async_data; - kmsg->msg.msg_control_user = sr->msg_control; - } else { - kmsg = io_msg_alloc_async(req, issue_flags); - if (unlikely(!kmsg)) - return -ENOMEM; - ret = io_sendmsg_copy_hdr(req, kmsg); - if (ret) - return ret; - } - if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) return -EAGAIN; @@ -436,58 +438,10 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -static struct io_async_msghdr *io_send_setup(struct io_kiocb *req, - unsigned int issue_flags) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg; - int ret; - - if (req_has_async_data(req)) { - kmsg = req->async_data; - } else { - kmsg = io_msg_alloc_async(req, issue_flags); - if (!kmsg) - return ERR_PTR(-ENOMEM); - - if (sr->addr) { - ret = move_addr_to_kernel(sr->addr, sr->addr_len, - &kmsg->addr); - if (unlikely(ret < 0)) - return ERR_PTR(ret); - } - - if (!io_do_buffer_select(req)) { - ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, - &kmsg->msg.msg_iter); - if (unlikely(ret)) - return ERR_PTR(ret); - } - } - - if (sr->addr) { - kmsg->msg.msg_name = &kmsg->addr; - kmsg->msg.msg_namelen = sr->addr_len; - } else { - kmsg->msg.msg_name = NULL; - kmsg->msg.msg_namelen = 0; - } - kmsg->msg.msg_control = NULL; - kmsg->msg.msg_controllen = 0; - kmsg->msg.msg_ubuf = NULL; - - if (!(req->flags & REQ_F_POLLED) && - (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return ERR_PTR(-EAGAIN); - - return kmsg; -} - int io_send(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg; - size_t len = sr->len; + struct io_async_msghdr *kmsg = req->async_data; struct socket *sock; unsigned flags; int min_ret = 0; @@ -497,13 +451,9 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!sock)) return -ENOTSOCK; - kmsg = io_send_setup(req, issue_flags); - if (IS_ERR(kmsg)) - return PTR_ERR(kmsg); - - ret = import_ubuf(ITER_SOURCE, sr->buf, len, &kmsg->msg.msg_iter); - if (unlikely(ret)) - return ret; + if (!(req->flags & REQ_F_POLLED) && + (sr->flags & IORING_RECVSEND_POLL_FIRST)) + return -EAGAIN; flags = sr->msg_flags; if (issue_flags & IO_URING_F_NONBLOCK) @@ -1085,7 +1035,7 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->ctx->compat) zc->msg_flags |= MSG_CMSG_COMPAT; #endif - return 0; + return io_sendmsg_prep_setup(req, req->opcode == IORING_OP_SENDMSG_ZC); } static int io_sg_from_iter_iovec(struct sock *sk, struct sk_buff *skb, @@ -1174,7 +1124,7 @@ static int io_send_zc_import(struct io_kiocb *req, struct io_async_msghdr *kmsg) int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg; + struct io_async_msghdr *kmsg = req->async_data; struct socket *sock; unsigned msg_flags; int ret, min_ret = 0; @@ -1185,9 +1135,9 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) if (!test_bit(SOCK_SUPPORT_ZC, &sock->flags)) return -EOPNOTSUPP; - kmsg = io_send_setup(req, issue_flags); - if (IS_ERR(kmsg)) - return PTR_ERR(kmsg); + if (!(req->flags & REQ_F_POLLED) && + (zc->flags & IORING_RECVSEND_POLL_FIRST)) + return -EAGAIN; if (!zc->done_io) { ret = io_send_zc_import(req, kmsg); @@ -1241,7 +1191,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *kmsg; + struct io_async_msghdr *kmsg = req->async_data; struct socket *sock; unsigned flags; int ret, min_ret = 0; @@ -1254,17 +1204,6 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) if (!test_bit(SOCK_SUPPORT_ZC, &sock->flags)) return -EOPNOTSUPP; - if (req_has_async_data(req)) { - kmsg = req->async_data; - } else { - kmsg = io_msg_alloc_async(req, issue_flags); - if (unlikely(!kmsg)) - return -ENOMEM; - ret = io_sendmsg_copy_hdr(req, kmsg); - if (ret) - return ret; - } - if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) return -EAGAIN; diff --git a/io_uring/net.h b/io_uring/net.h index 654324739346..2713dc90fdd4 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -33,13 +33,11 @@ struct io_async_connect { int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_shutdown(struct io_kiocb *req, unsigned int issue_flags); -int io_sendmsg_prep_async(struct io_kiocb *req); void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req); int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags); int io_send(struct io_kiocb *req, unsigned int issue_flags); -int io_sendrecv_prep_async(struct io_kiocb *req); int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 1368193edc57..dd4a1e1425e1 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -527,7 +527,6 @@ const struct io_cold_def io_cold_defs[] = { .name = "SENDMSG", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), - .prep_async = io_sendmsg_prep_async, .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, #endif @@ -603,7 +602,6 @@ const struct io_cold_def io_cold_defs[] = { .async_size = sizeof(struct io_async_msghdr), .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, - .prep_async = io_sendrecv_prep_async, #endif }, [IORING_OP_RECV] = { @@ -688,7 +686,6 @@ const struct io_cold_def io_cold_defs[] = { .name = "SEND_ZC", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), - .prep_async = io_sendrecv_prep_async, .cleanup = io_send_zc_cleanup, .fail = io_sendrecv_fail, #endif @@ -697,7 +694,6 @@ const struct io_cold_def io_cold_defs[] = { .name = "SENDMSG_ZC", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), - .prep_async = io_sendmsg_prep_async, .cleanup = io_send_zc_cleanup, .fail = io_sendrecv_fail, #endif From patchwork Wed Mar 20 01:17:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597171 Received: from mail-oo1-f53.google.com (mail-oo1-f53.google.com [209.85.161.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 010F56AB9 for ; Wed, 20 Mar 2024 01:23:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897792; cv=none; b=feMnV44JoB3l6mkBQm5wuTxqQflLQwawpRqnUEq0EeHFvYI3WP6ZUFM+Qz9FtfdahI8RgdD4R6WqmLCCa3hFM5x6tFMnZ/PcHpjYokabhYxOF4XsicMALOQ9oot4++i0TMG78HMmj1qqEZkvHc1ihoqpZej3Y8RYByWgRPlrK6E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897792; c=relaxed/simple; bh=GUq1YL1l99R26cr3XZNdacxQpTiKR95eVXArPQDbP6A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MrnKEvM6VNWyyPUR9xwu158EWYM/Ugp786f/ARmZYsuG7skFsQVE4CsdC5gMs8QmAVJUuac2Ixo1XXyUylfbv2wy4zjFPzVNVtvc5cRtLaYEdzC7WkdiiE/5wOY8jBTnZIxejDG2tMl7EsCdkT4mFqmo0P4Gi6G1b+mM7v4lcto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=S/NLWO5E; arc=none smtp.client-ip=209.85.161.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="S/NLWO5E" Received: by mail-oo1-f53.google.com with SMTP id 006d021491bc7-5a47680a806so1368978eaf.0 for ; Tue, 19 Mar 2024 18:23:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897789; x=1711502589; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Mwf5gpehLqrQJLmo4KvjUucibL5k2W580oU2DJPGaqY=; b=S/NLWO5Eob/dYga3fr2xesImJN57rJulWsnjjuWVFBIfZ+Y78k2YGChat7Eh9Y2U9s TXUr+d80iEryuxGK16ZeWd9xVVrMj/tULOnWuUzemxalcMy10Nv6KbOHvcq2u7MJIutS ggfUMgcZ5cyQ5smu67WioUd/vSgfrcg/gok+z1pVlWqdfPkik2D1uHPC0QAuxl9RAQDm 4LC0TCOfPpG1yhdcgZhbhcurzRJrS1x97xl/fKCpn4yYWIswcRUJf7HI/giiYBrrtxH7 j3IV3OJ65R8y78XKmRT8xRL+gN7PnMjp0+kgo6hwrTBN/bPpewM8TfHR2pFaYp26rvU1 ZAhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897789; x=1711502589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mwf5gpehLqrQJLmo4KvjUucibL5k2W580oU2DJPGaqY=; b=f9aFFAHk/t9VSqvICAyICh5u5Gbc/91844bmXq5WGMMoyTGG1viIS1hM7uA3do9LXR 68sK8j8arHB0duVkQ+hexevcyygm7a1D7qdKQkGrHf8Q7RWB3SoFBmy7xLD4ifZma+fv dy58pVbGHdRepbjS/sKOqYmZri7C61XBWZQkvcEGQ3U/2+UIIEAuAHJIE4Nxda8DcLot G3e0vEmqUv8KCFP6/4aHReVW8rs10GlMGLEBs9ukAEVgI5oTtJ54O1+YV0KcqC0m+wrc KTh8z9wWCQel6ql+aBgZC/KF55w3qm/0AXOMtd0ZNMj/4w6e2VfMfOhggj9s/XIglBQD 98yQ== X-Gm-Message-State: AOJu0YzEqBfuB4Js/0Gk1bOAuilXNVeezE7i+Asltb6xJ+Gdp0l4oP2R bAFk+LRpK3trxY2dT7AZYFXdFRnnkDsWd2BDS75VMphZyaTryEDuo+ZwBiMHvhHwXJm56BRNx/+ K X-Google-Smtp-Source: AGHT+IEncGfT0ro7fYY5iT2MZ6KOIWIb2GF7T+XWE9du76kVfDBcsS9v+/0NnYvRMR8xwwFPwWuFZQ== X-Received: by 2002:a05:6358:291d:b0:17e:b8b5:d6a7 with SMTP id y29-20020a056358291d00b0017eb8b5d6a7mr4258129rwb.1.1710897789590; Tue, 19 Mar 2024 18:23:09 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:07 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 07/15] io_uring: kill io_msg_alloc_async_prep() Date: Tue, 19 Mar 2024 19:17:35 -0600 Message-ID: <20240320012251.1120361-8-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We now ONLY call io_msg_alloc_async() from inside prep handling, which is always locked. No need for this helper anymore, or the check in io_msg_alloc_async() on whether the ring is locked or not. Signed-off-by: Jens Axboe --- io_uring/net.c | 31 ++++++++++--------------------- 1 file changed, 10 insertions(+), 21 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 94767d6c1946..21624b7ead8a 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -129,22 +129,19 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } } -static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req, - unsigned int issue_flags) +static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; struct io_cache_entry *entry; struct io_async_msghdr *hdr; - if (!(issue_flags & IO_URING_F_UNLOCKED)) { - entry = io_alloc_cache_get(&ctx->netmsg_cache); - if (entry) { - hdr = container_of(entry, struct io_async_msghdr, cache); - hdr->free_iov = NULL; - req->flags |= REQ_F_ASYNC_DATA; - req->async_data = hdr; - return hdr; - } + entry = io_alloc_cache_get(&ctx->netmsg_cache); + if (entry) { + hdr = container_of(entry, struct io_async_msghdr, cache); + hdr->free_iov = NULL; + req->flags |= REQ_F_ASYNC_DATA; + req->async_data = hdr; + return hdr; } if (!io_alloc_async_data(req)) { @@ -155,12 +152,6 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req, return NULL; } -static inline struct io_async_msghdr *io_msg_alloc_async_prep(struct io_kiocb *req) -{ - /* ->prep_async is always called from the submission context */ - return io_msg_alloc_async(req, 0); -} - #ifdef CONFIG_COMPAT static int io_compat_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, @@ -334,8 +325,7 @@ static int io_sendmsg_prep_setup(struct io_kiocb *req, int is_msg) struct io_async_msghdr *kmsg; int ret; - /* always locked for prep */ - kmsg = io_msg_alloc_async(req, 0); + kmsg = io_msg_alloc_async(req); if (unlikely(!kmsg)) return -ENOMEM; @@ -554,8 +544,7 @@ static int io_recvmsg_prep_setup(struct io_kiocb *req) struct io_async_msghdr *kmsg; int ret; - /* always locked for prep */ - kmsg = io_msg_alloc_async(req, 0); + kmsg = io_msg_alloc_async(req); if (unlikely(!kmsg)) return -ENOMEM; From patchwork Wed Mar 20 01:17:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597172 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BEE732582 for ; Wed, 20 Mar 2024 01:23:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897794; cv=none; b=sFPfdOmn+xzEiWc2WCgnKYceh6VCYmIQwAiEkCeM+qLMBKh1/+uHDo7PiofUnobL7CHhu8SAbP0Heffx+Ft2H2srPGYlsUZHnv0QZwlEWghCo7w3dOjwLRsl6KWeHYKiv0EivkttxDq5SRk71dZvitSRDj89wj/IdjFecsIdbck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897794; c=relaxed/simple; bh=UIhOmHPdU5wmx5HAhnDwKdfPc0rtoh0ZntsczzeHj5Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B5jTRB76O96q6L9TP0vj7430tKoNDoicMhTjMXxz8u909bgk/UgHiSxbtWspTIOQCIfTxUZBTD42YOxgiR09R+yWnVTaVHL2J0f4DHoeLnJk0AIr3th+DK12dHnqN/1+2eUCnn/yD5Ow0N03ohUJpuQxbMcaSIS1c+biuYPvnu8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=UMzWuVL4; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="UMzWuVL4" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1def81ee762so8810785ad.0 for ; Tue, 19 Mar 2024 18:23:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897792; x=1711502592; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ewoOwa6YRNZRmf7sqsHhcXXE2VqnL08XbKy4FXWlxnM=; b=UMzWuVL4fg3z0p67lsnSxmpGKgUkGgUpl0Dd9fMfdLCzPBH/0GFHQOPwoO2TSCpjXR fgqofZZDqElTY55ymTVQvlofksZ/CWzJZOl39TGLlaV71alLa4Mq1LtnDy74VtFg3Snn PiRFvxnGUaQG5/mAw4yHjew+HPp+eyHGpTZ4lflHJj1db05p101m2u2aF6ENYvw/FMJG jujh42R9ECVjUDhyR2jvnumbEDvuxETC5+fl74LiWhryAW7FA6ulX9V5PbzD9qc3tr0C 2w2cyTw2dkrMcXnGEdy66aqGGODyVRwPj4N0ubeHmoveBfZWQf2GkQW6A8nrKJxGQgDT SjBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897792; x=1711502592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ewoOwa6YRNZRmf7sqsHhcXXE2VqnL08XbKy4FXWlxnM=; b=Q0ucV1qxhhz+PR0wv89zC/Y0THq35uf1taN8Kg8IxUB10A20hNXvm0bCMwsgqSeFN3 E78Yx83X04vLNVebJW5aP1gMGrVC2srl2YmakeP8coH0XBgDi/809CIoTRmRMHOzpsGI wfzua8vEwBrkfDl2LP5082UaLS4HvqLcuZ6BLKMV6TAdQB0EKjnd5lJEXRAu8cB8BJBu sCeYBl6LDeoHClnsk7B2tzbrRpXco5N6mzpG1SO+55+lq2FckQk2aFJaV3tCJPpEH523 lrXLbaXUllpMpLCW6rE4oc/GZl4meIjE62a0UXGw9IN+Usxxq5chnvrwUIIZi6QRyV02 ZjGw== X-Gm-Message-State: AOJu0Yy7ixte+/dChq6qq/htmtRvTa94/Ap5oam7wjnCDMrDG3EJT14K ReMTMJDOoq7A1xlMzTR8VK/NmHgBEG2V3gxjtcbyzuwIvE1CLf3PjsshZ+X3iuYP885wpvzxxRk g X-Google-Smtp-Source: AGHT+IH8n3TjK+qWeEO5mTxlPmSnLTIaCmtaj1U78nc31c6s+8MCOK83yIybLwgmSf/fR+DWxtUTQA== X-Received: by 2002:a05:6a20:7f8f:b0:1a3:4721:ded3 with SMTP id d15-20020a056a207f8f00b001a34721ded3mr4417282pzj.1.1710897791600; Tue, 19 Mar 2024 18:23:11 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:10 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 08/15] io_uring/net: add iovec recycling Date: Tue, 19 Mar 2024 19:17:36 -0600 Message-ID: <20240320012251.1120361-9-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Right now the io_async_msghdr is recycled to avoid the overhead of allocating+freeing it for every request. But the iovec is not included, hence that will be allocated and freed for each transfer regardless. This commit enables recyling of the iovec between io_async_msghdr recycles. This avoids alloc+free for each one if we use it, and on top of that, it extends the cache hot nature of msg to the iovec as well. Enable KASAN for the iovec, if we're recycling it, to provide some extra protection against it being reused while in the cache. The io_async_msghdr also shrinks from 376 -> 272 bytes, a 104 byte saving (or ~28% smaller), as the fast_iovec entry is dropped from 8 entries to a single entry. There's no point keeping a big fast iovec entry, if we're keeping the iovec around. Signed-off-by: Jens Axboe --- io_uring/net.c | 107 +++++++++++++++++++++++++++++++------------------ io_uring/net.h | 6 ++- 2 files changed, 74 insertions(+), 39 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 21624b7ead8a..bae8cab08e36 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -115,14 +115,30 @@ static bool io_net_retry(struct socket *sock, int flags) return sock->type == SOCK_STREAM || sock->type == SOCK_SEQPACKET; } +static void io_netmsg_iovec_free(struct io_async_msghdr *kmsg) +{ + if (kmsg->free_iov) { + kfree(kmsg->free_iov); + kmsg->free_iov_nr = 0; + kmsg->free_iov = NULL; + } +} + static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) { struct io_async_msghdr *hdr = req->async_data; - if (!req_has_async_data(req) || issue_flags & IO_URING_F_UNLOCKED) + if (!req_has_async_data(req)) return; + /* can't recycle, ensure we free the iovec if we have one */ + if (issue_flags & IO_URING_F_UNLOCKED) { + io_netmsg_iovec_free(hdr); + return; + } /* Let normal cleanup path reap it if we fail adding to the cache */ + if (hdr->free_iov) + kasan_mempool_poison_object(hdr->free_iov); if (io_alloc_cache_put(&req->ctx->netmsg_cache, &hdr->cache)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -138,14 +154,17 @@ static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req) entry = io_alloc_cache_get(&ctx->netmsg_cache); if (entry) { hdr = container_of(entry, struct io_async_msghdr, cache); - hdr->free_iov = NULL; - req->flags |= REQ_F_ASYNC_DATA; + if (hdr->free_iov) + kasan_mempool_unpoison_object(hdr->free_iov, + hdr->free_iov_nr * sizeof(struct iovec)); + req->flags |= REQ_F_ASYNC_DATA | REQ_F_NEED_CLEANUP; req->async_data = hdr; return hdr; } if (!io_alloc_async_data(req)) { hdr = req->async_data; + hdr->free_iov_nr = 0; hdr->free_iov = NULL; return hdr; } @@ -159,7 +178,8 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct compat_iovec __user *uiov; - int ret; + struct iovec *iov; + int ret, nr_segs; if (copy_from_user(msg, sr->umsg_compat, sizeof(*msg))) return -EFAULT; @@ -168,7 +188,6 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, if (req->flags & REQ_F_BUFFER_SELECT) { compat_ssize_t clen; - iomsg->free_iov = NULL; if (msg->msg_iovlen == 0) { sr->len = 0; } else if (msg->msg_iovlen > 1) { @@ -186,13 +205,22 @@ static int io_compat_msg_copy_hdr(struct io_kiocb *req, return 0; } - iomsg->free_iov = iomsg->fast_iov; + if (iomsg->free_iov) { + nr_segs = iomsg->free_iov_nr; + iov = iomsg->free_iov; + } else { + iov = &iomsg->fast_iov; + nr_segs = 1; + } ret = __import_iovec(ddir, (struct iovec __user *)uiov, msg->msg_iovlen, - UIO_FASTIOV, &iomsg->free_iov, - &iomsg->msg.msg_iter, true); + nr_segs, &iov, &iomsg->msg.msg_iter, true); if (unlikely(ret < 0)) return ret; - + if (iov) { + iomsg->free_iov_nr = iomsg->msg.msg_iter.nr_segs; + kfree(iomsg->free_iov); + iomsg->free_iov = iov; + } return 0; } #endif @@ -201,7 +229,8 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct user_msghdr *msg, int ddir) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - int ret; + struct iovec *iov; + int ret, nr_segs; if (!user_access_begin(sr->umsg, sizeof(*sr->umsg))) return -EFAULT; @@ -217,9 +246,8 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, if (req->flags & REQ_F_BUFFER_SELECT) { if (msg->msg_iovlen == 0) { - sr->len = iomsg->fast_iov[0].iov_len = 0; - iomsg->fast_iov[0].iov_base = NULL; - iomsg->free_iov = NULL; + sr->len = iomsg->fast_iov.iov_len = 0; + iomsg->fast_iov.iov_base = NULL; } else if (msg->msg_iovlen > 1) { ret = -EINVAL; goto ua_end; @@ -227,10 +255,9 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, /* we only need the length for provided buffers */ if (!access_ok(&msg->msg_iov[0].iov_len, sizeof(__kernel_size_t))) goto ua_end; - unsafe_get_user(iomsg->fast_iov[0].iov_len, + unsafe_get_user(iomsg->fast_iov.iov_len, &msg->msg_iov[0].iov_len, ua_end); - sr->len = iomsg->fast_iov[0].iov_len; - iomsg->free_iov = NULL; + sr->len = iomsg->fast_iov.iov_len; } ret = 0; ua_end: @@ -239,12 +266,22 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, } user_access_end(); - iomsg->free_iov = iomsg->fast_iov; - ret = __import_iovec(ddir, msg->msg_iov, msg->msg_iovlen, UIO_FASTIOV, - &iomsg->free_iov, &iomsg->msg.msg_iter, false); + if (iomsg->free_iov) { + nr_segs = iomsg->free_iov_nr; + iov = iomsg->free_iov; + } else { + iov = &iomsg->fast_iov; + nr_segs = 1; + } + ret = __import_iovec(ddir, msg->msg_iov, msg->msg_iovlen, nr_segs, + &iov, &iomsg->msg.msg_iter, false); if (unlikely(ret < 0)) return ret; + if (nr_segs == 1 && iov) { + iomsg->free_iov_nr = iomsg->msg.msg_iter.nr_segs; + iomsg->free_iov = iov; + } return 0; } @@ -285,7 +322,7 @@ void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req) { struct io_async_msghdr *io = req->async_data; - kfree(io->free_iov); + io_netmsg_iovec_free(io); } static int io_send_setup(struct io_kiocb *req) @@ -374,9 +411,6 @@ static void io_req_msg_cleanup(struct io_kiocb *req, unsigned int issue_flags) { req->flags &= ~REQ_F_NEED_CLEANUP; - /* fast path, check for non-NULL to avoid function call */ - if (kmsg->free_iov) - kfree(kmsg->free_iov); io_netmsg_recycle(req, issue_flags); } @@ -625,11 +659,6 @@ static inline void io_recv_prep_retry(struct io_kiocb *req, { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - if (kmsg->free_iov) { - kfree(kmsg->free_iov); - kmsg->free_iov = NULL; - } - req->flags &= ~REQ_F_BL_EMPTY; sr->done_io = 0; sr->len = 0; /* get from the provided buffer */ @@ -943,13 +972,13 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) void io_send_zc_cleanup(struct io_kiocb *req) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); - struct io_async_msghdr *io; if (req_has_async_data(req)) { - io = req->async_data; + struct io_async_msghdr *io = req->async_data; + /* might be ->fast_iov if *msg_copy_hdr failed */ - if (io->free_iov != io->fast_iov) - kfree(io->free_iov); + if (io->free_iov != &io->fast_iov) + io_netmsg_iovec_free(io); } if (zc->notif) { io_notif_flush(zc->notif); @@ -1220,11 +1249,6 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) ret = -EINTR; req_set_fail(req); } - /* fast path, check for non-NULL to avoid function call */ - if (kmsg->free_iov) { - kfree(kmsg->free_iov); - kmsg->free_iov = NULL; - } io_netmsg_recycle(req, issue_flags); if (ret >= 0) @@ -1482,6 +1506,13 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags) void io_netmsg_cache_free(struct io_cache_entry *entry) { - kfree(container_of(entry, struct io_async_msghdr, cache)); + struct io_async_msghdr *kmsg; + + kmsg = container_of(entry, struct io_async_msghdr, cache); + if (kmsg->free_iov) + kasan_mempool_unpoison_object(kmsg->free_iov, + kmsg->free_iov_nr * sizeof(struct iovec)); + io_netmsg_iovec_free(kmsg); + kfree(kmsg); } #endif diff --git a/io_uring/net.h b/io_uring/net.h index 2713dc90fdd4..783dd601a432 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -8,7 +8,11 @@ struct io_async_msghdr { #if defined(CONFIG_NET) union { - struct iovec fast_iov[UIO_FASTIOV]; + struct { + struct iovec fast_iov; + /* entry size of ->free_iov, if valid */ + int free_iov_nr; + }; struct { __kernel_size_t controllen; int namelen; From patchwork Wed Mar 20 01:17:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597173 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6A502582 for ; Wed, 20 Mar 2024 01:23:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897796; cv=none; b=A9KBVSJu8ZIiMtQUf1OYhe9rscq25K0IqBI56ZNyMhSWVOz3rR2+KBYtQcA5bQSye2WGmd3yjrjY7/NcNpJRkOs+gg/7bIO4meVuwyIvSIAjaWeT/Q2mduKVR7TPqnTVbduYK3avEGqOjOlf5vhN3IxdO46k/5ki7u7iUqm3ZbI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897796; c=relaxed/simple; bh=b1Yw6AjBrKmLiNE5M50xDp0rL+hd2Brg3TM+ahttebo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kCR530dXDajVoJJ6MRl0wibBY/XYxoWf77yOxmrI0+eMOGRtdwKDH1zA2odXunALTNtCcIWbDIJEvM6w2PJ7ul0sHWeT0UIU8eG8qoFHmU5YEQXtWXf8AMdIVAuJ9F/rrHFwuf3RkQEbkfgHuCaEpolOq8r2jcYf4tgOv9Oqsmo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=OIfpdRdp; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="OIfpdRdp" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-6e2b466d213so95532a34.0 for ; Tue, 19 Mar 2024 18:23:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897793; x=1711502593; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=koYr3HDB0x7utiFY+HyI/2RGEMP2Y2Hy5qqDdHyPb68=; b=OIfpdRdpU5wTe/Ixh8XuP2DOR1mxkT7ZEcpvTb7EuGPId3IEo9O9q751hKqZTk6NYg vvniGYvU4CcQFoLnW80bk/jhcXzSFhn7fdGvnLBfCVD2QQmSjQNRtVYjkGwG6lW+LTLh 56fmCBYNxIOZlij6sYfyxsphUTQjg/+NxX0pFYq5Rmvf9AF0JIfdZYJqH7H1jGt20veI DB/sI3t1Y8LPXwdz1zAPgFsLsZq+VyNiWdHfCOzkIz6ivNLbHXZbaovym+QqnaT2YWsv 9t67+zMv1vKqmS9LcUHbUOkR3PO7QDhOml1CwAaS7HhJNJ5jLBGc3SqqgkWTpW/w/MdO W6eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897793; x=1711502593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=koYr3HDB0x7utiFY+HyI/2RGEMP2Y2Hy5qqDdHyPb68=; b=s+2vBpsXi7dGLwX5i0So+3meeRB2La/67nKfI+vQ9rfwiDuYEZ8fkb2soKzGWsSr3q rUcRCW9VDZaHBc73+QPFGIYhd90PQ6RLRNrl9p9z/QYmgBC5s5mpuR0VqIRFWqo8RRTk oUcLX0kGu6KRm8urV3X3+IZN+leBrbAnKPFkpLLHBRENQJ8GbI01LUNMHqKy0xNoTSrn kVUuqSBbDo0gRr6qf6J5Fs5aLKo8kBDR6owwpIwQcwya9Enh3sDr3SDxqkxiUGcNgty+ R3aYWK8I1E172qBSBC4ksI+NkvEzmgzUxumtrcC4Xn1MRZxHpsQbK8CmOozechiL+HMR I6Pw== X-Gm-Message-State: AOJu0Yy55jLhQhBwb/MONpPLsVZeF6MHlQ/Bon9Il25zqXuLUqAFhqGI s6aFHmXITqOxOFTL/dvhzXXvu+ebYtAYE8F4nVfM3QSt+ZY4SHixC11HMpQRGiQxNut7AA3N2fZ T X-Google-Smtp-Source: AGHT+IGxOrEBCX4a13yMrIxFpobyXOz7SJaqskBMYMIVeuNtHrzFbuwrdgnA4g21lHeqaHkIPLeSsQ== X-Received: by 2002:a05:6830:443:b0:6e6:a019:347f with SMTP id d3-20020a056830044300b006e6a019347fmr4077045otc.2.1710897793671; Tue, 19 Mar 2024 18:23:13 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:12 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 09/15] io_uring/net: drop 'kmsg' parameter from io_req_msg_cleanup() Date: Tue, 19 Mar 2024 19:17:37 -0600 Message-ID: <20240320012251.1120361-10-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Now that iovec recycling is being done, the iovec is no longer being freed in there. Hence the kmsg parameter is now useless. Signed-off-by: Jens Axboe --- io_uring/net.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index bae8cab08e36..82f314da1326 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -407,7 +407,6 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) } static void io_req_msg_cleanup(struct io_kiocb *req, - struct io_async_msghdr *kmsg, unsigned int issue_flags) { req->flags &= ~REQ_F_NEED_CLEANUP; @@ -453,7 +452,7 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret = -EINTR; req_set_fail(req); } - io_req_msg_cleanup(req, kmsg, issue_flags); + io_req_msg_cleanup(req, issue_flags); if (ret >= 0) ret += sr->done_io; else if (sr->done_io) @@ -505,7 +504,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; - io_req_msg_cleanup(req, kmsg, issue_flags); + io_req_msg_cleanup(req, issue_flags); io_req_set_res(req, ret, 0); return IOU_OK; } @@ -713,7 +712,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, *ret = IOU_STOP_MULTISHOT; else *ret = IOU_OK; - io_req_msg_cleanup(req, kmsg, issue_flags); + io_req_msg_cleanup(req, issue_flags); return true; } From patchwork Wed Mar 20 01:17:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597175 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E123747F for ; Wed, 20 Mar 2024 01:23:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897798; cv=none; b=q8NfGHRn8cj80biaHzDDHzZUXJtKAa7a5tuaZaU/YBSyukQRIYlnDXCQgmnGgXjlcYAtdQwxXZSjbZTBkdIsE1LEz5WipycchjGX6nJ3hi5laEba0K3Si9MTmFYsRQbDAUUS8DzAaz3uBk7E6uZXC+cFpu4qHxEA4zvyXbuWXkY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897798; c=relaxed/simple; bh=GVjBan1sVQJDTe0yOncPeyDdfBAgYLMqEdZl2LyXcPs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qyrdDNYWtrRLvzFg/HdsNEmFnh1ss0nwxLQPSknrxbbSsaw8iHqEI3WWq+MQDKK7Gh0Abo8jyqHWz5wInnZskMh26Nhr4kdGs1RVUufkpxEwOjg2T/j9OmlYuE6mkyyaDSErn8Lfj1cgAJLcyIrujlFtC0D3eAygmyXbKPy8RCk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=13+IYOn4; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="13+IYOn4" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-29d51cdde7eso1895409a91.0 for ; Tue, 19 Mar 2024 18:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897795; x=1711502595; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V29pEc9m1yxnT4gHi/AnDSuwzoP95IggpkzJ+sfeM0Y=; b=13+IYOn44EZ6ztE3cO/M1vev8PgG4m8r0WnT0zCMG6VfWhe/w36GbtseOtzHMbN0UC kJ0AdMGoEL9pW/U8n1aauXUibab9TWxIgWEme7z7pNkKF0xwpb06HbXHhkD+UbjYNS2d /d+vpK6JFaTzuhjPaDn4UHtVNM14IEEU2qrvIThJPK33S5Rh8ifv7YF5WdKeuJm7JKov ZUmF8MtPVoXMu3Y6CJLdVosLMWAzpwsFda5s+ZB/9JMY4+1cEFCbxnXLzfHAj3Yi74cD H7KGfLDSFGRbotgGx0ZYoY29CSQH0u76GG5AdS1Yf3BT1qZrNmZcUTpJNUuVd+FN3Tp8 Ep+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897795; x=1711502595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V29pEc9m1yxnT4gHi/AnDSuwzoP95IggpkzJ+sfeM0Y=; b=BFXkU0+C3FinhQ769NhaaaHTyFxWfJLJvCbZMYU9QxmQEljVaIdzi348k3b2PahNBE PyWpnrG7dd3G1tIcOtpy4Rdj7ojMLN7Oe7TMiZe5N4W2vspcGSAP8mlthiBdgPJd7wpE 4LECZQwNiry/HbIS523iExtSdVWpVXnzUHUffsPQ3HylOpAKqEplH6okDgkmti2+IZgA LxjWGNBP3eKDRnNE4EhLTsSzIR1qmJyPrmtHv5cZrOuSEaIdpIFNlYBopvxsGtmayfzP rmEjoVtVd66RhvF4Nzrf7sATiBgBV8sJpZRFkBNIkD0EUw7Vd07WFTwiRbqKF0TMuMqI 2Lfg== X-Gm-Message-State: AOJu0YyoOKWLEEVbPVxQJ+yoKl5YjAkG/lAdCcpEONjGaw60KDvHbj7+ 6Fvq4H5q5yfBoyaOyQhDbYndhS28ZE4GhXs4qzv4sDOKXpbwy1VA8od17mXKukW10ZqSk1B9jpj 4 X-Google-Smtp-Source: AGHT+IESjYKWKq0VHwaNE2Dh6/tOpauQB7VHh3SLhTzCZqVxS31qvPmZJepAJPkHW2o8BAxUMyVzvg== X-Received: by 2002:a05:6a00:9392:b0:6e7:362b:9873 with SMTP id ka18-20020a056a00939200b006e7362b9873mr4372263pfb.0.1710897794802; Tue, 19 Mar 2024 18:23:14 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:14 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 10/15] io_uring/rw: always setup io_async_rw for read/write requests Date: Tue, 19 Mar 2024 19:17:38 -0600 Message-ID: <20240320012251.1120361-11-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 read/write requests try to put everything on the stack, and then alloc and copy if we need to retry. This necessitates a bunch of nasty code that deals with intermediate state. Get rid of this, and have the prep side setup everything we need upfront, which greatly simplifies the opcode handlers. This includes adding an alloc cache for io_async_rw, to make it cheap to handle. In terms of cost, this should be basically free and transparent. For the worst case of {READ,WRITE}_FIXED which didn't need it before, performance is unaffected in the normal peak workload that is being used to test that. Still runs at 122M IOPS. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 3 + io_uring/opdef.c | 15 +- io_uring/rw.c | 507 +++++++++++++++------------------ io_uring/rw.h | 19 +- 5 files changed, 249 insertions(+), 296 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index f37caff64d05..2ba8676f83cc 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -300,6 +300,7 @@ struct io_ring_ctx { struct io_hash_table cancel_table_locked; struct io_alloc_cache apoll_cache; struct io_alloc_cache netmsg_cache; + struct io_alloc_cache rw_cache; /* * Any cancelable uring_cmd is added to this list in diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ff0e233ce3c9..cc8ce830ff4b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -308,6 +308,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) sizeof(struct async_poll)); io_alloc_cache_init(&ctx->netmsg_cache, IO_ALLOC_CACHE_MAX, sizeof(struct io_async_msghdr)); + io_alloc_cache_init(&ctx->rw_cache, IO_ALLOC_CACHE_MAX, + sizeof(struct io_async_rw)); io_futex_cache_init(ctx); init_completion(&ctx->ref_comp); xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); @@ -2898,6 +2900,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_eventfd_unregister(ctx); io_alloc_cache_free(&ctx->apoll_cache, io_apoll_cache_free); io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free); + io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free); io_futex_cache_free(ctx); io_destroy_buffers(ctx); mutex_unlock(&ctx->uring_lock); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index dd4a1e1425e1..fcae75a08f2c 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -67,7 +67,7 @@ const struct io_issue_def io_issue_defs[] = { .iopoll = 1, .iopoll_queue = 1, .vectored = 1, - .prep = io_prep_rwv, + .prep = io_prep_readv, .issue = io_read, }, [IORING_OP_WRITEV] = { @@ -81,7 +81,7 @@ const struct io_issue_def io_issue_defs[] = { .iopoll = 1, .iopoll_queue = 1, .vectored = 1, - .prep = io_prep_rwv, + .prep = io_prep_writev, .issue = io_write, }, [IORING_OP_FSYNC] = { @@ -99,7 +99,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, - .prep = io_prep_rw_fixed, + .prep = io_prep_read_fixed, .issue = io_read, }, [IORING_OP_WRITE_FIXED] = { @@ -112,7 +112,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, - .prep = io_prep_rw_fixed, + .prep = io_prep_write_fixed, .issue = io_write, }, [IORING_OP_POLL_ADD] = { @@ -239,7 +239,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, - .prep = io_prep_rw, + .prep = io_prep_read, .issue = io_read, }, [IORING_OP_WRITE] = { @@ -252,7 +252,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, - .prep = io_prep_rw, + .prep = io_prep_write, .issue = io_write, }, [IORING_OP_FADVISE] = { @@ -490,14 +490,12 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_READV] = { .async_size = sizeof(struct io_async_rw), .name = "READV", - .prep_async = io_readv_prep_async, .cleanup = io_readv_writev_cleanup, .fail = io_rw_fail, }, [IORING_OP_WRITEV] = { .async_size = sizeof(struct io_async_rw), .name = "WRITEV", - .prep_async = io_writev_prep_async, .cleanup = io_readv_writev_cleanup, .fail = io_rw_fail, }, @@ -699,6 +697,7 @@ const struct io_cold_def io_cold_defs[] = { #endif }, [IORING_OP_READ_MULTISHOT] = { + .async_size = sizeof(struct io_async_rw), .name = "READ_MULTISHOT", }, [IORING_OP_WAITID] = { diff --git a/io_uring/rw.c b/io_uring/rw.c index 35216e8adc29..054e6f381460 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -75,7 +75,124 @@ static int io_iov_buffer_select_prep(struct io_kiocb *req) return 0; } -int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) +static int __io_import_iovec(int ddir, struct io_kiocb *req, + struct io_async_rw *io, + unsigned int issue_flags) +{ + const struct io_issue_def *def = &io_issue_defs[req->opcode]; + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + void __user *buf; + size_t sqe_len; + + buf = u64_to_user_ptr(rw->addr); + sqe_len = rw->len; + + if (!def->vectored || req->flags & REQ_F_BUFFER_SELECT) { + if (io_do_buffer_select(req)) { + buf = io_buffer_select(req, &sqe_len, issue_flags); + if (!buf) + return -ENOBUFS; + rw->addr = (unsigned long) buf; + rw->len = sqe_len; + } + + return import_ubuf(ddir, buf, sqe_len, &io->s.iter); + } + + io->free_iovec = io->s.fast_iov; + return __import_iovec(ddir, buf, sqe_len, UIO_FASTIOV, &io->free_iovec, + &io->s.iter, req->ctx->compat); +} + +static inline int io_import_iovec(int rw, struct io_kiocb *req, + struct io_async_rw *io, + unsigned int issue_flags) +{ + int ret; + + ret = __io_import_iovec(rw, req, io, issue_flags); + if (unlikely(ret < 0)) + return ret; + + iov_iter_save_state(&io->s.iter, &io->s.iter_state); + return 0; +} + +static void io_rw_iovec_free(struct io_async_rw *rw) +{ + if (rw->free_iovec) { + kfree(rw->free_iovec); + rw->free_iovec = NULL; + } +} + +static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_async_rw *rw = req->async_data; + + if (issue_flags & IO_URING_F_UNLOCKED) { + io_rw_iovec_free(rw); + return; + } + if (io_alloc_cache_put(&req->ctx->rw_cache, &rw->cache)) { + req->async_data = NULL; + req->flags &= ~REQ_F_ASYNC_DATA; + } +} + +static void io_req_rw_cleanup(struct io_kiocb *req, unsigned int issue_flags) +{ + req->flags &= ~REQ_F_NEED_CLEANUP; + io_rw_recycle(req, issue_flags); +} + +static int io_rw_alloc_async(struct io_kiocb *req) +{ + struct io_ring_ctx *ctx = req->ctx; + struct io_cache_entry *entry; + struct io_async_rw *rw; + + entry = io_alloc_cache_get(&ctx->rw_cache); + if (entry) { + rw = container_of(entry, struct io_async_rw, cache); + req->flags |= REQ_F_ASYNC_DATA; + req->async_data = rw; + goto done; + } + + if (!io_alloc_async_data(req)) { + rw = req->async_data; +done: + rw->free_iovec = NULL; + rw->bytes_done = 0; + return 0; + } + + return -ENOMEM; +} + +static int io_prep_rw_setup(struct io_kiocb *req, int ddir, bool do_import) +{ + struct io_async_rw *rw; + int ret; + + if (io_rw_alloc_async(req)) + return -ENOMEM; + + if (!do_import || io_do_buffer_select(req)) + return 0; + + rw = req->async_data; + ret = io_import_iovec(ddir, req, rw, 0); + if (unlikely(ret < 0)) + return ret; + + iov_iter_save_state(&rw->s.iter, &rw->s.iter_state); + return 0; +} + +static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, + int ddir, bool do_import) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); unsigned ioprio; @@ -100,34 +217,58 @@ int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) rw->addr = READ_ONCE(sqe->addr); rw->len = READ_ONCE(sqe->len); rw->flags = READ_ONCE(sqe->rw_flags); - return 0; + return io_prep_rw_setup(req, ddir, do_import); } -int io_prep_rwv(struct io_kiocb *req, const struct io_uring_sqe *sqe) +int io_prep_read(struct io_kiocb *req, const struct io_uring_sqe *sqe) { + return io_prep_rw(req, sqe, ITER_DEST, true); +} + +int io_prep_write(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + return io_prep_rw(req, sqe, ITER_SOURCE, true); +} + +static int io_prep_rwv(struct io_kiocb *req, const struct io_uring_sqe *sqe, + int ddir) +{ + const bool do_import = !(req->flags & REQ_F_BUFFER_SELECT); int ret; - ret = io_prep_rw(req, sqe); + ret = io_prep_rw(req, sqe, ddir, do_import); if (unlikely(ret)) return ret; + if (do_import) + return 0; /* * Have to do this validation here, as this is in io_read() rw->len * might have chanaged due to buffer selection */ - if (req->flags & REQ_F_BUFFER_SELECT) - return io_iov_buffer_select_prep(req); + return io_iov_buffer_select_prep(req); +} - return 0; +int io_prep_readv(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + return io_prep_rwv(req, sqe, ITER_DEST); +} + +int io_prep_writev(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + return io_prep_rwv(req, sqe, ITER_SOURCE); } -int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe, + int ddir) { + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); struct io_ring_ctx *ctx = req->ctx; + struct io_async_rw *io; u16 index; int ret; - ret = io_prep_rw(req, sqe); + ret = io_prep_rw(req, sqe, ddir, false); if (unlikely(ret)) return ret; @@ -136,7 +277,21 @@ int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) index = array_index_nospec(req->buf_index, ctx->nr_user_bufs); req->imu = ctx->user_bufs[index]; io_req_set_rsrc_node(req, ctx, 0); - return 0; + + io = req->async_data; + ret = io_import_fixed(ddir, &io->s.iter, req->imu, rw->addr, rw->len); + iov_iter_save_state(&io->s.iter, &io->s.iter_state); + return ret; +} + +int io_prep_read_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + return io_prep_rw_fixed(req, sqe, ITER_DEST); +} + +int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + return io_prep_rw_fixed(req, sqe, ITER_SOURCE); } /* @@ -152,7 +307,7 @@ int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (!(req->flags & REQ_F_BUFFER_SELECT)) return -EINVAL; - ret = io_prep_rw(req, sqe); + ret = io_prep_rw(req, sqe, ITER_DEST, false); if (unlikely(ret)) return ret; @@ -165,9 +320,7 @@ int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) void io_readv_writev_cleanup(struct io_kiocb *req) { - struct io_async_rw *io = req->async_data; - - kfree(io->free_iovec); + io_rw_iovec_free(req->async_data); } static inline loff_t *io_kiocb_update_pos(struct io_kiocb *req) @@ -188,14 +341,11 @@ static inline loff_t *io_kiocb_update_pos(struct io_kiocb *req) } #ifdef CONFIG_BLOCK -static bool io_resubmit_prep(struct io_kiocb *req) +static void io_resubmit_prep(struct io_kiocb *req) { struct io_async_rw *io = req->async_data; - if (!req_has_async_data(req)) - return !io_req_prep_async(req); iov_iter_restore(&io->s.iter, &io->s.iter_state); - return true; } static bool io_rw_should_reissue(struct io_kiocb *req) @@ -224,9 +374,8 @@ static bool io_rw_should_reissue(struct io_kiocb *req) return true; } #else -static bool io_resubmit_prep(struct io_kiocb *req) +static void io_resubmit_prep(struct io_kiocb *req) { - return false; } static bool io_rw_should_reissue(struct io_kiocb *req) { @@ -308,6 +457,7 @@ void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts) if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) req->cqe.flags |= io_put_kbuf(req, 0); + io_req_rw_cleanup(req, 0); io_req_task_complete(req, ts); } @@ -388,6 +538,7 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret, io_req_io_end(req); io_req_set_res(req, final_ret, io_put_kbuf(req, issue_flags)); + io_req_rw_cleanup(req, issue_flags); return IOU_OK; } } else { @@ -396,71 +547,12 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret, if (req->flags & REQ_F_REISSUE) { req->flags &= ~REQ_F_REISSUE; - if (io_resubmit_prep(req)) - return -EAGAIN; - else - io_req_task_queue_fail(req, final_ret); + io_resubmit_prep(req); + return -EAGAIN; } return IOU_ISSUE_SKIP_COMPLETE; } -static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, - struct io_rw_state *s, - unsigned int issue_flags) -{ - struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); - struct iov_iter *iter = &s->iter; - u8 opcode = req->opcode; - struct iovec *iovec; - void __user *buf; - size_t sqe_len; - ssize_t ret; - - if (opcode == IORING_OP_READ_FIXED || opcode == IORING_OP_WRITE_FIXED) { - ret = io_import_fixed(ddir, iter, req->imu, rw->addr, rw->len); - if (ret) - return ERR_PTR(ret); - return NULL; - } - - buf = u64_to_user_ptr(rw->addr); - sqe_len = rw->len; - - if (!io_issue_defs[opcode].vectored || req->flags & REQ_F_BUFFER_SELECT) { - if (io_do_buffer_select(req)) { - buf = io_buffer_select(req, &sqe_len, issue_flags); - if (!buf) - return ERR_PTR(-ENOBUFS); - rw->addr = (unsigned long) buf; - rw->len = sqe_len; - } - - ret = import_ubuf(ddir, buf, sqe_len, iter); - if (ret) - return ERR_PTR(ret); - return NULL; - } - - iovec = s->fast_iov; - ret = __import_iovec(ddir, buf, sqe_len, UIO_FASTIOV, &iovec, iter, - req->ctx->compat); - if (unlikely(ret < 0)) - return ERR_PTR(ret); - return iovec; -} - -static inline int io_import_iovec(int rw, struct io_kiocb *req, - struct iovec **iovec, struct io_rw_state *s, - unsigned int issue_flags) -{ - *iovec = __io_import_iovec(rw, req, s, issue_flags); - if (IS_ERR(*iovec)) - return PTR_ERR(*iovec); - - iov_iter_save_state(&s->iter, &s->iter_state); - return 0; -} - static inline loff_t *io_kiocb_ppos(struct kiocb *kiocb) { return (kiocb->ki_filp->f_mode & FMODE_STREAM) ? NULL : &kiocb->ki_pos; @@ -532,89 +624,6 @@ static ssize_t loop_rw_iter(int ddir, struct io_rw *rw, struct iov_iter *iter) return ret; } -static void io_req_map_rw(struct io_kiocb *req, const struct iovec *iovec, - const struct iovec *fast_iov, struct iov_iter *iter) -{ - struct io_async_rw *io = req->async_data; - - memcpy(&io->s.iter, iter, sizeof(*iter)); - io->free_iovec = iovec; - io->bytes_done = 0; - /* can only be fixed buffers, no need to do anything */ - if (iov_iter_is_bvec(iter) || iter_is_ubuf(iter)) - return; - if (!iovec) { - unsigned iov_off = 0; - - io->s.iter.__iov = io->s.fast_iov; - if (iter->__iov != fast_iov) { - iov_off = iter_iov(iter) - fast_iov; - io->s.iter.__iov += iov_off; - } - if (io->s.fast_iov != fast_iov) - memcpy(io->s.fast_iov + iov_off, fast_iov + iov_off, - sizeof(struct iovec) * iter->nr_segs); - } else { - req->flags |= REQ_F_NEED_CLEANUP; - } -} - -static int io_setup_async_rw(struct io_kiocb *req, const struct iovec *iovec, - struct io_rw_state *s, bool force) -{ - if (!force && !io_cold_defs[req->opcode].prep_async) - return 0; - /* opcode type doesn't need async data */ - if (!io_cold_defs[req->opcode].async_size) - return 0; - if (!req_has_async_data(req)) { - struct io_async_rw *iorw; - - if (io_alloc_async_data(req)) { - kfree(iovec); - return -ENOMEM; - } - - io_req_map_rw(req, iovec, s->fast_iov, &s->iter); - iorw = req->async_data; - /* we've copied and mapped the iter, ensure state is saved */ - iov_iter_save_state(&iorw->s.iter, &iorw->s.iter_state); - } - return 0; -} - -static inline int io_rw_prep_async(struct io_kiocb *req, int rw) -{ - struct io_async_rw *iorw = req->async_data; - struct iovec *iov; - int ret; - - iorw->bytes_done = 0; - iorw->free_iovec = NULL; - - /* submission path, ->uring_lock should already be taken */ - ret = io_import_iovec(rw, req, &iov, &iorw->s, 0); - if (unlikely(ret < 0)) - return ret; - - if (iov) { - iorw->free_iovec = iov; - req->flags |= REQ_F_NEED_CLEANUP; - } - - return 0; -} - -int io_readv_prep_async(struct io_kiocb *req) -{ - return io_rw_prep_async(req, ITER_DEST); -} - -int io_writev_prep_async(struct io_kiocb *req) -{ - return io_rw_prep_async(req, ITER_SOURCE); -} - /* * This is our waitqueue callback handler, registered through __folio_lock_async() * when we initially tried to do the IO with the iocb armed our waitqueue. @@ -754,54 +763,28 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode) static int __io_read(struct io_kiocb *req, unsigned int issue_flags) { + bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); - struct io_rw_state __s, *s = &__s; - struct iovec *iovec; + struct io_async_rw *io = req->async_data; struct kiocb *kiocb = &rw->kiocb; - bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; - struct io_async_rw *io; - ssize_t ret, ret2; + ssize_t ret; loff_t *ppos; - if (!req_has_async_data(req)) { - ret = io_import_iovec(ITER_DEST, req, &iovec, s, issue_flags); + if (io_do_buffer_select(req)) { + ret = io_import_iovec(ITER_DEST, req, io, issue_flags); if (unlikely(ret < 0)) return ret; - } else { - io = req->async_data; - s = &io->s; - - /* - * Safe and required to re-import if we're using provided - * buffers, as we dropped the selected one before retry. - */ - if (io_do_buffer_select(req)) { - ret = io_import_iovec(ITER_DEST, req, &iovec, s, issue_flags); - if (unlikely(ret < 0)) - return ret; - } - - /* - * We come here from an earlier attempt, restore our state to - * match in case it doesn't. It's cheap enough that we don't - * need to make this conditional. - */ - iov_iter_restore(&s->iter, &s->iter_state); - iovec = NULL; } + ret = io_rw_init_file(req, FMODE_READ); - if (unlikely(ret)) { - kfree(iovec); + if (unlikely(ret)) return ret; - } - req->cqe.res = iov_iter_count(&s->iter); + req->cqe.res = iov_iter_count(&io->s.iter); if (force_nonblock) { /* If the file doesn't support async, just async punt */ - if (unlikely(!io_file_supports_nowait(req))) { - ret = io_setup_async_rw(req, iovec, s, true); - return ret ?: -EAGAIN; - } + if (unlikely(!io_file_supports_nowait(req))) + return -EAGAIN; kiocb->ki_flags |= IOCB_NOWAIT; } else { /* Ensure we clear previously set non-block flag */ @@ -811,20 +794,15 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ppos = io_kiocb_update_pos(req); ret = rw_verify_area(READ, req->file, ppos, req->cqe.res); - if (unlikely(ret)) { - kfree(iovec); + if (unlikely(ret)) return ret; - } - ret = io_iter_do_read(rw, &s->iter); + ret = io_iter_do_read(rw, &io->s.iter); if (ret == -EAGAIN || (req->flags & REQ_F_REISSUE)) { req->flags &= ~REQ_F_REISSUE; - /* - * If we can poll, just do that. For a vectored read, we'll - * need to copy state first. - */ - if (io_file_can_poll(req) && !io_issue_defs[req->opcode].vectored) + /* If we can poll, just do that. */ + if (io_file_can_poll(req)) return -EAGAIN; /* IOPOLL retry should happen for io-wq threads */ if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) @@ -834,8 +812,6 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) goto done; ret = 0; } else if (ret == -EIOCBQUEUED) { - if (iovec) - kfree(iovec); return IOU_ISSUE_SKIP_COMPLETE; } else if (ret == req->cqe.res || ret <= 0 || !force_nonblock || (req->flags & REQ_F_NOWAIT) || !need_complete_io(req)) { @@ -848,21 +824,7 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) * untouched in case of error. Restore it and we'll advance it * manually if we need to. */ - iov_iter_restore(&s->iter, &s->iter_state); - - ret2 = io_setup_async_rw(req, iovec, s, true); - iovec = NULL; - if (ret2) { - ret = ret > 0 ? ret : ret2; - goto done; - } - - io = req->async_data; - s = &io->s; - /* - * Now use our persistent iterator and state, if we aren't already. - * We've restored and mapped the iter to match. - */ + iov_iter_restore(&io->s.iter, &io->s.iter_state); do { /* @@ -870,11 +832,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) * above or inside this loop. Advance the iter by the bytes * that were consumed. */ - iov_iter_advance(&s->iter, ret); - if (!iov_iter_count(&s->iter)) + iov_iter_advance(&io->s.iter, ret); + if (!iov_iter_count(&io->s.iter)) break; io->bytes_done += ret; - iov_iter_save_state(&s->iter, &s->iter_state); + iov_iter_save_state(&io->s.iter, &io->s.iter_state); /* if we can retry, do so with the callbacks armed */ if (!io_rw_should_retry(req)) { @@ -882,24 +844,22 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) return -EAGAIN; } - req->cqe.res = iov_iter_count(&s->iter); + req->cqe.res = iov_iter_count(&io->s.iter); /* * Now retry read with the IOCB_WAITQ parts set in the iocb. If * we get -EIOCBQUEUED, then we'll get a notification when the * desired page gets unlocked. We can also get a partial read * here, and if we do, then just retry at the new offset. */ - ret = io_iter_do_read(rw, &s->iter); + ret = io_iter_do_read(rw, &io->s.iter); if (ret == -EIOCBQUEUED) return IOU_ISSUE_SKIP_COMPLETE; /* we got some bytes, but not all. retry. */ kiocb->ki_flags &= ~IOCB_WAITQ; - iov_iter_restore(&s->iter, &s->iter_state); + iov_iter_restore(&io->s.iter, &io->s.iter_state); } while (ret > 0); done: /* it's faster to check here then delegate to kfree */ - if (iovec) - kfree(iovec); return ret; } @@ -908,8 +868,9 @@ int io_read(struct io_kiocb *req, unsigned int issue_flags) int ret; ret = __io_read(req, issue_flags); - if (ret >= 0) - return kiocb_done(req, ret, issue_flags); + if (ret >= 0) { + ret = kiocb_done(req, ret, issue_flags); + } return ret; } @@ -974,6 +935,7 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) * multishot request, hitting overflow will terminate it. */ io_req_set_res(req, ret, cflags); + io_req_rw_cleanup(req, issue_flags); if (issue_flags & IO_URING_F_MULTISHOT) return IOU_STOP_MULTISHOT; return IOU_OK; @@ -981,42 +943,28 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) int io_write(struct io_kiocb *req, unsigned int issue_flags) { + bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); - struct io_rw_state __s, *s = &__s; - struct iovec *iovec; + struct io_async_rw *io = req->async_data; struct kiocb *kiocb = &rw->kiocb; - bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; ssize_t ret, ret2; loff_t *ppos; - if (!req_has_async_data(req)) { - ret = io_import_iovec(ITER_SOURCE, req, &iovec, s, issue_flags); - if (unlikely(ret < 0)) - return ret; - } else { - struct io_async_rw *io = req->async_data; - - s = &io->s; - iov_iter_restore(&s->iter, &s->iter_state); - iovec = NULL; - } ret = io_rw_init_file(req, FMODE_WRITE); - if (unlikely(ret)) { - kfree(iovec); + if (unlikely(ret)) return ret; - } - req->cqe.res = iov_iter_count(&s->iter); + req->cqe.res = iov_iter_count(&io->s.iter); if (force_nonblock) { /* If the file doesn't support async, just async punt */ if (unlikely(!io_file_supports_nowait(req))) - goto copy_iov; + goto ret_eagain; /* File path supports NOWAIT for non-direct_IO only for block devices. */ if (!(kiocb->ki_flags & IOCB_DIRECT) && !(kiocb->ki_filp->f_mode & FMODE_BUF_WASYNC) && (req->flags & REQ_F_ISREG)) - goto copy_iov; + goto ret_eagain; kiocb->ki_flags |= IOCB_NOWAIT; } else { @@ -1027,19 +975,17 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ppos = io_kiocb_update_pos(req); ret = rw_verify_area(WRITE, req->file, ppos, req->cqe.res); - if (unlikely(ret)) { - kfree(iovec); + if (unlikely(ret)) return ret; - } if (req->flags & REQ_F_ISREG) kiocb_start_write(kiocb); kiocb->ki_flags |= IOCB_WRITE; if (likely(req->file->f_op->write_iter)) - ret2 = call_write_iter(req->file, kiocb, &s->iter); + ret2 = call_write_iter(req->file, kiocb, &io->s.iter); else if (req->file->f_op->write) - ret2 = loop_rw_iter(WRITE, rw, &s->iter); + ret2 = loop_rw_iter(WRITE, rw, &io->s.iter); else ret2 = -EINVAL; @@ -1060,7 +1006,7 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) if (!force_nonblock || ret2 != -EAGAIN) { /* IOPOLL retry should happen for io-wq threads */ if (ret2 == -EAGAIN && (req->ctx->flags & IORING_SETUP_IOPOLL)) - goto copy_iov; + goto ret_eagain; if (ret2 != req->cqe.res && ret2 >= 0 && need_complete_io(req)) { struct io_async_rw *io; @@ -1073,33 +1019,22 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) * in the worker. Also update bytes_done to account for * the bytes already written. */ - iov_iter_save_state(&s->iter, &s->iter_state); - ret = io_setup_async_rw(req, iovec, s, true); - - io = req->async_data; - if (io) - io->bytes_done += ret2; + iov_iter_save_state(&io->s.iter, &io->s.iter_state); + io->bytes_done += ret2; if (kiocb->ki_flags & IOCB_WRITE) io_req_end_write(req); - return ret ? ret : -EAGAIN; + return -EAGAIN; } done: ret = kiocb_done(req, ret2, issue_flags); } else { -copy_iov: - iov_iter_restore(&s->iter, &s->iter_state); - ret = io_setup_async_rw(req, iovec, s, false); - if (!ret) { - if (kiocb->ki_flags & IOCB_WRITE) - io_req_end_write(req); - return -EAGAIN; - } - return ret; +ret_eagain: + iov_iter_restore(&io->s.iter, &io->s.iter_state); + if (kiocb->ki_flags & IOCB_WRITE) + io_req_end_write(req); + return -EAGAIN; } - /* it's reportedly faster than delegating the null check to kfree() */ - if (iovec) - kfree(iovec); return ret; } @@ -1174,6 +1109,8 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) break; nr_events++; req->cqe.flags = io_put_kbuf(req, 0); + if (req->opcode != IORING_OP_URING_CMD) + io_req_rw_cleanup(req, 0); } if (unlikely(!nr_events)) return 0; @@ -1187,3 +1124,11 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) __io_submit_flush_completions(ctx); return nr_events; } + +void io_rw_cache_free(struct io_cache_entry *entry) +{ + struct io_async_rw *rw; + + rw = container_of(entry, struct io_async_rw, cache); + kfree(rw); +} diff --git a/io_uring/rw.h b/io_uring/rw.h index f9e89b4fe4da..f7905070d10b 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -9,21 +9,26 @@ struct io_rw_state { }; struct io_async_rw { + union { + size_t bytes_done; + struct io_cache_entry cache; + }; struct io_rw_state s; - const struct iovec *free_iovec; - size_t bytes_done; + struct iovec *free_iovec; struct wait_page_queue wpq; }; -int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe); -int io_prep_rwv(struct io_kiocb *req, const struct io_uring_sqe *sqe); -int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_read_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_write_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_readv(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_writev(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_read(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_prep_write(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_read(struct io_kiocb *req, unsigned int issue_flags); -int io_readv_prep_async(struct io_kiocb *req); int io_write(struct io_kiocb *req, unsigned int issue_flags); -int io_writev_prep_async(struct io_kiocb *req); void io_readv_writev_cleanup(struct io_kiocb *req); void io_rw_fail(struct io_kiocb *req); void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts); int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags); +void io_rw_cache_free(struct io_cache_entry *entry); From patchwork Wed Mar 20 01:17:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597174 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5C907464 for ; Wed, 20 Mar 2024 01:23:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897798; cv=none; b=IjKQ1IPpgbb5eJeH6Xu9ntY8ocNF27+x6cyI652AYUtXK6AOqfhT3UdS8qY654wK3tJUULDiG9SPAoaeRzuzsYEkcwWuYMW47qTTlqRsFSvLmriJlwQjBIlQM/41X7062MMcLXiJIBdD1qDiP0MLfWmrDc/b1Ngn4KoEeCsUx50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897798; c=relaxed/simple; bh=+WgeA4b9DCM/hDBAIePgIIKsF1SEwqqNZprjMVCttqc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=czl2L7umlmKXKKBfajBPsbPzjsaTGIAF8hsfDsTsYrAFXbnTYPaIjze0nSOgH3fj4jBD3oKkxl9NpgkBiWrbX55/3FkkEPaqzQgKrl9fpbCDqALM3BuYteMgq7Lmwxn/ovnvZ3Jh4ANzI0BY5uNN5rIprLnBzjD1nWyAmsphPGk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=vyXi3EiI; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="vyXi3EiI" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6e6c38be762so941747b3a.1 for ; Tue, 19 Mar 2024 18:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897796; x=1711502596; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=746DtLrVdQ8v1/CM9YQD2PftzJISWXc3wAsFo4NIgVk=; b=vyXi3EiIXblFp4V2q2gvYbJ30rSAkq8x0EBGidekTD3aOwuSgpyaH/GDddInl64bM5 dmSBx6EIY+klQT40w5DWA6F0Ll+RzIMxxNM7YzKJtaqCXOpO1V+Hyl+eYxA1EYVW/WrI /dGDh5RmK13pZW4ZsGzl/hw+Y29aiivcBLt8swrX+lzQLKjDVnomQDboNUGd3xyZO5x/ Vcgx/l9Ek9db2ZBGEiuvs+D5iHkWqG00IQrNXorwEVB6VVFFkAyX+OEqWwc3oj8ldBWB npfceT8mtgvTAnRYKfOp11AccPMcICEh4LvWGWB2D9IKLcwIS7jTYN4QPuY+TajfmU2+ 036Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897796; x=1711502596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=746DtLrVdQ8v1/CM9YQD2PftzJISWXc3wAsFo4NIgVk=; b=DqBsUyRTgMzV2TvwnJ3Q78gOxm5Gxr+cOnBw6dEx1Ex8uBIB42TZAmuXfrCwDL4eAQ mhmV57XuWOCoogeV5eLMohjH3IS5A4ZaP/NUWjVFuDyrmjx5pjp1mcOiQ7fN413RP7Kd qjZBE9VR/T3EMYCrm3U/5i2ynYTn7aCRefW50R+kGxLRwknCogGPqWQJPvR7LFGexKic VraZF0gDo12SuH7TEoPXbgwX2G/gO5ojWeUdZ5QfGHGxa3BEUkFqkLZ+Qq3/Wogp1alL +bWgWMQ8F/2QL2xxBwx7CjFwGaoJTnMINiJZoAOw3tQE4yVYmAlWuRl2X7Bcyg9i7oJm EinA== X-Gm-Message-State: AOJu0Yy2MZPeVw4MHdA0phXJoW2VxOpGdVJODWJovhXgmldf0nXspzhs MSs6Fbn81aQDpIkkj06BX7rvpjOfxgEhyB0VhKOzTm8jLyyeEdOu4PL+GPRnEvZq+jYgkiX9dgt P X-Google-Smtp-Source: AGHT+IGRWiB3fvdEcbj8gox7/5nJ170+AoDmP1aaT7rlB/Owlg2mr6Wop/Z3Z9VedrTJReae85a7rQ== X-Received: by 2002:a05:6a00:9382:b0:6e6:864d:767 with SMTP id ka2-20020a056a00938200b006e6864d0767mr4279103pfb.3.1710897795733; Tue, 19 Mar 2024 18:23:15 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:15 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 11/15] io_uring: get rid of struct io_rw_state Date: Tue, 19 Mar 2024 19:17:39 -0600 Message-ID: <20240320012251.1120361-12-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A separate state struct is not needed anymore, just fold it in with io_async_rw. Signed-off-by: Jens Axboe --- io_uring/rw.c | 45 +++++++++++++++++++++++---------------------- io_uring/rw.h | 7 +++---- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index 054e6f381460..f26e1dd5acaf 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -96,12 +96,12 @@ static int __io_import_iovec(int ddir, struct io_kiocb *req, rw->len = sqe_len; } - return import_ubuf(ddir, buf, sqe_len, &io->s.iter); + return import_ubuf(ddir, buf, sqe_len, &io->iter); } - io->free_iovec = io->s.fast_iov; + io->free_iovec = io->fast_iov; return __import_iovec(ddir, buf, sqe_len, UIO_FASTIOV, &io->free_iovec, - &io->s.iter, req->ctx->compat); + &io->iter, req->ctx->compat); } static inline int io_import_iovec(int rw, struct io_kiocb *req, @@ -114,7 +114,7 @@ static inline int io_import_iovec(int rw, struct io_kiocb *req, if (unlikely(ret < 0)) return ret; - iov_iter_save_state(&io->s.iter, &io->s.iter_state); + iov_iter_save_state(&io->iter, &io->iter_state); return 0; } @@ -187,7 +187,7 @@ static int io_prep_rw_setup(struct io_kiocb *req, int ddir, bool do_import) if (unlikely(ret < 0)) return ret; - iov_iter_save_state(&rw->s.iter, &rw->s.iter_state); + iov_iter_save_state(&rw->iter, &rw->iter_state); return 0; } @@ -279,8 +279,8 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe io_req_set_rsrc_node(req, ctx, 0); io = req->async_data; - ret = io_import_fixed(ddir, &io->s.iter, req->imu, rw->addr, rw->len); - iov_iter_save_state(&io->s.iter, &io->s.iter_state); + ret = io_import_fixed(ddir, &io->iter, req->imu, rw->addr, rw->len); + iov_iter_save_state(&io->iter, &io->iter_state); return ret; } @@ -345,7 +345,7 @@ static void io_resubmit_prep(struct io_kiocb *req) { struct io_async_rw *io = req->async_data; - iov_iter_restore(&io->s.iter, &io->s.iter_state); + iov_iter_restore(&io->iter, &io->iter_state); } static bool io_rw_should_reissue(struct io_kiocb *req) @@ -779,7 +779,7 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) ret = io_rw_init_file(req, FMODE_READ); if (unlikely(ret)) return ret; - req->cqe.res = iov_iter_count(&io->s.iter); + req->cqe.res = iov_iter_count(&io->iter); if (force_nonblock) { /* If the file doesn't support async, just async punt */ @@ -797,7 +797,7 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret)) return ret; - ret = io_iter_do_read(rw, &io->s.iter); + ret = io_iter_do_read(rw, &io->iter); if (ret == -EAGAIN || (req->flags & REQ_F_REISSUE)) { req->flags &= ~REQ_F_REISSUE; @@ -824,7 +824,7 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) * untouched in case of error. Restore it and we'll advance it * manually if we need to. */ - iov_iter_restore(&io->s.iter, &io->s.iter_state); + iov_iter_restore(&io->iter, &io->iter_state); do { /* @@ -832,11 +832,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) * above or inside this loop. Advance the iter by the bytes * that were consumed. */ - iov_iter_advance(&io->s.iter, ret); - if (!iov_iter_count(&io->s.iter)) + iov_iter_advance(&io->iter, ret); + if (!iov_iter_count(&io->iter)) break; io->bytes_done += ret; - iov_iter_save_state(&io->s.iter, &io->s.iter_state); + iov_iter_save_state(&io->iter, &io->iter_state); /* if we can retry, do so with the callbacks armed */ if (!io_rw_should_retry(req)) { @@ -844,19 +844,19 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) return -EAGAIN; } - req->cqe.res = iov_iter_count(&io->s.iter); + req->cqe.res = iov_iter_count(&io->iter); /* * Now retry read with the IOCB_WAITQ parts set in the iocb. If * we get -EIOCBQUEUED, then we'll get a notification when the * desired page gets unlocked. We can also get a partial read * here, and if we do, then just retry at the new offset. */ - ret = io_iter_do_read(rw, &io->s.iter); + ret = io_iter_do_read(rw, &io->iter); if (ret == -EIOCBQUEUED) return IOU_ISSUE_SKIP_COMPLETE; /* we got some bytes, but not all. retry. */ kiocb->ki_flags &= ~IOCB_WAITQ; - iov_iter_restore(&io->s.iter, &io->s.iter_state); + iov_iter_restore(&io->iter, &io->iter_state); } while (ret > 0); done: /* it's faster to check here then delegate to kfree */ @@ -953,7 +953,7 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ret = io_rw_init_file(req, FMODE_WRITE); if (unlikely(ret)) return ret; - req->cqe.res = iov_iter_count(&io->s.iter); + req->cqe.res = iov_iter_count(&io->iter); if (force_nonblock) { /* If the file doesn't support async, just async punt */ @@ -983,9 +983,9 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) kiocb->ki_flags |= IOCB_WRITE; if (likely(req->file->f_op->write_iter)) - ret2 = call_write_iter(req->file, kiocb, &io->s.iter); + ret2 = call_write_iter(req->file, kiocb, &io->iter); else if (req->file->f_op->write) - ret2 = loop_rw_iter(WRITE, rw, &io->s.iter); + ret2 = loop_rw_iter(WRITE, rw, &io->iter); else ret2 = -EINVAL; @@ -1019,7 +1019,7 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) * in the worker. Also update bytes_done to account for * the bytes already written. */ - iov_iter_save_state(&io->s.iter, &io->s.iter_state); + iov_iter_save_state(&io->iter, &io->iter_state); io->bytes_done += ret2; if (kiocb->ki_flags & IOCB_WRITE) @@ -1030,7 +1030,7 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) ret = kiocb_done(req, ret2, issue_flags); } else { ret_eagain: - iov_iter_restore(&io->s.iter, &io->s.iter_state); + iov_iter_restore(&io->iter, &io->iter_state); if (kiocb->ki_flags & IOCB_WRITE) io_req_end_write(req); return -EAGAIN; @@ -1130,5 +1130,6 @@ void io_rw_cache_free(struct io_cache_entry *entry) struct io_async_rw *rw; rw = container_of(entry, struct io_async_rw, cache); + kfree(rw->free_iovec); kfree(rw); } diff --git a/io_uring/rw.h b/io_uring/rw.h index f7905070d10b..56fb1703dc5a 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -3,9 +3,6 @@ #include struct io_rw_state { - struct iov_iter iter; - struct iov_iter_state iter_state; - struct iovec fast_iov[UIO_FASTIOV]; }; struct io_async_rw { @@ -13,7 +10,9 @@ struct io_async_rw { size_t bytes_done; struct io_cache_entry cache; }; - struct io_rw_state s; + struct iov_iter iter; + struct iov_iter_state iter_state; + struct iovec fast_iov[UIO_FASTIOV]; struct iovec *free_iovec; struct wait_page_queue wpq; }; From patchwork Wed Mar 20 01:17:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597176 Received: from mail-oo1-f41.google.com (mail-oo1-f41.google.com [209.85.161.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 196862582 for ; Wed, 20 Mar 2024 01:23:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897801; cv=none; b=X82cYTbTWzzTxruPT45kW0130HF3oupv6i+fupz8ckiOVZ8CQPltct0rLa2hkmkai1I9cA91SKgo1bHEgBveiV4puNPOwDZeYljLD1zVZFbnmpZt6jrFFzO3UgBABqwTc4kZDDNKUnORwjtmvX512pIGgknhPHx/xqAmelGdg2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897801; c=relaxed/simple; bh=CC220oEvVsaltoIJxMcIegTRorS7Geg9OJonzp7wZq4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VW7oi1ot3z++UKRYHHpo6EPZ0a7VWpho2mLqY3OcxnQbvhAgIIq6pvDa6rBV4XeqOPak/atBEHfAbEL29gSdKrCJfPYpKGfN9Jd6raB5GNCPh0+kDC1Hsg3GHrKpgWl0MSa8iaz1+4KlGENkyQ6ygo1AJytFodRZqfTb3FWzJEw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=FkpqG+vZ; arc=none smtp.client-ip=209.85.161.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="FkpqG+vZ" Received: by mail-oo1-f41.google.com with SMTP id 006d021491bc7-5a4e0859b65so177425eaf.0 for ; Tue, 19 Mar 2024 18:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897799; x=1711502599; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EW4Jpwuf3NZTpANDCQrLOB/VQc3CBX3tLAnEuFHW/4c=; b=FkpqG+vZwTPEnpFERoo4xpEDiiCRadIWAKNT+FVfIp5t3AVMCHy+uExVXtjCYxX7gz lGVPXTOeKSRulE2CpaH313xgAZN1utSOgZK1mRjv281SfJ6OD63Qja2vE9mQG7anPz0f CIlk1K7l32sch5vEUgVrnaHpsb7Wt8LqoLtwXNFFmeo3Ut776VQXvTXzR+WT/4Hswccm eYuJx1lQhthyiIU9kJ05DbF8XMugw7DTcb1ltyznImyMBqs09WIwu/IAwSkwH1Qemcm4 uAG7CIqaBIwfjuOH6BssVIF0ltr0luDF7NqNwdEDb7Bf36hz48wfUYXc1nRiN7+bZjou bpqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897799; x=1711502599; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EW4Jpwuf3NZTpANDCQrLOB/VQc3CBX3tLAnEuFHW/4c=; b=mheLr1Od69iVCQoUV7qElk0EbR0T3MepOFKOsAdahCCaDsc+itUIkkJ8YoBEtaDgs0 sfYyPtiEjGnYZlNX0mfwsWH6VvsvSngDUzcVnwHFewxay6R61HsUy33u3M+izdXii9/X rH4p2CGYtB8uPbxnsQOkZV/pUj1jmkwkzqR2sE7ChLZkRTjLAlnURZgLpnZLBXux5K10 OEE+FUbE7SXqS4GPe8xhJ2ReBcZ7Xq9BElsaPdjTmMWNdNWxjZK88OeEf9XiP/yPCTM7 x4s0GzKQzPzQPQzcNwE9Op0fK6s7RN7dxuzwVYkhnbBoJqTD3OcOYFakD8c/HbHVmH1V /g+Q== X-Gm-Message-State: AOJu0YySwUNaia4fwpDMPEepRLuEE5UVvdlVBZ55YvvW4w3T0Uf3J3vR F4On4Tk/JFpaAIedo4WXAzD3824DfC4cVI+QitWKHrEu15Cl8hNigPKMOuOgRPtVfUq80AbUY52 n X-Google-Smtp-Source: AGHT+IH/k4OBrhSQSyZHfzzDnyJI3SiO7+gT93/MPAF23CueuN15BF8eoATFs15uARa7eZGvLA6+BQ== X-Received: by 2002:a05:6358:6f9d:b0:17e:bc8b:fbac with SMTP id s29-20020a0563586f9d00b0017ebc8bfbacmr4152526rwn.3.1710897798831; Tue, 19 Mar 2024 18:23:18 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:16 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 12/15] io_uring/rw: add iovec recycling Date: Tue, 19 Mar 2024 19:17:40 -0600 Message-ID: <20240320012251.1120361-13-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Let the io_async_rw hold on to the iovec and reuse it, rather than always allocate and free them. While doing so, shrink io_async_rw by getting rid of the bigger embedded fast iovec. Since iovecs are being recycled now, shrink it from 8 to 1. This reduces the io_async_rw size from 264 to 160 bytes, a 40% reduction. Includes KASAN tracking as well, if that is turned on. Signed-off-by: Jens Axboe --- io_uring/rw.c | 38 ++++++++++++++++++++++++++++++++------ io_uring/rw.h | 3 ++- 2 files changed, 34 insertions(+), 7 deletions(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index f26e1dd5acaf..71ef417373c2 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -81,7 +81,9 @@ static int __io_import_iovec(int ddir, struct io_kiocb *req, { const struct io_issue_def *def = &io_issue_defs[req->opcode]; struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + struct iovec *iov; void __user *buf; + int nr_segs, ret; size_t sqe_len; buf = u64_to_user_ptr(rw->addr); @@ -99,9 +101,23 @@ static int __io_import_iovec(int ddir, struct io_kiocb *req, return import_ubuf(ddir, buf, sqe_len, &io->iter); } - io->free_iovec = io->fast_iov; - return __import_iovec(ddir, buf, sqe_len, UIO_FASTIOV, &io->free_iovec, - &io->iter, req->ctx->compat); + if (io->free_iovec) { + nr_segs = io->free_iov_nr; + iov = io->free_iovec; + } else { + iov = &io->fast_iov; + nr_segs = 1; + } + ret = __import_iovec(ddir, buf, sqe_len, nr_segs, &iov, &io->iter, + req->ctx->compat); + if (unlikely(ret < 0)) + return ret; + if (iov) { + io->free_iov_nr = io->iter.nr_segs; + kfree(io->free_iovec); + io->free_iovec = iov; + } + return 0; } static inline int io_import_iovec(int rw, struct io_kiocb *req, @@ -122,6 +138,7 @@ static void io_rw_iovec_free(struct io_async_rw *rw) { if (rw->free_iovec) { kfree(rw->free_iovec); + rw->free_iov_nr = 0; rw->free_iovec = NULL; } } @@ -134,6 +151,8 @@ static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags) io_rw_iovec_free(rw); return; } + if (rw->free_iovec) + kasan_mempool_poison_object(rw->free_iovec); if (io_alloc_cache_put(&req->ctx->rw_cache, &rw->cache)) { req->async_data = NULL; req->flags &= ~REQ_F_ASYNC_DATA; @@ -155,15 +174,19 @@ static int io_rw_alloc_async(struct io_kiocb *req) entry = io_alloc_cache_get(&ctx->rw_cache); if (entry) { rw = container_of(entry, struct io_async_rw, cache); - req->flags |= REQ_F_ASYNC_DATA; + if (rw->free_iovec) + kasan_mempool_unpoison_object(rw->free_iovec, + rw->free_iov_nr * sizeof(struct iovec)); + req->flags |= REQ_F_ASYNC_DATA | REQ_F_NEED_CLEANUP; req->async_data = rw; goto done; } if (!io_alloc_async_data(req)) { rw = req->async_data; -done: rw->free_iovec = NULL; + rw->free_iov_nr = 0; +done: rw->bytes_done = 0; return 0; } @@ -1130,6 +1153,9 @@ void io_rw_cache_free(struct io_cache_entry *entry) struct io_async_rw *rw; rw = container_of(entry, struct io_async_rw, cache); - kfree(rw->free_iovec); + if (rw->free_iovec) + kasan_mempool_unpoison_object(rw->free_iovec, + rw->free_iov_nr * sizeof(struct iovec)); + io_rw_iovec_free(rw); kfree(rw); } diff --git a/io_uring/rw.h b/io_uring/rw.h index 56fb1703dc5a..26dfa12e2306 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -12,8 +12,9 @@ struct io_async_rw { }; struct iov_iter iter; struct iov_iter_state iter_state; - struct iovec fast_iov[UIO_FASTIOV]; + struct iovec fast_iov; struct iovec *free_iovec; + int free_iov_nr; struct wait_page_queue wpq; }; From patchwork Wed Mar 20 01:17:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597177 Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77124747F for ; Wed, 20 Mar 2024 01:23:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897804; cv=none; b=LGkFldNHTBy4ksmcMsnPfe6D3hJSORMJ7RoR0oZYFqIgysxMkDBlYoaDQ/T/0lMCOrUxoj/MfIgbxmKl/xqchSc7cdYJ6zrzBGXSO3xLRAvXzC0aBpavJZKALqbBM4x/yZZGXl45lyt3hpx6Fa+O6og8JWc2g7ZTkGq5I39l/cg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897804; c=relaxed/simple; bh=zOLolhSc9/Eqwo7Wy3b5pFoWLFFxhzd0SqPQBnjbOGc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aFKe8D+xCkxfJUVSeBtlZt540jT9P/DJJ5nt3zfv3m2OxQw2Nau9qKgoZXbZ6fwi+An1p9PJbF+KREGbMR1574xsvi3mXpSGfNy5IWdVSu5iJNq0ShsZEMNA7kBiiTa4LcJ6hpL7u6uPtL8wK7Xo3o/ewlZr9p64N+Ae7spPyho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=bry3kIQS; arc=none smtp.client-ip=209.85.161.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="bry3kIQS" Received: by mail-oo1-f44.google.com with SMTP id 006d021491bc7-5a46b30857bso871960eaf.1 for ; Tue, 19 Mar 2024 18:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897801; x=1711502601; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=L4rmuK/7dZjd4CaRMeFqboSSutQU3yibk6XIDIyy8gY=; b=bry3kIQS6+Rpz3ySR1HU3+tt/JJWgUc7wu1nKIKEIUxlVkE6PEWbC3D7pK9mo5AWKy pMko/WxjsKVFHyGBLj408L1H0R0Vm/OoDnWUGdNAumqWqxU8waDlKG34lc+/eS1pPf8k D+7q5M52lqQtj8g5IqH7SkaOC05nLBQoJ1BUlYxs2WNe+vLeU66QjDvBOUAmJgIHiypC UOAVZUrlX/BOQrniGaXlzqY0+Ay2pu9hSRRsYJaM2enveDFeedwq0ojNwKJpSCUdOAGb gCn2oCJPc3i2Ri5QreFn5AumRHWBEH1VsA26wc+NGCulNtcqFh48VuxyELsMD+6vGpDB 4nXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897801; x=1711502601; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L4rmuK/7dZjd4CaRMeFqboSSutQU3yibk6XIDIyy8gY=; b=rWznhzGGHpbX+Q3p7DIRbFc7jAZBoHLW6dEa938zFilKqZorIHmkRKcLR1gVEK7GWK EJA2p+gNP8jjVul1/qxIUAk/Yu+NY1GMOK5dX0K+Gb17bF6UkmTO6etUnZtiPs1MicYr 2z3E3nhRrtsWCF410bUweuGM8FVXDtraptIixQntsEHqxy0NxyelD8UYXzblMHjaQdq1 ByfW+ob8ETLYzHYPx9V58flrcuctQtDdCfwEffAX2nbzh6ed3GZUKEbAdEyqxKdedqMx FicAYQz0ui1wnm22tVoBqK609XleruUs8Md3XX7S9YuEmctVHoAyriYU/dcR5Ssf3xUJ p5lQ== X-Gm-Message-State: AOJu0YwPuZ9d6PqrPclnf+tOVGcy0icLEtcTeZjl49uXJVUXBRug4o+a EaDYE3xrKQpzj2pVDAym1KJnnW+hBPSNm/hvoCLCCOeGQFaPzaw3rw2Btn91/WhiBDqP73gFqIo y X-Google-Smtp-Source: AGHT+IGgxi/Xn4c4xaSD11DL2tExDZcrHmxd0qYoPVLGfvkKQB4UJhg+WwcRqDrp3pfH8OeSBPKpsA== X-Received: by 2002:a05:6820:288a:b0:5a2:26c7:397b with SMTP id dn10-20020a056820288a00b005a226c7397bmr356045oob.0.1710897801327; Tue, 19 Mar 2024 18:23:21 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:19 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 13/15] io_uring/net: move connect to always using async data Date: Tue, 19 Mar 2024 19:17:41 -0600 Message-ID: <20240320012251.1120361-14-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Not using an alloc cache, probably not that needed here. Signed-off-by: Jens Axboe --- io_uring/net.c | 38 ++++++++++---------------------------- io_uring/net.h | 1 - io_uring/opdef.c | 1 - 3 files changed, 10 insertions(+), 30 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 82f314da1326..60b8c7f58fd7 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1422,17 +1422,10 @@ int io_socket(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -int io_connect_prep_async(struct io_kiocb *req) -{ - struct io_async_connect *io = req->async_data; - struct io_connect *conn = io_kiocb_to_cmd(req, struct io_connect); - - return move_addr_to_kernel(conn->addr, conn->addr_len, &io->address); -} - int io_connect_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_connect *conn = io_kiocb_to_cmd(req, struct io_connect); + struct io_async_connect *io; if (sqe->len || sqe->buf_index || sqe->rw_flags || sqe->splice_fd_in) return -EINVAL; @@ -1440,28 +1433,22 @@ int io_connect_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) conn->addr = u64_to_user_ptr(READ_ONCE(sqe->addr)); conn->addr_len = READ_ONCE(sqe->addr2); conn->in_progress = conn->seen_econnaborted = false; - return 0; + + if (io_alloc_async_data(req)) + return -ENOMEM; + + io = req->async_data; + return move_addr_to_kernel(conn->addr, conn->addr_len, &io->address); } int io_connect(struct io_kiocb *req, unsigned int issue_flags) { struct io_connect *connect = io_kiocb_to_cmd(req, struct io_connect); - struct io_async_connect __io, *io; + struct io_async_connect *io = req->async_data; unsigned file_flags; int ret; bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; - if (req_has_async_data(req)) { - io = req->async_data; - } else { - ret = move_addr_to_kernel(connect->addr, - connect->addr_len, - &__io.address); - if (ret) - goto out; - io = &__io; - } - file_flags = force_nonblock ? O_NONBLOCK : 0; ret = __sys_connect_file(req->file, &io->address, @@ -1475,13 +1462,6 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags) goto out; connect->seen_econnaborted = true; } - if (req_has_async_data(req)) - return -EAGAIN; - if (io_alloc_async_data(req)) { - ret = -ENOMEM; - goto out; - } - memcpy(req->async_data, &__io, sizeof(__io)); return -EAGAIN; } if (connect->in_progress) { @@ -1497,6 +1477,8 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags) if (ret == -ERESTARTSYS) ret = -EINTR; out: + kfree(req->async_data); + req->flags &= ~REQ_F_ASYNC_DATA; if (ret < 0) req_set_fail(req); io_req_set_res(req, ret, 0); diff --git a/io_uring/net.h b/io_uring/net.h index 783dd601a432..9b47d61a9cf3 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -55,7 +55,6 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags); int io_socket_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_socket(struct io_kiocb *req, unsigned int issue_flags); -int io_connect_prep_async(struct io_kiocb *req); int io_connect_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_connect(struct io_kiocb *req, unsigned int issue_flags); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index fcae75a08f2c..065c92c57878 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -558,7 +558,6 @@ const struct io_cold_def io_cold_defs[] = { .name = "CONNECT", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_connect), - .prep_async = io_connect_prep_async, #endif }, [IORING_OP_FALLOCATE] = { From patchwork Wed Mar 20 01:17:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597178 Received: from mail-oa1-f41.google.com (mail-oa1-f41.google.com [209.85.160.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3C342582 for ; Wed, 20 Mar 2024 01:23:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897805; cv=none; b=VyOjSVYXC9jhc2DfO1garNOAmFt8I/USALHqX8pXvauncDduNO7ULjMfmm1S9bTyNzOqhPRxP8KnB8Z8KTAFtaLLLCzhY2+bwvC6/UbLjXUwwdKM3hxwJtbGzmxIEXoLOe+JfscLq1b8yQuy2F5JXjjuFxM4FA4GT6UNzmFDbBw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897805; c=relaxed/simple; bh=5v5AJa3hf+RSZ1LSk3M8sECmSGIhmGrRWTING2w7Xq4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tI6LNBMpRMmAqjY3U/GJ4Ac9cWo5PcNCMZuIqbwSV4hiOt5MMZAWK87GKyzoNXaSuL/ZZ/LhOLT+TwnO5zAUhwE6hxYiM58BZU/2C1KxZuFsmo1WKDQbMEPcHhZKv4CbfMxp2PA1OnmYExHOJVfO/9Ak6cNu+lUlhN8wRtZ3vdY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Z+GVLGxE; arc=none smtp.client-ip=209.85.160.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Z+GVLGxE" Received: by mail-oa1-f41.google.com with SMTP id 586e51a60fabf-221d4c52759so905131fac.1 for ; Tue, 19 Mar 2024 18:23:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897802; x=1711502602; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=blSLYhbZMEETgrYUItaGDGnFinyIemETqAcvMDs0Nsg=; b=Z+GVLGxEP25PM5w8UodY6LuMm4yBH0tfMOY6/1/W5yxVfA4NT8TAv4niNc7GH88Nk+ ThShtMeU119XMA+uJ7iMJT1yMgCUyYB4EyRMo9usUeux8ucENekteb5CVV+XNGL7iE0K 8vpsgE/0wRvn4IVEJjUEVjpS6KKGcRR33g5MS5A38gu1AJOX4wxmnqCDI0loLEwWNUs3 IAu6RDIKAoeLR6uf4qkGBE2y0HdfBmj186PYJNGIEt+2S+mg9pedDOEvSLK+Nfb3oV4r 8SM9k1aIjliT48XpCE86ODyXtJmf+MPxEdKOVeuTlekEP/bRkNqIfew1DltVheyMxdaI rHGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897802; x=1711502602; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=blSLYhbZMEETgrYUItaGDGnFinyIemETqAcvMDs0Nsg=; b=aGqAkemQs+YlJr2GkafMAvaoxzfT2nKm42L/UbAOPIOQPEav6ulU7tUUhxMjsZ0eGg AXS2hsMNQAmEmPbiM+iRZL+KEehGhipI2TSfC0R7ac3z+yQQHI9MBREh5F3Vny8AYIIc 4aTL+Zz1uhREBsotMwZ4pcCHbDwkEmpQyXMH0jKcCmom3u4PoccHO6iMvn8RNNkqMG2C TsJxCjoy3NX3u2TKBC/29oMcV0z2HtlpN/+7ivulYML8JWvLtsCj5PQKPybWm6Szxwss 5BHk0/vWg/dU5XYO/wuvwj4oZOWtdJQfGwg2IiJVPX8D2112jYQuOxOtGc4JGj/qm9nc sJsw== X-Gm-Message-State: AOJu0YzkK0bQQXB8ycLsx8vj+OVlHgWrjgLG+UFU5rHJ6BonBZ+qOaK4 RZQf44+DQFROKk5Kj131jRiqvK1bXrXT+0R9XlXejEBNGDkSqONi0NdESQv/Ar+sEZg7+Dzb1UT Q X-Google-Smtp-Source: AGHT+IH5J9fJce+ZSF1aMFQMTYaGSI1P1/6agnPf7c1nXWO99A2+vABUTnGpYMZY2HQPgO5lAknwPQ== X-Received: by 2002:a05:6358:7e16:b0:17e:b6bc:f73e with SMTP id o22-20020a0563587e1600b0017eb6bcf73emr264930rwm.0.1710897802586; Tue, 19 Mar 2024 18:23:22 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:21 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 14/15] io_uring/uring_cmd: switch to always allocating async data Date: Tue, 19 Mar 2024 19:17:42 -0600 Message-ID: <20240320012251.1120361-15-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Basic conversion ensuring async_data is allocated off the prep path. Adds a basic alloc cache as well, as passthrough IO can be quite high in rate. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 3 ++ io_uring/opdef.c | 1 - io_uring/uring_cmd.c | 77 ++++++++++++++++++++++++---------- io_uring/uring_cmd.h | 10 ++++- 5 files changed, 69 insertions(+), 23 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 2ba8676f83cc..e3ec84c43f1a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -301,6 +301,7 @@ struct io_ring_ctx { struct io_alloc_cache apoll_cache; struct io_alloc_cache netmsg_cache; struct io_alloc_cache rw_cache; + struct io_alloc_cache uring_cache; /* * Any cancelable uring_cmd is added to this list in diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cc8ce830ff4b..e2b9b00eedef 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -310,6 +310,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) sizeof(struct io_async_msghdr)); io_alloc_cache_init(&ctx->rw_cache, IO_ALLOC_CACHE_MAX, sizeof(struct io_async_rw)); + io_alloc_cache_init(&ctx->uring_cache, IO_ALLOC_CACHE_MAX, + sizeof(struct uring_cache)); io_futex_cache_init(ctx); init_completion(&ctx->ref_comp); xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); @@ -2901,6 +2903,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->apoll_cache, io_apoll_cache_free); io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free); io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free); + io_alloc_cache_free(&ctx->uring_cache, io_uring_cache_free); io_futex_cache_free(ctx); io_destroy_buffers(ctx); mutex_unlock(&ctx->uring_lock); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 065c92c57878..4c0e9688a159 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -677,7 +677,6 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_URING_CMD] = { .name = "URING_CMD", .async_size = 2 * sizeof(struct io_uring_sqe), - .prep_async = io_uring_cmd_prep_async, }, [IORING_OP_SEND_ZC] = { .name = "SEND_ZC", diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 4614ce734fee..9bd0ba87553f 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -14,6 +14,38 @@ #include "rsrc.h" #include "uring_cmd.h" +static struct uring_cache *io_uring_async_get(struct io_kiocb *req) +{ + struct io_ring_ctx *ctx = req->ctx; + struct io_cache_entry *entry; + struct uring_cache *cache; + + entry = io_alloc_cache_get(&ctx->uring_cache); + if (entry) { + cache = container_of(entry, struct uring_cache, cache); + req->flags |= REQ_F_ASYNC_DATA; + req->async_data = cache; + return cache; + } + if (!io_alloc_async_data(req)) + return req->async_data; + return NULL; +} + +static void io_req_uring_cleanup(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + struct uring_cache *cache = req->async_data; + + if (issue_flags & IO_URING_F_UNLOCKED) + return; + if (io_alloc_cache_put(&req->ctx->uring_cache, &cache->cache)) { + ioucmd->sqe = NULL; + req->async_data = NULL; + req->flags &= ~REQ_F_ASYNC_DATA; + } +} + bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx, struct task_struct *task, bool cancel_all) { @@ -128,6 +160,7 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, io_req_set_res(req, ret, 0); if (req->ctx->flags & IORING_SETUP_CQE32) io_req_set_cqe32_extra(req, res2, 0); + io_req_uring_cleanup(req, issue_flags); if (req->ctx->flags & IORING_SETUP_IOPOLL) { /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); @@ -142,13 +175,19 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, } EXPORT_SYMBOL_GPL(io_uring_cmd_done); -int io_uring_cmd_prep_async(struct io_kiocb *req) +static int io_uring_cmd_prep_setup(struct io_kiocb *req, + const struct io_uring_sqe *sqe) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + struct uring_cache *cache; - memcpy(req->async_data, ioucmd->sqe, uring_sqe_size(req->ctx)); - ioucmd->sqe = req->async_data; - return 0; + cache = io_uring_async_get(req); + if (cache) { + memcpy(cache->sqes, sqe, uring_sqe_size(req->ctx)); + ioucmd->sqe = req->async_data; + return 0; + } + return -ENOMEM; } int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -173,9 +212,9 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) req->imu = ctx->user_bufs[index]; io_req_set_rsrc_node(req, ctx, 0); } - ioucmd->sqe = sqe; ioucmd->cmd_op = READ_ONCE(sqe->cmd_op); - return 0; + + return io_uring_cmd_prep_setup(req, sqe); } int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) @@ -206,23 +245,14 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) } ret = file->f_op->uring_cmd(ioucmd, issue_flags); - if (ret == -EAGAIN) { - if (!req_has_async_data(req)) { - if (io_alloc_async_data(req)) - return -ENOMEM; - io_uring_cmd_prep_async(req); - } - return -EAGAIN; - } - - if (ret != -EIOCBQUEUED) { - if (ret < 0) - req_set_fail(req); - io_req_set_res(req, ret, 0); + if (ret == -EAGAIN || ret == -EIOCBQUEUED) return ret; - } - return IOU_ISSUE_SKIP_COMPLETE; + if (ret < 0) + req_set_fail(req); + io_req_uring_cleanup(req, issue_flags); + io_req_set_res(req, ret, 0); + return ret; } int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, @@ -311,3 +341,8 @@ int io_uring_cmd_sock(struct io_uring_cmd *cmd, unsigned int issue_flags) } EXPORT_SYMBOL_GPL(io_uring_cmd_sock); #endif + +void io_uring_cache_free(struct io_cache_entry *entry) +{ + kfree(container_of(entry, struct uring_cache, cache)); +} diff --git a/io_uring/uring_cmd.h b/io_uring/uring_cmd.h index 7356bf9aa655..b0ccff7091ee 100644 --- a/io_uring/uring_cmd.h +++ b/io_uring/uring_cmd.h @@ -1,8 +1,16 @@ // SPDX-License-Identifier: GPL-2.0 +struct uring_cache { + union { + struct io_cache_entry cache; + struct io_uring_sqe sqes[2]; + }; +}; + int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags); int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_uring_cmd_prep_async(struct io_kiocb *req); +void io_uring_cache_free(struct io_cache_entry *entry); bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx, - struct task_struct *task, bool cancel_all); \ No newline at end of file + struct task_struct *task, bool cancel_all); From patchwork Wed Mar 20 01:17:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13597179 Received: from mail-ot1-f54.google.com (mail-ot1-f54.google.com [209.85.210.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8061747F for ; Wed, 20 Mar 2024 01:23:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897807; cv=none; b=Sk0vTLZwXgtonZmXFJRSjXpk64P1pf93CY7YikHDQam12vzRnwB5TunSDcc5fBd+x+lX0mPICe57VECQTk2LvsUdCOF22mTSzVjfi8OH/5J2q5CSIRG2R5cN6C9umN+gFxPpcBoHy6ToeE+qaOU2jAMU/+/Dz9+6JO+xZaYEfUg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710897807; c=relaxed/simple; bh=ADwvS3dajz3v2D54m8dyWJSN7t8lj2nDbVxRCbQ+I3s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ILGy5ci+1bhd7z9ba7SXy0En03WKznJshiNoLdvneSN5wl+E9quuAIsS8qt1dIn59dEDLGB0XGmVe9bQStZ8g7/RTozhRqoRph8W1PEayBfiQ7ma8u01zRF5Ys4o+xSrM/+opmRCBNYS5jUgUCSooOdJUK7sZeX95aFWSRl36Og= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=F9uk2wbU; arc=none smtp.client-ip=209.85.210.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="F9uk2wbU" Received: by mail-ot1-f54.google.com with SMTP id 46e09a7af769-6e677ec9508so776715a34.1 for ; Tue, 19 Mar 2024 18:23:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1710897804; x=1711502604; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CPARoiYjtkNf6hhumA2MQ7GeD1lCOyuUTnmEortuRVI=; b=F9uk2wbUZMdypahMo2omQJadUWH6Wn1bjb9PuwfPPp1PovAE/vK9HincZ6E7JdHTjv fvhvQIHd6nSg4PiB8Kw7UcjudDvq4RPBUJYBYiGbrw9I6kUMSNjFHRWXA2NY/tD1Nhh0 tzayoPauUZm5noa9lJ5XXfgi1OaYC3285vCQ+v20xCTlkBg1BUMd2yBW0QpbO1w84saB I7Ngr2JYp/a65QJp+y/wD0lc2pMGZhgg69zAfmIswS0GeH5Q8l203eXVsVf5XGjAiJln ImRq2DyihPBmvoHfwcw7RRLir4ewBhiC+MK91UrlWWX7Luot54ekk1FsTBjEzuz4AQuW nfkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710897804; x=1711502604; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CPARoiYjtkNf6hhumA2MQ7GeD1lCOyuUTnmEortuRVI=; b=BuVF6HJhW6HJtM0ie7eoo9hHKJv3BxQTkZSyS+DIuO+UOdn/Gyzl2nfNwM89ZYYOow BpyRM9UCiCBbctQT33IgYFBCCjhclI85AGM18wcLz2bkC/zBzN1J+i+ItVmKxvKf455w 92MHLjdRWM8XWfgOzX0RnHuE2C/HHCxv7m25PxTAwNXn0qkXxCN1W9GrLChFXgmQJ7Ky PocWqv3n8ttv7wR0m+ECQ3Ew8T9gZ5duT15zd7GdeCWj1CTn7ybwmJRRPvCWsyWGa2X/ nkT68KDnBKLrk8jtTM60MgqGSlsQXVuyD0WMJxDXhwa6XSV0WHKLBM2oII4h/hq99+KD sxrg== X-Gm-Message-State: AOJu0YyavkKR1Wj669sbbgjM2OKqxH2UcxfobkecL++wnwdtxrNBSIvT 3T/UjlB6w76odGH4sfUhNE7+dF1qtpCQpCI9+zpukLNLtjZxbHgBZ5fvjVSeY4WVdrjrx+4J0RZ U X-Google-Smtp-Source: AGHT+IEWNk6cvWL2LwvsXfWzGpuLa2wZdzp37saL6nuGeMBvMG2uC9X0bo8gZGms+q8Ze/Avxz+6gA== X-Received: by 2002:a05:6358:7211:b0:17f:1d24:1432 with SMTP id h17-20020a056358721100b0017f1d241432mr844508rwa.3.1710897804374; Tue, 19 Mar 2024 18:23:24 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id v22-20020a634816000000b005dc26144d96sm9618007pga.75.2024.03.19.18.23.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Mar 2024 18:23:23 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 15/15] io_uring: drop ->prep_async() Date: Tue, 19 Mar 2024 19:17:43 -0600 Message-ID: <20240320012251.1120361-16-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240320012251.1120361-1-axboe@kernel.dk> References: <20240320012251.1120361-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's now unused, drop the code related to it. This includes the io_issue_defs->manual alloc field. While in there, and since ->async_size is now being used a bit more frequently and in the issue path, move it to io_issue_defs[]. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 36 ++++-------------------------------- io_uring/io_uring.h | 1 - io_uring/opdef.c | 44 +++++++++++++++++++------------------------- io_uring/opdef.h | 9 +++------ io_uring/uring_cmd.h | 1 - 5 files changed, 26 insertions(+), 65 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e2b9b00eedef..5eee07563079 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1709,8 +1709,10 @@ io_req_flags_t io_file_get_flags(struct file *file) bool io_alloc_async_data(struct io_kiocb *req) { - WARN_ON_ONCE(!io_cold_defs[req->opcode].async_size); - req->async_data = kmalloc(io_cold_defs[req->opcode].async_size, GFP_KERNEL); + const struct io_issue_def *def = &io_issue_defs[req->opcode]; + + WARN_ON_ONCE(!def->async_size); + req->async_data = kmalloc(def->async_size, GFP_KERNEL); if (req->async_data) { req->flags |= REQ_F_ASYNC_DATA; return false; @@ -1718,25 +1720,6 @@ bool io_alloc_async_data(struct io_kiocb *req) return true; } -int io_req_prep_async(struct io_kiocb *req) -{ - const struct io_cold_def *cdef = &io_cold_defs[req->opcode]; - const struct io_issue_def *def = &io_issue_defs[req->opcode]; - - /* assign early for deferred execution for non-fixed file */ - if (def->needs_file && !(req->flags & REQ_F_FIXED_FILE) && !req->file) - req->file = io_file_get_normal(req, req->cqe.fd); - if (!cdef->prep_async) - return 0; - if (WARN_ON_ONCE(req_has_async_data(req))) - return -EFAULT; - if (!def->manual_alloc) { - if (io_alloc_async_data(req)) - return -EAGAIN; - } - return cdef->prep_async(req); -} - static u32 io_get_sequence(struct io_kiocb *req) { u32 seq = req->ctx->cached_sq_head; @@ -2049,13 +2032,6 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) req->flags |= REQ_F_LINK; io_req_defer_failed(req, req->cqe.res); } else { - int ret = io_req_prep_async(req); - - if (unlikely(ret)) { - io_req_defer_failed(req, ret); - return; - } - if (unlikely(req->ctx->drain_active)) io_drain_req(req); else @@ -2265,10 +2241,6 @@ static inline int io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, * conditions are true (normal request), then just queue it. */ if (unlikely(link->head)) { - ret = io_req_prep_async(req); - if (unlikely(ret)) - return io_submit_fail_init(sqe, req, ret); - trace_io_uring_link(req, link->head); link->last->link = req; link->last = req; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ef9bf610734c..caf1f573bb87 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -101,7 +101,6 @@ int io_poll_issue(struct io_kiocb *req, struct io_tw_state *ts); int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr); int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin); void __io_submit_flush_completions(struct io_ring_ctx *ctx); -int io_req_prep_async(struct io_kiocb *req); struct io_wq_work *io_wq_free_work(struct io_wq_work *work); void io_wq_submit_work(struct io_wq_work *work); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 4c0e9688a159..ac98decf7891 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -67,6 +67,7 @@ const struct io_issue_def io_issue_defs[] = { .iopoll = 1, .iopoll_queue = 1, .vectored = 1, + .async_size = sizeof(struct io_async_rw), .prep = io_prep_readv, .issue = io_read, }, @@ -81,6 +82,7 @@ const struct io_issue_def io_issue_defs[] = { .iopoll = 1, .iopoll_queue = 1, .vectored = 1, + .async_size = sizeof(struct io_async_rw), .prep = io_prep_writev, .issue = io_write, }, @@ -99,6 +101,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .async_size = sizeof(struct io_async_rw), .prep = io_prep_read_fixed, .issue = io_read, }, @@ -112,6 +115,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .async_size = sizeof(struct io_async_rw), .prep = io_prep_write_fixed, .issue = io_write, }, @@ -138,8 +142,8 @@ const struct io_issue_def io_issue_defs[] = { .unbound_nonreg_file = 1, .pollout = 1, .ioprio = 1, - .manual_alloc = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_sendmsg_prep, .issue = io_sendmsg, #else @@ -152,8 +156,8 @@ const struct io_issue_def io_issue_defs[] = { .pollin = 1, .buffer_select = 1, .ioprio = 1, - .manual_alloc = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_recvmsg_prep, .issue = io_recvmsg, #else @@ -162,6 +166,7 @@ const struct io_issue_def io_issue_defs[] = { }, [IORING_OP_TIMEOUT] = { .audit_skip = 1, + .async_size = sizeof(struct io_timeout_data), .prep = io_timeout_prep, .issue = io_timeout, }, @@ -191,6 +196,7 @@ const struct io_issue_def io_issue_defs[] = { }, [IORING_OP_LINK_TIMEOUT] = { .audit_skip = 1, + .async_size = sizeof(struct io_timeout_data), .prep = io_link_timeout_prep, .issue = io_no_issue, }, @@ -199,6 +205,7 @@ const struct io_issue_def io_issue_defs[] = { .unbound_nonreg_file = 1, .pollout = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_connect), .prep = io_connect_prep, .issue = io_connect, #else @@ -239,6 +246,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .async_size = sizeof(struct io_async_rw), .prep = io_prep_read, .issue = io_read, }, @@ -252,6 +260,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .async_size = sizeof(struct io_async_rw), .prep = io_prep_write, .issue = io_write, }, @@ -272,8 +281,9 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .audit_skip = 1, .ioprio = 1, - .manual_alloc = 1, + .buffer_select = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_sendmsg_prep, .issue = io_send, #else @@ -288,6 +298,7 @@ const struct io_issue_def io_issue_defs[] = { .audit_skip = 1, .ioprio = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_recvmsg_prep, .issue = io_recv, #else @@ -403,6 +414,7 @@ const struct io_issue_def io_issue_defs[] = { .plug = 1, .iopoll = 1, .iopoll_queue = 1, + .async_size = 2 * sizeof(struct io_uring_sqe), .prep = io_uring_cmd_prep, .issue = io_uring_cmd, }, @@ -412,8 +424,8 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .audit_skip = 1, .ioprio = 1, - .manual_alloc = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_send_zc_prep, .issue = io_send_zc, #else @@ -425,8 +437,8 @@ const struct io_issue_def io_issue_defs[] = { .unbound_nonreg_file = 1, .pollout = 1, .ioprio = 1, - .manual_alloc = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_send_zc_prep, .issue = io_sendmsg_zc, #else @@ -439,10 +451,12 @@ const struct io_issue_def io_issue_defs[] = { .pollin = 1, .buffer_select = 1, .audit_skip = 1, + .async_size = sizeof(struct io_async_rw), .prep = io_read_mshot_prep, .issue = io_read_mshot, }, [IORING_OP_WAITID] = { + .async_size = sizeof(struct io_waitid_async), .prep = io_waitid_prep, .issue = io_waitid, }, @@ -488,13 +502,11 @@ const struct io_cold_def io_cold_defs[] = { .name = "NOP", }, [IORING_OP_READV] = { - .async_size = sizeof(struct io_async_rw), .name = "READV", .cleanup = io_readv_writev_cleanup, .fail = io_rw_fail, }, [IORING_OP_WRITEV] = { - .async_size = sizeof(struct io_async_rw), .name = "WRITEV", .cleanup = io_readv_writev_cleanup, .fail = io_rw_fail, @@ -503,12 +515,10 @@ const struct io_cold_def io_cold_defs[] = { .name = "FSYNC", }, [IORING_OP_READ_FIXED] = { - .async_size = sizeof(struct io_async_rw), .name = "READ_FIXED", .fail = io_rw_fail, }, [IORING_OP_WRITE_FIXED] = { - .async_size = sizeof(struct io_async_rw), .name = "WRITE_FIXED", .fail = io_rw_fail, }, @@ -524,7 +534,6 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_SENDMSG] = { .name = "SENDMSG", #if defined(CONFIG_NET) - .async_size = sizeof(struct io_async_msghdr), .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, #endif @@ -532,13 +541,11 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_RECVMSG] = { .name = "RECVMSG", #if defined(CONFIG_NET) - .async_size = sizeof(struct io_async_msghdr), .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, #endif }, [IORING_OP_TIMEOUT] = { - .async_size = sizeof(struct io_timeout_data), .name = "TIMEOUT", }, [IORING_OP_TIMEOUT_REMOVE] = { @@ -551,14 +558,10 @@ const struct io_cold_def io_cold_defs[] = { .name = "ASYNC_CANCEL", }, [IORING_OP_LINK_TIMEOUT] = { - .async_size = sizeof(struct io_timeout_data), .name = "LINK_TIMEOUT", }, [IORING_OP_CONNECT] = { .name = "CONNECT", -#if defined(CONFIG_NET) - .async_size = sizeof(struct io_async_connect), -#endif }, [IORING_OP_FALLOCATE] = { .name = "FALLOCATE", @@ -578,12 +581,10 @@ const struct io_cold_def io_cold_defs[] = { .cleanup = io_statx_cleanup, }, [IORING_OP_READ] = { - .async_size = sizeof(struct io_async_rw), .name = "READ", .fail = io_rw_fail, }, [IORING_OP_WRITE] = { - .async_size = sizeof(struct io_async_rw), .name = "WRITE", .fail = io_rw_fail, }, @@ -596,7 +597,6 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_SEND] = { .name = "SEND", #if defined(CONFIG_NET) - .async_size = sizeof(struct io_async_msghdr), .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, #endif @@ -604,7 +604,6 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_RECV] = { .name = "RECV", #if defined(CONFIG_NET) - .async_size = sizeof(struct io_async_msghdr), .cleanup = io_sendmsg_recvmsg_cleanup, .fail = io_sendrecv_fail, #endif @@ -676,12 +675,10 @@ const struct io_cold_def io_cold_defs[] = { }, [IORING_OP_URING_CMD] = { .name = "URING_CMD", - .async_size = 2 * sizeof(struct io_uring_sqe), }, [IORING_OP_SEND_ZC] = { .name = "SEND_ZC", #if defined(CONFIG_NET) - .async_size = sizeof(struct io_async_msghdr), .cleanup = io_send_zc_cleanup, .fail = io_sendrecv_fail, #endif @@ -689,18 +686,15 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_SENDMSG_ZC] = { .name = "SENDMSG_ZC", #if defined(CONFIG_NET) - .async_size = sizeof(struct io_async_msghdr), .cleanup = io_send_zc_cleanup, .fail = io_sendrecv_fail, #endif }, [IORING_OP_READ_MULTISHOT] = { - .async_size = sizeof(struct io_async_rw), .name = "READ_MULTISHOT", }, [IORING_OP_WAITID] = { .name = "WAITID", - .async_size = sizeof(struct io_waitid_async), }, [IORING_OP_FUTEX_WAIT] = { .name = "FUTEX_WAIT", diff --git a/io_uring/opdef.h b/io_uring/opdef.h index 9e5435ec27d0..7ee6f5aa90aa 100644 --- a/io_uring/opdef.h +++ b/io_uring/opdef.h @@ -27,22 +27,19 @@ struct io_issue_def { unsigned iopoll : 1; /* have to be put into the iopoll list */ unsigned iopoll_queue : 1; - /* opcode specific path will handle ->async_data allocation if needed */ - unsigned manual_alloc : 1; /* vectored opcode, set if 1) vectored, and 2) handler needs to know */ unsigned vectored : 1; + /* size of async data needed, if any */ + unsigned short async_size; + int (*issue)(struct io_kiocb *, unsigned int); int (*prep)(struct io_kiocb *, const struct io_uring_sqe *); }; struct io_cold_def { - /* size of async data needed, if any */ - unsigned short async_size; - const char *name; - int (*prep_async)(struct io_kiocb *); void (*cleanup)(struct io_kiocb *); void (*fail)(struct io_kiocb *); }; diff --git a/io_uring/uring_cmd.h b/io_uring/uring_cmd.h index b0ccff7091ee..477ea8865639 100644 --- a/io_uring/uring_cmd.h +++ b/io_uring/uring_cmd.h @@ -9,7 +9,6 @@ struct uring_cache { int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags); int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); -int io_uring_cmd_prep_async(struct io_kiocb *req); void io_uring_cache_free(struct io_cache_entry *entry); bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx,