From patchwork Sun Feb 25 00:35:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570683 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63AD710E4 for ; Sun, 25 Feb 2024 00:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821590; cv=none; b=ONpnLvuMgEbO5qfQOeWJSWhnvj9Qy2JGazlMFWNu8lw9nDgTgEYC812HjrHxdPviekoRoyk5nflp9LEDk/aRmwGBaPKLFoE9KPfq9QiSTFAfdfEmjXv0NadQVwgGqJn/JIV9vcqWZtPdK0nfUZE4LCCDjfUdfpitmCs0sI/zgIY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821590; c=relaxed/simple; bh=fCyD9l12nlVFS0uM8gSt7z43duSz7ZNJ2TW72BvnVw8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MaSsfds1/L37DXPJwnOmJe/Gz1ULGybiPrfEnI/Oz1UG8XbrcZYMxTh7LrIDTGfmzeM5fBlEeiF432eFliMMj08lqd8WAbdr1pOekzymjWiDyXcCRZ9isTJ9TdNGgtG0YgBsZ3ABV9PVCZjVtfJX3Ud+UReI/eNrVjFlMBMNwT0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=DJVPPn6w; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="DJVPPn6w" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5a0dc313058so169537a12.0 for ; Sat, 24 Feb 2024 16:39:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821586; x=1709426386; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8NMa6cdlS4mwHyd9OSd9n4AxIpngpcBlX+GZ3KJr7wY=; b=DJVPPn6wuFHI3Q8+wxj3z5kItw+L1j5PrnxAOUY9Fgif3m1+7YKkjgVojw+GOHUzD8 jseRdRSfIvNAdDA3cVDVzSmr1x+4d9Bh4NmZbZ2EBVTlrokZG7/rzhy8/II2nW3x36i0 tSr3n062//FhHOw8PGQwrTA2IXkk0Nr17wuIvmU3N10kfDNMDju5FA+f6Td6kau9UVhX A7P50U+qAlGXnq69geIF9Yj7azFRbHMVxOfLINy7gFRe4nDOKdir6YvnjO75dp116vV3 oSaRXSeId5S/M0MLJ+OCQxxuvv6QytvOcDaSrvToAPgSbvZZOOr/YvQzlnpn+du5c+Ou 3x7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821586; x=1709426386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8NMa6cdlS4mwHyd9OSd9n4AxIpngpcBlX+GZ3KJr7wY=; b=dPdxfhBJk58ZHAxMgHzSTaDu97ZIkEToYX+vRx627Ro3pKWIZxkUfmEYtcXp0Wasz3 c5orDJ+cQf0rpL1uDo6+SKhEu3K9ThunSZToCvnNLoDPnGRAfxRyFPUe7g0YHKOW1Eqe HNNE5a3YsP2f/j4CozGdw7JgPRxvngVUGBheZKUUKzQVMypU/nkciu/uovFt/+tdELv+ fGeT8mEyShXrTS+oxckMfPfnJmbThM/+TooHTUUYc3/8jN7NxxrXJPFPRH7ZzTr646lX DOSuYlyGVlHw/jku1BXZ9PP2Xm6KJ47vAelgzrENTGZItoJxhRBfK0CsT2NTat2Rc9cj 2pkQ== X-Gm-Message-State: AOJu0YxMKwzCNP3EoTcf2ognf+c0GlbT/F+Z3cMI8JStTJPF/UPDxNF9 ysi8IBZY3Zu1aD4HzgrthLRGUqJbXR9UdjmwNAgEFuaxXCBD+WpNJJ3TGtka7K99MzMbnw1mGXt w X-Google-Smtp-Source: AGHT+IHQL4QDx+B3z1baskFz/sg6EwH689IXC8swsgisY1Hc9dViRb8gAXn6eKTZPjWXnc1iQXBXhQ== X-Received: by 2002:a05:6a00:1792:b0:6e4:f431:847d with SMTP id s18-20020a056a00179200b006e4f431847dmr4016707pfg.2.1708821586226; Sat, 24 Feb 2024 16:39:46 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:44 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/8] io_uring/net: unify how recvmsg and sendmsg copy in the msghdr Date: Sat, 24 Feb 2024 17:35:47 -0700 Message-ID: <20240225003941.129030-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For recvmsg, we roll our own since we support buffer selections. This isn't the case for sendmsg right now, but in preparation for doing so, make the recvmsg copy helpers generic so we can call them from the sendmsg side as well. Signed-off-by: Jens Axboe --- io_uring/net.c | 275 +++++++++++++++++++++++++------------------------ 1 file changed, 140 insertions(+), 135 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 161622029147..fcbaeb7cc045 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -204,16 +204,148 @@ static int io_setup_async_msg(struct io_kiocb *req, return -EAGAIN; } +static bool io_recvmsg_multishot_overflow(struct io_async_msghdr *iomsg) +{ + int hdr; + + if (iomsg->namelen < 0) + return true; + if (check_add_overflow((int)sizeof(struct io_uring_recvmsg_out), + iomsg->namelen, &hdr)) + return true; + if (check_add_overflow(hdr, (int)iomsg->controllen, &hdr)) + return true; + + return false; +} + +#ifdef CONFIG_COMPAT +static int __io_compat_msg_copy_hdr(struct io_kiocb *req, + struct io_async_msghdr *iomsg, + struct sockaddr __user **addr, int ddir) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct compat_msghdr msg; + struct compat_iovec __user *uiov; + int ret; + + if (copy_from_user(&msg, sr->umsg_compat, sizeof(msg))) + return -EFAULT; + + ret = __get_compat_msghdr(&iomsg->msg, &msg, addr); + if (ret) + return ret; + + uiov = compat_ptr(msg.msg_iov); + if (req->flags & REQ_F_BUFFER_SELECT) { + compat_ssize_t clen; + + iomsg->free_iov = NULL; + if (msg.msg_iovlen == 0) { + sr->len = 0; + } else if (msg.msg_iovlen > 1) { + return -EINVAL; + } else { + if (!access_ok(uiov, sizeof(*uiov))) + return -EFAULT; + if (__get_user(clen, &uiov->iov_len)) + return -EFAULT; + if (clen < 0) + return -EINVAL; + sr->len = clen; + } + + if (ddir == ITER_DEST && req->flags & REQ_F_APOLL_MULTISHOT) { + iomsg->namelen = msg.msg_namelen; + iomsg->controllen = msg.msg_controllen; + if (io_recvmsg_multishot_overflow(iomsg)) + return -EOVERFLOW; + } + } else { + iomsg->free_iov = iomsg->fast_iov; + ret = __import_iovec(ddir, (struct iovec __user *)uiov, + msg.msg_iovlen, UIO_FASTIOV, + &iomsg->free_iov, &iomsg->msg.msg_iter, + true); + if (ret < 0) + return ret; + } + + return 0; +} +#endif + +static int __io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, + struct sockaddr __user **addr, int ddir) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct user_msghdr msg; + int ret; + + if (copy_from_user(&msg, sr->umsg, sizeof(*sr->umsg))) + return -EFAULT; + + ret = __copy_msghdr(&iomsg->msg, &msg, addr); + if (ret) + return ret; + + if (req->flags & REQ_F_BUFFER_SELECT) { + if (msg.msg_iovlen == 0) { + sr->len = iomsg->fast_iov[0].iov_len = 0; + iomsg->fast_iov[0].iov_base = NULL; + iomsg->free_iov = NULL; + } else if (msg.msg_iovlen > 1) { + return -EINVAL; + } else { + if (copy_from_user(iomsg->fast_iov, msg.msg_iov, + sizeof(*msg.msg_iov))) + return -EFAULT; + sr->len = iomsg->fast_iov[0].iov_len; + iomsg->free_iov = NULL; + } + + if (ddir == ITER_DEST && req->flags & REQ_F_APOLL_MULTISHOT) { + iomsg->namelen = msg.msg_namelen; + iomsg->controllen = msg.msg_controllen; + if (io_recvmsg_multishot_overflow(iomsg)) + return -EOVERFLOW; + } + } else { + iomsg->free_iov = iomsg->fast_iov; + ret = __import_iovec(ddir, msg.msg_iov, msg.msg_iovlen, + UIO_FASTIOV, &iomsg->free_iov, + &iomsg->msg.msg_iter, false); + if (ret > 0) + ret = 0; + } + + return ret; +} + +static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, + struct sockaddr __user **addr, int ddir) +{ + iomsg->msg.msg_name = &iomsg->addr; + iomsg->msg.msg_iter.nr_segs = 0; + +#ifdef CONFIG_COMPAT + if (req->ctx->compat) + return __io_compat_msg_copy_hdr(req, iomsg, addr, ddir); +#endif + + return __io_msg_copy_hdr(req, iomsg, addr, ddir); +} + static int io_sendmsg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->free_iov = iomsg->fast_iov; - ret = sendmsg_copy_msghdr(&iomsg->msg, sr->umsg, sr->msg_flags, - &iomsg->free_iov); + ret = io_msg_copy_hdr(req, iomsg, NULL, ITER_SOURCE); + if (ret) + return ret; + /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; return ret; @@ -435,142 +567,15 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -static bool io_recvmsg_multishot_overflow(struct io_async_msghdr *iomsg) -{ - int hdr; - - if (iomsg->namelen < 0) - return true; - if (check_add_overflow((int)sizeof(struct io_uring_recvmsg_out), - iomsg->namelen, &hdr)) - return true; - if (check_add_overflow(hdr, (int)iomsg->controllen, &hdr)) - return true; - - return false; -} - -static int __io_recvmsg_copy_hdr(struct io_kiocb *req, - struct io_async_msghdr *iomsg) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct user_msghdr msg; - int ret; - - if (copy_from_user(&msg, sr->umsg, sizeof(*sr->umsg))) - return -EFAULT; - - ret = __copy_msghdr(&iomsg->msg, &msg, &iomsg->uaddr); - if (ret) - return ret; - - if (req->flags & REQ_F_BUFFER_SELECT) { - if (msg.msg_iovlen == 0) { - sr->len = iomsg->fast_iov[0].iov_len = 0; - iomsg->fast_iov[0].iov_base = NULL; - iomsg->free_iov = NULL; - } else if (msg.msg_iovlen > 1) { - return -EINVAL; - } else { - if (copy_from_user(iomsg->fast_iov, msg.msg_iov, sizeof(*msg.msg_iov))) - return -EFAULT; - sr->len = iomsg->fast_iov[0].iov_len; - iomsg->free_iov = NULL; - } - - if (req->flags & REQ_F_APOLL_MULTISHOT) { - iomsg->namelen = msg.msg_namelen; - iomsg->controllen = msg.msg_controllen; - if (io_recvmsg_multishot_overflow(iomsg)) - return -EOVERFLOW; - } - } else { - iomsg->free_iov = iomsg->fast_iov; - ret = __import_iovec(ITER_DEST, msg.msg_iov, msg.msg_iovlen, UIO_FASTIOV, - &iomsg->free_iov, &iomsg->msg.msg_iter, - false); - if (ret > 0) - ret = 0; - } - - return ret; -} - -#ifdef CONFIG_COMPAT -static int __io_compat_recvmsg_copy_hdr(struct io_kiocb *req, - struct io_async_msghdr *iomsg) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct compat_msghdr msg; - struct compat_iovec __user *uiov; - int ret; - - if (copy_from_user(&msg, sr->umsg_compat, sizeof(msg))) - return -EFAULT; - - ret = __get_compat_msghdr(&iomsg->msg, &msg, &iomsg->uaddr); - if (ret) - return ret; - - uiov = compat_ptr(msg.msg_iov); - if (req->flags & REQ_F_BUFFER_SELECT) { - compat_ssize_t clen; - - iomsg->free_iov = NULL; - if (msg.msg_iovlen == 0) { - sr->len = 0; - } else if (msg.msg_iovlen > 1) { - return -EINVAL; - } else { - if (!access_ok(uiov, sizeof(*uiov))) - return -EFAULT; - if (__get_user(clen, &uiov->iov_len)) - return -EFAULT; - if (clen < 0) - return -EINVAL; - sr->len = clen; - } - - if (req->flags & REQ_F_APOLL_MULTISHOT) { - iomsg->namelen = msg.msg_namelen; - iomsg->controllen = msg.msg_controllen; - if (io_recvmsg_multishot_overflow(iomsg)) - return -EOVERFLOW; - } - } else { - iomsg->free_iov = iomsg->fast_iov; - ret = __import_iovec(ITER_DEST, (struct iovec __user *)uiov, msg.msg_iovlen, - UIO_FASTIOV, &iomsg->free_iov, - &iomsg->msg.msg_iter, true); - if (ret < 0) - return ret; - } - - return 0; -} -#endif - -static int io_recvmsg_copy_hdr(struct io_kiocb *req, - struct io_async_msghdr *iomsg) -{ - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - -#ifdef CONFIG_COMPAT - if (req->ctx->compat) - return __io_compat_recvmsg_copy_hdr(req, iomsg); -#endif - - return __io_recvmsg_copy_hdr(req, iomsg); -} - int io_recvmsg_prep_async(struct io_kiocb *req) { + struct io_async_msghdr *iomsg; int ret; if (!io_msg_alloc_async_prep(req)) return -ENOMEM; - ret = io_recvmsg_copy_hdr(req, req->async_data); + iomsg = req->async_data; + ret = io_msg_copy_hdr(req, iomsg, &iomsg->uaddr, ITER_DEST); if (!ret) req->flags |= REQ_F_NEED_CLEANUP; return ret; @@ -793,7 +798,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (req_has_async_data(req)) { kmsg = req->async_data; } else { - ret = io_recvmsg_copy_hdr(req, &iomsg); + ret = io_msg_copy_hdr(req, &iomsg, &iomsg.uaddr, ITER_DEST); if (ret) return ret; kmsg = &iomsg; From patchwork Sun Feb 25 00:35:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570684 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8217710F7 for ; Sun, 25 Feb 2024 00:39:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821591; cv=none; b=bauiF4J6+m/S2xhe+Y8FEkBLMRfRvdbOigUmFeCJqVMpsIOyzlzMpsisuDvXzcxOZRJkoZcYhg0AyOYR27nSJNnLTnmWzdH0gY6sIpJ19q2GLCTbnSVTsJ8ksfnmBK0qayWrIf8qfJjrKyCJ1N6FZ8riczBDaGh1T+w5MQqaSpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821591; c=relaxed/simple; bh=2ayUOfaEYiRowwWtv7K6yBwCX1wE9nF3iDBASOdcNs0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eeo5cr0y2JmuipJfJdD7YqTE0Y6mWhlp/8zM0YThuyXLlbeoV8K5XoZYnXV8zBk1p+YDva3bdHFDuc0milYfDmjZHCs4IluFsNdg+/ktIPqheZdY5hApJL4Uq3ExnlVlHotSGfTirD9g8uZjD4w5Iys+OSxo37zXQRN4Lq5v2/s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=CvFFqEj3; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="CvFFqEj3" Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-55b5a37acb6so861972a12.0 for ; Sat, 24 Feb 2024 16:39:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821588; x=1709426388; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jw0Je0tw5r/73HhCk/nGFBZB/5kheAEr7X7sZwheg6c=; b=CvFFqEj3nVzqx7eMV4b5DfXbfIV4CyHL7ngdALhfFUZtvM2Ckgw5x5SQfWgf6Z5UfP 7cMKvNcG6xP10X2fkwwa2zbXQtsB6bfyrZkqsnyHuDlrHyIy2c0SVzdh0+t8BrsirL3D hUshslxf8j233Qpw3CIWEQ739qCRER8tfU7D620LC0FdeQxroUAwDc3iH8Pbd1rQZazt 0bPekmgghI1tBFoWCPgp9NQQ0EUt3wmcPUVwLOOAOngxhiMqTX994TiOTR9eJ7eBAvfA Vmzfiiy6WmtuAYEdtrrd9PZyer5u7Q6gI4ji67nARty5VnUCcRt5TyDk6M1+bxV5Qo2K 3DmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821588; x=1709426388; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jw0Je0tw5r/73HhCk/nGFBZB/5kheAEr7X7sZwheg6c=; b=KUmz1yA2/hdZLKpIbWqo2rXqsVQajWYkSOr3/p0+MUMDJ6Oau10ZBC0uyXJ/I2kHDQ GhcwVWbeYQTMV25vuZgv2cryhdLbma92s/u62cnxLgXNAXj0arCAkVgTc726HK71fgIB v+1tEyBPqvNVPe318sveG8hlhFJqAJTNpOdR3E/RLXpj9GoUM6k9BZQz1xC0Np2LoiNd gvjKXdGlZCeq6A30P1OrVVSWksR+BqDhsDkWgPg/zu0fVaaBebQuww7Qq4cqq2aNDum6 7wz4cdHUw2Q0Bq1rQqndWmoYr88qly/ZBTEQreA7lpVbXGL0JUFB/z9JjJWBS9kQ/3Vn VHlg== X-Gm-Message-State: AOJu0YzMNwSQYedUmdOVt/s+MgXcryloy+CRj2rI94Cr4h0jz7E0CRme gKTGOidtku/8wNo53JdXhITlpcss5wLEjEDa+8M3Z1AOyrHWGtAnqnvVfWJMblXj8vbT5pycHzo M X-Google-Smtp-Source: AGHT+IELZ4VD+XueMRFyiRqpUYJPAJXStnJ+ZlOcLbf60lZEWC2dGGOi16ZEep1dPCAEdJeTXzTU4g== X-Received: by 2002:aa7:881a:0:b0:6e4:cdef:7d95 with SMTP id c26-20020aa7881a000000b006e4cdef7d95mr4557903pfo.3.1708821588158; Sat, 24 Feb 2024 16:39:48 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:46 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/8] net: remove {revc,send}msg_copy_msghdr() from exports Date: Sat, 24 Feb 2024 17:35:48 -0700 Message-ID: <20240225003941.129030-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The only user of these was io_uring, and it's not using them anymore. Make them static and remove them from the socket header file. Signed-off-by: Jens Axboe --- include/linux/socket.h | 7 ------- net/socket.c | 14 +++++++------- tools/perf/trace/beauty/include/linux/socket.h | 7 ------- 3 files changed, 7 insertions(+), 21 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index cfcb7e2c3813..139c330ccf2c 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -422,13 +422,6 @@ extern long __sys_recvmsg_sock(struct socket *sock, struct msghdr *msg, struct user_msghdr __user *umsg, struct sockaddr __user *uaddr, unsigned int flags); -extern int sendmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct iovec **iov); -extern int recvmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct sockaddr __user **uaddr, - struct iovec **iov); extern int __copy_msghdr(struct msghdr *kmsg, struct user_msghdr *umsg, struct sockaddr __user **save_addr); diff --git a/net/socket.c b/net/socket.c index ed3df2f749bf..0f5d5079fd91 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2600,9 +2600,9 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, return err; } -int sendmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct iovec **iov) +static int sendmsg_copy_msghdr(struct msghdr *msg, + struct user_msghdr __user *umsg, unsigned flags, + struct iovec **iov) { int err; @@ -2753,10 +2753,10 @@ SYSCALL_DEFINE4(sendmmsg, int, fd, struct mmsghdr __user *, mmsg, return __sys_sendmmsg(fd, mmsg, vlen, flags, true); } -int recvmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct sockaddr __user **uaddr, - struct iovec **iov) +static int recvmsg_copy_msghdr(struct msghdr *msg, + struct user_msghdr __user *umsg, unsigned flags, + struct sockaddr __user **uaddr, + struct iovec **iov) { ssize_t err; diff --git a/tools/perf/trace/beauty/include/linux/socket.h b/tools/perf/trace/beauty/include/linux/socket.h index cfcb7e2c3813..139c330ccf2c 100644 --- a/tools/perf/trace/beauty/include/linux/socket.h +++ b/tools/perf/trace/beauty/include/linux/socket.h @@ -422,13 +422,6 @@ extern long __sys_recvmsg_sock(struct socket *sock, struct msghdr *msg, struct user_msghdr __user *umsg, struct sockaddr __user *uaddr, unsigned int flags); -extern int sendmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct iovec **iov); -extern int recvmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct sockaddr __user **uaddr, - struct iovec **iov); extern int __copy_msghdr(struct msghdr *kmsg, struct user_msghdr *umsg, struct sockaddr __user **save_addr); From patchwork Sun Feb 25 00:35:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570685 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B49B10E4 for ; Sun, 25 Feb 2024 00:39:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821593; cv=none; b=NskJkk0qfdG0Fwd8dKkZdOSWOQKrZyx6AGZqB2fBVhFg7xRETJTaXGYEM0ihhZ2hQdFnHkMmGCjHAYKb3+1CU5KF48SAIo1naDAVnxcoocIKQXTaNSOptw314S/hz0rXEVtGkF+P0a2uyhlD3Y9sRGxxW67UTV9++DPyZke2bvQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821593; c=relaxed/simple; bh=A3OKKfft3DNeQRrP5felkjZSJPYhl3MGmCwrW5AVh74=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ngl9Axm30JDRt/gjxNPH+3SJptfME23h0R8xPgQ/lhQFuOUBSKbqQB6Ka+sDjK6RFye08UbH3iwPe10p7E5O0uJs/2pAcX/F3nwoqkSBcr93T9FHsPVrLlHTnnnPrX6PLwzNjUps/JPyVUmsnwmFFrUOc6rXwozuRYno46UEkJM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=s7ZbOtBc; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="s7ZbOtBc" Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-5cfb8126375so727215a12.1 for ; Sat, 24 Feb 2024 16:39:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821590; x=1709426390; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YVNqlUfD9EN4SXvPUo1xiT2FLfu9522ZGHPisvqay5M=; b=s7ZbOtBcRN7VI/mbIUES2mzQ0sQDSGm35fqfW1LnYH6u23N1tSLFRwYe2PNd/IAWZ8 X7Psr9UwNf02tElSSS9ghrNPGRsC5DEWUAGAdVawNNR6f1xcA5/uvc2sffvy3VxpLJ0K WZSiYz6WClCDapb/2YteXtIq4TpF7CJY2z0/jF1xBqorrzgCWwV6XhG1c4Qja3DORRC6 KJ0nSuYI26XPrXEmHghw49ONIntdBZdhiGoxETvytiMrnMO+Mp39U0f5CVciKSpe+yn+ MT9UwiB65VrOCmKXbW0QNdUzKPyTFBHkOLFFgDZvnAD+Nywfc3Xmbf0O/8BuUCnK0Eah eAXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821590; x=1709426390; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YVNqlUfD9EN4SXvPUo1xiT2FLfu9522ZGHPisvqay5M=; b=Sda8LTWeds0ZYdoZ3liFNih+SGCbic5/HazToKnFFa40rzrMYCN8sQVNZAxMTNYe/t IpsmM287PlckkNOAv7UCu3yS2lYB3UYmVrK3PDEuf3WlWwX+xfk4ZUgWr+anV6tOf0AL aXhB0GCeIDJSI3cjcONp+8yPwnr7PuSexLWu7MoJnNYRPZLlRUEdCS6IIPiYEERq9Os3 Ws4wwu69IN3LPIB0koAD3TpGiE66cOQAwNLUaIde+VVRFqiMHdWzQml4hwUmEqXeCdjj 7uuModFYvS4rEEF6eOG4kECPOx64zKOAUasgddfIho9anFulAAdu32g4s0vOvnthYDyU AC0g== X-Gm-Message-State: AOJu0YxwTlBc7gqWJRIvnwPebubOoiZKyZ+5ktp1B+xA3CB9vOa17U9e 2n77A0gjGrUy3pIWVhjtKSVeuuUZoP7ngdWcAq19YgOpPUF8l1/ROB3jUGnjApHiKvSlM1DeadY 9 X-Google-Smtp-Source: AGHT+IEJNCL4QZ1GEDZC45uKYZSn2CuqU31lflHNwHXd+g7xSqnUZzp7zEgiottDTfTGz+WvXIXPxA== X-Received: by 2002:a05:6a00:91:b0:6e4:e7c3:f0c6 with SMTP id c17-20020a056a00009100b006e4e7c3f0c6mr4469922pfj.0.1708821590141; Sat, 24 Feb 2024 16:39:50 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:48 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/8] io_uring/net: add provided buffer support for IORING_OP_SEND Date: Sat, 24 Feb 2024 17:35:49 -0700 Message-ID: <20240225003941.129030-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's pretty trivial to wire up provided buffer support for the send side, just like we do on the receive side. This enables setting up a buffer ring that an application can use to push pending sends to, and then have a send pick a buffer from that ring. One of the challenges with async IO and networking sends is that you can get into reordering conditions if you have more than one inflight at the same time. Consider the following scenario where everything is fine: 1) App queues sendA for socket1 2) App queues sendB for socket1 3) App does io_uring_submit() 4) sendA is issued, completes successfully, posts CQE 5) sendB is issued, completes successfully, posts CQE All is fine. Requests are always issued in-order, and both complete inline as most sends do. However, if we're flooding socket1 with sends, the following could also result from the same sequence: 1) App queues sendA for socket1 2) App queues sendB for socket1 3) App does io_uring_submit() 4) sendA is issued, socket1 is full, poll is armed for retry 5) Space frees up in socket1, this triggers sendA retry via task_work 6) sendB is issued, completes successfully, posts CQE 7) sendA is retried, completes successfully, posts CQE Now we've sent sendB before sendA, which can make things unhappy. If both sendA and sendB had been using provided buffers, then it would look as follows instead: 1) App queues dataA for sendA, queues sendA for socket1 2) App queues dataB for sendB queues sendB for socket1 3) App does io_uring_submit() 4) sendA is issued, socket1 is full, poll is armed for retry 5) Space frees up in socket1, this triggers sendA retry via task_work 6) sendB is issued, picks first buffer (dataA), completes successfully, posts CQE (which says "I sent dataA") 7) sendA is retried, picks first buffer (dataB), completes successfully, posts CQE (which says "I sent dataB") Now we've sent the data in order, and everybody is happy. It's worth noting that this also opens the door for supporting multishot sends, as provided buffers would be a prerequisite for that. Those can trigger either when new buffers are added to the outgoing ring, or (if stalled due to lack of space) when space frees up in the socket. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 1 + io_uring/io_uring.c | 3 ++- io_uring/net.c | 19 ++++++++++++++++--- io_uring/opdef.c | 1 + 4 files changed, 20 insertions(+), 4 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 7bd10201a02b..74c3afac9c63 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -522,6 +522,7 @@ struct io_uring_params { #define IORING_FEAT_CQE_SKIP (1U << 11) #define IORING_FEAT_LINKED_FILE (1U << 12) #define IORING_FEAT_REG_REG_RING (1U << 13) +#define IORING_FEAT_SEND_BUFS (1U << 14) /* * io_uring_register(2) opcodes and arguments diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cf2f514b7cc0..f6332fc56bed 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3962,7 +3962,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, IORING_FEAT_POLL_32BITS | IORING_FEAT_SQPOLL_NONFIXED | IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS | IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP | - IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING; + IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING | + IORING_FEAT_SEND_BUFS; if (copy_to_user(params, p, sizeof(*p))) { ret = -EFAULT; diff --git a/io_uring/net.c b/io_uring/net.c index fcbaeb7cc045..10b6d8caf4da 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -489,7 +489,8 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; - io_req_set_res(req, ret, 0); + cflags = io_put_kbuf(req, issue_flags); + io_req_set_res(req, ret, cflags); return IOU_OK; } @@ -497,8 +498,10 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) { struct sockaddr_storage __address; struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct msghdr msg; + size_t len = sr->len; + unsigned int cflags; struct socket *sock; + struct msghdr msg; unsigned flags; int min_ret = 0; int ret; @@ -531,7 +534,17 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!sock)) return -ENOTSOCK; - ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &msg.msg_iter); + if (io_do_buffer_select(req)) { + void __user *buf; + + buf = io_buffer_select(req, &len, issue_flags); + if (!buf) + return -ENOBUFS; + sr->buf = buf; + sr->len = len; + } + + ret = import_ubuf(ITER_SOURCE, sr->buf, len, &msg.msg_iter); if (unlikely(ret)) return ret; diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 9c080aadc5a6..88fbe5cfd379 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -273,6 +273,7 @@ const struct io_issue_def io_issue_defs[] = { .audit_skip = 1, .ioprio = 1, .manual_alloc = 1, + .buffer_select = 1, #if defined(CONFIG_NET) .prep = io_sendmsg_prep, .issue = io_send, From patchwork Sun Feb 25 00:35:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570686 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C39011C06 for ; Sun, 25 Feb 2024 00:39:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821594; cv=none; b=Ltt8E7QUQKpxvnjzstT2yAH9HOsvDdYm2KrxHl6PFpAH4/QrNSJ31VUA3wyQ6mWI68CZf2neXfcDbr6FaVRXl82uGmDOosrMHpIK5Huwb/BQ+5v0EaDZ7vBfBoRXi9umSuzopDwOSSNDmv/tZCnffRXTdO3fcapbDDlRMn2GfoQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821594; c=relaxed/simple; bh=taZLZYe9/lGhZlFpg6RgyuB6JMR1eg7+Ku/sLaHYol0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aGPDMJiAp9P4Kk8rivkNyf85BZ58dKHDML8ZH/iW3svJPB3Nu4Vc9ahIq6Fip8/9ERIRXyG3VVK1kCkyLwa8HqAS3QOvnRbNEG5r+Ga51D+MlWYr5pyFiqwy6cw3OudGxBd6Q/iJiGn1ykrrqlkIDARwQZTdPG8Afj/IfrjPADQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=NGHUNsSy; arc=none smtp.client-ip=209.85.215.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="NGHUNsSy" Received: by mail-pg1-f175.google.com with SMTP id 41be03b00d2f7-5cdbc42f5efso863980a12.0 for ; Sat, 24 Feb 2024 16:39:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821592; x=1709426392; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iQJvsw/J6sK+/X03ucFK8E4fqOrLrmHIUymPpkqPOW8=; b=NGHUNsSypXujgus++eHegLs4nUakh4DOn0bw2mTTTLgDyAWrLPC+0n0KKgG2W/jZwN qvXalRE0XMN3N10h3ZFVLzKz3LntBr4SvTnmnoDOOim9cq0nt0ty5uiJxnn/vQ/x0yVt 47DPY/d3J+rJ3r7WiH6UigRWXt57Hxgvr4xqMFli4VWQrBzXrmX3BubjJkUsFWCc/5/B eHD5bBOlsDjDxq9UuHtayrZbgDp8FHrL9OYgdHPYVL6xXO33TmibqKoVtHLRigrvBY4f MW4gZgVGS7XOH+t3VKpoQpzQESHPgA8i4pkea4f3KcjjOgskZSMuOhx/xDPTGXchGYw7 LkyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821592; x=1709426392; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iQJvsw/J6sK+/X03ucFK8E4fqOrLrmHIUymPpkqPOW8=; b=BQw1rZMp7PmRufQPQuQPHzacS6ffQUbdAN2r5+Z+Knl2MSjygBUhecBgemUSUXROlz C04Eo3kD/CMdbFsQZtVU8iAwzzsLwbv/zSuImugeS/M2yWyaofVqt3VHtm+T9iTUNj6x LPF7S2OSYLhT1+kguSC8GkwS8LhbiZUmNYjL/AGON2n64CoXZZ9/+hkAzNQF3PBdHBMc PhpXMfttRurqFO9SMPbSLAxHrCUCgsaXdGaVrIxNK6sdwWu3TktGXIYSd6wf2W33rhYs LC2YPZvBz7R0XGDGi1y+YH7sxvy/8Cu0+8yevCVmTjjMb3O1RUFZgU4f5vNhzrS3Xtss t3QA== X-Gm-Message-State: AOJu0Yy4osMcfWb/M08pTqtY6ZuKfcIHXu/ZME3vzvKQ3UBS5UGTc7jk qP1SIVgYK22DEU/ckijUbp0pw6lDA980gcJV566eH6oLALZXMbOIsPJo16R+/pJpvPDNyP4re51 2 X-Google-Smtp-Source: AGHT+IE64TcOG5PUyWvD7CcpbPJ/6kW4ZzMogc2I1KCjkG9uigooJ7l8ukPjAX+5V5UCZSSi0hGN+w== X-Received: by 2002:a05:6a20:9186:b0:1a0:dcfd:feda with SMTP id v6-20020a056a20918600b001a0dcfdfedamr5168120pzd.5.1708821591851; Sat, 24 Feb 2024 16:39:51 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:50 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/8] io_uring/net: add provided buffer support for IORING_OP_SENDMSG Date: Sat, 24 Feb 2024 17:35:50 -0700 Message-ID: <20240225003941.129030-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Adds provided buffer support for sendmsg as well, see the previous commit that added it to IORING_OP_SEND for a longer explanation of why this makes sense. Signed-off-by: Jens Axboe --- io_uring/net.c | 15 ++++++++++++++- io_uring/opdef.c | 1 + 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/io_uring/net.c b/io_uring/net.c index 10b6d8caf4da..30afb394efd7 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -436,6 +436,7 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr iomsg, *kmsg; struct socket *sock; + unsigned int cflags; unsigned flags; int min_ret = 0; int ret; @@ -458,6 +459,17 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_msg(req, kmsg, issue_flags); + if (io_do_buffer_select(req)) { + void __user *buf; + size_t len = sr->len; + + buf = io_buffer_select(req, &len, issue_flags); + if (!buf) + return -ENOBUFS; + + iov_iter_ubuf(&kmsg->msg.msg_iter, ITER_SOURCE, buf, len); + } + flags = sr->msg_flags; if (issue_flags & IO_URING_F_NONBLOCK) flags |= MSG_DONTWAIT; @@ -576,7 +588,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; - io_req_set_res(req, ret, 0); + cflags = io_put_kbuf(req, issue_flags); + io_req_set_res(req, ret, cflags); return IOU_OK; } diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 88fbe5cfd379..1f6b09e61ef8 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -139,6 +139,7 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .ioprio = 1, .manual_alloc = 1, + .buffer_select = 1, #if defined(CONFIG_NET) .prep = io_sendmsg_prep, .issue = io_sendmsg, From patchwork Sun Feb 25 00:35:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570687 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA10910E4 for ; Sun, 25 Feb 2024 00:39:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821596; cv=none; b=g+KZphFXNiwzopHydm8+AuzLcTiIeU70EmM41axWR0qA5CotqwzCRj5Fibh97sPMOznofnkAditwHeoyntI8Ki4FgBaaIQ+FIoSbwdUqhbz8EoGnUovfpIQxZ92PJg6NIgp5Eic/qZcCwJWW/FG/Cr2+cfEZUPAsvGlH2ZzAVk0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821596; c=relaxed/simple; bh=Bwz1XCGG8sAa4hB/q27XYGr3ZK5UY47pzg7bfBwJ80o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r03SefJYqNaGlT+6bjYjf4S8RbdGiD9d+En5JfCnSdDXbE8/j9VMxuEU+CLSQgUizT7aDYtQTxfSiixh+gVVBFGffVdjgckj+jDtEGy+R1k2vKI0eQChQjV+hnEavKYCmM7Iv1XlCNO3PlUEUoBilCB8hNsO2UZ0AbLEnhdLuZM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=tF/Iawto; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="tF/Iawto" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1dbff00beceso5517375ad.1 for ; Sat, 24 Feb 2024 16:39:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821593; x=1709426393; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qke7VBOcxahFVWc17Wbuzh6Bf+gTJGXU/ImftE7zCJs=; b=tF/IawtouPaBKcuuVO0B/tIMyjCFArNiC3ReYv0UT6YUGYBSq3eUU0pcMDa84SBhUu YIxgHTsubHmJskGPR4cvjRDKZFhsY0ThJ4NAC/bDADIU6A0QMtUqPfz6pjLT3DsbJm3J 9NUedxQ66DIki0w6mb7BEsRM21P9x6U/Grasnx3GPsABaBdVNz/puJKliPT+EiYtN3lI ekSKh089GdCXyOsJnmshznVZiCOSAJ89ULSWhgraPdP8snUee8sjHX385dbjQNl4mP4Z J2U5mg9OA17kUlWkCTRDvF4/b1JKXJHcMp+v/mCFcw4iLf7HmmsQzsVH+gQZrGUlgg/y paZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821593; x=1709426393; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qke7VBOcxahFVWc17Wbuzh6Bf+gTJGXU/ImftE7zCJs=; b=Zcvqx4FUgieRebZ0tkREuIVSKiUGE734DhaNPlfG1+tUlwdEmsr1V/h4JpRHYsAzic cJJZDk9QcQS16ugDiWRZsDi+PQptLLFj7aJjs56Mb/JHESWpxg2l6y9h5ri1mLtu7iKN wN78kFNsszp4Q0ALmJdh2Vfe6ee+lQkdRE2BDK2/c5XfTnwbgxeeck5mDne2l455BtJo eF1J79J8nNv2+pHzqs/AhK6QBJfQBQyU0lDkwKVZJXWbY18I0bIN43JN7kbJ4PP/p/D+ 7rRl5HCgVoZEZkHVAYGjiMLdRImjTvb+pmK7QqIWAdKhVW2/9sC9RP1hiBWDc/yurxpq FHtg== X-Gm-Message-State: AOJu0YzUohlA1VaBphMUFNg2Xv8NyUSSCYYtRtWnYaKkFLpFlhy2XOrX hrUWJv6Cy7nl5shN548qPN2Vohcv0EzAFfe977pJwerkn+YzbgUJKktAHy3f/nX8GYUJAM3mnio I X-Google-Smtp-Source: AGHT+IH893jI7KqrZ/KeccH7Ixf2UWgtIzOlw5ZL4Z/Vm3TEKNcRKgI/mq1dMnpldFsmWIbaTBQ/oQ== X-Received: by 2002:a05:6a20:158b:b0:1a0:7fa7:52b with SMTP id h11-20020a056a20158b00b001a07fa7052bmr4862131pzj.5.1708821593558; Sat, 24 Feb 2024 16:39:53 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:52 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/8] io_uring/kbuf: flag request if buffer pool is empty after buffer pick Date: Sat, 24 Feb 2024 17:35:51 -0700 Message-ID: <20240225003941.129030-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Normally we do an extra roundtrip for retries even if the buffer pool has depleted, as we don't check that upfront. Rather than add this check, have the buffer selection methods mark the request with REQ_F_BL_EMPTY if the used buffer group is out of buffers after this selection. This is very cheap to do once we're all the way inside there anyway, and it gives the caller a chance to make better decisions on how to proceed. For example, recv/recvmsg multishot could check this flag when it decides whether to keep receiving or not. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +++ io_uring/kbuf.c | 10 ++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index bd7071aeec5d..d8111d64812b 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -480,6 +480,7 @@ enum { REQ_F_POLL_NO_LAZY_BIT, REQ_F_CANCEL_SEQ_BIT, REQ_F_CAN_POLL_BIT, + REQ_F_BL_EMPTY_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -556,6 +557,8 @@ enum { REQ_F_CANCEL_SEQ = IO_REQ_FLAG(REQ_F_CANCEL_SEQ_BIT), /* file is pollable */ REQ_F_CAN_POLL = IO_REQ_FLAG(REQ_F_CAN_POLL_BIT), + /* buffer list was empty after selection of buffer */ + REQ_F_BL_EMPTY = IO_REQ_FLAG(REQ_F_BL_EMPTY_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index ee866d646997..3d257ed9031b 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -139,6 +139,8 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, list_del(&kbuf->list); if (*len == 0 || *len > kbuf->len) *len = kbuf->len; + if (list_empty(&bl->buf_list)) + req->flags |= REQ_F_BL_EMPTY; req->flags |= REQ_F_BUFFER_SELECTED; req->kbuf = kbuf; req->buf_index = kbuf->bid; @@ -152,12 +154,16 @@ static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len, unsigned int issue_flags) { struct io_uring_buf_ring *br = bl->buf_ring; + __u16 tail, head = bl->head; struct io_uring_buf *buf; - __u16 head = bl->head; - if (unlikely(smp_load_acquire(&br->tail) == head)) + tail = smp_load_acquire(&br->tail); + if (unlikely(tail == head)) return NULL; + if (head + 1 == tail) + req->flags |= REQ_F_BL_EMPTY; + head &= bl->mask; /* mmaped buffers are always contig */ if (bl->is_mmap || head < IO_BUFFER_LIST_BUF_PER_PAGE) { From patchwork Sun Feb 25 00:35:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570688 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A35BE1113 for ; Sun, 25 Feb 2024 00:39:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821598; cv=none; b=A3AR8vnGYs5Dd5oSPfECUbN+tuUC1h1qc1jPb0c5zoN81sLex7uzEV8LB4BXvgD6N2NdLfisq1AZIlmouXdd8U1OTXDN5U56soXuDwUxwXKrUN3RBlIuF0oYmJg7WYd8XcM08MSHtlaXUueXuP1AhTsG9KZvNySCNYMST8j3cMw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821598; c=relaxed/simple; bh=Bf1KrOzbD/EJPal5T4Nb2NAmhQ56nL/WYBD96g0Ps5A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rz2mPxx/kcuN2Hqr96J2s/AN2goFyDLaon3lIrh8xKtrQ6IMQItbMSE3DGG5TWYbhRNEeisvpiOyquUNsovM3lrgjyGuvVr9Kr10pLgeyhO6Y0YGmj9T0wNhN7BrS78+czainNMuh1bqa6WwjTXJ4/scmBzgDmyZKVfIAskb4zA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=JTVekTyI; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="JTVekTyI" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6e50c36a08cso544b3a.0 for ; Sat, 24 Feb 2024 16:39:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821595; x=1709426395; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lwWlvBNSsBfW5G92Au6ltP4hlZyM4QgGqnUIJdx9PlE=; b=JTVekTyIbvGnTOjLBBVV02Uru3f5cs/levZrKiL6CRaI4Am46G4PphWPKs5EBPmoc0 p902H8d4GiTo1v0eWUTump5ctiVWIQfDDSHAZu0e8R9UlWnZrGArvuczMo4YCjeU99kI Vi/nbsf9bcAriHLvQIPgxo9c1m5LdiEgUNyGFSu3gRlBM9rwIQwHH8R557ceYbK83BCZ 4xPlzoYVegdqyVeJQM8oqgc6OKxFlk/EJhgt0W5s8g1X648PzMRa8bdEl2Gq05fH2lI2 mVUwnmB3VWauwd4B83EkB/yqiFBkpNctTL3d0MSeoUjtlnlZUKZq4hQc5wpjQem89Fac MJ+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821595; x=1709426395; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lwWlvBNSsBfW5G92Au6ltP4hlZyM4QgGqnUIJdx9PlE=; b=S5CJeW+J7v1IC3vi0w9zkgwYUlon058UZoZSzUiTsNsgX2Mz40erXOUb3jvCiIU78n r7KICZg2EaMBfu/5yPqj5N/uj7MrVQBu18c49f1WoGsX626zH1lhN4ILV9qsoWrshjEh wEFyMPJxVMii1WR0a5M8ONYK9pQb++v5GQKP716rXjoKZH2fuRJyRZ882BHht51PpUBy vEM5OjO5yKkI7wrHnhgJmpoXaD1fyrQDepF1ZgZTCp31TG0j9bLX4aOSOWoCj54NtCDY chtNcZkL/NAZcfJqgeKhPgHNRyZ/nuf6DWuPhNJutNy40tUvln88DC29Bo7xPUgK3tEg YRWQ== X-Gm-Message-State: AOJu0YyTKT8UEAzjhegjpZc4Vc9BCJt0ih8+9orQWutFnKjNIAAJi7OJ TFkPZ0mTbGA7oqFA9NSXKpAYqyXwQzWRqj9twaUZjzAL/P2LdBYOIW55Gb4HC2W2q1jfDwav30Q u X-Google-Smtp-Source: AGHT+IFe6nlEjzl1y0QKd2HNRWG38DPLqQ1t5GGloOpxTUGeZrF1itOcNd8ousLvn7xwSMl/+fz4Rw== X-Received: by 2002:aa7:850b:0:b0:6e5:3e:9eb8 with SMTP id v11-20020aa7850b000000b006e5003e9eb8mr1871797pfn.2.1708821595541; Sat, 24 Feb 2024 16:39:55 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:53 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/8] io_uring/net: support multishot for send Date: Sat, 24 Feb 2024 17:35:52 -0700 Message-ID: <20240225003941.129030-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This works very much like the receive side, except for sends. The idea is that an application can fill outgoing buffers in a provided buffer group, and then arm a single send that will service them all. For now this variant just terminates when we are out of buffers to send, and hence the application needs to re-arm it if IORING_CQE_F_MORE isn't set, as per usual for multishot requests. This only enables it for IORING_OP_SEND, IORING_OP_SENDMSG is coming in a separate patch. However, this patch does do a lot of the prep work that makes wiring up the sendmsg variant pretty trivial. They share the prep side. Enabling multishot for sends is, again, identical to the receive side. The app sets IORING_SEND_MULTISHOT in sqe->ioprio. This flag is also the same as IORING_RECV_MULTISHOT. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 8 +++ io_uring/net.c | 98 +++++++++++++++++++++++++++++------ 2 files changed, 89 insertions(+), 17 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 74c3afac9c63..6766e78ee03b 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -351,9 +351,17 @@ enum io_uring_op { * 0 is reported if zerocopy was actually possible. * IORING_NOTIF_USAGE_ZC_COPIED if data was copied * (at least partially). + * + * IORING_SEND_MULTISHOT Multishot send. Like the recv equivalent, must + * be used with provided buffers. Keeps sending + * from the given buffer group ID until it is + * empty. Sets IORING_CQE_F_MORE if more + * completions should be expected on behalf of + * the same SQE. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) #define IORING_RECV_MULTISHOT (1U << 1) +#define IORING_SEND_MULTISHOT IORING_RECV_MULTISHOT #define IORING_RECVSEND_FIXED_BUF (1U << 2) #define IORING_SEND_ZC_REPORT_USAGE (1U << 3) diff --git a/io_uring/net.c b/io_uring/net.c index 30afb394efd7..8237ac5c957f 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -204,6 +204,16 @@ static int io_setup_async_msg(struct io_kiocb *req, return -EAGAIN; } +static inline void io_mshot_prep_retry(struct io_kiocb *req) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + + req->flags &= ~REQ_F_BL_EMPTY; + sr->done_io = 0; + sr->len = 0; /* get from the provided buffer */ + req->buf_index = sr->buf_group; +} + static bool io_recvmsg_multishot_overflow(struct io_async_msghdr *iomsg) { int hdr; @@ -401,6 +411,8 @@ void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req) kfree(io->free_iov); } +#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_SEND_MULTISHOT) + int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -417,11 +429,19 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); sr->len = READ_ONCE(sqe->len); sr->flags = READ_ONCE(sqe->ioprio); - if (sr->flags & ~IORING_RECVSEND_POLL_FIRST) + if (sr->flags & ~SENDMSG_FLAGS) return -EINVAL; sr->msg_flags = READ_ONCE(sqe->msg_flags) | MSG_NOSIGNAL; if (sr->msg_flags & MSG_DONTWAIT) req->flags |= REQ_F_NOWAIT; + if (sr->flags & IORING_SEND_MULTISHOT) { + if (!(req->flags & REQ_F_BUFFER_SELECT)) + return -EINVAL; + if (sr->msg_flags & MSG_WAITALL) + return -EINVAL; + req->flags |= REQ_F_APOLL_MULTISHOT; + sr->buf_group = req->buf_index; + } #ifdef CONFIG_COMPAT if (req->ctx->compat) @@ -431,6 +451,44 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static inline bool io_send_finish(struct io_kiocb *req, int *ret, + struct msghdr *msg, unsigned issue_flags) +{ + bool mshot_finished = *ret <= 0; + unsigned int cflags; + + cflags = io_put_kbuf(req, issue_flags); + + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { + io_req_set_res(req, *ret, cflags); + *ret = IOU_OK; + return true; + } + + if (mshot_finished || req->flags & REQ_F_BL_EMPTY) + goto finish; + + /* + * Fill CQE for this receive and see if we should keep trying to + * receive from this socket. + */ + if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, + *ret, cflags | IORING_CQE_F_MORE)) { + io_mshot_prep_retry(req); + *ret = IOU_ISSUE_SKIP_COMPLETE; + return false; + } + + /* Otherwise stop multishot but use the current result. */ +finish: + io_req_set_res(req, *ret, cflags); + if (issue_flags & IO_URING_F_MULTISHOT) + *ret = IOU_STOP_MULTISHOT; + else + *ret = IOU_OK; + return true; +} + int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -511,7 +569,6 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) struct sockaddr_storage __address; struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); size_t len = sr->len; - unsigned int cflags; struct socket *sock; struct msghdr msg; unsigned flags; @@ -542,10 +599,14 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_addr(req, &__address, issue_flags); + if (!io_check_multishot(req, issue_flags)) + return -EAGAIN; + sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; +retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; @@ -570,8 +631,16 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) msg.msg_flags = flags; ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { - if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_addr(req, &__address, issue_flags); + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) { + ret = io_setup_async_addr(req, &__address, issue_flags); + if (ret != -EAGAIN) + return ret; + if (issue_flags & IO_URING_F_MULTISHOT) { + io_kbuf_recycle(req, issue_flags); + return IOU_ISSUE_SKIP_COMPLETE; + } + return -EAGAIN; + } if (ret > 0 && io_net_retry(sock, flags)) { sr->len -= ret; @@ -588,9 +657,13 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; - cflags = io_put_kbuf(req, issue_flags); - io_req_set_res(req, ret, cflags); - return IOU_OK; + else + io_kbuf_recycle(req, issue_flags); + + if (!io_send_finish(req, &ret, &msg, issue_flags)) + goto retry_multishot; + + return ret; } int io_recvmsg_prep_async(struct io_kiocb *req) @@ -654,15 +727,6 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } -static inline void io_recv_prep_retry(struct io_kiocb *req) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - - sr->done_io = 0; - sr->len = 0; /* get from the provided buffer */ - req->buf_index = sr->buf_group; -} - /* * Finishes io_recv and io_recvmsg. * @@ -697,7 +761,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; - io_recv_prep_retry(req); + io_mshot_prep_retry(req); /* Known not-empty or unknown state, retry */ if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq == -1) { if (sr->nr_multishot_loops++ < MULTISHOT_MAX_RETRY) From patchwork Sun Feb 25 00:35:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570689 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85F2B1373 for ; Sun, 25 Feb 2024 00:39:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821600; cv=none; b=Mdc2M5akSvqaYDbvgT3UehH8PWsGYdkkplPMpjTlyrTZThp/lyAagu5sLVW1LG38Qp0eocqiBNwHyM422qfVgf6Rv2XaGTR+4JEBlUYCxpyxKx0NEXdRIW8apHLg5b6+gINsbRbifTk3EeP2kTEKAFoBDsx6hISWeVJSJ2Mkn8w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821600; c=relaxed/simple; bh=P5ixUEnPDqgCgetkEGXKZH60xYxsyTjwl9mK1THuS/Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TMUOYunDUC8hRBdagvYdDWVh1+ee4DOxrekNPtBaWMx+sSFFfyv2plQgwU7s471nna8YvYi9J3AHfwBnMfrC3ERX05b/J5F+3NnhNADnQj8/kfXNksoeFaeP80QdENn/jNjQOUfpPdLQSuEAW/+9KoWPE95gNFGCAtUgexrbV9o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=flxHyOLO; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="flxHyOLO" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-58962bf3f89so754894a12.0 for ; Sat, 24 Feb 2024 16:39:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821597; x=1709426397; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8nERVJumMFm9j/dfRWnpXO7RdH3oxXK+/zTrBj/X2q4=; b=flxHyOLO+QR/6CDFrRrpbMNIuftUZfwHAchosWpnKHonwiTh38jG+uMjndsYq25a5z cNWbe9Jt8FcmadKhKtdp8DWaOB0yLyo6tqd0udRIBPycVh/cT0K/bz6Tgn5++Sue33Je xMwZydag1c9iK9f8OwbMXnpzC1prm6Nh4flQOY3ZiYY62QT7mInGwirlcrh7m+9+ekxB Y1xkEdnX8fKsJJeCr0VosFZ0IrjQcnPESNYpo30DGP+iSM0i78KjlgQeCCMuVGxegCdL R5qHgPfuIuN4TS0KmGJjXMkMiboj8UPgOJELZXEj+vWxtoRdp2H5CSW1GqT2x8Px5rJN ICgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821597; x=1709426397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8nERVJumMFm9j/dfRWnpXO7RdH3oxXK+/zTrBj/X2q4=; b=TfC9nno53gU8kiaE1wnYJZkRw56Au5X/C8l0qh/EwwC6944tZo/Fmj/ZqPXhjI6Dy+ 4KjU6zW4ISgm7YabKIgQW83eKcWK59S8zBEk/KQ7+68HcbziO1mDLRzMV14SldKfEdVU HfvV4t0sm7S2UBHP0uUsDwH2pa4wFefrywwAKdiQ4pmWrzzp/V+4KlOvh+MZu87lW49O OBE6ydeFYJhJjzLO0oaC4vkCgMDwRTC+qww+OSLYJYRzHgte3pERdjqxZeRLdMUsuL4x CIA/bn3/BukpUytM+21mlaFV+OOMGdCa9XK4BHgw9U5ZElEakwYppz3gUojTk1epvbg6 eIIw== X-Gm-Message-State: AOJu0YxhSDp7U6yOrICH2zjjBTJOsajqzEWLvEUbWq2DQRvyrRWVYX97 1YgDBeH8bONtPeIiXZCuy1/UGfQv3y9xPLqVfqiCXgJeebuEj8lLQ/e+8Nz1WUMt+C0acIGYBCW o X-Google-Smtp-Source: AGHT+IFaAfw+8zMkjQQlObRVu9V53qgpLdtsXDIupcJCxd0ZgAiWAp8h7llNJv4SmhU+wPCNjAXwsw== X-Received: by 2002:a05:6a20:4283:b0:1a0:e557:7ec3 with SMTP id o3-20020a056a20428300b001a0e5577ec3mr5203630pzj.1.1708821597391; Sat, 24 Feb 2024 16:39:57 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:55 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 7/8] io_uring/net: support multishot for sendmsg Date: Sat, 24 Feb 2024 17:35:53 -0700 Message-ID: <20240225003941.129030-8-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Same as the IORING_OP_SEND multishot mode. Needs further work, but it's functional and can be tested. Signed-off-by: Jens Axboe --- io_uring/net.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 8237ac5c957f..240b8eff1a78 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -494,7 +494,6 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr iomsg, *kmsg; struct socket *sock; - unsigned int cflags; unsigned flags; int min_ret = 0; int ret; @@ -517,6 +516,10 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_msg(req, kmsg, issue_flags); + if (!io_check_multishot(req, issue_flags)) + return io_setup_async_msg(req, kmsg, issue_flags); + +retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; size_t len = sr->len; @@ -537,8 +540,14 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); if (ret < min_ret) { - if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_msg(req, kmsg, issue_flags); + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) { + ret = io_setup_async_msg(req, kmsg, issue_flags); + if (ret == -EAGAIN && (issue_flags & IO_URING_F_MULTISHOT)) { + io_kbuf_recycle(req, issue_flags); + return IOU_ISSUE_SKIP_COMPLETE; + } + return ret; + } if (ret > 0 && io_net_retry(sock, flags)) { kmsg->msg.msg_controllen = 0; kmsg->msg.msg_control = NULL; @@ -550,18 +559,22 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret = -EINTR; req_set_fail(req); } + if (ret >= 0) + ret += sr->done_io; + else if (sr->done_io) + ret = sr->done_io; + else + io_kbuf_recycle(req, issue_flags); + + if (!io_send_finish(req, &ret, &kmsg->msg, issue_flags)) + goto retry_multishot; + /* fast path, check for non-NULL to avoid function call */ if (kmsg->free_iov) kfree(kmsg->free_iov); req->flags &= ~REQ_F_NEED_CLEANUP; io_netmsg_recycle(req, issue_flags); - if (ret >= 0) - ret += sr->done_io; - else if (sr->done_io) - ret = sr->done_io; - cflags = io_put_kbuf(req, issue_flags); - io_req_set_res(req, ret, cflags); - return IOU_OK; + return ret; } int io_send(struct io_kiocb *req, unsigned int issue_flags) From patchwork Sun Feb 25 00:35:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13570690 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21CA410F4 for ; Sun, 25 Feb 2024 00:39:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821601; cv=none; b=uizVzAMyf6GYzEK2irlho6unqUQBCypv42qS5+YGn9luL5M5Os1GRbraJuCcx0AnTzcOlhgUgNI/WWjq02r9CpiwH5P4nJSEbrsmwsRs7Yhq6P/8Eu1twsLn6yV4Xr6BxNY6Dby9x47lTyxfzp2m0a/QlQP/OdaVSXNrtRXCa8w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708821601; c=relaxed/simple; bh=peO494qYRoBM2DluyJlzyCy7oFDWhquPo33wmiwRUlg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BrSKuaUIpwvmqMHs5rmPVO0X0ruq8s39t90ynmxnC2PJWqO0nzJJMqmp7jTuKbadXEHKDIn9vXtbkzegxRsBOPcqQkhQChk7ghs1iRaRN9JZP5iP/dPziJtjbsSPI1STO5sLbB+8SKqHySOEG1mFIlnd2SSXmKDLoZMF7Ju0RO0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ja6c1Zqf; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ja6c1Zqf" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-5cdbc42f5efso864009a12.0 for ; Sat, 24 Feb 2024 16:39:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708821599; x=1709426399; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZqgkzRD5fxa2R5gbrgNqkGIM72hLGfqmZT6jWLnrvH8=; b=ja6c1ZqfXizpihLvyd7K+zRMda0fyxKSrvv4SbNcdm38lEklCK/I2vbK9vt00xlC/z 5YI96q5zMe2X9mVKgpZfGPq6VMJ0kdkM8dENDFJ5qVZfecIBSpVZuqTIuz/ivdttQWt7 RnUqjKYGOwxkytFUHv9DbZnj2Gg9S+1F+2PK3RQ4ghoGlIttciFcdqEb5X2PguKAyin3 tQrC0DLfitkIEs1B1es057tlecplMRFAfhnY2hRii7j/P4OQao+cbracSw7jBuXXKDq+ /5OsH6fEhigY7bUTDAKS98YIDO1HT+pP/aPQP00zJ79FLrlKT9ZLqT/D+IAolVubCACo RjAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708821599; x=1709426399; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZqgkzRD5fxa2R5gbrgNqkGIM72hLGfqmZT6jWLnrvH8=; b=cZ1ujDi2+GNgdJ+oEKFNf/rlTAM8so4ShWoXc8ygEBwK0lzdrnaMCRJxi9m/M8FXl5 lX0glrPFHPODrLBE/HTUHSY4TF6gFBXCVDeBkAXtrmFCeZwJOmiQ5RTSwoa63rkZqh+O DKqDTp+lZdGfWGqsuHoDgqwIum1O1Om2US27xtdFaGkfN3x6+PGiemXLUocmLYJbCdFL n/4x3OE7P+JsWXMoAqZqDxbV90NMcwPH9zsfuhDUUbqG3+DN/FUDSeA9Ux8ksPrPCoQs PkFevfOOWy2rlYB9RG09uk0EtaBvCO5xQBHkvysWr28aqhoNGZQdyNnlu3IA9LU0Axt8 xO/A== X-Gm-Message-State: AOJu0YzIdhM9XNSGNW7fth9bTtAwfYJYxixiR1zugr7OgdMtVKD6MCGK H9V1IbYuFrGrXK+Gv/ZOBBIYGoabkLakrID0DnDR4jwH+r/eetoA1toeGsRgxID46bVpOd5eywy i X-Google-Smtp-Source: AGHT+IElM3CLEysaVY4HXkVtCTWJMxzeNVK2W+wCpCRyfIrvHo+uxy5XIbjCGyOd19Jr1iKosDRl9Q== X-Received: by 2002:a05:6a20:938f:b0:1a0:f897:738a with SMTP id x15-20020a056a20938f00b001a0f897738amr138488pzh.0.1708821599135; Sat, 24 Feb 2024 16:39:59 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id u3-20020a62d443000000b006e24991dd5bsm1716170pfl.98.2024.02.24.16.39.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 24 Feb 2024 16:39:57 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 8/8] io_uring/net: set MSG_MORE if we're doing multishot send and have more Date: Sat, 24 Feb 2024 17:35:54 -0700 Message-ID: <20240225003941.129030-9-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240225003941.129030-1-axboe@kernel.dk> References: <20240225003941.129030-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If we have more data pending, we know we're going to do one more loop. If that's the case, then set MSG_MORE to inform the networking stack that there's more data coming shortly for this socket. Signed-off-by: Jens Axboe --- io_uring/net.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 240b8eff1a78..07307dd5a077 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -519,6 +519,10 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) if (!io_check_multishot(req, issue_flags)) return io_setup_async_msg(req, kmsg, issue_flags); + flags = sr->msg_flags; + if (issue_flags & IO_URING_F_NONBLOCK) + flags |= MSG_DONTWAIT; + retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; @@ -528,12 +532,12 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) if (!buf) return -ENOBUFS; + if ((req->flags & (REQ_F_BL_EMPTY|REQ_F_APOLL_MULTISHOT)) == + REQ_F_APOLL_MULTISHOT) + flags |= MSG_MORE; iov_iter_ubuf(&kmsg->msg.msg_iter, ITER_SOURCE, buf, len); } - flags = sr->msg_flags; - if (issue_flags & IO_URING_F_NONBLOCK) - flags |= MSG_DONTWAIT; if (flags & MSG_WAITALL) min_ret = iov_iter_count(&kmsg->msg.msg_iter);