From patchwork Tue Jun 28 15:02:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 526B3C43334 for ; Tue, 28 Jun 2022 15:02:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347668AbiF1PCr (ORCPT ); Tue, 28 Jun 2022 11:02:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347665AbiF1PCr (ORCPT ); Tue, 28 Jun 2022 11:02:47 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57F8033E3C for ; Tue, 28 Jun 2022 08:02:46 -0700 (PDT) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25SAE7ZE013208 for ; Tue, 28 Jun 2022 08:02:46 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=/R5f40SX0myY+sLsV9T01ncFIwUnVx4ToYywqLxN6Ws=; b=nAmkmqYcEnu6uf8RChzrghN6IZgh+xpBiH/buD1YNVUMEa+yV/kZt21audq0wyzLI2Wa SA0G56fqKIuuDXji0ryQ8f3qg9zbr/mdZr9IhIXKNRj+Nuvsd16DBJ1u/9TDwlbUxlzZ Zs0yocp8KDIYQ6MXftU0nA52/ZvaUbkzA1Y= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gyxx31y7r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:02:45 -0700 Received: from twshared25107.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:44 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 95F71244BBC5; Tue, 28 Jun 2022 08:02:37 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 1/8] io_uring: allow 0 length for buffer select Date: Tue, 28 Jun 2022 08:02:21 -0700 Message-ID: <20220628150228.1379645-2-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 4pRyPUqkcOetCK3N0_Ij-P_DEaR608vA X-Proofpoint-ORIG-GUID: 4pRyPUqkcOetCK3N0_Ij-P_DEaR608vA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_08,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org If user gives 0 for length, we can set it from the available buffer size. Signed-off-by: Dylan Yudaken --- io_uring/kbuf.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 8e4f1e8aaf4a..4ed5102461bf 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -115,7 +115,7 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, kbuf = list_first_entry(&bl->buf_list, struct io_buffer, list); list_del(&kbuf->list); - if (*len > kbuf->len) + if (*len == 0 || *len > kbuf->len) *len = kbuf->len; req->flags |= REQ_F_BUFFER_SELECTED; req->kbuf = kbuf; @@ -145,7 +145,7 @@ static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len, buf = page_address(bl->buf_pages[index]); buf += off; } - if (*len > buf->len) + if (*len == 0 || *len > buf->len) *len = buf->len; req->flags |= REQ_F_BUFFER_RING; req->buf_list = bl; From patchwork Tue Jun 28 15:02:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8CDACCA479 for ; Tue, 28 Jun 2022 15:02:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347692AbiF1PCy (ORCPT ); Tue, 28 Jun 2022 11:02:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347686AbiF1PCx (ORCPT ); Tue, 28 Jun 2022 11:02:53 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C48CA33E8C for ; Tue, 28 Jun 2022 08:02:52 -0700 (PDT) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25SELS8B006984 for ; Tue, 28 Jun 2022 08:02:52 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=Q0rmH/1tlJvFVDKrfJqrbwktOSP5EyNGr2kAA/EaMHs=; b=J3hUyIENMYVeE/OMKS/mvqaVWcUx4cLSRrtyimUL4g9wOhpgYCe2JOjuMXivCZFLKu1B slj6vTNYMTzvrZVcd2lpvjcaL1YW7Ahc09ViOeieTCB44MTU7z7fH3UnU9y6EDTMnfx1 myw8jV7mSq+iZoLjcxmollcsn31gtf0BzgU= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h03ax0a8x-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:02:52 -0700 Received: from twshared17349.03.ash7.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:51 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id C0AD9244BBC9; Tue, 28 Jun 2022 08:02:37 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 2/8] io_uring: restore bgid in io_put_kbuf Date: Tue, 28 Jun 2022 08:02:22 -0700 Message-ID: <20220628150228.1379645-3-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: ZNo6u6eAUdQg4N5BKKbvCOUnD7ajPoi9 X-Proofpoint-ORIG-GUID: ZNo6u6eAUdQg4N5BKKbvCOUnD7ajPoi9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_08,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Attempt to restore bgid. This is needed when recycling unused buffers as the next time around it will want the correct bgid. Signed-off-by: Dylan Yudaken --- io_uring/kbuf.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 3d48f1ab5439..c64f02ea1c30 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -96,16 +96,20 @@ static inline void io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags) static inline unsigned int __io_put_kbuf_list(struct io_kiocb *req, struct list_head *list) { + unsigned int ret = IORING_CQE_F_BUFFER | (req->buf_index << IORING_CQE_BUFFER_SHIFT); if (req->flags & REQ_F_BUFFER_RING) { - if (req->buf_list) + if (req->buf_list) { + req->buf_index = req->buf_list->bgid; req->buf_list->head++; + } req->flags &= ~REQ_F_BUFFER_RING; } else { + req->buf_index = req->kbuf->bgid; list_add(&req->kbuf->list, list); req->flags &= ~REQ_F_BUFFER_SELECTED; } - return IORING_CQE_F_BUFFER | (req->buf_index << IORING_CQE_BUFFER_SHIFT); + return ret; } static inline unsigned int io_put_kbuf_comp(struct io_kiocb *req) From patchwork Tue Jun 28 15:02:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0BF0CCA479 for ; Tue, 28 Jun 2022 15:03:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347730AbiF1PDK (ORCPT ); Tue, 28 Jun 2022 11:03:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347712AbiF1PDC (ORCPT ); Tue, 28 Jun 2022 11:03:02 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9323833E9F for ; Tue, 28 Jun 2022 08:03:01 -0700 (PDT) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25SAE75B013218 for ; Tue, 28 Jun 2022 08:03:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=yNGWca9EZYptSu02EU61qEdS1WccUVHJPmvgfuLJvJI=; b=PT5C3RKF+bqqbTqcVYEy+XGiQto7YKGGtENqdyg9k/8eyAl4PcD5o1GLCI3d0bcHEiGn 2S4V14Zj70GmXEVnnKDXoePpJF1hxiaC41MDdzd+urLPeHgpg9mdocvLyPYhFJ2SKQtY RmiV/ABNzw4ABTJ42O156r6IXbKSk8MzWYI= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gyxx31y98-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:03:01 -0700 Received: from twshared25107.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:55 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id DAC6E244BBD1; Tue, 28 Jun 2022 08:02:37 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 3/8] io_uring: allow iov_len = 0 for recvmsg and buffer select Date: Tue, 28 Jun 2022 08:02:23 -0700 Message-ID: <20220628150228.1379645-4-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: lyYIeo1IPJduEjJ32lTdzwqBWSxttULD X-Proofpoint-ORIG-GUID: lyYIeo1IPJduEjJ32lTdzwqBWSxttULD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_08,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org When using BUFFER_SELECT there is no technical requirement that the user actually provides iov, and this removes one copy_from_user call. So allow iov_len to be 0. Signed-off-by: Dylan Yudaken --- io_uring/net.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 19a805c3814c..5e84f7ab92a3 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -300,12 +300,18 @@ static int __io_recvmsg_copy_hdr(struct io_kiocb *req, return ret; if (req->flags & REQ_F_BUFFER_SELECT) { - if (iov_len > 1) + if (iov_len == 0) { + sr->len = iomsg->fast_iov[0].iov_len = 0; + iomsg->fast_iov[0].iov_base = NULL; + iomsg->free_iov = NULL; + } else if (iov_len > 1) { return -EINVAL; - if (copy_from_user(iomsg->fast_iov, uiov, sizeof(*uiov))) - return -EFAULT; - sr->len = iomsg->fast_iov[0].iov_len; - iomsg->free_iov = NULL; + } else { + if (copy_from_user(iomsg->fast_iov, uiov, sizeof(*uiov))) + return -EFAULT; + sr->len = iomsg->fast_iov[0].iov_len; + iomsg->free_iov = NULL; + } } else { iomsg->free_iov = iomsg->fast_iov; ret = __import_iovec(READ, uiov, iov_len, UIO_FASTIOV, From patchwork Tue Jun 28 15:02:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEC12C433EF for ; Tue, 28 Jun 2022 15:02:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347677AbiF1PCu (ORCPT ); Tue, 28 Jun 2022 11:02:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347673AbiF1PCt (ORCPT ); Tue, 28 Jun 2022 11:02:49 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5725333E38 for ; Tue, 28 Jun 2022 08:02:48 -0700 (PDT) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25SAE7ZI013208 for ; Tue, 28 Jun 2022 08:02:48 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=htEEzCcE4766+UwdC1leFCQGb+BeCyqpCEFz/0YnVKE=; b=CDx4QR6emayqwiddP+de7/TmBe6lA0yOj4AW7JjXjypzFlQGOZkBWukq8CeAJfeeSDnV iB4hdJZUxAqklfngOqnBVYpD/FrvUAWgZ7HaBFEghtP63VmUHqhHTWx8nbStHBj7tMNY +SiHDFqyRbMRwnMDXLZokCDJqqcsprMTpT4= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gyxx31y7r-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:02:48 -0700 Received: from twshared25107.07.ash9.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:45 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id EF0E8244BBD3; Tue, 28 Jun 2022 08:02:37 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 4/8] io_uring: recycle buffers on error Date: Tue, 28 Jun 2022 08:02:24 -0700 Message-ID: <20220628150228.1379645-5-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 3n5km1FxofacmGiWCLxiucOLTcGKsVjf X-Proofpoint-ORIG-GUID: 3n5km1FxofacmGiWCLxiucOLTcGKsVjf X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_08,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Rather than passing an error back to the user with a buffer attached, recycle the buffer immediately. Signed-off-by: Dylan Yudaken --- io_uring/net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 5e84f7ab92a3..0268c4603f5d 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -481,10 +481,13 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (kmsg->free_iov) kfree(kmsg->free_iov); req->flags &= ~REQ_F_NEED_CLEANUP; - if (ret >= 0) + if (ret > 0) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; + else + io_kbuf_recycle(req, issue_flags); + cflags = io_put_kbuf(req, issue_flags); if (kmsg->msg.msg_inq) cflags |= IORING_CQE_F_SOCK_NONEMPTY; @@ -557,10 +560,13 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) req_set_fail(req); } - if (ret >= 0) + if (ret > 0) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; + else + io_kbuf_recycle(req, issue_flags); + cflags = io_put_kbuf(req, issue_flags); if (msg.msg_inq) cflags |= IORING_CQE_F_SOCK_NONEMPTY; From patchwork Tue Jun 28 15:02:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E7C8CCA47E for ; Tue, 28 Jun 2022 15:02:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347675AbiF1PC4 (ORCPT ); Tue, 28 Jun 2022 11:02:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347674AbiF1PCx (ORCPT ); Tue, 28 Jun 2022 11:02:53 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74EA033E3C for ; Tue, 28 Jun 2022 08:02:52 -0700 (PDT) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25SELS8A006984 for ; Tue, 28 Jun 2022 08:02:51 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=vTcz7CZ5Xy/EbxsuhtZpx3SZHM8cYYbE6e4zhycs9lc=; b=fXoi82n7Qk/U7XG6cdk5KxgMKxBwHCXZGAPtFBItNNao/N5A918riJ2mnEuwHFrM5lLr WYMnfodlkzgJJkWvxtwSFJwZtncpWL1E87eOfOcrq5DOz4+XvtarWkCFrSKfbPCoP9tv ZQvbXjkev9lF0HQPTYJOU86cszH41xRYn2E= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h03ax0a8x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:02:51 -0700 Received: from twshared14577.08.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:50 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 07168244BBD5; Tue, 28 Jun 2022 08:02:37 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 5/8] io_uring: clean up io_poll_check_events return values Date: Tue, 28 Jun 2022 08:02:25 -0700 Message-ID: <20220628150228.1379645-6-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: I37z561YIfZzfeZ6ENYjWojgiU0PS8In X-Proofpoint-ORIG-GUID: I37z561YIfZzfeZ6ENYjWojgiU0PS8In X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_08,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The values returned are a bit confusing, where 0 and 1 have implied meaning, so add some definitions for them. Signed-off-by: Dylan Yudaken --- io_uring/poll.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index fa25b88a7b93..fc3a5789d303 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -192,6 +192,11 @@ static void io_poll_remove_entries(struct io_kiocb *req) rcu_read_unlock(); } +enum { + IOU_POLL_DONE = 0, + IOU_POLL_NO_ACTION = 1, +}; + /* * All poll tw should go through this. Checks for poll events, manages * references, does rewait, etc. @@ -214,10 +219,11 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) /* tw handler should be the owner, and so have some references */ if (WARN_ON_ONCE(!(v & IO_POLL_REF_MASK))) - return 0; + return IOU_POLL_DONE; if (v & IO_POLL_CANCEL_FLAG) return -ECANCELED; + /* the mask was stashed in __io_poll_execute */ if (!req->cqe.res) { struct poll_table_struct pt = { ._key = req->apoll_events }; req->cqe.res = vfs_poll(req->file, &pt) & req->apoll_events; @@ -226,7 +232,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) if ((unlikely(!req->cqe.res))) continue; if (req->apoll_events & EPOLLONESHOT) - return 0; + return IOU_POLL_DONE; /* multishot, just fill a CQE and proceed */ if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { @@ -238,7 +244,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) return -ECANCELED; } else { ret = io_poll_issue(req, locked); - if (ret) + if (ret < 0) return ret; } @@ -248,7 +254,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) */ } while (atomic_sub_return(v & IO_POLL_REF_MASK, &req->poll_refs)); - return 1; + return IOU_POLL_NO_ACTION; } static void io_poll_task_func(struct io_kiocb *req, bool *locked) @@ -256,12 +262,11 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) int ret; ret = io_poll_check_events(req, locked); - if (ret > 0) + if (ret == IOU_POLL_NO_ACTION) return; - if (!ret) { + if (ret == IOU_POLL_DONE) { struct io_poll *poll = io_kiocb_to_cmd(req); - req->cqe.res = mangle_poll(req->cqe.res & poll->events); } else { req->cqe.res = ret; @@ -280,7 +285,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) int ret; ret = io_poll_check_events(req, locked); - if (ret > 0) + if (ret == IOU_POLL_NO_ACTION) return; io_poll_remove_entries(req); From patchwork Tue Jun 28 15:02:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ADD6C43334 for ; Tue, 28 Jun 2022 15:03:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347727AbiF1PDJ (ORCPT ); Tue, 28 Jun 2022 11:03:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347709AbiF1PDC (ORCPT ); Tue, 28 Jun 2022 11:03:02 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D89433E3C for ; Tue, 28 Jun 2022 08:03:00 -0700 (PDT) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 25SEooR9009892 for ; Tue, 28 Jun 2022 08:02:59 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=6OnrcbHVM3UEKAIV92LdKB+Frnz6Bl7ou2nLFHOv2Zc=; b=N4IVZi4jJCHrLBIj6fN6fiD8lY7YCuCq8F7ImS9e7pQJS3UBxgDNc6XV+TSVu+ac1dlC 7beisqKyzNSWV7zu5WYOklHemB576GXjzuroEzVRn7FkTpuQXyORTZeakwmZMdI7+LCG j82m0UpyBlTTirbiT6FJpSLUPd6MoECfCv8= Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3h03ru0386-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:02:59 -0700 Received: from snc-exhub201.TheFacebook.com (2620:10d:c085:21d::7) by snc-exhub103.TheFacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:56 -0700 Received: from twshared25107.07.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:55 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 0B8A4244BBD7; Tue, 28 Jun 2022 08:02:38 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 6/8] io_uring: add IOU_STOP_MULTISHOT return code Date: Tue, 28 Jun 2022 08:02:26 -0700 Message-ID: <20220628150228.1379645-7-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: Vwrlnq8E3N0yiQPQsPvpzOI0umY9KStC X-Proofpoint-GUID: Vwrlnq8E3N0yiQPQsPvpzOI0umY9KStC X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_08,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org For multishot we want a way to signal the caller that multishot has ended but also this might not be an error return. For example sockets return 0 when closed, which should end a multishot recv, but still have a CQE with result 0 Introduce IOU_STOP_MULTISHOT which does this and indicates that the return code is stored inside req->cqe Signed-off-by: Dylan Yudaken --- io_uring/io_uring.h | 7 +++++++ io_uring/poll.c | 9 +++++++-- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index f77e4a5403e4..e8da70781fa3 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -15,6 +15,13 @@ enum { IOU_OK = 0, IOU_ISSUE_SKIP_COMPLETE = -EIOCBQUEUED, + + /* + * Intended only when both REQ_F_POLLED and REQ_F_APOLL_MULTISHOT + * are set to indicate to the poll runner that multishot should be + * removed and the result is set on req->cqe.res. + */ + IOU_STOP_MULTISHOT = -ECANCELED, }; struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx); diff --git a/io_uring/poll.c b/io_uring/poll.c index fc3a5789d303..2054df9af291 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -195,6 +195,7 @@ static void io_poll_remove_entries(struct io_kiocb *req) enum { IOU_POLL_DONE = 0, IOU_POLL_NO_ACTION = 1, + IOU_POLL_REMOVE_POLL_USE_RES = 2, }; /* @@ -244,6 +245,8 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) return -ECANCELED; } else { ret = io_poll_issue(req, locked); + if (ret == IOU_STOP_MULTISHOT) + return IOU_POLL_REMOVE_POLL_USE_RES; if (ret < 0) return ret; } @@ -268,7 +271,7 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) if (ret == IOU_POLL_DONE) { struct io_poll *poll = io_kiocb_to_cmd(req); req->cqe.res = mangle_poll(req->cqe.res & poll->events); - } else { + } else if (ret != IOU_POLL_REMOVE_POLL_USE_RES) { req->cqe.res = ret; req_set_fail(req); } @@ -291,7 +294,9 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) io_poll_remove_entries(req); io_poll_tw_hash_eject(req, locked); - if (!ret) + if (ret == IOU_POLL_REMOVE_POLL_USE_RES) + io_req_complete_post(req); + else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else io_req_complete_failed(req, ret); From patchwork Tue Jun 28 15:02:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A299C43334 for ; Tue, 28 Jun 2022 15:02:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347696AbiF1PC5 (ORCPT ); Tue, 28 Jun 2022 11:02:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347694AbiF1PCy (ORCPT ); Tue, 28 Jun 2022 11:02:54 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1B9B33E9F for ; Tue, 28 Jun 2022 08:02:53 -0700 (PDT) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25SEdFhO030169 for ; Tue, 28 Jun 2022 08:02:52 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=GhUO5GqGv+DbKPtClNBIaCznVrmxio+cvLQXkR92xOo=; b=P7Zo9DXOH/pSsHHphilUequlMJmNVXTEOODghSi9E/AM2iAY0D/2jtx4h01a+fyRzOZ0 tiEFWtulHfEsPcW27dlvVSOzKSPwsRrz2M3ZJlRKjMMUtMD60rRTBwyMJe+IQYxynlyt EeWEJ+qs9GC8lnCWML3ZY8P5GNSYg/X6tUM= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3h03kf05qx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:02:52 -0700 Received: from twshared17349.03.ash7.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:51 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 10BBC244BBD9; Tue, 28 Jun 2022 08:02:38 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 7/8] io_uring: add IORING_RECV_MULTISHOT flag Date: Tue, 28 Jun 2022 08:02:27 -0700 Message-ID: <20220628150228.1379645-8-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: ZGubI1Skm2MdgQsk4hBtCHz3ztXYZIO8 X-Proofpoint-ORIG-GUID: ZGubI1Skm2MdgQsk4hBtCHz3ztXYZIO8 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_07,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Introduce multishot recv flag which will be used for multishot recv/recvmsg Signed-off-by: Dylan Yudaken --- include/uapi/linux/io_uring.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 09e7c3b13d2d..1e5bdb323184 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -259,8 +259,13 @@ enum io_uring_op { * or receive and arm poll if that yields an * -EAGAIN result, arm poll upfront and skip * the initial transfer attempt. + * + * IORING_RECV_MULTISHOT Multishot recv. Sets IORING_CQE_F_MORE if + * the handler will continue to report + * CQEs on behalf of the same SQE. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) +#define IORING_RECV_MULTISHOT (1U << 1) /* * accept flags stored in sqe->ioprio From patchwork Tue Jun 28 15:02:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12898425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 488B5C433EF for ; Tue, 28 Jun 2022 15:03:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347712AbiF1PDK (ORCPT ); Tue, 28 Jun 2022 11:03:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347710AbiF1PDC (ORCPT ); Tue, 28 Jun 2022 11:03:02 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F100233E91 for ; Tue, 28 Jun 2022 08:03:00 -0700 (PDT) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 25SEooRA009892 for ; Tue, 28 Jun 2022 08:03:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=XcSPD6iqW0QERWybzBYEyeXKnxx4udfiL5IBrKJFiyA=; b=DANhO+5t14dsEcpfq83Xhj87X5Uk3UqgTkLscjJ2V+FbBZQT5Pz/PwebdS/NbQK9qi3t ToIVonaWSUWTcJfyiYEyGRc3QNRr6qDY9udEudjSVXj5jgLETFk3kqD8KywBLaeGYLBQ XLH1SZCr2x+iB7Y2RxAmfRmm9321hTbIlyQ= Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3h03ru0386-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Jun 2022 08:03:00 -0700 Received: from twshared25107.07.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Tue, 28 Jun 2022 08:02:56 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 15830244BBDB; Tue, 28 Jun 2022 08:02:38 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov , CC: , , Dylan Yudaken Subject: [PATCH for-next 8/8] io_uring: multishot recv Date: Tue, 28 Jun 2022 08:02:28 -0700 Message-ID: <20220628150228.1379645-9-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220628150228.1379645-1-dylany@fb.com> References: <20220628150228.1379645-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: JejFK1ZmaBEwc1nhV5kjf_329Y1s2Uaq X-Proofpoint-GUID: JejFK1ZmaBEwc1nhV5kjf_329Y1s2Uaq X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-28_08,2022-06-28_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Support multishot receive for io_uring. Typical server applications will run a loop where for each recv CQE it requeues another recv/recvmsg. This can be simplified by using the existing multishot functionality combined with io_uring's provided buffers. The API is to add the IORING_RECV_MULTISHOT flag to the SQE. CQEs will then be posted (with IORING_CQE_F_MORE flag set) when data is available and is read. Once an error occurs or the socket ends, the multishot will be removed and a completion without IORING_CQE_F_MORE will be posted. The benefit to this is that the recv is much more performant. * Subsequent receives are queued up straight away without requiring the application to finish a processing loop. * If there are more data in the socket (sat the provided buffer size is smaller than the socket buffer) then the data is immediately returned, improving batching. * Poll is only armed once and reused, saving CPU cycles Signed-off-by: Dylan Yudaken --- io_uring/net.c | 93 +++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 81 insertions(+), 12 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 0268c4603f5d..9bf8c6c0b549 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -389,6 +389,8 @@ int io_recvmsg_prep_async(struct io_kiocb *req) return ret; } +#define RECVMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECV_MULTISHOT) + int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *sr = io_kiocb_to_cmd(req); @@ -399,13 +401,22 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); sr->len = READ_ONCE(sqe->len); sr->flags = READ_ONCE(sqe->addr2); - if (sr->flags & ~IORING_RECVSEND_POLL_FIRST) + if (sr->flags & ~(RECVMSG_FLAGS)) return -EINVAL; sr->msg_flags = READ_ONCE(sqe->msg_flags) | MSG_NOSIGNAL; if (sr->msg_flags & MSG_DONTWAIT) req->flags |= REQ_F_NOWAIT; if (sr->msg_flags & MSG_ERRQUEUE) req->flags |= REQ_F_CLEAR_POLLIN; + if (sr->flags & IORING_RECV_MULTISHOT) { + if (!(req->flags & REQ_F_BUFFER_SELECT)) + return -EINVAL; + if (sr->msg_flags & MSG_WAITALL) + return -EINVAL; + if (req->opcode == IORING_OP_RECV && sr->len) + return -EINVAL; + req->flags |= REQ_F_APOLL_MULTISHOT; + } #ifdef CONFIG_COMPAT if (req->ctx->compat) @@ -415,6 +426,14 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static inline void io_recv_prep_retry(struct io_kiocb *req) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req); + + sr->done_io = 0; + sr->len = 0; /* get from the provided buffer */ +} + int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req); @@ -424,6 +443,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) unsigned flags; int ret, min_ret = 0; bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; + size_t len = sr->len; sock = sock_from_file(req->file); if (unlikely(!sock)) @@ -442,16 +462,17 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_msg(req, kmsg); +retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; - buf = io_buffer_select(req, &sr->len, issue_flags); + buf = io_buffer_select(req, &len, issue_flags); if (!buf) return -ENOBUFS; kmsg->fast_iov[0].iov_base = buf; - kmsg->fast_iov[0].iov_len = sr->len; + kmsg->fast_iov[0].iov_len = len; iov_iter_init(&kmsg->msg.msg_iter, READ, kmsg->fast_iov, 1, - sr->len); + len); } flags = sr->msg_flags; @@ -463,8 +484,15 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_get_inq = 1; ret = __sys_recvmsg_sock(sock, &kmsg->msg, sr->umsg, kmsg->uaddr, flags); if (ret < min_ret) { - if (ret == -EAGAIN && force_nonblock) - return io_setup_async_msg(req, kmsg); + if (ret == -EAGAIN && force_nonblock) { + ret = io_setup_async_msg(req, kmsg); + if (ret == -EAGAIN && (req->flags & IO_APOLL_MULTI_POLLED) == + IO_APOLL_MULTI_POLLED) { + io_kbuf_recycle(req, issue_flags); + ret = IOU_ISSUE_SKIP_COMPLETE; + } + return ret; + } if (ret == -ERESTARTSYS) ret = -EINTR; if (ret > 0 && io_net_retry(sock, flags)) { @@ -491,8 +519,24 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) cflags = io_put_kbuf(req, issue_flags); if (kmsg->msg.msg_inq) cflags |= IORING_CQE_F_SOCK_NONEMPTY; + + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { + io_req_set_res(req, ret, cflags); + return IOU_OK; + } + + if (ret > 0) { + if (io_post_aux_cqe(req->ctx, req->cqe.user_data, ret, + cflags | IORING_CQE_F_MORE)) { + io_recv_prep_retry(req); + goto retry_multishot; + } else { + ret = -ECANCELED; + } + } + io_req_set_res(req, ret, cflags); - return IOU_OK; + return req->flags & REQ_F_POLLED ? IOU_STOP_MULTISHOT : ret; } int io_recv(struct io_kiocb *req, unsigned int issue_flags) @@ -505,6 +549,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) unsigned flags; int ret, min_ret = 0; bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; + size_t len = sr->len; if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) @@ -514,16 +559,17 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!sock)) return -ENOTSOCK; +retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; - buf = io_buffer_select(req, &sr->len, issue_flags); + buf = io_buffer_select(req, &len, issue_flags); if (!buf) return -ENOBUFS; sr->buf = buf; } - ret = import_single_range(READ, sr->buf, sr->len, &iov, &msg.msg_iter); + ret = import_single_range(READ, sr->buf, len, &iov, &msg.msg_iter); if (unlikely(ret)) goto out_free; @@ -543,8 +589,14 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) ret = sock_recvmsg(sock, &msg, flags); if (ret < min_ret) { - if (ret == -EAGAIN && force_nonblock) - return -EAGAIN; + if (ret == -EAGAIN && force_nonblock) { + if ((req->flags & IO_APOLL_MULTI_POLLED) == IO_APOLL_MULTI_POLLED) { + io_kbuf_recycle(req, issue_flags); + ret = IOU_ISSUE_SKIP_COMPLETE; + } + + return ret; + } if (ret == -ERESTARTSYS) ret = -EINTR; if (ret > 0 && io_net_retry(sock, flags)) { @@ -570,8 +622,25 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) cflags = io_put_kbuf(req, issue_flags); if (msg.msg_inq) cflags |= IORING_CQE_F_SOCK_NONEMPTY; + + + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { + io_req_set_res(req, ret, cflags); + return IOU_OK; + } + + if (ret > 0) { + if (io_post_aux_cqe(req->ctx, req->cqe.user_data, ret, + cflags | IORING_CQE_F_MORE)) { + io_recv_prep_retry(req); + goto retry_multishot; + } else { + ret = -ECANCELED; + } + } + io_req_set_res(req, ret, cflags); - return IOU_OK; + return req->flags & REQ_F_POLLED ? IOU_STOP_MULTISHOT : ret; } int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)