From patchwork Thu Nov 24 09:35:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80F76C4332F for ; Thu, 24 Nov 2022 09:36:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229693AbiKXJga (ORCPT ); Thu, 24 Nov 2022 04:36:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229669AbiKXJg3 (ORCPT ); Thu, 24 Nov 2022 04:36:29 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 41E07109589 for ; Thu, 24 Nov 2022 01:36:29 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANHsDVY024539 for ; Thu, 24 Nov 2022 01:36:29 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=+buD+PM5wUy6zaCH/kJrPBrEOBD7JPENkycFACOudHc=; b=SZTJmXXzhmC1o6eBBrAjan8GSl0FXzlbVZkL/GW2GDhd/1nx+KM9kqbDiQqNL3f2thTF ACnW9mh0VrEMyhlntAUTzR5C0pIfo+Titwabk4PKqWesc13+BsMGXjniHJtGnpTYFiyM DS2ApSVQTpY60LXgY07Z5Nf/laLjoFtq+bmUMBn2+qY++lKO6nVoSPOMzcIjlmlyPAlJ Wq7AW1T7sHexY+X71xnYg7YS8MSNhASu0PLTlYyR1hgqOx5DL1MpTzZibMsgcN7KuRAz LySFRTbXrb2MbAjV4oUNJsqv9YXRJj2NnrAr/b4xfd8301toZiI7tWcxO6rDavejoWQf lg== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m1cg3s8wk-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:29 -0800 Received: from twshared7043.05.ash9.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:28 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id D7124A173A0A; Thu, 24 Nov 2022 01:36:18 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 1/9] io_uring: io_req_complete_post should defer if available Date: Thu, 24 Nov 2022 01:35:51 -0800 Message-ID: <20221124093559.3780686-2-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: MxDcMgOR_tS4DbI-l14HKJb0RQkgPzd2 X-Proofpoint-GUID: MxDcMgOR_tS4DbI-l14HKJb0RQkgPzd2 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org For consistency always defer completion if specified in the issue flags. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cc27413129fc..ec23ebb63489 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -852,7 +852,9 @@ static void __io_req_complete_post(struct io_kiocb *req) void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { - if (!(issue_flags & IO_URING_F_UNLOCKED) || + if (issue_flags & IO_URING_F_COMPLETE_DEFER) { + io_req_complete_defer(req); + } else if (!(issue_flags & IO_URING_F_UNLOCKED) || !(req->ctx->flags & IORING_SETUP_IOPOLL)) { __io_req_complete_post(req); } else { From patchwork Thu Nov 24 09:35:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE78AC4332F for ; Thu, 24 Nov 2022 09:36:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229632AbiKXJgj (ORCPT ); Thu, 24 Nov 2022 04:36:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229755AbiKXJge (ORCPT ); Thu, 24 Nov 2022 04:36:34 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2F3C122941 for ; Thu, 24 Nov 2022 01:36:32 -0800 (PST) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2ANHsAvp030074 for ; Thu, 24 Nov 2022 01:36:32 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=mgxNfn3aBEluRQnXqAqEqXdRp2XqDR3ffSyAA3dJMNQ=; b=XsbTsTznG+V6sJCTbTb54X3PL4fSjXBt9gn2cVdbhJRdunff9PHO0ZRtDYLKZx0ja4jj zaezayMfDnAYlM4WqYHl8Vu36O1oZ20o/+8KCV9koF2a2uCrd6bA22NsyvJU5VkVNjw4 iT0Eew6CeBg3EAMyb7iTcM+BWbK5ZBUIA+Q7PXqsjieG7kRs5Y7XjdewqW/3r5QfL7AV 3V2AveSxnU3yRstMxjFpUwl6HlhUI9sRuoUN/RMfrKsdRb22ThRolPWGaKXN9PV6INWB qFNVJ5ucmLBVLT0E7Me6D5aRx+nLR3AZ2DLK4xY3FoPKauX0xWyQrm0mzz6qCbqwmLlw Dg== Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net (PPS) with ESMTPS id 3m1c7rhj30-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:31 -0800 Received: from twshared41876.03.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:30 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 29262A173A0D; Thu, 24 Nov 2022 01:36:19 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 2/9] io_uring: always lock in io_apoll_task_func Date: Thu, 24 Nov 2022 01:35:52 -0800 Message-ID: <20221124093559.3780686-3-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: Z_0JnM4zdQdfLXbI23D1qMg6k-nthnHX X-Proofpoint-ORIG-GUID: Z_0JnM4zdQdfLXbI23D1qMg6k-nthnHX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This is required for the failure case (io_req_complete_failed) and is missing. The alternative would be to only lock in the failure path, however all of the non-error paths in io_poll_check_events that do not do not return IOU_POLL_NO_ACTION end up locking anyway. The only extraneous lock would be for the multishot poll overflowing the CQE ring, however multishot poll would probably benefit from being locked as it will allow completions to be batched. So it seems reasonable to lock always. Signed-off-by: Dylan Yudaken --- io_uring/poll.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 4624e5eba63e..42aa10b50f6c 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -308,11 +308,12 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) if (ret == IOU_POLL_NO_ACTION) return; + io_tw_lock(req->ctx, locked); io_poll_remove_entries(req); io_poll_tw_hash_eject(req, locked); if (ret == IOU_POLL_REMOVE_POLL_USE_RES) - io_req_complete_post_tw(req, locked); + io_req_task_complete(req, locked); else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else From patchwork Thu Nov 24 09:35:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4453C433FE for ; Thu, 24 Nov 2022 09:36:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229652AbiKXJg3 (ORCPT ); Thu, 24 Nov 2022 04:36:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229632AbiKXJg3 (ORCPT ); Thu, 24 Nov 2022 04:36:29 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A5E3109589 for ; Thu, 24 Nov 2022 01:36:28 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANHsDrB024521 for ; Thu, 24 Nov 2022 01:36:28 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=uJ6b5V/7s0WTTTLrp0iHQXvNPAcNVs0y33SVziYDgp0=; b=WXAC2kvsumo91eFezRKPoNJ5yOqQW8dyVqlkV/Tf8aXA+ypo8VMf0w30/hSPVuGlfc/K V5cyVFX/uPPumV5XwoWrgzZFUNo9VDXcOkeDWkDrm9Q+vig5mt6S+5UqiVsZUAfSoKoR mZ9+c5VizhwxFug0BkPFnThePiTyC7IkU3IYOZrWtwwWZH8oDKdHed9NkSPnLrn10Qlt Y7AW5ohnt2EIdhYIG3SN86a66ZkXrcRhYs017QG1JF6YKW3/XfjiIlVlbxX5lRHKX3cL BbuLhpjp/asuPuuxSU8mXE1OD2ezfyunnPGUEXvSwuCVN0YI/QUazJFPCgPBIo9W0EdZ Jw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m1cg3s8wj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:28 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:27 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 1B33FA173A10; Thu, 24 Nov 2022 01:36:19 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 3/9] io_uring: defer all io_req_complete_failed Date: Thu, 24 Nov 2022 01:35:53 -0800 Message-ID: <20221124093559.3780686-4-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: IuVVdh6wHwPL9oQ6hJX4K5pjxq1Qt0-0 X-Proofpoint-GUID: IuVVdh6wHwPL9oQ6hJX4K5pjxq1Qt0-0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org All failures happen under lock now, and can be deferred. To be consistent when the failure has happened after some multishot cqe has been deferred (and keep ordering), always defer failures. To make this obvious at the caller (and to help prevent a future bug) rename io_req_complete_failed to io_req_defer_failed. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 17 ++++++++--------- io_uring/io_uring.h | 2 +- io_uring/poll.c | 2 +- 3 files changed, 10 insertions(+), 11 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ec23ebb63489..cbd271b6188a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -866,7 +866,7 @@ void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) } } -void io_req_complete_failed(struct io_kiocb *req, s32 res) +void io_req_defer_failed(struct io_kiocb *req, s32 res) __must_hold(&ctx->uring_lock) { const struct io_op_def *def = &io_op_defs[req->opcode]; @@ -877,7 +877,7 @@ void io_req_complete_failed(struct io_kiocb *req, s32 res) io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); if (def->fail) def->fail(req); - io_req_complete_post(req, 0); + io_req_complete_defer(req); } /* @@ -1233,9 +1233,8 @@ int io_run_local_work(struct io_ring_ctx *ctx) static void io_req_task_cancel(struct io_kiocb *req, bool *locked) { - /* not needed for normal modes, but SQPOLL depends on it */ io_tw_lock(req->ctx, locked); - io_req_complete_failed(req, req->cqe.res); + io_req_defer_failed(req, req->cqe.res); } void io_req_task_submit(struct io_kiocb *req, bool *locked) @@ -1245,7 +1244,7 @@ void io_req_task_submit(struct io_kiocb *req, bool *locked) if (likely(!(req->task->flags & PF_EXITING))) io_queue_sqe(req); else - io_req_complete_failed(req, -EFAULT); + io_req_defer_failed(req, -EFAULT); } void io_req_task_queue_fail(struct io_kiocb *req, int ret) @@ -1632,7 +1631,7 @@ static __cold void io_drain_req(struct io_kiocb *req) ret = io_req_prep_async(req); if (ret) { fail: - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); return; } io_prep_async_link(req); @@ -1862,7 +1861,7 @@ static void io_queue_async(struct io_kiocb *req, int ret) struct io_kiocb *linked_timeout; if (ret != -EAGAIN || (req->flags & REQ_F_NOWAIT)) { - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); return; } @@ -1912,14 +1911,14 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) */ req->flags &= ~REQ_F_HARDLINK; req->flags |= REQ_F_LINK; - io_req_complete_failed(req, req->cqe.res); + io_req_defer_failed(req, req->cqe.res); } else if (unlikely(req->ctx->drain_active)) { io_drain_req(req); } else { int ret = io_req_prep_async(req); if (unlikely(ret)) - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); else io_queue_iowq(req, NULL); } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index b5b80bf03385..a26d5aa7f3f3 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -30,7 +30,7 @@ bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); -void io_req_complete_failed(struct io_kiocb *req, s32 res); +void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow); diff --git a/io_uring/poll.c b/io_uring/poll.c index 42aa10b50f6c..4bd43e6f5b72 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -317,7 +317,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); } static void __io_poll_execute(struct io_kiocb *req, int mask) From patchwork Thu Nov 24 09:35:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C76AC4332F for ; Thu, 24 Nov 2022 09:36:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229753AbiKXJgm (ORCPT ); Thu, 24 Nov 2022 04:36:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229773AbiKXJgl (ORCPT ); Thu, 24 Nov 2022 04:36:41 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6EC112294A for ; Thu, 24 Nov 2022 01:36:39 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AO7r6NK026397 for ; Thu, 24 Nov 2022 01:36:39 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=s2048-2021-q4; bh=lGhq9H6BOQHa/c7KCccQkOPCfll7ETMh/0j2/9zqCf4=; b=AwOxUNVFCz+bP/FRh/NEL8vdW70s2q4prq6EwuCGSUegWtIAtmNhTfuRDh2QP1FJG9TM w0XbB7uUN1anEtqqKvegLpTLEX2TeUJoLwizZd+pDoMEUTKdGL0DOf21ZPztlT8aJ6tI TvEUZkxhWWzRzQW37nzZOZS1AIeaWru0WWCfB27ZtfNRcd/Wr26FfzxWQfDmk+mfJeq4 KLngB8RhIeOBhIXIvzCK4fvyvDcjNCm3+cItS0CFpg93bkGYdoMSZ59P0bF1iZUOucK0 NGyvZw+96Evmbo9g22iwaMYo5UDUY5vNv4JeFSVhFy4Dhwuftg/tU/3FHbrhL7rRGi1Z oA== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m1cg3s8x9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:39 -0800 Received: from twshared7043.05.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:38 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id C360BA173A14; Thu, 24 Nov 2022 01:36:20 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 4/9] io_uring: allow defer completion for aux posted cqes Date: Thu, 24 Nov 2022 01:35:54 -0800 Message-ID: <20221124093559.3780686-5-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: -YVLCDuxVHbCkYTYeHhBj8PMqWd-TCsX X-Proofpoint-GUID: -YVLCDuxVHbCkYTYeHhBj8PMqWd-TCsX X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Multishot ops cannot use the compl_reqs list as the request must stay in the poll list, but that means they need to run each completion without benefiting from batching. Here introduce batching infrastructure for only small (ie 16 byte) CQEs. This restriction is ok because there are no use cases posting 32 byte CQEs. In the ring keep a batch of up to 16 posted results, and flush in the same way as compl_reqs. 16 was chosen through experimentation on a microbenchmark ([1]), as well as trying not to increase the size of the ring too much. This increases the size to 1472 bytes from 1216. [1]: https://github.com/DylanZA/liburing/commit/9ac66b36bcf4477bfafeff1c5f107896b7ae31cf Run with $ make -j && ./benchmark/reg.b -s 1 -t 2000 -r 10 Gives results: baseline 8309 k/s 8 18807 k/s 16 19338 k/s 32 20134 k/s Suggested-by: Pavel Begunkov Signed-off-by: Dylan Yudaken --- include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 27 ++++++++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index f5b687a787a3..accdfecee953 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -174,7 +174,9 @@ struct io_submit_state { bool plug_started; bool need_plug; unsigned short submit_nr; + unsigned int cqes_count; struct blk_plug plug; + struct io_uring_cqe cqes[16]; }; struct io_ev_fd { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cbd271b6188a..53b61b5cde80 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -167,7 +167,8 @@ EXPORT_SYMBOL(io_uring_get_socket); static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { - if (!wq_list_empty(&ctx->submit_state.compl_reqs)) + if (!wq_list_empty(&ctx->submit_state.compl_reqs) || + ctx->submit_state.cqes_count) __io_submit_flush_completions(ctx); } @@ -802,6 +803,21 @@ bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags return false; } +static void __io_flush_post_cqes(struct io_ring_ctx *ctx) + __must_hold(&ctx->uring_lock) +{ + struct io_submit_state *state = &ctx->submit_state; + unsigned int i; + + lockdep_assert_held(&ctx->uring_lock); + for (i = 0; i < state->cqes_count; i++) { + struct io_uring_cqe *cqe = &state->cqes[i]; + + io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags, true); + } + state->cqes_count = 0; +} + bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow) @@ -1325,6 +1341,9 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_submit_state *state = &ctx->submit_state; io_cq_lock(ctx); + /* must come first to preserve CQE ordering in failure cases */ + if (state->cqes_count) + __io_flush_post_cqes(ctx); wq_list_for_each(node, prev, &state->compl_reqs) { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); @@ -1334,8 +1353,10 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) } io_cq_unlock_post(ctx); - io_free_batch_list(ctx, state->compl_reqs.first); - INIT_WQ_LIST(&state->compl_reqs); + if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { + io_free_batch_list(ctx, state->compl_reqs.first); + INIT_WQ_LIST(&state->compl_reqs); + } } /* From patchwork Thu Nov 24 09:35:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DCE6C43219 for ; Thu, 24 Nov 2022 09:36:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229755AbiKXJgj (ORCPT ); Thu, 24 Nov 2022 04:36:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229766AbiKXJge (ORCPT ); Thu, 24 Nov 2022 04:36:34 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 521E8122954 for ; Thu, 24 Nov 2022 01:36:33 -0800 (PST) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2ANHsBM4030099 for ; Thu, 24 Nov 2022 01:36:32 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=jmOelkJnNh5qVKmW4sT7iMOybubcd1zgJvtXLSygZ1o=; b=nReQ+TAg5X4FaP3daLRc47xEeQO+jpRY5nvtXch9uAh+SGEMvpX19J272cJMig633xrW FooAwTVxnMfozRhE9IGF682VdLiCNgm/eBrjFODDt9dA5JPOPMkoEoBTB1kzEYZqXD1X D1ZAkGp76+lQMdO/cHXFM1egPBJqoiO4UTh56HjcXPHevzp/Th3IzqZC//+qiDz0fUMQ Vc8ItCHgY9+sr71prX3gAMWxqg6+gmDPK3fSNy8/rHebXUg9dPe8qF0IbcCtsgZFP8Bv F7T+a9+HkX3aPK784neqA+td8nM8FRgESjeajFCY2PIOX8SiMFAvvn4td+0lZ8W5DFZ2 jw== Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net (PPS) with ESMTPS id 3m1c7rhj33-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:32 -0800 Received: from twshared41876.03.ash8.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:31 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id E93EAA173A16; Thu, 24 Nov 2022 01:36:20 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 5/9] io_uring: add io_aux_cqe which allows deferred completion Date: Thu, 24 Nov 2022 01:35:55 -0800 Message-ID: <20221124093559.3780686-6-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: IfeZgjiiZen9I_TWB9DLPdhWtuLubLZS X-Proofpoint-ORIG-GUID: IfeZgjiiZen9I_TWB9DLPdhWtuLubLZS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use the just introduced deferred post cqe completion state when possible in io_aux_cqe. If not possible fallback to io_post_aux_cqe. This introduces a complication because of allow_overflow. For deferred completions we cannot know without locking the completion_lock if it will overflow (and even if we locked it, another post could sneak in and cause this cqe to be in overflow). However since overflow protection is mostly a best effort defence in depth to prevent infinite loops of CQEs for poll, just checking the overflow bit is going to be good enough and will result in at most 16 (array size of deferred cqes) overflows. Suggested-by: Pavel Begunkov Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 34 ++++++++++++++++++++++++++++++++++ io_uring/io_uring.h | 2 ++ io_uring/net.c | 7 ++++--- io_uring/poll.c | 4 ++-- 4 files changed, 42 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 53b61b5cde80..4f48e8a919a2 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -830,6 +830,40 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, return filled; } +bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, + bool allow_overflow) +{ + struct io_uring_cqe *cqe; + unsigned int length; + + if (!defer) + return io_post_aux_cqe(ctx, user_data, res, cflags, allow_overflow); + + length = ARRAY_SIZE(ctx->submit_state.cqes); + + lockdep_assert_held(&ctx->uring_lock); + + if (ctx->submit_state.cqes_count == length) { + io_cq_lock(ctx); + __io_flush_post_cqes(ctx); + /* no need to flush - flush is deferred */ + spin_unlock(&ctx->completion_lock); + } + + /* For defered completions this is not as strict as it is otherwise, + * however it's main job is to prevent unbounded posted completions, + * and in that it works just as well. + */ + if (!allow_overflow && test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) + return false; + + cqe = &ctx->submit_state.cqes[ctx->submit_state.cqes_count++]; + cqe->user_data = user_data; + cqe->res = res; + cqe->flags = cflags; + return true; +} + static void __io_req_complete_post(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index a26d5aa7f3f3..dd02adf3d0df 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -36,6 +36,8 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags bool allow_overflow); bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow); +bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, + bool allow_overflow); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); static inline void io_req_complete_post_tw(struct io_kiocb *req, bool *locked) diff --git a/io_uring/net.c b/io_uring/net.c index 0de6f78ad978..90342dcb6b1d 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -601,8 +601,8 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, } if (!mshot_finished) { - if (io_post_aux_cqe(req->ctx, req->cqe.user_data, *ret, - cflags | IORING_CQE_F_MORE, true)) { + if (io_aux_cqe(req->ctx, issue_flags & IO_URING_F_COMPLETE_DEFER, + req->cqe.user_data, *ret, cflags | IORING_CQE_F_MORE, true)) { io_recv_prep_retry(req); return false; } @@ -1320,7 +1320,8 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags) if (ret < 0) return ret; - if (io_post_aux_cqe(ctx, req->cqe.user_data, ret, IORING_CQE_F_MORE, true)) + if (io_aux_cqe(ctx, issue_flags & IO_URING_F_COMPLETE_DEFER, + req->cqe.user_data, ret, IORING_CQE_F_MORE, true)) goto retry; return -ECANCELED; diff --git a/io_uring/poll.c b/io_uring/poll.c index 4bd43e6f5b72..922c1a366c41 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -252,8 +252,8 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_post_aux_cqe(ctx, req->cqe.user_data, - mask, IORING_CQE_F_MORE, false)) { + if (!io_aux_cqe(ctx, *locked, req->cqe.user_data, + mask, IORING_CQE_F_MORE, false)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; } From patchwork Thu Nov 24 09:35:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1440C4332F for ; Thu, 24 Nov 2022 09:36:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229766AbiKXJgk (ORCPT ); Thu, 24 Nov 2022 04:36:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229829AbiKXJgi (ORCPT ); Thu, 24 Nov 2022 04:36:38 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 577AA12297E for ; Thu, 24 Nov 2022 01:36:37 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANHsDVK024532 for ; Thu, 24 Nov 2022 01:36:37 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=ypQnsRu7Buxort9bAj70n6tcg67nT7jZNP2MzV/nQ6Y=; b=hBe8Rr9MLcPbOyhk5sWaEXtWeKUl4+l8/4duBGh/31+4dk/k0FuD9h4e8wdC9d4uSYVR n+soqGwUFaCz6lOt95br0H5azok4W7CbFKgcGRepTG9NijOztgAUx8cULTR6zVOS4tPY H+ftY7bpDVXAV+78CJzidTgZvsoslp3qaYBqdwcCKlw+W6kIxe1OxDyc8StaYo3oZsow 7Yysw3mX5me/EaKjtBK9PE92Z016z7j9BW3UGcEpbCsWYvthEtEbKRF+gXvamh9yZFu8 8UTuH3e5MWNQjzsCqek/e9MF3rW246onjJqjF3VPDVYN3xLMdE+Tqa45dEw67o3vvAYq Cg== Received: from maileast.thefacebook.com ([163.114.130.3]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m1cg3s8x6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:36 -0800 Received: from twshared8047.05.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:35 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 0019BA173A18; Thu, 24 Nov 2022 01:36:20 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 6/9] io_uring: make io_fill_cqe_aux static Date: Thu, 24 Nov 2022 01:35:56 -0800 Message-ID: <20221124093559.3780686-7-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: oTAvkL_OUhYR54syrn1gtWFhAaQqVVIA X-Proofpoint-GUID: oTAvkL_OUhYR54syrn1gtWFhAaQqVVIA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This is only used in io_uring.c Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 4 ++-- io_uring/io_uring.h | 2 -- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4f48e8a919a2..92a7d6deacb6 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -770,8 +770,8 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow) return &rings->cqes[off]; } -bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow) +static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, + bool allow_overflow) { struct io_uring_cqe *cqe; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index dd02adf3d0df..46694f40bf72 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -34,8 +34,6 @@ void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow); -bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow); bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, bool allow_overflow); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); From patchwork Thu Nov 24 09:35:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEEDCC4332F for ; Thu, 24 Nov 2022 09:37:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229794AbiKXJgz (ORCPT ); Thu, 24 Nov 2022 04:36:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229791AbiKXJgq (ORCPT ); Thu, 24 Nov 2022 04:36:46 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE33A125216 for ; Thu, 24 Nov 2022 01:36:44 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANHsNZi025297 for ; Thu, 24 Nov 2022 01:36:44 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=JSS5CL/30b52rzl+j3X2NYB+cHO3TOtfcWoEpuDvd48=; b=ZKUHakm7eEjMuGF5u/lHVdFLT8SdKHYZMghKYkLernKohw0Kf15dWhFCTB4VUqq6BUIh IIXTlMBhrT6Ht7zr8If9ej8qdWl7kzLdKsv04myCc54DhkKVjUWhmdvpvfR45eaUXYnK U3+Eev/RUvrIY5nrn3fLYYZo6JHeCi9ol3MtmpK4a3YsKIWa57gm1mwHmxJM5FoEecib LnPNvv1whTETOTHNFcB00q6mDOwplniqzXQkXqNj87u5a3hGZR97ipaF9fBuZExwgvqz dmozI2aNLHep7m+q4Ef0TDj8TOCIyx2v734eqfnT3l94/5kDdlb+p5VKs9zpUjklZxXk Hw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m0m1a3193-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:44 -0800 Received: from twshared41876.03.ash8.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:42 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 08955A173A1A; Thu, 24 Nov 2022 01:36:21 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 7/9] io_uring: add lockdep assertion in io_fill_cqe_aux Date: Thu, 24 Nov 2022 01:35:57 -0800 Message-ID: <20221124093559.3780686-8-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: hp1FH_P30e73qoFkqyQWexkK6L-EYcZj X-Proofpoint-GUID: hp1FH_P30e73qoFkqyQWexkK6L-EYcZj X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add an assertion for the completion lock to io_fill_cqe_aux Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 92a7d6deacb6..e937e5d68b22 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -775,6 +775,8 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 { struct io_uring_cqe *cqe; + lockdep_assert_held(&ctx->completion_lock); + ctx->cq_extra++; /* From patchwork Thu Nov 24 09:35:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10A6CC4332F for ; Thu, 24 Nov 2022 09:36:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229883AbiKXJgt (ORCPT ); Thu, 24 Nov 2022 04:36:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229870AbiKXJgq (ORCPT ); Thu, 24 Nov 2022 04:36:46 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE2B412296E for ; Thu, 24 Nov 2022 01:36:44 -0800 (PST) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2ANHsBhW030115 for ; Thu, 24 Nov 2022 01:36:44 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=iN/SL2jD0KHAuMW4C6ATqB31yLcyixW7ZJWCOlrAY5A=; b=faPKYbxoWBsXMLAkii4rfbrOoheZ7MEyy1tJv9CQvB5mdv8nukVgRx8UDk1U/jovNqe8 HArKxqoopaCtkP9ydELrPKbyuu2pHLkWwdkq/lY6lgv7KNT1j6Zu9cFKravxzVJCOlmZ LIJ8QZVSKob3HVEGAqG0CZhNdKumZ41KM6DLv7JmFOYtDYohQLOvrobCsfzW4GrXcne+ 9bQdVc5Ke1Rax/bRDz8qUCGZGYLHEQVyh+2Vk0jHvUvj+hC8p3thGejpMPAIvLiw2bR0 LuYPZHC7VCApKz43Wiam/LzzFUgvLHPm4QHdozYvRF4DML8dt+Ypsbll0rSDBbyIXOG3 Bw== Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net (PPS) with ESMTPS id 3m1c7rhj3q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:44 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:42 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 0DBEFA173A1C; Thu, 24 Nov 2022 01:36:21 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 8/9] io_uring: remove overflow param from io_post_aux_cqe Date: Thu, 24 Nov 2022 01:35:58 -0800 Message-ID: <20221124093559.3780686-9-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: PYPK1MMlYVAhvatOFneQsLjc29HVoRt1 X-Proofpoint-ORIG-GUID: PYPK1MMlYVAhvatOFneQsLjc29HVoRt1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The only call sites which would not allow overflow are also call sites which would use the io_aux_cqe as they care about ordering. So remove this parameter from io_post_aux_cqe. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 12 ++++++++---- io_uring/io_uring.h | 3 +-- io_uring/msg_ring.c | 4 ++-- io_uring/rsrc.c | 4 ++-- 4 files changed, 13 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e937e5d68b22..f0e5ae7740cf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -820,9 +820,8 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) state->cqes_count = 0; } -bool io_post_aux_cqe(struct io_ring_ctx *ctx, - u64 user_data, s32 res, u32 cflags, - bool allow_overflow) +static bool __io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, + bool allow_overflow) { bool filled; @@ -832,6 +831,11 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, return filled; } +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) +{ + return __io_post_aux_cqe(ctx, user_data, res, cflags, true); +} + bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, bool allow_overflow) { @@ -839,7 +843,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 unsigned int length; if (!defer) - return io_post_aux_cqe(ctx, user_data, res, cflags, allow_overflow); + return __io_post_aux_cqe(ctx, user_data, res, cflags, allow_overflow); length = ARRAY_SIZE(ctx->submit_state.cqes); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 46694f40bf72..dcb8e3468f1d 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -32,8 +32,7 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); -bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow); +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, bool allow_overflow); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 90d2fc6fd80e..afb543aab9f6 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -31,7 +31,7 @@ static int io_msg_ring_data(struct io_kiocb *req) if (msg->src_fd || msg->dst_fd || msg->flags) return -EINVAL; - if (io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0, true)) + if (io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0)) return 0; return -EOVERFLOW; @@ -116,7 +116,7 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) * completes with -EOVERFLOW, then the sender must ensure that a * later IORING_OP_MSG_RING delivers the message. */ - if (!io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0, true)) + if (!io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0)) ret = -EOVERFLOW; out_unlock: io_double_unlock_ctx(ctx, target_ctx, issue_flags); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 187f1c83e779..133608200769 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -170,10 +170,10 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) if (prsrc->tag) { if (ctx->flags & IORING_SETUP_IOPOLL) { mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0, true); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); mutex_unlock(&ctx->uring_lock); } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0, true); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); } } From patchwork Thu Nov 24 09:35:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13054745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08D6FC352A1 for ; Thu, 24 Nov 2022 09:36:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229758AbiKXJgm (ORCPT ); Thu, 24 Nov 2022 04:36:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229787AbiKXJgl (ORCPT ); Thu, 24 Nov 2022 04:36:41 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73FB011E72A for ; Thu, 24 Nov 2022 01:36:40 -0800 (PST) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANNUF14005926 for ; Thu, 24 Nov 2022 01:36:40 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=xdIQVOTGgm/gk39ZeUZ9hdqKda+zof3w1vIvgJkVPm4=; b=HWdueoh8mM1O+2YRKophKISAr5RKoOtA80KZNi1p1JgSVuIQwe0bsQIG9k4/o0PMLlvT qDCUmf3Smgj173G57tMRFK7CfShwtBonREVitok3drFIiFmxSGYsKGcTeyHX449fL77N os9O0CC2mgg1yCZrAKNWSN82ayMmcf2MkYFsIXih6X1hVS6qRKwjxYriB8evy4cgU+2f rfe4iT/dbbkzif3jrzD+aLP2tiZ0HoolD3Aswye4Q5iAZGAAGRwKsp+U17bEZda51et+ YfqqSFJmNskjfKhqTkAy3OXLuQlLCFKe8VRgxM1wgqLl49oTzAr0NNoNvtJALha8c5nQ Mw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m1w88tr5t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 24 Nov 2022 01:36:39 -0800 Received: from snc-exhub201.TheFacebook.com (2620:10d:c085:21d::7) by snc-exhub203.TheFacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:39 -0800 Received: from twshared7043.05.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 24 Nov 2022 01:36:38 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 2603DA173A1F; Thu, 24 Nov 2022 01:36:21 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v3 9/9] io_uring: allow multishot polled reqs to defer completion Date: Thu, 24 Nov 2022 01:35:59 -0800 Message-ID: <20221124093559.3780686-10-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221124093559.3780686-1-dylany@meta.com> References: <20221124093559.3780686-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: KZMBmWDFU1LPRUYwCbpuvyy6EU8KmNG9 X-Proofpoint-GUID: KZMBmWDFU1LPRUYwCbpuvyy6EU8KmNG9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-24_07,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Until now there was no reason for multishot polled requests to defer completions as there was no functional difference. However now this will actually defer the completions, for a performance win. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index f0e5ae7740cf..164904e7da25 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1805,7 +1805,8 @@ int io_poll_issue(struct io_kiocb *req, bool *locked) io_tw_lock(req->ctx, locked); if (unlikely(req->task->flags & PF_EXITING)) return -EFAULT; - return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_MULTISHOT); + return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_MULTISHOT| + IO_URING_F_COMPLETE_DEFER); } struct io_wq_work *io_wq_free_work(struct io_wq_work *work)