From patchwork Wed Nov 23 11:06:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053464 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9C1AC4332F for ; Wed, 23 Nov 2022 11:07:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236671AbiKWLHO (ORCPT ); Wed, 23 Nov 2022 06:07:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237621AbiKWLGx (ORCPT ); Wed, 23 Nov 2022 06:06:53 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A55CBBC for ; Wed, 23 Nov 2022 03:06:33 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB5BBs011769 for ; Wed, 23 Nov 2022 03:06:32 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=kf1uoMbGwpTuBYF/6IuVpWIqcEDg7pjfwlKTh2oGRlM=; b=lGtEROq3lfPPI1ZfLYqM1NtyadHTUlSX68U3dglvZMEm56djP6iabA7D8AN3sovaojCy zp3aiC0wdqmSgASypMHJxzNykwu39vF44jyqbHsb6ROhqUx/3myqP39c7MsD/loVBjjN +ASgXpBp+CWzaOCoPAaCoG+1CbL0n4IK2ZDk5JJgwwW1aaDrClj+tqnjc8nHxuPKIxjw 0q4cRwbSyaZrY15kbV3nNMVarwBL+quoeidJlYAkIxUo9lc8viW1H4xLRIgR+w1tonUM 3Xy10/aw/7flqEeGutBtdavBnOitjRHOsbqtNkmbZrq8B1soL46IaMhhOzxRND625cbB cQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m0m19unef-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:32 -0800 Received: from twshared0551.06.ash8.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:30 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 808F4A0804B8; Wed, 23 Nov 2022 03:06:26 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 01/13] io_uring: merge io_req_tw_post and io_req_task_complete Date: Wed, 23 Nov 2022 03:06:02 -0800 Message-ID: <20221123110614.3297343-2-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: j1UmGiDJyZxLTCtHpztDbkr0iRltyUbl X-Proofpoint-GUID: j1UmGiDJyZxLTCtHpztDbkr0iRltyUbl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_05,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Merge these functions that have the same logic Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2260fb7aa7f2..e40d7b3404eb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1223,15 +1223,18 @@ int io_run_local_work(struct io_ring_ctx *ctx) return ret; } -static void io_req_tw_post(struct io_kiocb *req, bool *locked) +void io_req_task_complete(struct io_kiocb *req, bool *locked) { - io_req_complete_post(req); + if (*locked) + io_req_complete_defer(req); + else + io_req_complete_post(req); } void io_req_tw_post_queue(struct io_kiocb *req, s32 res, u32 cflags) { io_req_set_res(req, res, cflags); - req->io_task_work.func = io_req_tw_post; + req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); } @@ -1460,14 +1463,6 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) return ret; } -void io_req_task_complete(struct io_kiocb *req, bool *locked) -{ - if (*locked) - io_req_complete_defer(req); - else - io_req_complete_post(req); -} - /* * After the iocb has been issued, it's safe to be found on the poll list. * Adding the kiocb to the list AFTER submission ensures that we don't From patchwork Wed Nov 23 11:06:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BA5DC4332F for ; Wed, 23 Nov 2022 11:07:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237415AbiKWLHb (ORCPT ); Wed, 23 Nov 2022 06:07:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237680AbiKWLG6 (ORCPT ); Wed, 23 Nov 2022 06:06:58 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A802156EF3 for ; Wed, 23 Nov 2022 03:06:42 -0800 (PST) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB4PXp007756 for ; Wed, 23 Nov 2022 03:06:42 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=xrpdzGsOSo95/SQ7TUl7Ka4Y4KT5Qjp35Cxs0HYJzks=; b=frfT+yJOmmO3XGeBCnr0KAZgGqnXtFZl9h2WYsNW8QAJZ52G8p5WfbWsieDFVq0x5hYR oixAm/VjIM0yEyZVb9XHZeu5qVIW5XFw3mig4nnklsfO3UAadgoZQhtUHxyeeyKcCjr1 VH8uEOD4NDye+TWHKBnibfHeosoOeXBmvNblXebESlXjzTMtuNp1MLJrL4Fu2yWa7uUX 9S/w+Jf/EHtcZk3cLELTar6Sb+tivmakfu7xt9WxtWKyZ0Yt/8qYBPjMo9/YaucAPCuL A6dKkvy9Nakl9FNlFvsC9OJgeqeS+0f7ywo4pSqOjjrWUeuC5D/lar3ZIiu8myrC58lN uA== Received: from maileast.thefacebook.com ([163.114.130.8]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m17esbjnq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:42 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:40 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 81F2BA0804BA; Wed, 23 Nov 2022 03:06:26 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 02/13] io_uring: __io_req_complete should defer if available Date: Wed, 23 Nov 2022 03:06:03 -0800 Message-ID: <20221123110614.3297343-3-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: ChHLSGDhSq5mK2ou5QAxPDtQjnXG1B4X X-Proofpoint-ORIG-GUID: ChHLSGDhSq5mK2ou5QAxPDtQjnXG1B4X X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org For consistency always defer completion if specified in the issue flags. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e40d7b3404eb..0741a728fb6a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -858,7 +858,10 @@ void io_req_complete_post(struct io_kiocb *req) inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) { - io_req_complete_post(req); + if (issue_flags & IO_URING_F_COMPLETE_DEFER) + io_req_complete_defer(req); + else + io_req_complete_post(req); } void io_req_complete_failed(struct io_kiocb *req, s32 res) From patchwork Wed Nov 23 11:06:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 933F3C433FE for ; Wed, 23 Nov 2022 11:07:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237535AbiKWLHf (ORCPT ); Wed, 23 Nov 2022 06:07:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237709AbiKWLHF (ORCPT ); Wed, 23 Nov 2022 06:07:05 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FEFD5E9C3 for ; Wed, 23 Nov 2022 03:06:45 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB4Hcw001800 for ; Wed, 23 Nov 2022 03:06:45 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=brQDbixN8YISRVn8bXZa4Sh7F8LuNzI2WbiGkYXarKo=; b=l3LWc6LpxCjvgD+dRKGdOXBk1ZZukVH4uE51XtZH6Ypm+OM6q3eu+C4XpIp1srZj58+7 klUk2Ad6JtKpgJ2nVSCz/a9u8ilg940c8gutYtr72rscOEyNW5i8CvY+e6sZ6joYqbaF ddDlQMoODFNNhNB/Hn1JelumiJm08jYLZ9r0hfLTpySksjVS1+bocbyjKZm+HNruISmw LpeIcSkXHK6+dTJVupwuSPmLH5VTf2QPmnANkrlN5QGteMilQwO7VL40AtPxDjJMzs6u xom5olhueG9TrxB2JVr2l86XQ8YedBMFSJbVbihVM6BXZ3Ei1p/T1aK8c1nheoug91kv 9w== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m0javkw8w-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:45 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:42 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id C7F89A0804BD; Wed, 23 Nov 2022 03:06:26 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 03/13] io_uring: split io_req_complete_failed into post/defer Date: Wed, 23 Nov 2022 03:06:04 -0800 Message-ID: <20221123110614.3297343-4-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: cufm7jwLAk6w4zDgp0s54PxgJkc-91SE X-Proofpoint-GUID: cufm7jwLAk6w4zDgp0s54PxgJkc-91SE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Different use cases might want to defer failure completion if available, or post the completion immediately if the lock is not definitely taken. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 28 ++++++++++++++++++++-------- io_uring/io_uring.h | 2 +- io_uring/poll.c | 2 +- 3 files changed, 22 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0741a728fb6a..1e23adb7b0c5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -864,7 +864,7 @@ inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) io_req_complete_post(req); } -void io_req_complete_failed(struct io_kiocb *req, s32 res) +static inline void io_req_prep_failed(struct io_kiocb *req, s32 res) { const struct io_op_def *def = &io_op_defs[req->opcode]; @@ -872,6 +872,18 @@ void io_req_complete_failed(struct io_kiocb *req, s32 res) io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); if (def->fail) def->fail(req); +} + +static void io_req_defer_failed(struct io_kiocb *req, s32 res) + __must_hold(&ctx->uring_lock) +{ + io_req_prep_failed(req, res); + io_req_complete_defer(req); +} + +void io_req_post_failed(struct io_kiocb *req, s32 res) +{ + io_req_prep_failed(req, res); io_req_complete_post(req); } @@ -1245,7 +1257,7 @@ static void io_req_task_cancel(struct io_kiocb *req, bool *locked) { /* not needed for normal modes, but SQPOLL depends on it */ io_tw_lock(req->ctx, locked); - io_req_complete_failed(req, req->cqe.res); + io_req_defer_failed(req, req->cqe.res); } void io_req_task_submit(struct io_kiocb *req, bool *locked) @@ -1255,7 +1267,7 @@ void io_req_task_submit(struct io_kiocb *req, bool *locked) if (likely(!(req->task->flags & PF_EXITING))) io_queue_sqe(req); else - io_req_complete_failed(req, -EFAULT); + io_req_defer_failed(req, -EFAULT); } void io_req_task_queue_fail(struct io_kiocb *req, int ret) @@ -1633,7 +1645,7 @@ static __cold void io_drain_req(struct io_kiocb *req) ret = io_req_prep_async(req); if (ret) { fail: - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); return; } io_prep_async_link(req); @@ -1863,7 +1875,7 @@ static void io_queue_async(struct io_kiocb *req, int ret) struct io_kiocb *linked_timeout; if (ret != -EAGAIN || (req->flags & REQ_F_NOWAIT)) { - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); return; } @@ -1913,14 +1925,14 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) */ req->flags &= ~REQ_F_HARDLINK; req->flags |= REQ_F_LINK; - io_req_complete_failed(req, req->cqe.res); + io_req_defer_failed(req, req->cqe.res); } else if (unlikely(req->ctx->drain_active)) { io_drain_req(req); } else { int ret = io_req_prep_async(req); if (unlikely(ret)) - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); else io_queue_iowq(req, NULL); } @@ -2847,7 +2859,7 @@ static __cold bool io_cancel_defer_files(struct io_ring_ctx *ctx, while (!list_empty(&list)) { de = list_first_entry(&list, struct io_defer_entry, list); list_del_init(&de->list); - io_req_complete_failed(de->req, -ECANCELED); + io_req_post_failed(de->req, -ECANCELED); kfree(de); } return true; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 83013ee584d6..4d2d0926a42b 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -30,7 +30,7 @@ bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); -void io_req_complete_failed(struct io_kiocb *req, s32 res); +void io_req_post_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); void io_req_complete_post(struct io_kiocb *req); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, diff --git a/io_uring/poll.c b/io_uring/poll.c index cd4d98d622d2..ceb8255b54eb 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -316,7 +316,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else - io_req_complete_failed(req, ret); + io_req_post_failed(req, ret); } static void __io_poll_execute(struct io_kiocb *req, int mask) From patchwork Wed Nov 23 11:06:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 636B3C433FE for ; Wed, 23 Nov 2022 11:07:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237428AbiKWLHd (ORCPT ); Wed, 23 Nov 2022 06:07:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237693AbiKWLHD (ORCPT ); Wed, 23 Nov 2022 06:07:03 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91CF33F077 for ; Wed, 23 Nov 2022 03:06:44 -0800 (PST) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB4PXq007756 for ; Wed, 23 Nov 2022 03:06:44 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=pfgkninkean9r6uL5du/awvzK1aW9ARm3AW/GquuILQ=; b=jC6RRrlMvD/tBtkeheiO/fvMjiuKsJBT6Ys34tTCD5W0yNyF1yFNPIzM6fMUsWB3Vno9 ahcX0TyHWX6A8ep2FYgat7CJqZ/quXUjuAs/mpkmpcyWzpuhC8FWERt7oc2DU4dYGOMU DfMQJiHx2tn0ffDBqCx1VhNBOtJLT9MJltoLUrjY1o5I/tbwHDx+jEfJTPqCLXBf1Eg0 bZ3M7GqYgc//2Xu3dlCQ6rybsiZlIzEGWNPyniTVTEdoWHhutXdOFN2hwdU9Xr5GycT7 wtRxDSUWK+X3ZlZN0jmbfCGTxYEjbig/5qQrcGmK2WTAIBnk+oPyXL2rYZpuAX8fiHZY iQ== Received: from maileast.thefacebook.com ([163.114.130.8]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m17esbjnq-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:43 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:42 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 0FA7AA0804BF; Wed, 23 Nov 2022 03:06:26 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 04/13] io_uring: lock on remove in io_apoll_task_func Date: Wed, 23 Nov 2022 03:06:05 -0800 Message-ID: <20221123110614.3297343-5-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: plmTKqt8W2bY1k81VIdMxCVbVA55k4-q X-Proofpoint-ORIG-GUID: plmTKqt8W2bY1k81VIdMxCVbVA55k4-q X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This allows using io_req_defer_failed rather than post in all cases. The alternative would be to branch based on *locked and decide whether to post or defer the completion. However all of the non-error paths in io_poll_check_events that do not do not return IOU_POLL_NO_ACTION end up locking anyway, and locking here does reduce the logic complexity, so it seems reasonable to lock always and then also defer the completion on failure always. This also means that only io_req_defer_failed needs exporting from io_uring.h Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 4 ++-- io_uring/io_uring.h | 2 +- io_uring/poll.c | 5 +++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 1e23adb7b0c5..5a620001df2e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -874,14 +874,14 @@ static inline void io_req_prep_failed(struct io_kiocb *req, s32 res) def->fail(req); } -static void io_req_defer_failed(struct io_kiocb *req, s32 res) +void io_req_defer_failed(struct io_kiocb *req, s32 res) __must_hold(&ctx->uring_lock) { io_req_prep_failed(req, res); io_req_complete_defer(req); } -void io_req_post_failed(struct io_kiocb *req, s32 res) +static void io_req_post_failed(struct io_kiocb *req, s32 res) { io_req_prep_failed(req, res); io_req_complete_post(req); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 4d2d0926a42b..ffab0d2d33c0 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -30,7 +30,7 @@ bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); -void io_req_post_failed(struct io_kiocb *req, s32 res); +void io_req_defer_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); void io_req_complete_post(struct io_kiocb *req); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, diff --git a/io_uring/poll.c b/io_uring/poll.c index ceb8255b54eb..4bd43e6f5b72 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -308,15 +308,16 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) if (ret == IOU_POLL_NO_ACTION) return; + io_tw_lock(req->ctx, locked); io_poll_remove_entries(req); io_poll_tw_hash_eject(req, locked); if (ret == IOU_POLL_REMOVE_POLL_USE_RES) - io_req_complete_post(req); + io_req_task_complete(req, locked); else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else - io_req_post_failed(req, ret); + io_req_defer_failed(req, ret); } static void __io_poll_execute(struct io_kiocb *req, int mask) From patchwork Wed Nov 23 11:06:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0240CC4332F for ; Wed, 23 Nov 2022 11:07:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237272AbiKWLHa (ORCPT ); Wed, 23 Nov 2022 06:07:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237673AbiKWLG4 (ORCPT ); Wed, 23 Nov 2022 06:06:56 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E305A449 for ; Wed, 23 Nov 2022 03:06:40 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB6532016526 for ; Wed, 23 Nov 2022 03:06:40 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=N47RFG6rqxzhK9mIsH6lCWq+xQ4nDOqdeytRX15d14s=; b=CGizRgCg+hu9i+OFn8iVpRQGwwWQEivfCwv5Ufs0IhG3UJp6KRe9a1ykm0LjPXdqxOlf X7OcE3sOmGdfemayTE2Kd7LTJgVHhE+X7aqN0MKsYFk3jmxTSfEn7BVCCPIl1ZrfNvMs qTtHSOrhZ/DVuk4Ina6OKSjlUs7F8ouBMLDDR6BiHMwU2z91PwfrtMihtneTZ2Yx+FtA dVnQe0pSOKfWQyolgl0V+hhC5maraVs2q5UFQWm4xq+OryD5G4my5tBY8+D/xsOa8FFB EyEVJyKUvwqPCrzjFASXUc1TEd2HfX++5UZJwt83zHbksqewBBauFZ7qpcV+d6GfTwn7 pw== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m1cg3hy35-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:39 -0800 Received: from twshared8047.05.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:36 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 76D3BA0804D0; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 05/13] io_uring: timeout should use io_req_task_complete Date: Wed, 23 Nov 2022 03:06:06 -0800 Message-ID: <20221123110614.3297343-6-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: H5dLzbt71QmYMcQ_hBHcojO8pwF-YQKx X-Proofpoint-GUID: H5dLzbt71QmYMcQ_hBHcojO8pwF-YQKx X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Allow timeouts to defer completions if the ring is locked Signed-off-by: Dylan Yudaken --- io_uring/timeout.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/timeout.c b/io_uring/timeout.c index e8a8c2099480..26b61e62aa9a 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -282,12 +282,11 @@ static void io_req_task_link_timeout(struct io_kiocb *req, bool *locked) ret = io_try_cancel(req->task->io_uring, &cd, issue_flags); } io_req_set_res(req, ret ?: -ETIME, 0); - io_req_complete_post(req); io_put_req(prev); } else { io_req_set_res(req, -ETIME, 0); - io_req_complete_post(req); } + io_req_task_complete(req, locked); } static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer) From patchwork Wed Nov 23 11:06:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89138C433FE for ; Wed, 23 Nov 2022 11:07:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237567AbiKWLHl (ORCPT ); Wed, 23 Nov 2022 06:07:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237732AbiKWLHH (ORCPT ); Wed, 23 Nov 2022 06:07:07 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FD2F20B for ; Wed, 23 Nov 2022 03:06:50 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB4GP9001733 for ; Wed, 23 Nov 2022 03:06:49 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=SPE3OypV1PIZXo3fdcwq8SDYDd8wDO+IXZtK61ofzVA=; b=IuWwxo8Jj3Bo6nB+eOU6/OwoTIExkGgNUiPWrIk1Vhuw+04rolMEqjnioLMmxkWPRQzM l9StjHzgF2CCxG7F7LQ9B7uePn2k2cnMxWoYvskwQMVMBcF+eMzZkcbx/CEeGWQ4uhDS lGBSe7OG1By85gNzFEzHYlgB385jpCrLa3HUZAtmuztfa5EOh33UCCZYsv+CyfXHMcU2 em1n7zGHtBsMbgDhydHO9PKk48ISa+//l2yEHXkA6Wu02t2NDjMwVkgu3dM3rgtFTnrK JvJp3p0WHaSlBdikHb1VLA+zKET3ND+G9AKEdfCR4yAddPUFYlgbK0hIcAIfsMAq3bGS kQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m0javkw99-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:49 -0800 Received: from twshared10308.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:48 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 81F2FA0804D2; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 06/13] io_uring: simplify io_issue_sqe Date: Wed, 23 Nov 2022 03:06:07 -0800 Message-ID: <20221123110614.3297343-7-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: mlhsMtwr7LFLUAiYBQyaRsokI8ilF6nj X-Proofpoint-GUID: mlhsMtwr7LFLUAiYBQyaRsokI8ilF6nj X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_issue_sqe can reuse __io_req_complete for completion logic Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5a620001df2e..912f6fefc665 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1738,12 +1738,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) if (creds) revert_creds(creds); - if (ret == IOU_OK) { - if (issue_flags & IO_URING_F_COMPLETE_DEFER) - io_req_complete_defer(req); - else - io_req_complete_post(req); - } else if (ret != IOU_ISSUE_SKIP_COMPLETE) + if (ret == IOU_OK) + __io_req_complete(req, issue_flags); + else if (ret != IOU_ISSUE_SKIP_COMPLETE) return ret; /* If the op doesn't have a file, we're not polling for it */ From patchwork Wed Nov 23 11:06:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3435EC43219 for ; Wed, 23 Nov 2022 11:07:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237420AbiKWLHd (ORCPT ); Wed, 23 Nov 2022 06:07:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237685AbiKWLHA (ORCPT ); Wed, 23 Nov 2022 06:07:00 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 561AEF597 for ; Wed, 23 Nov 2022 03:06:42 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB5AJa011730 for ; Wed, 23 Nov 2022 03:06:42 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=tT9SXXgL4qAiH0CLnlUBy9F9WnDDxv+tpt2rXyxWWtY=; b=bCts4UVhLfCOUGOFD8dpZDNkS5PcedwkeCPTeiWXx5nQiWqopEtOg4Z7IkqVNOo5vP7Y gtz6LqZad0/UdzamGgMD+cjsGGVNdDjn99L2oBRAY7G81PGO49/dXwWfI8t8xE53mKqt xEsIcZQmv0ZSOw50dqPtcHkIqvMwtVc6d3dbToFmLmINV9TthuX+VvvNj02P3C+H6ux/ 1240xcwwlAwWNFeLHNkSk1JuVhU9zxTVkHqx1tR3xspeTvsusBPl/ZGXqpC94xKrl+oE +4cBpcHqfWOFATVBTj/3WZOXgZO7z1h5Kes31wKQZfFgjHG+HbTY9sx3fbYZK/ydQx/5 9w== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m0m19unf1-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:42 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:40 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 88050A0804D4; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 07/13] io_uring: make io_req_complete_post static Date: Wed, 23 Nov 2022 03:06:08 -0800 Message-ID: <20221123110614.3297343-8-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: oNDQgSNfRKOnL7-vGmRGpwhc3Ti0SVTt X-Proofpoint-GUID: oNDQgSNfRKOnL7-vGmRGpwhc3Ti0SVTt X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_05,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This is only called from two functions in io_uring.c so remove the header export. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 912f6fefc665..43db84fe001d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -845,7 +845,7 @@ static void __io_req_complete_put(struct io_kiocb *req) } } -void io_req_complete_post(struct io_kiocb *req) +static void io_req_complete_post(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ffab0d2d33c0..3c3a93493239 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -32,7 +32,6 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); -void io_req_complete_post(struct io_kiocb *req); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow); bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, From patchwork Wed Nov 23 11:06:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA1C0C4332F for ; Wed, 23 Nov 2022 11:07:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237674AbiKWLHq (ORCPT ); Wed, 23 Nov 2022 06:07:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237749AbiKWLHI (ORCPT ); Wed, 23 Nov 2022 06:07:08 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FE03B7A for ; Wed, 23 Nov 2022 03:06:55 -0800 (PST) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2ANB5YK9007011 for ; Wed, 23 Nov 2022 03:06:54 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=s2048-2021-q4; bh=AdN0224fAZ77xd7LmyPfs1YCRBOz2SV+0/LexFe/noo=; b=SPEXEkeNZWZpriCIKj5FSneorTsbTodm8J4l1D5vJUQIBC83FHRvU0T2oCZv31jHAENk 2rInh9PDeWsYXpJkL1x6+8THUOsaVvEV75KVAnIw6Wx/+v0c+m5y5olHgf/IUKk9iZN2 QMFAARUzGocQUohFWymoklvWjwiIGtWtN9HJ/kUSMTP3wjIlidEtw20x/qFmBgsTn3bS BCMqQ0j0J8dON+8tyE3P3opOsG3DdMDLjXN99aq9eamZMM2fdrXAV0UpFMZAv0AwaLhH oZRH/c7EUNaijbMNvri1D7nrbOWzHJQxZ5toUwlJtWfgIfUc33saadx17ajA6ijrKW9E 1w== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3m1c7ra43e-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:54 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:53 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 8E91FA0804D6; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 08/13] io_uring: allow defer completion for aux posted cqes Date: Wed, 23 Nov 2022 03:06:09 -0800 Message-ID: <20221123110614.3297343-9-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> X-FB-Internal: Safe X-Proofpoint-GUID: 77YBq-kpC80KKXjCzYKsd0BC_bXuTTZh X-Proofpoint-ORIG-GUID: 77YBq-kpC80KKXjCzYKsd0BC_bXuTTZh X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Multishot ops cannot use the compl_reqs list as the request must stay in the poll list, but that means they need to run each completion without benefiting from batching. Here introduce batching infrastructure for only small (ie 16 byte) CQEs. This restriction is ok because there are no use cases posting 32 byte CQEs. In the ring keep a batch of up to 16 posted results, and flush in the same way as compl_reqs. 16 was chosen through experimentation on a microbenchmark ([1]), as well as trying not to increase the size of the ring too much. This increases the size to 1472 bytes from 1216. [1]: https://github.com/DylanZA/liburing/commit/9ac66b36bcf4477bfafeff1c5f107896b7ae31cf Run with $ make -j && ./benchmark/reg.b -s 1 -t 2000 -r 10 Gives results: baseline 8309 k/s 8 18807 k/s 16 19338 k/s 32 20134 k/s Suggested-by: Pavel Begunkov Signed-off-by: Dylan Yudaken --- include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 27 ++++++++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index f5b687a787a3..accdfecee953 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -174,7 +174,9 @@ struct io_submit_state { bool plug_started; bool need_plug; unsigned short submit_nr; + unsigned int cqes_count; struct blk_plug plug; + struct io_uring_cqe cqes[16]; }; struct io_ev_fd { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 43db84fe001d..39f80d68d31c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -167,7 +167,8 @@ EXPORT_SYMBOL(io_uring_get_socket); static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { - if (!wq_list_empty(&ctx->submit_state.compl_reqs)) + if (!wq_list_empty(&ctx->submit_state.compl_reqs) || + ctx->submit_state.cqes_count) __io_submit_flush_completions(ctx); } @@ -802,6 +803,21 @@ bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags return false; } +static void __io_flush_post_cqes(struct io_ring_ctx *ctx) + __must_hold(&ctx->uring_lock) +{ + struct io_submit_state *state = &ctx->submit_state; + unsigned int i; + + lockdep_assert_held(&ctx->uring_lock); + for (i = 0; i < state->cqes_count; i++) { + struct io_uring_cqe *cqe = &state->cqes[i]; + + io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags, true); + } + state->cqes_count = 0; +} + bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow) @@ -1348,6 +1364,9 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_submit_state *state = &ctx->submit_state; io_cq_lock(ctx); + /* post must come first to preserve CQE ordering */ + if (state->cqes_count) + __io_flush_post_cqes(ctx); wq_list_for_each(node, prev, &state->compl_reqs) { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); @@ -1357,8 +1376,10 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) } io_cq_unlock_post(ctx); - io_free_batch_list(ctx, state->compl_reqs.first); - INIT_WQ_LIST(&state->compl_reqs); + if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { + io_free_batch_list(ctx, state->compl_reqs.first); + INIT_WQ_LIST(&state->compl_reqs); + } } /* From patchwork Wed Nov 23 11:06:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56D68C433FE for ; Wed, 23 Nov 2022 11:07:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237642AbiKWLHo (ORCPT ); Wed, 23 Nov 2022 06:07:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237744AbiKWLHI (ORCPT ); Wed, 23 Nov 2022 06:07:08 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A9A1B13 for ; Wed, 23 Nov 2022 03:06:54 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2ANB59K9009094 for ; Wed, 23 Nov 2022 03:06:53 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=Z/qs6/5RRxSeziWegAEgCbtVCatuTGgy+6L8E6H9i1Y=; b=eQIvvN2JNVVW1soG4pqRjwxJ/9gvicN46oRtRiU/IMaR60dZHfgjOrtMxjYiok1kwcWy IwdIKu/u540Pr8A3Mbogd75graQQJhYNMHpOCTkWpGdYFBAQZK29K4/2krNzH5gcA1Bj 5b5q3AQy/h/pCww6qt+PuazrR5t2jPXxCwz8/I3FKDRt9UrrjtQUwoq5qA/rDiFUpRfp haQOE6uqbzRmyNK/XEwCTuqGKn6RfM3nCA2ni2D/R9ZG2hC1Dwh7VsFYhuCqdIeyWjhl WPDvBa/rieaAaNARTB0yH824lhUy0GrHmmO4DWK0qvwHo2hAhGqV2sTjust/V4qvCIBm 4A== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net (PPS) with ESMTPS id 3m0kkdut8r-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:53 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:52 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id A93C3A0804D9; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 09/13] io_uring: add io_aux_cqe which allows deferred completion Date: Wed, 23 Nov 2022 03:06:10 -0800 Message-ID: <20221123110614.3297343-10-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 7g3ZnYyspYyvOIjw2yis4l5CDBysdkhi X-Proofpoint-GUID: 7g3ZnYyspYyvOIjw2yis4l5CDBysdkhi X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use the just introduced deferred post cqe completion state when possible in io_aux_cqe. If not possible fallback to io_post_aux_cqe. This introduces a complication because of allow_overflow. For deferred completions we cannot know without locking the completion_lock if it will overflow (and even if we locked it, another post could sneak in and cause this cqe to be in overflow). However since overflow protection is mostly a best effort defence in depth to prevent infinite loops of CQEs for poll, just checking the overflow bit is going to be good enough and will result in at most 16 (array size of deferred cqes) overflows. Suggested-by: Pavel Begunkov Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 34 ++++++++++++++++++++++++++++++++++ io_uring/io_uring.h | 2 ++ io_uring/net.c | 7 ++++--- io_uring/poll.c | 4 ++-- 4 files changed, 42 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 39f80d68d31c..37b195d85f32 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -830,6 +830,40 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, return filled; } +bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, + bool allow_overflow) +{ + struct io_uring_cqe *cqe; + unsigned int length; + + if (!defer) + return io_post_aux_cqe(ctx, user_data, res, cflags, allow_overflow); + + length = ARRAY_SIZE(ctx->submit_state.cqes); + + lockdep_assert_held(&ctx->uring_lock); + + if (ctx->submit_state.cqes_count == length) { + io_cq_lock(ctx); + __io_flush_post_cqes(ctx); + /* no need to flush - flush is deferred */ + spin_unlock(&ctx->completion_lock); + } + + /* For defered completions this is not as strict as it is otherwise, + * however it's main job is to prevent unbounded posted completions, + * and in that it works just as well. + */ + if (!allow_overflow && test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) + return false; + + cqe = ctx->submit_state.cqes + ctx->submit_state.cqes_count++; + cqe->user_data = user_data; + cqe->res = res; + cqe->flags = cflags; + return true; +} + static void __io_req_complete_put(struct io_kiocb *req) { /* diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 3c3a93493239..e075c4fb70c9 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -36,6 +36,8 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags bool allow_overflow); bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow); +bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, + bool allow_overflow); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); diff --git a/io_uring/net.c b/io_uring/net.c index 0de6f78ad978..90342dcb6b1d 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -601,8 +601,8 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, } if (!mshot_finished) { - if (io_post_aux_cqe(req->ctx, req->cqe.user_data, *ret, - cflags | IORING_CQE_F_MORE, true)) { + if (io_aux_cqe(req->ctx, issue_flags & IO_URING_F_COMPLETE_DEFER, + req->cqe.user_data, *ret, cflags | IORING_CQE_F_MORE, true)) { io_recv_prep_retry(req); return false; } @@ -1320,7 +1320,8 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags) if (ret < 0) return ret; - if (io_post_aux_cqe(ctx, req->cqe.user_data, ret, IORING_CQE_F_MORE, true)) + if (io_aux_cqe(ctx, issue_flags & IO_URING_F_COMPLETE_DEFER, + req->cqe.user_data, ret, IORING_CQE_F_MORE, true)) goto retry; return -ECANCELED; diff --git a/io_uring/poll.c b/io_uring/poll.c index 4bd43e6f5b72..922c1a366c41 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -252,8 +252,8 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_post_aux_cqe(ctx, req->cqe.user_data, - mask, IORING_CQE_F_MORE, false)) { + if (!io_aux_cqe(ctx, *locked, req->cqe.user_data, + mask, IORING_CQE_F_MORE, false)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; } From patchwork Wed Nov 23 11:06:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BC18C46467 for ; Wed, 23 Nov 2022 11:07:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237678AbiKWLHr (ORCPT ); Wed, 23 Nov 2022 06:07:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237748AbiKWLHI (ORCPT ); Wed, 23 Nov 2022 06:07:08 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B6285F9F for ; Wed, 23 Nov 2022 03:06:55 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB4GCA001713 for ; Wed, 23 Nov 2022 03:06:54 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=bLNGZhlpyHKXbaemEZ2JEZ10+s6g9IYZV+VGqpv0h5Y=; b=muB/CnN0rNhdfWlIvkMyM1Oam9JhaMK65D+2gksTJBeIib+KRelqvoytO1CViKEShR66 Zer+F6trRETd94J5ieUAhkVlqjDtlktqjMPVB2o0Ud/YnYcfE63x99vgM9LnV2cIeTv8 3XAk6PBPp2GUYSxNEruJiFOuYpHfsm5iiVlP2lceceXGcosYV/C58F5s/giCCLzndrn8 OC3tyJ432bz9MWcdnvzcg8TZUqdZQALme3Gj/3wcELLn5OhJm0Q7RysNtr8ySF/5HOe6 JnL7LBWA4MBz/f62upIt3PyzM/wtZK4wHdLIZkvMch73IEQsoZAFUXb2D9Y5Bku8CJ/w AQ== Received: from maileast.thefacebook.com ([163.114.130.3]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m0javkw9g-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:54 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:52 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id BF395A0804DB; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 10/13] io_uring: make io_fill_cqe_aux static Date: Wed, 23 Nov 2022 03:06:11 -0800 Message-ID: <20221123110614.3297343-11-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: loFo_cCB6iNEiqg6-DHTgco1cwyeV9fh X-Proofpoint-GUID: loFo_cCB6iNEiqg6-DHTgco1cwyeV9fh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This is only used in io_uring.c Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 4 ++-- io_uring/io_uring.h | 2 -- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 37b195d85f32..42c7383ead91 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -770,8 +770,8 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow) return &rings->cqes[off]; } -bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow) +static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, + bool allow_overflow) { struct io_uring_cqe *cqe; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index e075c4fb70c9..4519d91008de 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -34,8 +34,6 @@ void io_req_defer_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, bool allow_overflow); -bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow); bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, bool allow_overflow); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); From patchwork Wed Nov 23 11:06:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95F65C4332F for ; Wed, 23 Nov 2022 11:07:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237637AbiKWLHn (ORCPT ); Wed, 23 Nov 2022 06:07:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237739AbiKWLHI (ORCPT ); Wed, 23 Nov 2022 06:07:08 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61594F51 for ; Wed, 23 Nov 2022 03:06:50 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB4GC9001713 for ; Wed, 23 Nov 2022 03:06:50 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=2sN06wr+1KcWtoE/vbRTMxGMhpyxRaDQtzzs4SI6w2E=; b=Y/tLe9EC15fLJzoLwa93IuXsSn0Q23rfIrfIxYBo1F1iQ0RtOnpvuAWiA+FPXdaGJxZX wO/q7rNkAmzjkaWvIa79B16BjfhSn5jJdiya1m9ZTFB+Fx4EBI9847c7rUMyMzBPpW+I ix8pRKi1LP1Zd3WEranh+25m+NoRys9ZaoiN/v9g8KvOk+gb6ELnHZs1KfTHOdNgbSMz glFEQ+wQjzOOopSAvKUAT5eXODoGlqzV6b807ot+SI0qWI8vdE0t7cNJedjh1bY55tUD XhlU07UdJdOYEpYe40bOMeqCfD682J6bNGaKY8329sSmjz3HVk5dazlRF7BvsQARJFeK FQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m0javkw9b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:50 -0800 Received: from twshared10308.07.ash9.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:49 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id ECAFCA0804DD; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 11/13] io_uring: add lockdep assertion in io_fill_cqe_aux Date: Wed, 23 Nov 2022 03:06:12 -0800 Message-ID: <20221123110614.3297343-12-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: KP_DF3J6lA2Qs8nHPeqJ97v92V51cxw_ X-Proofpoint-GUID: KP_DF3J6lA2Qs8nHPeqJ97v92V51cxw_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add an assertion for the completion lock to io_fill_cqe_aux Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 42c7383ead91..6e1139a11fbf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -775,6 +775,8 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 { struct io_uring_cqe *cqe; + lockdep_assert_held(&ctx->completion_lock); + ctx->cq_extra++; /* From patchwork Wed Nov 23 11:06:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15108C43219 for ; Wed, 23 Nov 2022 11:07:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237669AbiKWLHp (ORCPT ); Wed, 23 Nov 2022 06:07:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237746AbiKWLHI (ORCPT ); Wed, 23 Nov 2022 06:07:08 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D251B2DF6 for ; Wed, 23 Nov 2022 03:06:54 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2ANB59KA009094 for ; Wed, 23 Nov 2022 03:06:54 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=mCk5YgmYT77ygWxC1yw8ZOTOMT0u0chAtLdcatMD1o0=; b=g6UOGEasdbhDygdAxZyVbIabF+JKQyFcmXD6FOX/FoMg5RRI4fdT7SA6y9QEYZiwfSNt 6ybJz64PBr2S7BASdCbjSAWmB7Ti3uNaERdT4h6kFzaJPEhbj+XzoohfIlsFjbaSP58Q F2hD3gC5ibK1VeYq5azEcqzfg22XXLJVk12DJ2cA47PEQU6z01D19Z4p4As7/4oer2mP svZJMfKuY0VlrS15Fh4as454uYCdQhq4kPpEZvtemvi1A42Ap+UnisU+DCgDaXrytA+r gmyano7g0oWRfKH2wFBx82zTtDUM1ofxTtcNGC+6qOJTEKDueWeWZ+Qw2gvBu8GKDA9g sQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net (PPS) with ESMTPS id 3m0kkdut8r-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:06:54 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:06:53 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 15EC2A0804DF; Wed, 23 Nov 2022 03:06:27 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 12/13] io_uring: remove overflow param from io_post_aux_cqe Date: Wed, 23 Nov 2022 03:06:13 -0800 Message-ID: <20221123110614.3297343-13-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: OIWLvfWLm3mGira4uL-JAKKXBn-cw0E- X-Proofpoint-GUID: OIWLvfWLm3mGira4uL-JAKKXBn-cw0E- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The only call sites which would not allow overflow are also call sites which would use the io_aux_cqe as they care about ordering. So remove this parameter from io_post_aux_cqe. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 12 ++++++++---- io_uring/io_uring.h | 3 +-- io_uring/msg_ring.c | 4 ++-- io_uring/rsrc.c | 4 ++-- 4 files changed, 13 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6e1139a11fbf..87ea497590b5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -820,9 +820,8 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) state->cqes_count = 0; } -bool io_post_aux_cqe(struct io_ring_ctx *ctx, - u64 user_data, s32 res, u32 cflags, - bool allow_overflow) +static bool __io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, + bool allow_overflow) { bool filled; @@ -832,6 +831,11 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, return filled; } +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) +{ + return __io_post_aux_cqe(ctx, user_data, res, cflags, true); +} + bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, bool allow_overflow) { @@ -839,7 +843,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 unsigned int length; if (!defer) - return io_post_aux_cqe(ctx, user_data, res, cflags, allow_overflow); + return __io_post_aux_cqe(ctx, user_data, res, cflags, allow_overflow); length = ARRAY_SIZE(ctx->submit_state.cqes); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 4519d91008de..d61dd9c5030a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -32,8 +32,7 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); -bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow); +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags, bool allow_overflow); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 90d2fc6fd80e..afb543aab9f6 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -31,7 +31,7 @@ static int io_msg_ring_data(struct io_kiocb *req) if (msg->src_fd || msg->dst_fd || msg->flags) return -EINVAL; - if (io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0, true)) + if (io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0)) return 0; return -EOVERFLOW; @@ -116,7 +116,7 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) * completes with -EOVERFLOW, then the sender must ensure that a * later IORING_OP_MSG_RING delivers the message. */ - if (!io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0, true)) + if (!io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0)) ret = -EOVERFLOW; out_unlock: io_double_unlock_ctx(ctx, target_ctx, issue_flags); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 187f1c83e779..133608200769 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -170,10 +170,10 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) if (prsrc->tag) { if (ctx->flags & IORING_SETUP_IOPOLL) { mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0, true); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); mutex_unlock(&ctx->uring_lock); } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0, true); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); } } From patchwork Wed Nov 23 11:06:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13053477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29455C433FE for ; Wed, 23 Nov 2022 11:11:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237038AbiKWLLP (ORCPT ); Wed, 23 Nov 2022 06:11:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237454AbiKWLKp (ORCPT ); Wed, 23 Nov 2022 06:10:45 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3957FD08A3 for ; Wed, 23 Nov 2022 03:09:06 -0800 (PST) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANB65dN016531 for ; Wed, 23 Nov 2022 03:09:05 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=AjHrb9o1Ma8k9hUtX6XDKWMtxRJPGN6YQTuzWdDYyNc=; b=LhkHqLghNziT+b4xefIjkftn1hJA0lCcGwbMywNn0dRszCW3u5TbqwmAAd+AAzj4yDQL g4uV2OuNaOMDCoTRQQjdHNfZCufnUcsYW0w7V+AV7C4io9KLAD1QsCYXtKxHzy/1LXp2 Wde6/S3TDkv8hpGPmNpsFSGyH8yFeeVIOBApKPkWBNHfnFmP4v29mCiRqukHp+eM7K6t 6ueO9jMsSHcmlzcHPHg+1ZrkzqWAMVdxwIkLyXRsWHu8Ssl2K/kWKl6gKUzGCodm6FLG i2q3zYFP1NVrYaHI5OVfloP8pDv5uoyX132BKaobA6UI6gZsqxEFkzem2GcGMEr/bk5c nQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3m1cg3hyey-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 23 Nov 2022 03:09:05 -0800 Received: from twshared41876.03.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 23 Nov 2022 03:09:04 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 1B0BFA0804E1; Wed, 23 Nov 2022 03:06:28 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next v2 13/13] io_uring: allow multishot polled reqs to defer completion Date: Wed, 23 Nov 2022 03:06:14 -0800 Message-ID: <20221123110614.3297343-14-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221123110614.3297343-1-dylany@meta.com> References: <20221123110614.3297343-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: dcWpwaLje8gkprnlWmztj9yzNu3kMWS0 X-Proofpoint-GUID: dcWpwaLje8gkprnlWmztj9yzNu3kMWS0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_06,2022-11-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Until now there was no reason for multishot polled requests to defer completions as there was no functional difference. However now this will actually defer the completions, for a performance win. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 87ea497590b5..e3f0b4728db3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1816,7 +1816,8 @@ int io_poll_issue(struct io_kiocb *req, bool *locked) io_tw_lock(req->ctx, locked); if (unlikely(req->task->flags & PF_EXITING)) return -EFAULT; - return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_MULTISHOT); + return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_MULTISHOT| + IO_URING_F_COMPLETE_DEFER); } struct io_wq_work *io_wq_free_work(struct io_wq_work *work)