From patchwork Mon Nov 21 10:03:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C1ACC4332F for ; Mon, 21 Nov 2022 10:04:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230089AbiKUKEX (ORCPT ); Mon, 21 Nov 2022 05:04:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231254AbiKUKEF (ORCPT ); Mon, 21 Nov 2022 05:04:05 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 194D49B7C7 for ; Mon, 21 Nov 2022 02:04:02 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AL1UWDk030968 for ; Mon, 21 Nov 2022 02:04:02 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=zqE7wnuHH2jfEJzRt8kiWCSTGyh0h4w3yFgaDz5TXDE=; b=eQxRUmRo/LQhOxcfhHNmd2ng9Db8+srbsywNDIFJAH8Ul/ETxj4GvAE6Pz+d1mkLPhHC mX1kx9JLfc3RFQUbPE9lGItbrF2ibT2rtTFNCBHRLKjFQe+KCHYIld1uHDzIJ6v/pYNf X2ma2mfw4VKfW8J3MPMx8D/TVLaY8faAMIknuO4m4vPYy4lA6xuSTKzyYYv1e0tTETbM bGp/AG2iOIAvYv0LFpdqg1/XUBWzcOZ7qYZQVZSEnwEAQqOerJ5S1O3D5YFYVjb2L9VC 1T6e4ctz4IJFczRK3rBdgORytpPC5TkH06uZrCRXhQyQWSJCwQyTzXNT06y+PzvlFuCr Uw== Received: from maileast.thefacebook.com ([163.114.130.3]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxwj43g33-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:04:02 -0800 Received: from twshared41876.03.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:04:01 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id DFE649E66F69; Mon, 21 Nov 2022 02:03:54 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 01/10] io_uring: merge io_req_tw_post and io_req_task_complete Date: Mon, 21 Nov 2022 02:03:44 -0800 Message-ID: <20221121100353.371865-2-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: F-BlaF81HpGVW4iKLJyPOj7wIC_oy5_V X-Proofpoint-ORIG-GUID: F-BlaF81HpGVW4iKLJyPOj7wIC_oy5_V X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Merge these functions that have the same logic Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9e868a83e472..f15aca039db6 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1227,15 +1227,18 @@ int io_run_local_work(struct io_ring_ctx *ctx) return ret; } -static void io_req_tw_post(struct io_kiocb *req, bool *locked) +void io_req_task_complete(struct io_kiocb *req, bool *locked) { - io_req_complete_post(req); + if (*locked) + io_req_complete_defer(req); + else + io_req_complete_post(req); } void io_req_tw_post_queue(struct io_kiocb *req, s32 res, u32 cflags) { io_req_set_res(req, res, cflags); - req->io_task_work.func = io_req_tw_post; + req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); } @@ -1464,14 +1467,6 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) return ret; } -void io_req_task_complete(struct io_kiocb *req, bool *locked) -{ - if (*locked) - io_req_complete_defer(req); - else - io_req_complete_post(req); -} - /* * After the iocb has been issued, it's safe to be found on the poll list. * Adding the kiocb to the list AFTER submission ensures that we don't From patchwork Mon Nov 21 10:03:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050594 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5936CC433FE for ; Mon, 21 Nov 2022 10:42:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229547AbiKUKmN (ORCPT ); Mon, 21 Nov 2022 05:42:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229607AbiKUKmM (ORCPT ); Mon, 21 Nov 2022 05:42:12 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C37DEE0E6 for ; Mon, 21 Nov 2022 02:42:10 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AKNK3o6031066 for ; Mon, 21 Nov 2022 02:42:10 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=qSmCXqBquJPBd/4bCImrTAvqiQHok6YchcnHrDYQvYI=; b=LWIQr6TtLeoid+UH3bc6e6P+5EVRVv4FDHHB59ReXXCj9UuOUrFBmNSkDT5EK9ekMYNi kK3vIyXtC8gl+rIUJcMzerM6xbg65LSzopsh8i7CIeCuOQI56YNtuaMGyqWEe7l15FKW 8IQrDQzGWycC6DosYJqW2nOQpXLJ85QYCRK56bvQN7I4GjVDToCfiIXXUjgbtdznjjoF cfC0najcIXjUqfmmYUjeS0HYUL6sb2yWo2VwMt1pLaSvG7hnKsScGPkijwTjuZGcGBwg K0E+zPd4VQR2KPEEX4FHqg6no2S7p+XS5495pw/nan3nUxRha4eUSIO4C95tPqG9DAXX Mw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxwj43p6f-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:42:09 -0800 Received: from twshared9088.05.ash9.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:42:08 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 1ACED9E66F6B; Mon, 21 Nov 2022 02:03:54 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 02/10] io_uring: __io_req_complete should defer if available Date: Mon, 21 Nov 2022 02:03:45 -0800 Message-ID: <20221121100353.371865-3-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: m6jFbUheQYypzU8c2Ym-9MuCjoJEekSZ X-Proofpoint-ORIG-GUID: m6jFbUheQYypzU8c2Ym-9MuCjoJEekSZ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org For consistency always defer completion if specified in the issue flags. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index f15aca039db6..208afb944b0c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -862,7 +862,10 @@ void io_req_complete_post(struct io_kiocb *req) inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) { - io_req_complete_post(req); + if (issue_flags & IO_URING_F_COMPLETE_DEFER) + io_req_complete_defer(req); + else + io_req_complete_post(req); } void io_req_complete_failed(struct io_kiocb *req, s32 res) From patchwork Mon Nov 21 10:03:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11850C433FE for ; Mon, 21 Nov 2022 10:48:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229843AbiKUKso (ORCPT ); Mon, 21 Nov 2022 05:48:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229894AbiKUKsP (ORCPT ); Mon, 21 Nov 2022 05:48:15 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D6321001 for ; Mon, 21 Nov 2022 02:48:14 -0800 (PST) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AKNmo8b024223 for ; Mon, 21 Nov 2022 02:48:14 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=8yNetuLxtxYQ4CH6zKHEwvu3Z56G4Zn4RgL48kPZoKw=; b=KQSqXeoPs9jOScsaUqM4gNWHQ+0vrYDHdkPigH9uC4WHEU3NHGyT1r/5LlN+jbHho2Wb S2ZXBIzijPXg3NAE93ONxW8c3+sZZ13M6/XnY9ZhbWipcGRtwNZ1NGmyZO1XJIXMTTaj lE0jenYUA+Q1Ar+Ox0fG48h4tZIQyaoL+e6CcTxTRgw+qKSI6hg3wRMHrGIgU5tyV9nd vfoIpg7wywpP5czjtoffAMJhi/jUrOvJYZDqFCWpibbi+56QwBId6ERuwgI1OJJY6tGC SA/JT/8+NCqAby4/8vajd2v7gOP9qFnUPtLrxKY+qXXCreLVe0zSnD8bENOYAPSdGbDy 1g== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxyb9b64p-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:48:14 -0800 Received: from twshared0705.02.ash8.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:48:13 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 586999E66F6D; Mon, 21 Nov 2022 02:03:55 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 03/10] io_uring: split io_req_complete_failed into post/defer Date: Mon, 21 Nov 2022 02:03:46 -0800 Message-ID: <20221121100353.371865-4-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: qeJMVzEJkraU2CyT4h20jx0aZI-jrd7U X-Proofpoint-ORIG-GUID: qeJMVzEJkraU2CyT4h20jx0aZI-jrd7U X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Different use cases might want to defer failure completion if available, or post the completion immediately if the lock is not definitely taken. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 28 ++++++++++++++++++++-------- io_uring/io_uring.h | 2 +- io_uring/poll.c | 2 +- 3 files changed, 22 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 208afb944b0c..d9bd18e3a603 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -868,7 +868,7 @@ inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) io_req_complete_post(req); } -void io_req_complete_failed(struct io_kiocb *req, s32 res) +static inline void io_req_prep_failed(struct io_kiocb *req, s32 res) { const struct io_op_def *def = &io_op_defs[req->opcode]; @@ -876,6 +876,18 @@ void io_req_complete_failed(struct io_kiocb *req, s32 res) io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); if (def->fail) def->fail(req); +} + +static void io_req_defer_failed(struct io_kiocb *req, s32 res) + __must_hold(&ctx->uring_lock) +{ + io_req_prep_failed(req, res); + io_req_complete_defer(req); +} + +void io_req_post_failed(struct io_kiocb *req, s32 res) +{ + io_req_prep_failed(req, res); io_req_complete_post(req); } @@ -1249,7 +1261,7 @@ static void io_req_task_cancel(struct io_kiocb *req, bool *locked) { /* not needed for normal modes, but SQPOLL depends on it */ io_tw_lock(req->ctx, locked); - io_req_complete_failed(req, req->cqe.res); + io_req_defer_failed(req, req->cqe.res); } void io_req_task_submit(struct io_kiocb *req, bool *locked) @@ -1259,7 +1271,7 @@ void io_req_task_submit(struct io_kiocb *req, bool *locked) if (likely(!(req->task->flags & PF_EXITING))) io_queue_sqe(req); else - io_req_complete_failed(req, -EFAULT); + io_req_defer_failed(req, -EFAULT); } void io_req_task_queue_fail(struct io_kiocb *req, int ret) @@ -1637,7 +1649,7 @@ static __cold void io_drain_req(struct io_kiocb *req) ret = io_req_prep_async(req); if (ret) { fail: - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); return; } io_prep_async_link(req); @@ -1867,7 +1879,7 @@ static void io_queue_async(struct io_kiocb *req, int ret) struct io_kiocb *linked_timeout; if (ret != -EAGAIN || (req->flags & REQ_F_NOWAIT)) { - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); return; } @@ -1917,14 +1929,14 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) */ req->flags &= ~REQ_F_HARDLINK; req->flags |= REQ_F_LINK; - io_req_complete_failed(req, req->cqe.res); + io_req_defer_failed(req, req->cqe.res); } else if (unlikely(req->ctx->drain_active)) { io_drain_req(req); } else { int ret = io_req_prep_async(req); if (unlikely(ret)) - io_req_complete_failed(req, ret); + io_req_defer_failed(req, ret); else io_queue_iowq(req, NULL); } @@ -2851,7 +2863,7 @@ static __cold bool io_cancel_defer_files(struct io_ring_ctx *ctx, while (!list_empty(&list)) { de = list_first_entry(&list, struct io_defer_entry, list); list_del_init(&de->list); - io_req_complete_failed(de->req, -ECANCELED); + io_req_post_failed(de->req, -ECANCELED); kfree(de); } return true; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index af3f82bd4017..ee3139947fcc 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -29,7 +29,7 @@ bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); -void io_req_complete_failed(struct io_kiocb *req, s32 res); +void io_req_post_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); void io_req_complete_post(struct io_kiocb *req); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); diff --git a/io_uring/poll.c b/io_uring/poll.c index 989b72a47331..e0a4faa010b3 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -304,7 +304,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else - io_req_complete_failed(req, ret); + io_req_post_failed(req, ret); } static void __io_poll_execute(struct io_kiocb *req, int mask) From patchwork Mon Nov 21 10:03:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A1F0C4332F for ; Mon, 21 Nov 2022 10:42:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229512AbiKUKmG (ORCPT ); Mon, 21 Nov 2022 05:42:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229505AbiKUKmF (ORCPT ); Mon, 21 Nov 2022 05:42:05 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FC4BAFE73 for ; Mon, 21 Nov 2022 02:42:05 -0800 (PST) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AKNUqfi027349 for ; Mon, 21 Nov 2022 02:42:04 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=fSLTEJetbWr2pbtuuZssjhD39zwbW2dgI2JxFNwjEkk=; b=M+N7ML8yT95cGOnRt79lueyS5NX+zTSKdBof+bK6UAjyTe/KJ/sXj/Q1MobOEiRr05CV pMETGe4jwcAQYesyoQn2uCStlcMFCMv38PsX/Z3n+fMEL3MLqMmc/6C69h7Cts36h5gd Uf7WcGgw6WB6XKyDrQsudkCMhy//jVn4aMlHrQNDwPgft/DZOwF0hnPQwTtc4L9dVx81 hBEZFg8VYYdkSC+9t3b5f2Zh9YrwsH/SdqN3HHzANFP/ZpjVg3Cpm5pr3Er/cSoWcFkc 4LLNxdZz1//99N0DN22tUW7wWkhvi2zuZs3GfT+qzFS5lcQJ00qpe5S2ooNRoEfbThej xQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxubu435t-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:42:04 -0800 Received: from twshared3131.02.ash7.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:42:03 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id C6C849E66F70; Mon, 21 Nov 2022 02:03:55 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 04/10] io_uring: lock on remove in io_apoll_task_func Date: Mon, 21 Nov 2022 02:03:47 -0800 Message-ID: <20221121100353.371865-5-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: HlxjMDcGpPcRKW7ZpksaLayt6FX4D-WW X-Proofpoint-ORIG-GUID: HlxjMDcGpPcRKW7ZpksaLayt6FX4D-WW X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This allows using io_req_defer_failed rather than post in all cases. The alternative would be to branch based on *locked and decide whether to post or defer the completion. However all of the non-error paths in io_poll_check_events that do not do not return IOU_POLL_NO_ACTION end up locking anyway, and locking here does reduce the logic complexity, so it seems reasonable to lock always and then also defer the completion on failure always. This also means that only io_req_defer_failed needs exporting from io_uring.h Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 4 ++-- io_uring/io_uring.h | 2 +- io_uring/poll.c | 5 +++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d9bd18e3a603..03946f46dadc 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -878,14 +878,14 @@ static inline void io_req_prep_failed(struct io_kiocb *req, s32 res) def->fail(req); } -static void io_req_defer_failed(struct io_kiocb *req, s32 res) +void io_req_defer_failed(struct io_kiocb *req, s32 res) __must_hold(&ctx->uring_lock) { io_req_prep_failed(req, res); io_req_complete_defer(req); } -void io_req_post_failed(struct io_kiocb *req, s32 res) +static void io_req_post_failed(struct io_kiocb *req, s32 res) { io_req_prep_failed(req, res); io_req_complete_post(req); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ee3139947fcc..1daf236513cc 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -29,7 +29,7 @@ bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); -void io_req_post_failed(struct io_kiocb *req, s32 res); +void io_req_defer_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); void io_req_complete_post(struct io_kiocb *req); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); diff --git a/io_uring/poll.c b/io_uring/poll.c index e0a4faa010b3..2b77d18a67a7 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -296,15 +296,16 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) if (ret == IOU_POLL_NO_ACTION) return; + io_tw_lock(req->ctx, locked); io_poll_remove_entries(req); io_poll_tw_hash_eject(req, locked); if (ret == IOU_POLL_REMOVE_POLL_USE_RES) - io_req_complete_post(req); + io_req_task_complete(req, locked); else if (ret == IOU_POLL_DONE) io_req_task_submit(req, locked); else - io_req_post_failed(req, ret); + io_req_defer_failed(req, ret); } static void __io_poll_execute(struct io_kiocb *req, int mask) From patchwork Mon Nov 21 10:03:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050604 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28DA4C4321E for ; Mon, 21 Nov 2022 10:48:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229840AbiKUKso (ORCPT ); Mon, 21 Nov 2022 05:48:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229893AbiKUKsO (ORCPT ); Mon, 21 Nov 2022 05:48:14 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC072B842 for ; Mon, 21 Nov 2022 02:48:13 -0800 (PST) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AKMvgXl009589 for ; Mon, 21 Nov 2022 02:48:12 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=N47RFG6rqxzhK9mIsH6lCWq+xQ4nDOqdeytRX15d14s=; b=YkJ4KadjdI/OXHVp3VCZx8zW6X0QM4cLlDTzvaq/Nac17Po1WMxKCwcU4YtQDFNvEnRM 0Km/nXUgZX4QFmzV2dhqj1ZiMuJs2oLhbE1N+/cydDXpBYYUjJzE4QUgEjp7O42WGiCH CGjlL8SNn5rtNa/c1DV71Dnm3xIMsmlGQ3De4ysonyRQycUoHfx1f0YO1Tsq4c4/xHFC RYn4FaZg3nO68olwHhNly6oym9JFm32XxKcCnNAxTkBadVDLL5dLMO3xbp3uBxn3FSQM qvLdL50Qg7WJPylh8Hu9aFOMvOlIoLjxJ2+k98al0mXPk+ebC5aWdpbyNlXVc8XloOJ1 2g== Received: from maileast.thefacebook.com ([163.114.130.8]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxwj43q7v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:48:12 -0800 Received: from twshared9088.05.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:48:11 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 301279E66F74; Mon, 21 Nov 2022 02:03:56 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 05/10] io_uring: timeout should use io_req_task_complete Date: Mon, 21 Nov 2022 02:03:48 -0800 Message-ID: <20221121100353.371865-6-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 7J6JfRFT6jhzcP7PV59UiD1Cri5hL8jl X-Proofpoint-ORIG-GUID: 7J6JfRFT6jhzcP7PV59UiD1Cri5hL8jl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Allow timeouts to defer completions if the ring is locked Signed-off-by: Dylan Yudaken --- io_uring/timeout.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/timeout.c b/io_uring/timeout.c index e8a8c2099480..26b61e62aa9a 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -282,12 +282,11 @@ static void io_req_task_link_timeout(struct io_kiocb *req, bool *locked) ret = io_try_cancel(req->task->io_uring, &cd, issue_flags); } io_req_set_res(req, ret ?: -ETIME, 0); - io_req_complete_post(req); io_put_req(prev); } else { io_req_set_res(req, -ETIME, 0); - io_req_complete_post(req); } + io_req_task_complete(req, locked); } static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer) From patchwork Mon Nov 21 10:03:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 289B9C4332F for ; Mon, 21 Nov 2022 10:36:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229495AbiKUKg3 (ORCPT ); Mon, 21 Nov 2022 05:36:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230172AbiKUKgI (ORCPT ); Mon, 21 Nov 2022 05:36:08 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AFCF776EC for ; Mon, 21 Nov 2022 02:36:04 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2AKFORoA031703 for ; Mon, 21 Nov 2022 02:36:03 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=dvEu+E82RJiWZKGF3K/9stvmM4gp2hwml7gpW0WZ6aI=; b=PMAUEzsQn5/33ke5XMgF+AuCU5bBohiidQnKntL5uNR1iCGGV/E9JZheFUD6kolNLN16 ZB3yF4201UIC7oSbkR8SZVYN5ettsrSeggV7ehlbyI4gFl1yDWkfxSjbFyhwryrQYQT0 6hVge0enxGslUu/sTPbgPuB1SejsN0zy1y0CaJefvO23jIfECjtsjFZM7NCMa0iVSWJV q/8GSW1K+Na3C2Un/nE2GIFY3hRIRTof6qNY5wL3GhKZN48vQdgChKKWV4QS5Al0n9p5 39HO/b6Xr+BFDcNLZTSjpViLdlIG2fxfKC/HbMEObMnhuCjldErC7lDyvEOCqUAASuME nw== Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3kxuq0c52k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:36:03 -0800 Received: from twshared9088.05.ash9.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:36:02 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 8093C9E66F79; Mon, 21 Nov 2022 02:03:56 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 06/10] io_uring: simplify io_issue_sqe Date: Mon, 21 Nov 2022 02:03:49 -0800 Message-ID: <20221121100353.371865-7-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: iIWs-Q1iESQobUmksV3jlTdLMcpqhYx7 X-Proofpoint-GUID: iIWs-Q1iESQobUmksV3jlTdLMcpqhYx7 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_issue_sqe can reuse __io_req_complete for completion logic Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 03946f46dadc..2177b3ef094a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1742,12 +1742,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) if (creds) revert_creds(creds); - if (ret == IOU_OK) { - if (issue_flags & IO_URING_F_COMPLETE_DEFER) - io_req_complete_defer(req); - else - io_req_complete_post(req); - } else if (ret != IOU_ISSUE_SKIP_COMPLETE) + if (ret == IOU_OK) + __io_req_complete(req, issue_flags); + else if (ret != IOU_ISSUE_SKIP_COMPLETE) return ret; /* If the op doesn't have a file, we're not polling for it */ From patchwork Mon Nov 21 10:03:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EB9AC433FE for ; Mon, 21 Nov 2022 10:08:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230105AbiKUKIw (ORCPT ); Mon, 21 Nov 2022 05:08:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231265AbiKUKIY (ORCPT ); Mon, 21 Nov 2022 05:08:24 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B7674384D for ; Mon, 21 Nov 2022 02:08:09 -0800 (PST) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AKBn6B3032730 for ; Mon, 21 Nov 2022 02:08:02 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=gCcaYV5ZDcJMmdmNHT3t2urXtrSvZCQor+dsB0KT/HA=; b=U6mnar8pSrS2MM8nRc1prMf7qNXSMYaa84qiyXcXqBU2U+C8xsjzueo4im8RIPJnmXTX 67skiZott2VUhJVyuDaaRHmIu689BAPHfqwjRyUjae1EVFnp7StpA/3VoY81vO3XGdPG LSD1zobqPABon34vg01d/H1jQ1/XypF9H1zJ8dG1KGi9OvFr3kCpjyRMEXBHfnJKq64G v5JFLsvZf3Un+203wjRwhuzbsx6npQbaJtfxW1zITV1pS1nlCLhXB1C66LwvYGaHDV+Y z5OLpPbOBOEOwShOo7Bi/K6rppURoItWF8rxiC3Yo3z1PimXBvE+cOppEsSwHGqdpJbF LA== Received: from maileast.thefacebook.com ([163.114.130.8]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxws6uje6-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:08:02 -0800 Received: from twshared41876.03.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:08:01 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 923B09E66F80; Mon, 21 Nov 2022 02:03:56 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 07/10] io_uring: make io_req_complete_post static Date: Mon, 21 Nov 2022 02:03:50 -0800 Message-ID: <20221121100353.371865-8-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: F_9n4XbSI9bex2EyoZenSyIru7dyxkJo X-Proofpoint-ORIG-GUID: F_9n4XbSI9bex2EyoZenSyIru7dyxkJo X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This is only called from two functions in io_uring.c so remove the header export. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2177b3ef094a..715ded749110 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -849,7 +849,7 @@ static void __io_req_complete_put(struct io_kiocb *req) } } -void io_req_complete_post(struct io_kiocb *req) +static void io_req_complete_post(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 1daf236513cc..bfe1b5488c25 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -31,7 +31,6 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); -void io_req_complete_post(struct io_kiocb *req); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); From patchwork Mon Nov 21 10:03:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC50EC433FE for ; Mon, 21 Nov 2022 10:13:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231288AbiKUKNQ (ORCPT ); Mon, 21 Nov 2022 05:13:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231252AbiKUKNH (ORCPT ); Mon, 21 Nov 2022 05:13:07 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86B7D12D1D for ; Mon, 21 Nov 2022 02:13:06 -0800 (PST) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AKJpiPE021694 for ; Mon, 21 Nov 2022 02:13:06 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=s2048-2021-q4; bh=lFA7gxh8JLEUjTCsdy5/lYT3GGleQZs5DE6/1DvT8dc=; b=AdZSetHqyy3HFihGOhMsElpztQDPD6JQAGuT8YzCzA3M8LZznvA2oVY++j4965YBRKph FkiqGKyzWrvhpoN6Qkjb4/5876i2EMDah9T5URBd9UZfvmshwuccm0eEXoOOtuHiVaIN N7Osf+5k3lTs1QaW6L5y8P0G+VqsZZrjCECt/XptutveyKXc1MBvyTccGc/5C31VMWID MgyDrB4y+hBAOxP4p+bukKqHTa9phXekw/EJMQcTDk3BUINGaZVEBWVoLftGQX3QjGST V/hfEPRYH+qShwgvNKu6L7J6nEwphlwAagDrAyDCCJ4KYXtC4VrDWtXRJnIN9D5Um5c5 hA== Received: from maileast.thefacebook.com ([163.114.130.3]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxw4xuh2k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:13:06 -0800 Received: from twshared3131.02.ash7.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:13:04 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 985829E66F82; Mon, 21 Nov 2022 02:03:56 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 08/10] io_uring: allow defer completion for aux posted cqes Date: Mon, 21 Nov 2022 02:03:51 -0800 Message-ID: <20221121100353.371865-9-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> X-FB-Internal: Safe X-Proofpoint-GUID: ugKVrQB18DUmtejcS6IngGVnYkv3C0oP X-Proofpoint-ORIG-GUID: ugKVrQB18DUmtejcS6IngGVnYkv3C0oP X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Multishot ops cannot use the compl_reqs list as the request must stay in the poll list, but that means they need to run each completion without benefiting from batching. Here introduce batching infrastructure for only small (ie 16 byte) CQEs. This restriction is ok because there are no use cases posting 32 byte CQEs. In the ring keep a batch of up to 16 posted results, and flush in the same way as compl_reqs. 16 was chosen through experimentation on a microbenchmark ([1]), as well as trying not to increase the size of the ring too much. This increases the size to 1472 bytes from 1216. [1]: https://github.com/DylanZA/liburing/commit/9ac66b36bcf4477bfafeff1c5f107896b7ae31cf Run with $ make -j && ./benchmark/reg.b -s 1 -t 2000 -r 10 Gives results: baseline 8309 k/s 8 18807 k/s 16 19338 k/s 32 20134 k/s Signed-off-by: Dylan Yudaken --- include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 49 +++++++++++++++++++++++++++++++--- 2 files changed, 48 insertions(+), 3 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index f5b687a787a3..accdfecee953 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -174,7 +174,9 @@ struct io_submit_state { bool plug_started; bool need_plug; unsigned short submit_nr; + unsigned int cqes_count; struct blk_plug plug; + struct io_uring_cqe cqes[16]; }; struct io_ev_fd { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 715ded749110..c797f9a75dfe 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -167,7 +167,8 @@ EXPORT_SYMBOL(io_uring_get_socket); static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { - if (!wq_list_empty(&ctx->submit_state.compl_reqs)) + if (!wq_list_empty(&ctx->submit_state.compl_reqs) || + ctx->submit_state.cqes_count) __io_submit_flush_completions(ctx); } @@ -807,6 +808,43 @@ bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags return io_cqring_event_overflow(ctx, user_data, res, cflags, 0, 0); } +static bool __io_fill_cqe_small(struct io_ring_ctx *ctx, + struct io_uring_cqe *cqe) +{ + struct io_uring_cqe *cqe_out; + + cqe_out = io_get_cqe(ctx); + if (unlikely(!cqe_out)) { + return io_cqring_event_overflow(ctx, cqe->user_data, + cqe->res, cqe->flags, + 0, 0); + } + + trace_io_uring_complete(ctx, NULL, cqe->user_data, + cqe->res, cqe->flags, + 0, 0); + + memcpy(cqe_out, cqe, sizeof(*cqe_out)); + + if (ctx->flags & IORING_SETUP_CQE32) { + WRITE_ONCE(cqe_out->big_cqe[0], 0); + WRITE_ONCE(cqe_out->big_cqe[1], 0); + } + return true; +} + +static void __io_flush_post_cqes(struct io_ring_ctx *ctx) + __must_hold(&ctx->uring_lock) +{ + struct io_submit_state *state = &ctx->submit_state; + unsigned int i; + + lockdep_assert_held(&ctx->uring_lock); + for (i = 0; i < state->cqes_count; i++) + __io_fill_cqe_small(ctx, state->cqes + i); + state->cqes_count = 0; +} + bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) { @@ -1352,6 +1390,9 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_submit_state *state = &ctx->submit_state; io_cq_lock(ctx); + /* post must come first to preserve CQE ordering */ + if (state->cqes_count) + __io_flush_post_cqes(ctx); wq_list_for_each(node, prev, &state->compl_reqs) { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); @@ -1361,8 +1402,10 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) } __io_cq_unlock_post(ctx); - io_free_batch_list(ctx, state->compl_reqs.first); - INIT_WQ_LIST(&state->compl_reqs); + if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { + io_free_batch_list(ctx, state->compl_reqs.first); + INIT_WQ_LIST(&state->compl_reqs); + } } /* From patchwork Mon Nov 21 10:03:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62067C4332F for ; Mon, 21 Nov 2022 10:30:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229688AbiKUKaM (ORCPT ); Mon, 21 Nov 2022 05:30:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229798AbiKUKaL (ORCPT ); Mon, 21 Nov 2022 05:30:11 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49A1A1E71C for ; Mon, 21 Nov 2022 02:30:10 -0800 (PST) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2AKMBPBW030760 for ; Mon, 21 Nov 2022 02:30:09 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=HwgaHRmiIMIwClOOrtv+vxnBg6LeAzsV7GXTaGlkfao=; b=Zq6zkarrbxUhc+fL4C3Lcxqy0xlY5k+ZIOxHuAWWsMeU5myd7vM/bBoa78H8Mg90dSm4 306dJIRGO+FKWfxlv4ZonLKTrQo6pGhZGVG8xSOFAmdMRyiUc4nxRkf8kqLl6JjU/NyN MBq+n2AbbEvQYMgrR+GBF9CGERvgdreoXr5ltGH11Z3TGQMnS/my3m/3RK8bjXStz/vW 3ZRXGjYOh5R5w3S2QMD4U6efpBOMVYy6p3efP32mcFGgztrvOxS1fw9oAtAG52sIWCKj rGwLIvhGTgq7b2fuRm/wsfNFc9j2HozHykzl2BaqgPqOMRRPZr93OVXyc30hW9nLW+am Wg== Received: from maileast.thefacebook.com ([163.114.130.3]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kxws6up56-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:30:09 -0800 Received: from twshared10308.07.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:30:08 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id A0AAF9E66F84; Mon, 21 Nov 2022 02:03:56 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 09/10] io_uring: allow io_post_aux_cqe to defer completion Date: Mon, 21 Nov 2022 02:03:52 -0800 Message-ID: <20221121100353.371865-10-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: DfneB-bHSpHXr78JR260iJYGao6O5pR_ X-Proofpoint-ORIG-GUID: DfneB-bHSpHXr78JR260iJYGao6O5pR_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use the just introduced deferred post cqe completion state when possible in io_post_aux_cqe. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 21 ++++++++++++++++++++- io_uring/io_uring.h | 2 +- io_uring/msg_ring.c | 10 ++++++---- io_uring/net.c | 15 ++++++++------- io_uring/poll.c | 2 +- io_uring/rsrc.c | 4 ++-- 6 files changed, 38 insertions(+), 16 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c797f9a75dfe..5c240d01278a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -845,11 +845,30 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) state->cqes_count = 0; } -bool io_post_aux_cqe(struct io_ring_ctx *ctx, +bool io_post_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags) { bool filled; + if (defer) { + unsigned int length = ARRAY_SIZE(ctx->submit_state.cqes); + struct io_uring_cqe *cqe; + + lockdep_assert_held(&ctx->uring_lock); + + if (ctx->submit_state.cqes_count == length) { + io_cq_lock(ctx); + __io_flush_post_cqes(ctx); + /* no need to flush - flush is deferred */ + spin_unlock(&ctx->completion_lock); + } + + cqe = ctx->submit_state.cqes + ctx->submit_state.cqes_count++; + cqe->user_data = user_data; + cqe->res = res; + cqe->flags = cflags; + return true; + } io_cq_lock(ctx); filled = io_fill_cqe_aux(ctx, user_data, res, cflags); io_cq_unlock_post(ctx); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index bfe1b5488c25..979a223286bd 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -31,7 +31,7 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); -bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); +bool io_post_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 cflags); bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index afb543aab9f6..c5e831e3dcfc 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -23,7 +23,7 @@ struct io_msg { u32 flags; }; -static int io_msg_ring_data(struct io_kiocb *req) +static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) { struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); @@ -31,7 +31,8 @@ static int io_msg_ring_data(struct io_kiocb *req) if (msg->src_fd || msg->dst_fd || msg->flags) return -EINVAL; - if (io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0)) + if (io_post_aux_cqe(target_ctx, false, + msg->user_data, msg->len, 0)) return 0; return -EOVERFLOW; @@ -116,7 +117,8 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) * completes with -EOVERFLOW, then the sender must ensure that a * later IORING_OP_MSG_RING delivers the message. */ - if (!io_post_aux_cqe(target_ctx, msg->user_data, msg->len, 0)) + if (!io_post_aux_cqe(target_ctx, false, + msg->user_data, msg->len, 0)) ret = -EOVERFLOW; out_unlock: io_double_unlock_ctx(ctx, target_ctx, issue_flags); @@ -153,7 +155,7 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags) switch (msg->cmd) { case IORING_MSG_DATA: - ret = io_msg_ring_data(req); + ret = io_msg_ring_data(req, issue_flags); break; case IORING_MSG_SEND_FD: ret = io_msg_send_fd(req, issue_flags); diff --git a/io_uring/net.c b/io_uring/net.c index a1a0b8f223e0..8c5154b05344 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -592,8 +592,8 @@ static inline void io_recv_prep_retry(struct io_kiocb *req) * Returns true if it is actually finished, or false if it should run * again (for multishot). */ -static inline bool io_recv_finish(struct io_kiocb *req, int *ret, - unsigned int cflags, bool mshot_finished) +static inline bool io_recv_finish(struct io_kiocb *req, unsigned int issue_flags, + int *ret, unsigned int cflags, bool mshot_finished) { if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { io_req_set_res(req, *ret, cflags); @@ -602,8 +602,8 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, } if (!mshot_finished) { - if (io_post_aux_cqe(req->ctx, req->cqe.user_data, *ret, - cflags | IORING_CQE_F_MORE)) { + if (io_post_aux_cqe(req->ctx, issue_flags & IO_URING_F_COMPLETE_DEFER, + req->cqe.user_data, *ret, cflags | IORING_CQE_F_MORE)) { io_recv_prep_retry(req); return false; } @@ -801,7 +801,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (kmsg->msg.msg_inq) cflags |= IORING_CQE_F_SOCK_NONEMPTY; - if (!io_recv_finish(req, &ret, cflags, mshot_finished)) + if (!io_recv_finish(req, issue_flags, &ret, cflags, mshot_finished)) goto retry_multishot; if (mshot_finished) { @@ -900,7 +900,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) if (msg.msg_inq) cflags |= IORING_CQE_F_SOCK_NONEMPTY; - if (!io_recv_finish(req, &ret, cflags, ret <= 0)) + if (!io_recv_finish(req, issue_flags, &ret, cflags, ret <= 0)) goto retry_multishot; return ret; @@ -1323,7 +1323,8 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags) if (ret < 0) return ret; - if (io_post_aux_cqe(ctx, req->cqe.user_data, ret, IORING_CQE_F_MORE)) + if (io_post_aux_cqe(ctx, issue_flags & IO_URING_F_COMPLETE_DEFER, + req->cqe.user_data, ret, IORING_CQE_F_MORE)) goto retry; return -ECANCELED; } diff --git a/io_uring/poll.c b/io_uring/poll.c index 2b77d18a67a7..c4865dd58862 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -245,7 +245,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_post_aux_cqe(ctx, req->cqe.user_data, + if (!io_post_aux_cqe(ctx, *locked, req->cqe.user_data, mask, IORING_CQE_F_MORE)) return -ECANCELED; } else { diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 133608200769..f37cdd8cfc95 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -170,10 +170,10 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) if (prsrc->tag) { if (ctx->flags & IORING_SETUP_IOPOLL) { mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + io_post_aux_cqe(ctx, false, prsrc->tag, 0, 0); mutex_unlock(&ctx->uring_lock); } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + io_post_aux_cqe(ctx, false, prsrc->tag, 0, 0); } } From patchwork Mon Nov 21 10:03:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13050577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E56E2C433FE for ; Mon, 21 Nov 2022 10:30:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229599AbiKUKaM (ORCPT ); Mon, 21 Nov 2022 05:30:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229779AbiKUKaL (ORCPT ); Mon, 21 Nov 2022 05:30:11 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52494B0439 for ; Mon, 21 Nov 2022 02:30:10 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 2AKMcOjg009034 for ; Mon, 21 Nov 2022 02:30:09 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=5sUvSezzyVnUch2+Aj1tHqEtCms1S8nBX2WWVDfbM50=; b=Bkks/xSrwGeJbHNOs6lqXcDDDpFBRsTHyf/YMkusRlakm4+kK3Xlhc/FjFUdGCOWoRUn pu/Iupnyslerxaun1TZuYQcPfGYFsWx3RsCD40VM4UaHJBNGRhlspYlgA+v5avwpGiz7 swY7kznryjqC6xfjlZGezQgaMKs9GzAhpBHXv8RhGjWxAFGZczrYilofKbyT21OtE9o7 uIHeD6IyzjQpdmDdxoByyyapFX0HW1FroaW8WvlZZc6x4p9QaTAfuduHYZNBV3+2ISQh z/EliYXpmSU2IrcnLPNlV6KsUZai4T5pO/7+tcMTDYqHMEphXj9qJl0iYQajdYDLdIuj Ew== Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3kxuq0c45s-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 21 Nov 2022 02:30:09 -0800 Received: from twshared2003.08.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 21 Nov 2022 02:30:07 -0800 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id A8EA19E66F86; Mon, 21 Nov 2022 02:03:56 -0800 (PST) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH for-next 10/10] io_uring: allow multishot polled reqs to defer completion Date: Mon, 21 Nov 2022 02:03:53 -0800 Message-ID: <20221121100353.371865-11-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221121100353.371865-1-dylany@meta.com> References: <20221121100353.371865-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: Rw5DwMxuApp38tK-6Y1b4wMIPKsjjnsU X-Proofpoint-GUID: Rw5DwMxuApp38tK-6Y1b4wMIPKsjjnsU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-21_06,2022-11-18_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Until now there was no reason for multishot polled requests to defer completions as there was no functional difference. However now this will actually defer the completions, for a performance win. Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5c240d01278a..2e12bddcfb2c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1821,7 +1821,7 @@ int io_poll_issue(struct io_kiocb *req, bool *locked) io_tw_lock(req->ctx, locked); if (unlikely(req->task->flags & PF_EXITING)) return -EFAULT; - return io_issue_sqe(req, IO_URING_F_NONBLOCK); + return io_issue_sqe(req, IO_URING_F_NONBLOCK | IO_URING_F_COMPLETE_DEFER); } struct io_wq_work *io_wq_free_work(struct io_wq_work *work)