From patchwork Thu Oct 27 14:44:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13022234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45C34FA3742 for ; Thu, 27 Oct 2022 14:45:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235638AbiJ0Opg (ORCPT ); Thu, 27 Oct 2022 10:45:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235705AbiJ0OpS (ORCPT ); Thu, 27 Oct 2022 10:45:18 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AB635143E for ; Thu, 27 Oct 2022 07:44:47 -0700 (PDT) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29RCQiFv028594 for ; Thu, 27 Oct 2022 07:44:47 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=12liYapi0VFx3n1Mqiv8jke831v291xACgfAGon9CDk=; b=m/2rF6IMe66uOzT8PJ3zddaB/4TXTLkLha5ZBYjs9Qfu1Bi0SYL3vnziXPiB5Rlv1dQ1 CzCNtBr8lmSWgmtkMqFvws9XclSZwEfowqjqC32RgybjVGAS2J5OsAh2z376pN0l0g2b 0eydZv7nXH0wqBtbVqfat4KwAVA9/355pxjo+u/wOYZjwKwtIVhVmCw7yYzhSyk+O4DK ZhGu77cyIsOikrc41q3g/dPmk7EDyEFF6U8KZpfRYxc9G2omt8hPEMqY8V/gZUBquCxB dY8PdawBa9TprvsKINKadUTAeb0RecRIUY9vZCh/WMMj3QJKe/riPycg+74S+FPlN8aH QQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kfagyhnrm-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 27 Oct 2022 07:44:47 -0700 Received: from twshared3704.02.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 27 Oct 2022 07:44:46 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id EDD038673CB2; Thu, 27 Oct 2022 07:44:34 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH 1/2] io_uring: use io_run_local_work_locked helper Date: Thu, 27 Oct 2022 07:44:28 -0700 Message-ID: <20221027144429.3971400-2-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221027144429.3971400-1-dylany@meta.com> References: <20221027144429.3971400-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: UXjdHdXg-nLDqc5pNC3udn8wXRyNgqD4 X-Proofpoint-ORIG-GUID: UXjdHdXg-nLDqc5pNC3udn8wXRyNgqD4 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-27_07,2022-10-27_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org prefer to use io_run_local_work_locked helper for consistency Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6cc16e39b27f..8a0ce7379e89 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1446,8 +1446,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) io_task_work_pending(ctx)) { u32 tail = ctx->cached_cq_tail; - if (!llist_empty(&ctx->work_llist)) - __io_run_local_work(ctx, true); + (void) io_run_local_work_locked(ctx); if (task_work_pending(current) || wq_list_empty(&ctx->iopoll_list)) { From patchwork Thu Oct 27 14:44:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 13022235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE9FAFA3741 for ; Thu, 27 Oct 2022 14:45:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235637AbiJ0Opf (ORCPT ); Thu, 27 Oct 2022 10:45:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235659AbiJ0OpR (ORCPT ); Thu, 27 Oct 2022 10:45:17 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A6BD50538 for ; Thu, 27 Oct 2022 07:44:46 -0700 (PDT) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29RCQoDL016566 for ; Thu, 27 Oct 2022 07:44:46 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=05X4luiSjwJIKYwwo1UsKsWxuMGPsXi1tm04DzyeZZQ=; b=fL/PKXNi+gugVCYoOOgs069UltQW6NdN8xZBwnLQGnoR4cIk9CwZr5BzAQBzH2EiqeCb zHhECtV6MtM6pHTZKZ/cCBb8dhzncRRK1iOFE6xOnXswXZ8gttK/FZ1wSX3eXpcty44R 2Vwa+9rcfTcphq9L+IMSFxjG4Ltr8D/FvzkcGozMnn7IkIVH4gsKx9LQBRDFaXYxAMag BVvya3WXDyl4kKwUXfdPxpDQ1yizX+zEOSDLJeaiya1ElQ1RLLx0UjIXlJd/oK+GLtsL KaUfYSVfVChpA4q5lEoOJmr8u/Us9zlevDw8MLMc21yPGY9oPbgjdXIjc87uEsdXIKf0 AA== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3kfagwhr8n-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 27 Oct 2022 07:44:46 -0700 Received: from twshared2001.03.ash8.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 27 Oct 2022 07:44:45 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 39F5B8673CD6; Thu, 27 Oct 2022 07:44:36 -0700 (PDT) From: Dylan Yudaken To: Jens Axboe , Pavel Begunkov CC: , , Dylan Yudaken Subject: [PATCH 2/2] io_uring: unlock if __io_run_local_work locked inside Date: Thu, 27 Oct 2022 07:44:29 -0700 Message-ID: <20221027144429.3971400-3-dylany@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221027144429.3971400-1-dylany@meta.com> References: <20221027144429.3971400-1-dylany@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: ZsB2Vflt7F5MPhu43pvRT7_ij4CtElnt X-Proofpoint-ORIG-GUID: ZsB2Vflt7F5MPhu43pvRT7_ij4CtElnt X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-27_07,2022-10-27_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org It is possible for tw to lock the ring, and this was not propogated out to io_run_local_work. This can cause an unlock to be missed. Instead pass a pointer to locked into __io_run_local_work. Fixes: 8ac5d85a89b4 ("io_uring: add local task_work run helper that is entered locked") Signed-off-by: Dylan Yudaken --- io_uring/io_uring.c | 8 ++++---- io_uring/io_uring.h | 12 ++++++++++-- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8a0ce7379e89..ac8c488e3077 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1173,7 +1173,7 @@ static void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx) } } -int __io_run_local_work(struct io_ring_ctx *ctx, bool locked) +int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) { struct llist_node *node; struct llist_node fake; @@ -1192,7 +1192,7 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool locked) struct io_kiocb *req = container_of(node, struct io_kiocb, io_task_work.node); prefetch(container_of(next, struct io_kiocb, io_task_work.node)); - req->io_task_work.func(req, &locked); + req->io_task_work.func(req, locked); ret++; node = next; } @@ -1208,7 +1208,7 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool locked) goto again; } - if (locked) + if (*locked) io_submit_flush_completions(ctx); trace_io_uring_local_work_run(ctx, ret, loops); return ret; @@ -1225,7 +1225,7 @@ int io_run_local_work(struct io_ring_ctx *ctx) __set_current_state(TASK_RUNNING); locked = mutex_trylock(&ctx->uring_lock); - ret = __io_run_local_work(ctx, locked); + ret = __io_run_local_work(ctx, &locked); if (locked) mutex_unlock(&ctx->uring_lock); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ef77d2aa3172..331ec2869212 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -27,7 +27,7 @@ enum { struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow); bool io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); -int __io_run_local_work(struct io_ring_ctx *ctx, bool locked); +int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked); int io_run_local_work(struct io_ring_ctx *ctx); void io_req_complete_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); @@ -277,9 +277,17 @@ static inline int io_run_task_work_ctx(struct io_ring_ctx *ctx) static inline int io_run_local_work_locked(struct io_ring_ctx *ctx) { + bool locked; + int ret; + if (llist_empty(&ctx->work_llist)) return 0; - return __io_run_local_work(ctx, true); + + locked = true; + ret = __io_run_local_work(ctx, &locked); + if (WARN_ON(!locked)) + mutex_lock(&ctx->uring_lock); + return ret; } static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked)