From patchwork Tue Jan 3 03:03:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5454C3DA7A for ; Tue, 3 Jan 2023 03:05:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236605AbjACDFZ (ORCPT ); Mon, 2 Jan 2023 22:05:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231327AbjACDFX (ORCPT ); Mon, 2 Jan 2023 22:05:23 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EB0E2733 for ; Mon, 2 Jan 2023 19:05:22 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id c65-20020a1c3544000000b003cfffd00fc0so25298834wma.1 for ; Mon, 02 Jan 2023 19:05:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bqsHZn+qHgHSoeDc737BfWQdCtKG04zCxIEm2ZdOgbg=; b=XYfImAobV6SpW4yIiOnkaG5twtuWr2HOPDjb9XnIRhoyhH+2ZdnJWao4X5foK2UTnv qBukGaeDbhGg3KHyro99j7NReZYteSFy5TvVuSSp2MYk4G8yuLf99kWT3bW5gJpvTCqF yacGe3MGc7tJL7VLynGY/5NP9MrocvZeM/tXaRy4CJ5hFWgP130QZsg00VQHcYh/D3lw 0UoU+G5mrWWbiJydB+zc1OqNvuRyALJhpHF9DoiAgh0/Db+JUgqaUkrjI0wEQ8mpOIBs Kbb27KTWEy5fHd4lttDzz4Ur0EY1AmVPSFTkposgCQvcLMwna+C0tY1954gtL7GW8Gg7 lgGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bqsHZn+qHgHSoeDc737BfWQdCtKG04zCxIEm2ZdOgbg=; b=kc4+yZVSEk4Lsl45d5n5JU/mf6jRbPtLcWiFPe+0FxPE0U1gETcg2dQ5vjHDlmeFO8 u/pQeSLRXyIHjXHa/TFOWKib1+JIdswXl202Uin5+noI008/1pGs9kmBQ1VMMLcvZKLf bCldz7M6bCrgvYdHnqIkUQa2FxN0i38DEnizNN394yE9xUQi702bZwIvkPsMUIc/BwQY EM52F+NTwAFaD8ZBNgh7Dpz2zV8DCfwRvtXlGywPks0/Euumn3UV7KM5Kq/mKsqXpWPF BVJpcObDqlN7BfGU1iCR7kK4e9k9J51Fd1WBEGp6DiylojH9FVcN2L5Fg5Hzo0GwKGpF mWdg== X-Gm-Message-State: AFqh2kp8IDxF6eqV97YPjZZp27ZeKLFB/RNxDXFugMv2ewl7FENUlear SpSXI0qAst639pIbPn2oV0LjzXBxVNc= X-Google-Smtp-Source: AMrXdXuJROY/4NMzvRLSzXPqTY0zlSYsIndEYdqBhMppZKbl09QekFmMoGn+gB7yXcbQCUbvieAFuQ== X-Received: by 2002:a05:600c:18a1:b0:3d2:3ec4:7eed with SMTP id x33-20020a05600c18a100b003d23ec47eedmr33243527wmp.10.1672715120818; Mon, 02 Jan 2023 19:05:20 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:20 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 01/13] io_uring: rearrange defer list checks Date: Tue, 3 Jan 2023 03:03:52 +0000 Message-Id: <52e8192f6e36b5918371aa139a4f62a084757622.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There should be nothing in the ->work_llist for non DEFER_TASKRUN rings, so we can skip flag checks and test the list emptiness directly. Also move it out of io_run_local_work() for inlining. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 3 --- io_uring/io_uring.h | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 58ac13b69dc8..4f12619f9f21 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1338,9 +1338,6 @@ int io_run_local_work(struct io_ring_ctx *ctx) bool locked; int ret; - if (llist_empty(&ctx->work_llist)) - return 0; - __set_current_state(TASK_RUNNING); locked = mutex_trylock(&ctx->uring_lock); ret = __io_run_local_work(ctx, &locked); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index e9f0d41ebb99..46c0f765a77a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -274,7 +274,7 @@ static inline int io_run_task_work_ctx(struct io_ring_ctx *ctx) int ret = 0; int ret2; - if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) + if (!llist_empty(&ctx->work_llist)) ret = io_run_local_work(ctx); /* want to run this after in case more is added */ From patchwork Tue Jan 3 03:03:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4378FC5479D for ; Tue, 3 Jan 2023 03:05:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236616AbjACDF0 (ORCPT ); Mon, 2 Jan 2023 22:05:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236576AbjACDFY (ORCPT ); Mon, 2 Jan 2023 22:05:24 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AEB0B7FA for ; Mon, 2 Jan 2023 19:05:23 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id l26so20181953wme.5 for ; Mon, 02 Jan 2023 19:05:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O/tD7Lk7IpV2oZF2JCtq4S9GMjdkUZ3nZbiGPCh9Wao=; b=Oi+RWNvNe9Q7BVmb62RkLBMK4r5cUX6Qze2/0Ea3nKYBeFu2D0CfxNfjQ8ljRXr65l gIqjLF3L3+dgdQGsTX9UJYNOfk9NRsHgoLltmhmB8UfojmUaQAJFEsVOl7Z3y6PXA4OJ gyqjtTWMS8JN8ZEIVGECtPPYheRWmz3UESqktRjWwzLRHXUmMXQe3CxA2K2Jw/VOoQv+ crj9ahlKm/jBbpJTMruTred3WK0cgKswXM/UzyWkKJVI1hSkdvPPclNrum9o83A0eUEH 8NR8BjpYGdOTXP9ZmJ6mbBIugU2+up1NRFn9Kne2kIrBBXq1lsZrOKvZD25KBw8A7Oul VPDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O/tD7Lk7IpV2oZF2JCtq4S9GMjdkUZ3nZbiGPCh9Wao=; b=ZSicY5Sr6o0oizCAuvJXjbEYgPXuadp/aqMbZcUwPqvl3dnh4sC6wR12+MxsbOr86v zwN7J5QJzZ5TV5jW12lGbvYqg72BgAaRq9FqSXSH416Qddw1PlOdbGf5Pvaf2cJUb28O 3mqhBbWsV4i9ErXgSI5HGKy7E0bqrjCnoeeuABCWqVfYPD4BptDBnTMxFbK+JLAaNruI pmbpkAQw9Cj4qEBXTG6JhOBTSEpXd+5q2oAilmAKlka9T/52zIEHxyOa1ZGTtMavVd+I xdtsoJxl9SO3UCIuVjpQserbNxJTM4PpH7gfO4tulFRxI9l8R57GKyDloQaKx43xhRim mOeQ== X-Gm-Message-State: AFqh2koOfq+fqBhOYd2mnffepp/yqSH13Id++omcIbU2Ch/uTauQEheL 8TcutybhmJ9HWDZGFPmrsNbIvtYyg2M= X-Google-Smtp-Source: AMrXdXvujv1P6DSg+NEpZWpt5jmB8XwxZRf8P2jWyRPzgJBBKNXbG8GWdcG5A6gWXfkAzDhvpn78dA== X-Received: by 2002:a05:600c:34cf:b0:3d0:7415:c5a9 with SMTP id d15-20020a05600c34cf00b003d07415c5a9mr31074631wmq.21.1672715121496; Mon, 02 Jan 2023 19:05:21 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:21 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 02/13] io_uring: don't iterate cq wait fast path Date: Tue, 3 Jan 2023 03:03:53 +0000 Message-Id: <53f32543f0907a73f1e7c201dd78765485f26213.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Task work runners keep running until all queues tw items are exhausted. It's also rare for defer tw to queue normal tw and vise versa. Taking it into account, there is only a dim chance that further iterating the io_cqring_wait() fast path will get us anything and so we can remove the loop there. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4f12619f9f21..d9a2cf061acc 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2507,18 +2507,16 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (!io_allowed_run_tw(ctx)) return -EEXIST; - - do { - /* always run at least 1 task work to process local work */ - ret = io_run_task_work_ctx(ctx); + if (!llist_empty(&ctx->work_llist)) { + ret = io_run_local_work(ctx); if (ret < 0) return ret; - io_cqring_overflow_flush(ctx); - - /* if user messes with these they will just get an early return */ - if (__io_cqring_events_user(ctx) >= min_events) - return 0; - } while (ret > 0); + } + io_run_task_work(); + io_cqring_overflow_flush(ctx); + /* if user messes with these they will just get an early return */ + if (__io_cqring_events_user(ctx) >= min_events) + return 0; if (sig) { #ifdef CONFIG_COMPAT From patchwork Tue Jan 3 03:03:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3044C54EBC for ; Tue, 3 Jan 2023 03:05:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236589AbjACDF1 (ORCPT ); Mon, 2 Jan 2023 22:05:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236572AbjACDFY (ORCPT ); Mon, 2 Jan 2023 22:05:24 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BADCCB7F8 for ; Mon, 2 Jan 2023 19:05:23 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id z8-20020a05600c220800b003d33b0bda11so15897308wml.0 for ; Mon, 02 Jan 2023 19:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dKZTw9ynuvwT3BWG3xafuTprhv2oUDeCYdXR4dPur1U=; b=T76KYE5D2srVZxZ4dqgcU1yjFSxOtZb4TabU7uTAgZxEykx6FG0YBdxK5Uvodyzc3e 2QGejU1hg91WqfK1HzwhMN8p5656VRhHq+So/DvEXzCee6w89yXqMo+jjVPXrMJnB4Lp WvJfwY5/7ON7l/7rxPIPc6OezaALb/fn7hajJoHe5D6rT/mZWL0UKH4zBwG+2jWKyqxr flKJYrQRLZHZlVS8Ju//iEsEKibJuLm9RRkcDKeM8UJ/zCjl+irb+yHiWkv9tGoe0MZN I/8MaLrL0kmIjRLX8JWSjyrcRnki8S3x3y1yTAzNNwBdB48sXXw4ZvnAcacFp0XIwNjC pmtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dKZTw9ynuvwT3BWG3xafuTprhv2oUDeCYdXR4dPur1U=; b=oe5N8gLOvtWo0PK21O6QBy0Rwp6w9qtXM+hSqB3mazSswI7pkvIheKbcUUGmU5WA9A +mxLg122LemN5Xv2I8IDBhEEsrBSG5lm0uJXcoqZQV7534HUGxvcbtlYvAMYnuoyCsf6 l6qF7iezC1YH6HyjMg0sLlgXflNjL6VXR6NDbOUICBJcdzAZElecB3dYMYZPihuu/wkj /IbIaDvdKFhRXHiW+WZPID6QNyZjGeDgAL2E4XPtwZ3oVrFEu8vdeJJJp4dhWD1UTrd1 13Sr/T1TfFGhNYhzxAHbwz0EZIRG0oMHdZ44X8NCo/7OX23ULrGHtQ+Q0Zwd4iRFH5ov /yiQ== X-Gm-Message-State: AFqh2koob/7JDHMlNIuQqf7TOwwmmypsOuFkWvZn4xL5mLNmV1rd1UPn ooWVCPQrdZEgsd8CT84alithQBM3Jh8= X-Google-Smtp-Source: AMrXdXsnki7NheOUxQa1ahD9scmYHdyL9v84iHE+m66Y8o7pVqAlxoj5ks8f4dWJbjmh7nCPwmEuUw== X-Received: by 2002:a05:600c:43ca:b0:3d0:73f5:b2c0 with SMTP id f10-20020a05600c43ca00b003d073f5b2c0mr31897276wmn.20.1672715122176; Mon, 02 Jan 2023 19:05:22 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:21 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 03/13] io_uring: kill io_run_task_work_ctx Date: Tue, 3 Jan 2023 03:03:54 +0000 Message-Id: <8a6592ceb47d808d3e83507c133845c9029e7a2f.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is only one user of io_run_task_work_ctx(), inline it. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 6 +++++- io_uring/io_uring.h | 20 -------------------- 2 files changed, 5 insertions(+), 21 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d9a2cf061acc..a22c6778a988 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2452,7 +2452,11 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode, int io_run_task_work_sig(struct io_ring_ctx *ctx) { - if (io_run_task_work_ctx(ctx) > 0) + if (!llist_empty(&ctx->work_llist)) { + if (io_run_local_work(ctx) > 0) + return 1; + } + if (io_run_task_work() > 0) return 1; if (task_sigpending(current)) return -EINTR; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 46c0f765a77a..8a5c3affd724 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -269,26 +269,6 @@ static inline bool io_task_work_pending(struct io_ring_ctx *ctx) return task_work_pending(current) || !wq_list_empty(&ctx->work_llist); } -static inline int io_run_task_work_ctx(struct io_ring_ctx *ctx) -{ - int ret = 0; - int ret2; - - if (!llist_empty(&ctx->work_llist)) - ret = io_run_local_work(ctx); - - /* want to run this after in case more is added */ - ret2 = io_run_task_work(); - - /* Try propagate error in favour of if tasks were run, - * but still make sure to run them if requested - */ - if (ret >= 0) - ret += ret2; - - return ret; -} - static inline int io_run_local_work_locked(struct io_ring_ctx *ctx) { bool locked; From patchwork Tue Jan 3 03:03:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D297EC53210 for ; Tue, 3 Jan 2023 03:05:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236572AbjACDF2 (ORCPT ); Mon, 2 Jan 2023 22:05:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236604AbjACDFZ (ORCPT ); Mon, 2 Jan 2023 22:05:25 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B5132733 for ; Mon, 2 Jan 2023 19:05:24 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id g25-20020a7bc4d9000000b003d97c8d4941so16615381wmk.4 for ; Mon, 02 Jan 2023 19:05:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4jaEqFc+iCMwpGJhqH81jmmQEa3DfQMsul7/hQkLGvk=; b=QrXjygBzewKh7F65FBgvbLSfl3m7OReHrurOGMcLIr2WY1CCEjQ8/np/H9h8WBivSF RjQ3S5PPwZ8zJL2j6ErMy+qjKuupTNFq7J3efQspd//WLzKFZXGSAWJtcpVL0jECYYyf vI+BJwp8ph1NURcNbgeOCNnTyoelEIkdzxqiTpm6nXE+JIz4upp5GItQwaoaouBbt/5Q MLzqI+3GR2xCzkolJAcs6bcgqdRKCyIBRJn2ehrsxabB6Vbf+p8ypacI+88nQKgu+hxz FQCM8K7i8w/jK3SZ+0X9Q7F+yRLP5S3Y7ur2kqB8GHMn+hPLQNe3P4BxSUvzup2Pady/ dI9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4jaEqFc+iCMwpGJhqH81jmmQEa3DfQMsul7/hQkLGvk=; b=6KFxZihYRlJsJH8mysSYjJgXDZJw6vznXYtg+eL+fAx0TJoKJaJrjKmWTMREbH2rGa Yr+bnQULJ08QPyFycJ1GvtipX319BJyc7rHcw3NMLBChnhm6icxYwtQBea5vVqvgfg9j 3HhJ27q2WUOaLC+hSt6C9W08dFdlMW13OLq5udZ1JZKVSHTtYojX/sp2yWm0M+J+4mFE VPccE4xX6j9HQGA3xjJybI8M/e04Ip4KvcZMnptnlZdtxC+29CJydr8m3Tm1o9CVYEOC iodN3//We4CvjG/qoZYixL5nFQvaTqU8ImduP7HKNBElq8toVQeGt9rYez1Xzn/+SjSk V5Jw== X-Gm-Message-State: AFqh2kqwKbMQ1Zk8Ox1VUhDfWHVUd103028XJRbucq9F98Tp45OzKMPK 6cACNVYNfEe2gNWqUr9WfNEs3oU0d6I= X-Google-Smtp-Source: AMrXdXvXf4fhySgApQv83YXUzQAwQNtCj/QbnRNin5K1rkVbHGcBUe1aoLS/gbH8GqfIKxXovFGQzA== X-Received: by 2002:a05:600c:4307:b0:3d3:494f:6a39 with SMTP id p7-20020a05600c430700b003d3494f6a39mr30219769wme.16.1672715123015; Mon, 02 Jan 2023 19:05:23 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:22 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 04/13] io_uring: move defer tw task checks Date: Tue, 3 Jan 2023 03:03:55 +0000 Message-Id: <5ec52d3ab11d92bd8fef0376aca9f0fde38ab4e7.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Most places that want to run local tw explicitly and in advance check if they are allowed to do so. Don't rely on a similar check in __io_run_local_work(), leave it as a just-in-case warning and make sure callers checks capabilities themselves. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 15 ++++++--------- io_uring/io_uring.h | 5 +++++ 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a22c6778a988..ff457e525e7c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1296,14 +1296,13 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) struct llist_node *node; struct llist_node fake; struct llist_node *current_final = NULL; - int ret; + int ret = 0; unsigned int loops = 1; - if (unlikely(ctx->submitter_task != current)) + if (WARN_ON_ONCE(ctx->submitter_task != current)) return -EEXIST; node = io_llist_xchg(&ctx->work_llist, &fake); - ret = 0; again: while (node != current_final) { struct llist_node *next = node->next; @@ -2511,11 +2510,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (!io_allowed_run_tw(ctx)) return -EEXIST; - if (!llist_empty(&ctx->work_llist)) { - ret = io_run_local_work(ctx); - if (ret < 0) - return ret; - } + if (!llist_empty(&ctx->work_llist)) + io_run_local_work(ctx); io_run_task_work(); io_cqring_overflow_flush(ctx); /* if user messes with these they will just get an early return */ @@ -3052,7 +3048,8 @@ static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, } } - if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) + if ((ctx->flags & IORING_SETUP_DEFER_TASKRUN) && + io_allowed_defer_tw_run(ctx)) ret |= io_run_local_work(ctx) > 0; ret |= io_cancel_defer_files(ctx, task, cancel_all); mutex_lock(&ctx->uring_lock); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 8a5c3affd724..9b7baeff5a1c 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -352,6 +352,11 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) return container_of(node, struct io_kiocb, comp_list); } +static inline bool io_allowed_defer_tw_run(struct io_ring_ctx *ctx) +{ + return likely(ctx->submitter_task == current); +} + static inline bool io_allowed_run_tw(struct io_ring_ctx *ctx) { return likely(!(ctx->flags & IORING_SETUP_DEFER_TASKRUN) || From patchwork Tue Jan 3 03:03:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8426CC5479D for ; Tue, 3 Jan 2023 03:05:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231327AbjACDF3 (ORCPT ); Mon, 2 Jan 2023 22:05:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236614AbjACDF0 (ORCPT ); Mon, 2 Jan 2023 22:05:26 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FA61BCA5 for ; Mon, 2 Jan 2023 19:05:25 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id b24-20020a05600c4a9800b003d21efdd61dso22171382wmp.3 for ; Mon, 02 Jan 2023 19:05:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VLXUDtmB9EhqaBSicBvqsDUUk5cQrRfzMcjLmxZc7BY=; b=L5SbiIgoHeppEKCFwQXswQdD5PzZ0eWPeE5MuUe9+TXe21U44GZzhQTNBD7AXDibEw bP57zDZIXlnuqgpCctkJBsLynqSd0rJuWf+3wKNdRcdx9pKGMDL/P3BjdnOlKrpP4fMs vktEY1Gxijc0scQlK1kyDIslpQTOh6pATrlgGQKs7xcg75OWv0bfHikuFou+rIXGPt6V tecKjNvDTzmZsc5TBca/nbp1OBkRWFQ/CzXmSSC9aGfmnmuz7JDFsTIhX0bDBL1zel2w xZyH++FNVWu+3pa/anNfT8WQmQTSdfzInB3+8sjyMScUDcWLCJZOGIxxYWfhEBf+r/Ry JQbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VLXUDtmB9EhqaBSicBvqsDUUk5cQrRfzMcjLmxZc7BY=; b=dKFfW3D5yRXhtzJNNrxXHroBTUzLSr/T8BbGvcEEhnUr759MqSU02oKr9r3Bq3WUoy JHagcYL7S1tACW2/dsL61Is2md+Bi6bZIRgigy4RFy82QPqhqpkRzQS+KaiVUiNvrZrB tzzZH1uwrChikBQuGXuulTr6rf+2EqpIus1UUW/DdrIr/CkZDskyHmpyQZ0u7pdmczxn 3MlZHbiRQKsBkj3HtO+BNgUv7ikfAqr5hA2fK2WRiIFnXzgYI41pUrBTCdze4mfXfj7j xXJJBlB/oS37GBcN1Zm0PMJxUYxrJMDa02o0eFKknCsY5hVcGdaQygDqSFk5/pucxaIQ DXgw== X-Gm-Message-State: AFqh2kpoHl/SjJGd2mBitBDEcDh62nYpTBTxCVfehLjm2oOshc5uOmkn YxNpOwAANh5WV1wXisk/Y4FPui0BGEk= X-Google-Smtp-Source: AMrXdXsuOPivGIsMjJPK0wQoQVm5CtjKGDP6bCSMjIPcAhMKEIl+mVmwoLYt9itL8oyUesD2lSZEAQ== X-Received: by 2002:a05:600c:4e07:b0:3d3:5319:b6d3 with SMTP id b7-20020a05600c4e0700b003d35319b6d3mr30453558wmq.38.1672715123768; Mon, 02 Jan 2023 19:05:23 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:23 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 05/13] io_uring: parse check_cq out of wq waiting Date: Tue, 3 Jan 2023 03:03:56 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We already avoid flushing overflows in io_cqring_wait_schedule() but only return an error for the outer loop to handle it. Minimise it even further by moving all ->check_cq parsing there. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ff457e525e7c..e3c5de299baa 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2468,21 +2468,13 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, ktime_t timeout) { int ret; - unsigned long check_cq; + if (unlikely(READ_ONCE(ctx->check_cq))) + return 1; /* make sure we run task_work before checking for signals */ ret = io_run_task_work_sig(ctx); if (ret || io_should_wake(iowq)) return ret; - - check_cq = READ_ONCE(ctx->check_cq); - if (unlikely(check_cq)) { - /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - return 1; - if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) - return -EBADR; - } if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS)) return -ETIME; @@ -2548,13 +2540,25 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, trace_io_uring_cqring_wait(ctx, min_events); do { - if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) { - finish_wait(&ctx->cq_wait, &iowq.wq); - io_cqring_do_overflow_flush(ctx); - } + unsigned long check_cq; + prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); ret = io_cqring_wait_schedule(ctx, &iowq, timeout); + + check_cq = READ_ONCE(ctx->check_cq); + if (unlikely(check_cq)) { + /* let the caller flush overflows, retry */ + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) { + finish_wait(&ctx->cq_wait, &iowq.wq); + io_cqring_do_overflow_flush(ctx); + } + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) { + ret = -EBADR; + break; + } + } + if (__io_cqring_events_user(ctx) >= min_events) break; cond_resched(); From patchwork Tue Jan 3 03:03:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3A36C53210 for ; Tue, 3 Jan 2023 03:05:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236635AbjACDFa (ORCPT ); Mon, 2 Jan 2023 22:05:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236622AbjACDF1 (ORCPT ); Mon, 2 Jan 2023 22:05:27 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35C3EBE3C for ; Mon, 2 Jan 2023 19:05:26 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id bi26-20020a05600c3d9a00b003d3404a89faso15758521wmb.1 for ; Mon, 02 Jan 2023 19:05:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Df0rkeviyDF2yto0MQKu8Bc1IpWSypehWf9m4+kV1uM=; b=c+cbOcL2HcVB6jlLiv0EROiSEgODmRzVb6dfFjWtkhDC2Aj/grTJEkjFZW6F76lw8W uU+csCclI9xgAo5+cmX6mi8pq+OBQn0ViWB9aRzpA5MkwVa9AytuPVA2Aw1jLQtDfc1g X4WeYILNsSUyl1nT0AzI+oj/bO6va0NK0Z5ZQiFcv7V8q5BYoapqXlYR6zzFXagUMpsO xWjalh8CKaT7WDW3USGy7vLbQn8/4U705Ti6U0egp/jf5lA98AdGc/g19fGsuXPT3STJ AruNzkciXhk2OtSTnpBTzhw2jjCKZytsUNiJx19lay3glsnp5GkwNAdwEAdQWam0al63 o21A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Df0rkeviyDF2yto0MQKu8Bc1IpWSypehWf9m4+kV1uM=; b=N0Vp/jGkrPByrieXiSYgthKay5QtZ7GBTpylx8eLB2Vb8zjY4pJOYSVl4PMtxkDeyk 80rUjv7NXtLuzx+VK7d1qGQcw1SVWxP5hXEP48Wqw2s0V4eeWEfwwx8n+NIbqQHWbtC4 3NM0I2SuxTfpuTjSR1mpsenLpdkkw7WNJAt+D7QQko9mtzg5pdztGvfapcQme3Qgq010 41WyuMzKlW88S/vrt4xH1UsF6ODkNxbGnfgeTEMiHGX0QkWZQDPp1Fbs43NaskDG4J5A pTXgT+aryO9ojf5alAeNewqU1kNIbCxOvadQ/oZLky56VuE/XnvSZuLTr98Y6DvbHQLF MYCg== X-Gm-Message-State: AFqh2kr2iNXDn9QRmhsZS/RHoQwc9eEpbQgD1Wuf5JIzI84KZtdUK87k h9WXtK4i5qiMzTmcZjQ8R48ByI6C4hY= X-Google-Smtp-Source: AMrXdXtWCDN8VeB5S+gHQNgaT/B7HYWEsXCL/zjGwa/eoIRLs+mjipzTWoHm86N54C8TbMd70Zasqw== X-Received: by 2002:a05:600c:214a:b0:3cf:6910:51d4 with SMTP id v10-20020a05600c214a00b003cf691051d4mr29612671wml.29.1672715124434; Mon, 02 Jan 2023 19:05:24 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:24 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 06/13] io_uring: mimimise io_cqring_wait_schedule Date: Tue, 3 Jan 2023 03:03:57 +0000 Message-Id: <4d299f7d0c3007bed5f62e2a53a928013e8c19e4.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_cqring_wait_schedule() is called after we started waiting on the cq wq and set the state to TASK_INTERRUPTIBLE, for that reason we have to constantly worry whether we has returned the state back to running or not. Leave only quick checks in io_cqring_wait_schedule() and move the rest including running task work to the callers. Note, we run tw in the loop after the sched checks because of the fast path in the beginning of the function. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e3c5de299baa..fc9604848bbb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2467,24 +2467,19 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, struct io_wait_queue *iowq, ktime_t timeout) { - int ret; - if (unlikely(READ_ONCE(ctx->check_cq))) return 1; - /* make sure we run task_work before checking for signals */ - ret = io_run_task_work_sig(ctx); - if (ret || io_should_wake(iowq)) - return ret; + if (unlikely(!llist_empty(&ctx->work_llist))) + return 1; + if (unlikely(test_thread_flag(TIF_NOTIFY_SIGNAL))) + return 1; + if (unlikely(task_sigpending(current))) + return -EINTR; + if (unlikely(io_should_wake(iowq))) + return 0; if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS)) return -ETIME; - - /* - * Run task_work after scheduling. If we got woken because of - * task_work being processed, run it now rather than let the caller - * do another wait loop. - */ - ret = io_run_task_work_sig(ctx); - return ret < 0 ? ret : 1; + return 0; } /* @@ -2545,6 +2540,16 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); ret = io_cqring_wait_schedule(ctx, &iowq, timeout); + if (ret < 0) + break; + /* + * Run task_work after scheduling and before io_should_wake(). + * If we got woken because of task_work being processed, run it + * now rather than let the caller do another wait loop. + */ + io_run_task_work(); + if (!llist_empty(&ctx->work_llist)) + io_run_local_work(ctx); check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { @@ -2559,10 +2564,12 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, } } - if (__io_cqring_events_user(ctx) >= min_events) + if (io_should_wake(&iowq)) { + ret = 0; break; + } cond_resched(); - } while (ret > 0); + } while (1); finish_wait(&ctx->cq_wait, &iowq.wq); restore_saved_sigmask_unless(ret == -EINTR); From patchwork Tue Jan 3 03:03:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 234D8C53210 for ; Tue, 3 Jan 2023 03:05:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236622AbjACDFb (ORCPT ); Mon, 2 Jan 2023 22:05:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236576AbjACDF1 (ORCPT ); Mon, 2 Jan 2023 22:05:27 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A305BBF59 for ; Mon, 2 Jan 2023 19:05:26 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id ay2-20020a05600c1e0200b003d22e3e796dso22196992wmb.0 for ; Mon, 02 Jan 2023 19:05:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P+EQRTaDuT5Ag8ayOXl3BanL3YIC2TQpRoWl91YOkYw=; b=LUd8Ax58JoWPH+xsb0kt6PRIXVdauaIvdhT6CBr6IZG9EIfvn4Hq3NP9QWNZ5oGb5T JNWCn93/2Q/vjIDxQ04sUH91JaJBRFnPfP3JsCIv50yTmSzT1z92pTgO32ZWOxCEgJb8 TBXvgR7pYXViXEHDMqsveX+kN7QI1CcKb/tlQIHZQ1FHnn7yN+4lR17GFGK73DkUxd1z dyUFqJ0B6otMPpplSPEGfQl1/J0RBN74hcVvMPxp/xlk1KYsLysAHLPnqvNarvWB5+BE AxZaHrRJjFYRVv7iJei0fHiYpHI/6jsNDXSOE5kly9zul5IhplO2Vrrj3xRZnV4/Zced oaOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P+EQRTaDuT5Ag8ayOXl3BanL3YIC2TQpRoWl91YOkYw=; b=x33F06uTh9PJWQ4/ybYRDraRlXuD6Vcss8AZGiXNr3sYXIPdjbLuB7Cd/a7izi9zOk onOngx07NtovBN8VmVb/j62XPgENNSiJVvK2ICmN84WGCiruGzmM2VZrpvQvMW4c0FJ3 Sj7Lqvw9ei4xtXf0oiu5NJA9Q87X5FvkJACwiQPFVhBsIadHCQzLQJ41ZYPOhy20ubpf iQ6IKO9+USwx823Cahu7HT0IMrJqN4o9SAXLBxylfV6MRU0Cya0xYlxEgq3ZHXBXah27 iQvCy7HwkWulas73PbEhTaxGMi3H2fPMEjOcw25IJ7csH0nF7iRPB6YZqYCLbM/7AeUR o+yQ== X-Gm-Message-State: AFqh2kqGVZ2TAguYEQ99fo7rPHYvyUj/W/K/BRaxALk1ECKc4swERjCo K8QpLhl0n8pezk02kBZUBqeZWc+2Zms= X-Google-Smtp-Source: AMrXdXutjqoipSI+zkKApC13atGLI0G/vroWPZs4dg6wEzF1CW7cpwm72TNRCJE+3nUWBJfSQjPRBg== X-Received: by 2002:a05:600c:2252:b0:3d3:5d8b:7af with SMTP id a18-20020a05600c225200b003d35d8b07afmr32615459wmm.41.1672715125097; Mon, 02 Jan 2023 19:05:25 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:24 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 07/13] io_uring: simplify io_has_work Date: Tue, 3 Jan 2023 03:03:58 +0000 Message-Id: <8a275bf82c72132862aba3f268718555caa81a53.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org ->work_llist should never be non-empty for a non DEFER_TASKRUN ring, so we can safely skip checking the flag. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index fc9604848bbb..a8d3826f3d17 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2416,8 +2416,7 @@ struct io_wait_queue { static inline bool io_has_work(struct io_ring_ctx *ctx) { return test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq) || - ((ctx->flags & IORING_SETUP_DEFER_TASKRUN) && - !llist_empty(&ctx->work_llist)); + !llist_empty(&ctx->work_llist); } static inline bool io_should_wake(struct io_wait_queue *iowq) From patchwork Tue Jan 3 03:03:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34DC4C3DA7A for ; Tue, 3 Jan 2023 03:05:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232024AbjACDFa (ORCPT ); Mon, 2 Jan 2023 22:05:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236625AbjACDF1 (ORCPT ); Mon, 2 Jan 2023 22:05:27 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 534FEBF58 for ; Mon, 2 Jan 2023 19:05:26 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id l26so20182036wme.5 for ; Mon, 02 Jan 2023 19:05:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=F6vtilktCsxzq342UKZKm+kG0VKB3HCbCG+3kPZMYhc=; b=BggrJkyG1pkd0tBFBd02i9j0xdx1kxDaNBp6M0uZuQp3UUoeCSDX/cMDq0JqtoEzjy QMyDrjyz02157vdnvuJ9zMH4utjf+3FVxoSOWkxKjNwzpIOp6zcGv9+VM6AFy8Gmjz4N 2e5lxTAN3HUOivnRpzZYGKJTqjkVfx3XxpxOLaWtdxomfZCs6ikkBO5kbs/L+A01XUzg viW5YfOU5GaL/3NHNSNpiLqnOtCwoOoZteLFuEBKGDQNgWtbDGT4FfltmsXpI8Zotg2i H2Kt5nlvboaKYJxbvAk26gIXL3UHkBTwi/drZ9O0HFIJC6rOTWSfC/FtThGFBgsXWOhD l2Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F6vtilktCsxzq342UKZKm+kG0VKB3HCbCG+3kPZMYhc=; b=i9Whn9BSiD7hpx+sw9UgLbmEoabxnb4gumMHVKi3GByijrFxDJ/zjiSfySrgZieKJL zPmOJZJ+iastvJ4AiPCm6IPuWhbOnPqouMh4RvOa91hRfbQDCDzioxOVFqsTM7igSeyB Ukt3tE58fdBlEMtwlFynk09P8DPwRZ+I1D7H24p/7ZHW1XXRb2KgUi0agM2w7CW4aXIm JkY7nLbSWyBxDuw+LRpoWF03MCfp/cQ85dFOiZViljNpweseRatWdtSeEbCBlMKNOWQ5 X9DivpuHRNEcDTe9sIupNc0Tt/HjvN2n72QBQerZqeUSlNze0hNuCzW01W+vo20YS93a WDDw== X-Gm-Message-State: AFqh2koE4ZzsKNHxZa39VzP78VgoOM/997TMkH0Q9gTX7F1QPakKh/SJ 9tN/rzzCjuhqW8jc7W3NjRl0Lw+FkeI= X-Google-Smtp-Source: AMrXdXvdJ+LSHxNvDp9FwxmPumyMZ4UMm8oIaSAr43nXY8mLQfmK4bkMDdV6S/XL3AcQfVyEbFu0hw== X-Received: by 2002:a05:600c:4da2:b0:3d2:39dc:f50e with SMTP id v34-20020a05600c4da200b003d239dcf50emr29852055wmp.7.1672715125751; Mon, 02 Jan 2023 19:05:25 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:25 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 08/13] io_uring: set TASK_RUNNING right after schedule Date: Tue, 3 Jan 2023 03:03:59 +0000 Message-Id: <590f0b0e2fa5529e975f3f3ecbb588ff7e67e9c5.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of constantly watching that the state of the task is running before executing tw or taking locks in io_cqring_wait(), switch it back to TASK_RUNNING immediately. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a8d3826f3d17..682f4b086f09 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2541,6 +2541,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ret = io_cqring_wait_schedule(ctx, &iowq, timeout); if (ret < 0) break; + __set_current_state(TASK_RUNNING); /* * Run task_work after scheduling and before io_should_wake(). * If we got woken because of task_work being processed, run it @@ -2553,10 +2554,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) { - finish_wait(&ctx->cq_wait, &iowq.wq); + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) io_cqring_do_overflow_flush(ctx); - } if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) { ret = -EBADR; break; From patchwork Tue Jan 3 03:04:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F5F2C3DA7A for ; Tue, 3 Jan 2023 03:05:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236508AbjACDFc (ORCPT ); Mon, 2 Jan 2023 22:05:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236604AbjACDF3 (ORCPT ); Mon, 2 Jan 2023 22:05:29 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23E142733 for ; Mon, 2 Jan 2023 19:05:28 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id bg13-20020a05600c3c8d00b003d9712b29d2so19644539wmb.2 for ; Mon, 02 Jan 2023 19:05:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8kWWRsaw0qWaWK3TMMd0psKCqbvCHyf3bRkImyBw87Y=; b=PeqjXJRPOh/XzUOUqP2a/+mgxbfrGWBGGflwslugDL/NFr0Z+1acrP9RJBsjLW6qe3 QEUBbqFRG2IhCfyQWucWYfFleaRsymKiujYHFKy39TTkLXTnppEdLUrOlZIvD94hQi8t 867CoLg1OcQswtXihHuc8mwWdujXmYwI12IiRvcJWwNLhlGy5dgBzpuYw4NwmbKLdtR2 Mb8r5TQUhzn+p4z0fBVDKfJt+wF6/jS0NoGZIi7aS9N/s2287kHMgrAYvyfB+rKx1G9b pqlFJbgU4pUcyrDg1WcVE6qZ90ZhhZXzZ796gNcX1qHLCfnbC74T0qyNkdF4+RRkwr23 g0xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8kWWRsaw0qWaWK3TMMd0psKCqbvCHyf3bRkImyBw87Y=; b=w7wLTWjdqdzbFSvmz0nCy47OxJgJD1qzpEcQESdrn6JQ/YK1RiYc4J5gWI053ICMY2 IbgSobiYuyRyYAb9SS6Wb+InOubFIWUVbe+ZvUdcHlpLtwGSxP7KiibhHEjTu/hJnYED GjcnW9d9Y631DcIxAzt4YT3LM8sdoaLq1BamDy3Lh+LqDtl4t8i6IAugRcZh4H62ItZ5 CAgdp0a+JzDPPgNGNK3scmKIvY8eD44+7LL/lQK68dXmID7tGyykUJdlsRCtMoYXlxkk 0pM5UXCWgcaxRetlKmH47QbErFt0mEbkH82OdhWPVIrKo0bnrAtXgK5jqFkf/FxUIAjk mNGw== X-Gm-Message-State: AFqh2kpdXiv3VYkBINrPwuwI8GB8D8vtd3vtWWFBD8vrvuopt5DOEBJn cQbNLsS7pkxQa+FsMYZspZsaYJpece4= X-Google-Smtp-Source: AMrXdXuN/cj/sPMgeuIHfHjZ/wUFqT3WyHMfVbCOdxY/MoPqCvNDqMCX0iOG7cYsAzvATwy8+z9l4w== X-Received: by 2002:a05:600c:254:b0:3d2:2c86:d2b2 with SMTP id 20-20020a05600c025400b003d22c86d2b2mr37417590wmj.24.1672715126476; Mon, 02 Jan 2023 19:05:26 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:26 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 09/13] io_uring: separate wq for ring polling Date: Tue, 3 Jan 2023 03:04:00 +0000 Message-Id: <0fbee0baf170cbfb8488773e61890fc78ed48d1e.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't use ->cq_wait for ring polling but add a separate wait queue for it. We need it for following patches. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 3 ++- io_uring/io_uring.h | 9 +++++++++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index dcd8a563ab52..cbcd3aaddd9d 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -286,6 +286,7 @@ struct io_ring_ctx { unsigned cq_entries; struct io_ev_fd __rcu *io_ev_fd; struct wait_queue_head cq_wait; + struct wait_queue_head poll_wq; unsigned cq_extra; } ____cacheline_aligned_in_smp; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 682f4b086f09..42f512c42099 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -316,6 +316,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); mutex_init(&ctx->uring_lock); init_waitqueue_head(&ctx->cq_wait); + init_waitqueue_head(&ctx->poll_wq); spin_lock_init(&ctx->completion_lock); spin_lock_init(&ctx->timeout_lock); INIT_WQ_LIST(&ctx->iopoll_list); @@ -2768,7 +2769,7 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait) struct io_ring_ctx *ctx = file->private_data; __poll_t mask = 0; - poll_wait(file, &ctx->cq_wait, wait); + poll_wait(file, &ctx->poll_wq, wait); /* * synchronizes with barrier from wq_has_sleeper call in * io_commit_cqring diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9b7baeff5a1c..645ace377d7e 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -207,9 +207,18 @@ static inline void io_commit_cqring(struct io_ring_ctx *ctx) smp_store_release(&ctx->rings->cq.tail, ctx->cached_cq_tail); } +static inline void io_poll_wq_wake(struct io_ring_ctx *ctx) +{ + if (waitqueue_active(&ctx->poll_wq)) + __wake_up(&ctx->poll_wq, TASK_NORMAL, 0, + poll_to_key(EPOLL_URING_WAKE | EPOLLIN)); +} + /* requires smb_mb() prior, see wq_has_sleeper() */ static inline void __io_cqring_wake(struct io_ring_ctx *ctx) { + io_poll_wq_wake(ctx); + /* * Trigger waitqueue handler on all waiters on our waitqueue. This * won't necessarily wake up all the tasks, io_should_wake() will make From patchwork Tue Jan 3 03:04:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD2D5C5479D for ; Tue, 3 Jan 2023 03:05:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236576AbjACDFe (ORCPT ); Mon, 2 Jan 2023 22:05:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236614AbjACDF3 (ORCPT ); Mon, 2 Jan 2023 22:05:29 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE314B84B for ; Mon, 2 Jan 2023 19:05:28 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id m26-20020a05600c3b1a00b003d9811fcaafso15637412wms.5 for ; Mon, 02 Jan 2023 19:05:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3isi2VSN68XETOdIovNlf5UDSVV24D7okjNbrEgjk1w=; b=DQj5EpaoYldJ2OIcu3ZY8D4w4fF3wVbSwm4ovnFW/OfnYsyESHvHY+J13I0n2cINzv ta8cn7ezpET6FGezls9RTN8hGonNi4ccmsAhTnG6O1S/42Jll/b3tY7zmeB4Br25MfGA qWwul0dllukBHqZXc1Ju1NXWGiPq6+bBGxUTUyEb6bGEYfTo9iX+rnt+oFadCugr6DQh kLN+rnwDku4y+ZBjXxC76h5V+ntM5F5JSJYy9LtZFPrFNc8A+tMzkj4626gLf2EQfWeH hxUE4oblYqUxWZZsKo0Sueoi1kxcCLjLP/sUbKQAQp01m7JhOZ0pRuFxGnNc0JBdI4D1 P6FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3isi2VSN68XETOdIovNlf5UDSVV24D7okjNbrEgjk1w=; b=I4k3YuFnAyVf2zkKVThF82vD7t3JRXYaPeaC3e0JdRlyVJuhK0CAbh8wC8LWzwNyOi eSYblTcDG91LouAkF8obdJhFG9IQxyHYDK31qPkXyfV9CbyYxgwSbBkR93UX5lHqCAjA nDFdv0OBl6mpKerh0kDm99G4zrVU7auWzZN3cOukVbsfOagsYd6+xIbFUfCHWN6Fn13o P1m6IpG33AtvubgJIpakyf48KU+JlhfjBQcu02WTFT1RNSpoxF5omAcTOxoPuJNlCcTT 7BbjufETX6IFK7cZ4ndDv7QpSqgNb+GCcMstyQuKGDGy/eHzGIA8NdwDwwFllqp9kiEF c3+w== X-Gm-Message-State: AFqh2kq72wtdQLeN4tMwpOe934fdGcvfM0MUicrF8vdKqLb78chwCBne U20Cj6SqqB7FGG+wkMJpb9HgEoXS0DA= X-Google-Smtp-Source: AMrXdXveNHlRPS2HEBsmQKG8c9uU8+zjvR77ByryE1mQpmVPjwcdqRLziLBlFgifKPiC0r+HDb2GsQ== X-Received: by 2002:a05:600c:1f0e:b0:3cf:497c:c59e with SMTP id bd14-20020a05600c1f0e00b003cf497cc59emr30541872wmb.6.1672715127225; Mon, 02 Jan 2023 19:05:27 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:27 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 10/13] io_uring: add lazy poll_wq activation Date: Tue, 3 Jan 2023 03:04:01 +0000 Message-Id: <81e49bfc364b3b385fa405adf4065a41dcaf9141.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Even though io_poll_wq_wake()'s waitqueue_active reuses a barrier we do for another waitqueue, it's not going to be the case in the future and so we want to have a fast path for it when the ring has never been polled. Move poll_wq wake ups into __io_commit_cqring_flush() using a new flag called ->poll_activated. The idea behind the flag is to set it when the ring was polled for the first time. This requires additional sync to not miss events, which is done here by using task_work for ->task_complete rings, and by default enabling the flag for all other types of rings. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 ++ io_uring/io_uring.c | 40 ++++++++++++++++++++++++++++++++++ io_uring/io_uring.h | 7 +++--- 3 files changed, 45 insertions(+), 4 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index cbcd3aaddd9d..1452ff745e5c 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -210,6 +210,7 @@ struct io_ring_ctx { unsigned int syscall_iopoll: 1; /* all CQEs should be posted only by the submitter task */ unsigned int task_complete: 1; + unsigned int poll_activated: 1; } ____cacheline_aligned_in_smp; /* submission data */ @@ -357,6 +358,7 @@ struct io_ring_ctx { u32 iowq_limits[2]; bool iowq_limits_set; + struct callback_head poll_wq_task_work; struct list_head defer_list; unsigned sq_thread_idle; /* protected by ->completion_lock */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 42f512c42099..d2a3d9928ba3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -573,6 +573,8 @@ static void io_eventfd_flush_signal(struct io_ring_ctx *ctx) void __io_commit_cqring_flush(struct io_ring_ctx *ctx) { + if (ctx->poll_activated) + io_poll_wq_wake(ctx); if (ctx->off_timeout_used) io_flush_timeouts(ctx); if (ctx->drain_active) { @@ -2764,11 +2766,42 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) kfree(ctx); } +static __cold void io_lazy_activate_poll(struct callback_head *cb) +{ + struct io_ring_ctx *ctx = container_of(cb, struct io_ring_ctx, + poll_wq_task_work); + + mutex_lock(&ctx->uring_lock); + ctx->poll_activated = true; + mutex_unlock(&ctx->uring_lock); + + /* + * Wake ups for some events between start of polling and activation + * might've been lost due to loose synchronisation. + */ + io_poll_wq_wake(ctx); + percpu_ref_put(&ctx->refs); +} + static __poll_t io_uring_poll(struct file *file, poll_table *wait) { struct io_ring_ctx *ctx = file->private_data; __poll_t mask = 0; + if (unlikely(!ctx->poll_activated)) { + spin_lock(&ctx->completion_lock); + if (!ctx->poll_activated && !ctx->poll_wq_task_work.func && + ctx->submitter_task) { + init_task_work(&ctx->poll_wq_task_work, io_lazy_activate_poll); + percpu_ref_get(&ctx->refs); + + if (task_work_add(ctx->submitter_task, + &ctx->poll_wq_task_work, TWA_SIGNAL)) + percpu_ref_put(&ctx->refs); + } + spin_unlock(&ctx->completion_lock); + } + poll_wait(file, &ctx->poll_wq, wait); /* * synchronizes with barrier from wq_has_sleeper call in @@ -3575,6 +3608,13 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, !(ctx->flags & IORING_SETUP_SQPOLL)) ctx->task_complete = true; + /* + * Lazy poll_wq activation requires sync with all potential completors, + * ->task_complete guarantees a single completor + */ + if (!ctx->task_complete) + ctx->poll_activated = true; + /* * When SETUP_IOPOLL and SETUP_SQPOLL are both enabled, user * space applications don't need to do io completion events diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 645ace377d7e..e9819872c186 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -209,7 +209,7 @@ static inline void io_commit_cqring(struct io_ring_ctx *ctx) static inline void io_poll_wq_wake(struct io_ring_ctx *ctx) { - if (waitqueue_active(&ctx->poll_wq)) + if (wq_has_sleeper(&ctx->poll_wq)) __wake_up(&ctx->poll_wq, TASK_NORMAL, 0, poll_to_key(EPOLL_URING_WAKE | EPOLLIN)); } @@ -217,8 +217,6 @@ static inline void io_poll_wq_wake(struct io_ring_ctx *ctx) /* requires smb_mb() prior, see wq_has_sleeper() */ static inline void __io_cqring_wake(struct io_ring_ctx *ctx) { - io_poll_wq_wake(ctx); - /* * Trigger waitqueue handler on all waiters on our waitqueue. This * won't necessarily wake up all the tasks, io_should_wake() will make @@ -319,7 +317,8 @@ static inline void io_req_complete_defer(struct io_kiocb *req) static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx) { - if (unlikely(ctx->off_timeout_used || ctx->drain_active || ctx->has_evfd)) + if (unlikely(ctx->off_timeout_used || ctx->drain_active || + ctx->has_evfd || ctx->poll_activated)) __io_commit_cqring_flush(ctx); } From patchwork Tue Jan 3 03:04:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C236CC54EBD for ; Tue, 3 Jan 2023 03:05:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236631AbjACDFf (ORCPT ); Mon, 2 Jan 2023 22:05:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236625AbjACDFa (ORCPT ); Mon, 2 Jan 2023 22:05:30 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88F56BF5F for ; Mon, 2 Jan 2023 19:05:29 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id k22-20020a05600c1c9600b003d1ee3a6289so22169831wms.2 for ; Mon, 02 Jan 2023 19:05:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MoTHAaXqKBt52ZvHVgZSMQyuUe+Z8C2Fhn5NtXMvC9E=; b=mm5r2qFhbJcOlDXjGU+IglpBpoWiPA1/gsJKN4EB9iVSwi5y89uhdlDvWVNZZVGMkx WJpRjLphpUvmF6IuHeOpLDEDRalW+9NA7331XlkhrWPGMj75OanjHjSznNpdVPUf6d1G +z9YZRBnydcpJ8q3gsiDMopB5GphQ89zuswTW9dFX2tkQrJTpAKsIXOnxmOOjUXRmA88 FqlQdeaVav9xJyX4M7zEr1sk56GRFNtTefMBbaS26Md0EG9WulAETfpvAd6OS+15lIB7 wusQiIudNcpdPL6Jzqt1hGYd/u2YE80yIE6bblcn1cV+ksKJaC40ST0K8XcWRfV0tsEW aarA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MoTHAaXqKBt52ZvHVgZSMQyuUe+Z8C2Fhn5NtXMvC9E=; b=sDOV0VOB1nVfyYcph1YPvMzQdhQZM3gs7Q6U20mXOIlN0sWe9g61MvFw5LoankIewK j7QfFqN/9kICV96HnafJMvn/eCXTa+RFvrgelBtk7rk5RCZxbnsd71eQB/4HUvxfdfsW KJ2uuyXtI52XBB0nRTmcpPeWe0OqL5JWJDIgzW8m+oifzn9DwOEp/aJ/7HbIeee1z2sr hp16c7xYZWNBBoZLMHEa7FE24zG67Sr303NQP/U59kBFuTm1lVIj0Vq8tV+2QUzcLYag YhaqnKr9SalymJ2sxhpJVRqPbMRW1hOTHg2/ZHmYRMb7C0X6Hy+J5byaJrtbhLHHDX6W qFcA== X-Gm-Message-State: AFqh2ko8dNxOZ06D7nvBevpb3k+g2WVFWLuewa+prbziVNRmtuRXWjmA GTj662YFf/iSoIET3GNv9woz4QPpzjc= X-Google-Smtp-Source: AMrXdXv8/sKjyIRjsipPHhnNj8YicQz5CvOVA1qBeYsUPAKl+saptt3rhESiHbYNmgc1F4e6NKvRoA== X-Received: by 2002:a05:600c:3b82:b0:3d3:4877:e560 with SMTP id n2-20020a05600c3b8200b003d34877e560mr29520586wms.27.1672715127914; Mon, 02 Jan 2023 19:05:27 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:27 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 11/13] io_uring: wake up optimisations Date: Tue, 3 Jan 2023 03:04:02 +0000 Message-Id: <1b09f9674e9140ef9623c5a26ab7b826d4d4fe69.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Flush completions is done either from the submit syscall or by the task_work, both are in the context of the submitter task, and when it goes for a single threaded rings like implied by ->task_complete, there won't be any waiters on ->cq_wait but the master task. That means that there can be no tasks sleeping on cq_wait while we run __io_submit_flush_completions() and so waking up can be skipped. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d2a3d9928ba3..98d0d9e49be0 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -621,6 +621,25 @@ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) io_cqring_wake(ctx); } +static inline void __io_cq_unlock_post_flush(struct io_ring_ctx *ctx) + __releases(ctx->completion_lock) +{ + io_commit_cqring(ctx); + __io_cq_unlock(ctx); + io_commit_cqring_flush(ctx); + + /* + * As ->task_complete implies that the ring is single tasked, cq_wait + * may only be waited on by the current in io_cqring_wait(), but since + * it will re-check the wakeup conditions once we return we can safely + * skip waking it up. + */ + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) { + smp_mb(); + __io_cqring_wake(ctx); + } +} + void io_cq_unlock_post(struct io_ring_ctx *ctx) __releases(ctx->completion_lock) { @@ -1461,7 +1480,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) } } } - __io_cq_unlock_post(ctx); + __io_cq_unlock_post_flush(ctx); if (!wq_list_empty(&ctx->submit_state.compl_reqs)) { io_free_batch_list(ctx, state->compl_reqs.first); From patchwork Tue Jan 3 03:04:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47F7DC53210 for ; Tue, 3 Jan 2023 03:05:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236604AbjACDFe (ORCPT ); Mon, 2 Jan 2023 22:05:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236638AbjACDFb (ORCPT ); Mon, 2 Jan 2023 22:05:31 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60B2EB7F8 for ; Mon, 2 Jan 2023 19:05:30 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id p1-20020a05600c1d8100b003d8c9b191e0so22155887wms.4 for ; Mon, 02 Jan 2023 19:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Qv2lXExtqdHyonmaWjP9naQwvPOWtKueXxlt47AT7+Q=; b=PnsIDb18B/vxcSI4wedC/aaATrKp56nZJMOKZlDzYLQy0dK/RQTsbGcgzzNIssn7vp 3UjFVy5ONbgRVTPehSymUh9TymG2oWFx/A/8dBFu2XjutgC86DKDt2xqFVvoekxjxyG0 pry/IHPmrcZDbZqMSwXfoas6bkJUtdsGb0LM/cWu/4E/LCErg2Xp9Ge8wGJ0fwgiHrFh eY3LK8M1UXmFJiWzxTdhossiiVSZN86PictO2NROia3/CiXxQMyxPeVZKVGo5oVkRQ8p tIbYdauFQKjjZuQ0obPH4jLJ+4tAsXZkI8+RJRwEM+kydHVeg7rqaAo6OuRGWdJVueBN Qv2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qv2lXExtqdHyonmaWjP9naQwvPOWtKueXxlt47AT7+Q=; b=8PsEbYEj0ogmxqXUViW0r+YNoCDoQQCOlVQAOl0Npf7l5jWU/n2h8hXhdDCo6hUTmd zwNz/gQ/s7m8BMSjrHgMvoDOW+uNWYi70+3FoH2c4MFOj08jiiKELkBOfIC38pbA8Zho 9oK0lYBl1CT8j/YOLEq56Y5tvb2103iiFAVKsD+T8mQeiPuJw36xFOr0K0k4forQ+NHb uB3y6JdYTOhGvA8jPxsTpWVpvm8rMbNm4y2b7HIEZmNeqT6Df+ZrQLaDbrtdF/KaBeOY UxO2CtP0MZhMQXeSOA0MTd2MJQvOlBDgfU35wdaxIu+zwo8I0lhlN8AE/mKhS+rS1EtJ bSNg== X-Gm-Message-State: AFqh2kpfygWO5OcBHXXu1YqRdWgpauog74LT4hLD3zhB9Y0ImuosfavA 0orgO5Lc3zVPIhaLrsNZcNsHxix9T08= X-Google-Smtp-Source: AMrXdXsYu6VxU2QovbAAqeVYmUIapDtiNJ3vlnGNW7QLrG1ysRcgXWOYLWZRHbKWtYHPoga0lkHNoQ== X-Received: by 2002:a05:600c:500a:b0:3d3:5b56:b834 with SMTP id n10-20020a05600c500a00b003d35b56b834mr30081030wmr.5.1672715128693; Mon, 02 Jan 2023 19:05:28 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:28 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 12/13] io_uring: waitqueue-less cq waiting Date: Tue, 3 Jan 2023 03:04:03 +0000 Message-Id: <304db555100334593675eba81fd07f95d7ec02f5.1672713341.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org With DEFER_TASKRUN only ctx->submitter_task might be waiting for CQEs, we can use this to optimise io_cqring_wait(). Replace ->cq_wait waitqueue with waking the task directly. It works but misses an important optimisation covered by the following patch, so this patch without follow ups might hurt performance. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 98d0d9e49be0..943032d2fd21 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1273,7 +1273,12 @@ static void io_req_local_work_add(struct io_kiocb *req) if (ctx->has_evfd) io_eventfd_signal(ctx); - __io_cqring_wake(ctx); + + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); + } else { + __io_cqring_wake(ctx); + } } void __io_req_task_work_add(struct io_kiocb *req, bool allow_local) @@ -2558,12 +2563,17 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, do { unsigned long check_cq; - prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, - TASK_INTERRUPTIBLE); + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + set_current_state(TASK_INTERRUPTIBLE); + } else { + prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, + TASK_INTERRUPTIBLE); + } + ret = io_cqring_wait_schedule(ctx, &iowq, timeout); + __set_current_state(TASK_RUNNING); if (ret < 0) break; - __set_current_state(TASK_RUNNING); /* * Run task_work after scheduling and before io_should_wake(). * If we got woken because of task_work being processed, run it @@ -2591,7 +2601,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, cond_resched(); } while (1); - finish_wait(&ctx->cq_wait, &iowq.wq); + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) + finish_wait(&ctx->cq_wait, &iowq.wq); restore_saved_sigmask_unless(ret == -EINTR); return READ_ONCE(rings->cq.head) == READ_ONCE(rings->cq.tail) ? ret : 0; From patchwork Tue Jan 3 03:04:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13087127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B20FC3DA7A for ; Tue, 3 Jan 2023 03:05:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236606AbjACDFg (ORCPT ); Mon, 2 Jan 2023 22:05:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236626AbjACDFb (ORCPT ); Mon, 2 Jan 2023 22:05:31 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E247BF7A for ; Mon, 2 Jan 2023 19:05:30 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id k22-20020a05600c1c9600b003d1ee3a6289so22169865wms.2 for ; Mon, 02 Jan 2023 19:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o2BAv+ryfLgNKGIDpPfwqLcUzObKBG5bNKUNy7ao14I=; b=Nv/W5O9e7W7oRhEdBbdwqEv8xeJIw3okvbCkU3OaDsLlX1cg4cMnTpIjIQAtEW2f5j OWAODFuZouSQyekHhYczc04o18rze/KG2lWj7PRK4SwzZDPrrDHzN6DFxZwTVc4MV/g+ xeB9bL5GrJSM+1W5AsIZ0PwBPYIlPK8EI6MaC79Xq+OAQCKIq3QfAUaGfNSmyXNmkiU5 ZMVe4ElFy4Kf5cbvkujmRyJadYUEQdhMvTiMJ0z3fyEv6xTn9xBnoGcQFSOXkq1+tTBC rdMz5xDtDsC+jyk8yBc0Y3hYIIhOTIj7/bvRuu/V8pzEKsauWCSgfbvAix6+3cCuRWgo M/9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o2BAv+ryfLgNKGIDpPfwqLcUzObKBG5bNKUNy7ao14I=; b=t0kRKT9QCzgqS2N8pU/X93MNLzPBhQZyEw66ztizCjpTPC1+2hIQhKsPtMbfio4n8x bXnhIp/QasTi7eWPXmP1HrQmHyFXrP3HMLspL669pKmdy8J/rYf4cMqzWEhQwbgd6a3V 5lXvAB+fBIMHlKgxD6IiFH9GBZo0FF77KF0O/Wou6e/5GkjHutpUE84W+ql80RtdvUaV 8S1WIsR27SfpSAfbgc5bxVH7zrhf97engw5oUf6W1Ra7HbTM0tf9mQSUPshcC121E5v2 PWJYPT9Gnm12U2C9p+d4b6gXF0EVxMJHqd2OkDnp1rbjxxMGglbY7bHznM/8NLscW7+M CimQ== X-Gm-Message-State: AFqh2kqZeRzoEJs0b6RsmR0IpRDpnBpyIewYfLidmwv/8MkY15cXe86l 5//fdI+n+wCc3OS7o9fC1yDkpp0bqp4= X-Google-Smtp-Source: AMrXdXuB2yV9Z6wVqpkBqJnmFMBi165wJ6YYLXjvPr4TP059qEyszWrt5ELzBThwLJf243a1kW1I7Q== X-Received: by 2002:a7b:c4da:0:b0:3d3:864a:1173 with SMTP id g26-20020a7bc4da000000b003d3864a1173mr29167489wmk.18.1672715129396; Mon, 02 Jan 2023 19:05:29 -0800 (PST) Received: from 127.0.0.1localhost (188.28.229.101.threembb.co.uk. [188.28.229.101]) by smtp.gmail.com with ESMTPSA id m1-20020a7bca41000000b003d1de805de5sm39967839wml.16.2023.01.02.19.05.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Jan 2023 19:05:29 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [RFC v2 13/13] io_uring: add io_req_local_work_add wake fast path Date: Tue, 3 Jan 2023 03:04:04 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't wake the master task after queueing a deferred tw unless it's currently waiting in io_cqring_wait. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 1452ff745e5c..332a29cfe076 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -285,6 +285,7 @@ struct io_ring_ctx { unsigned cached_cq_tail; unsigned cq_entries; + bool cq_waiting; struct io_ev_fd __rcu *io_ev_fd; struct wait_queue_head cq_wait; struct wait_queue_head poll_wq; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 943032d2fd21..e436fe73becf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1275,7 +1275,8 @@ static void io_req_local_work_add(struct io_kiocb *req) io_eventfd_signal(ctx); if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { - wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); + if (READ_ONCE(ctx->cq_waiting)) + wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); } else { __io_cqring_wake(ctx); } @@ -2565,6 +2566,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { set_current_state(TASK_INTERRUPTIBLE); + smp_store_mb(ctx->cq_waiting, 1); } else { prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); @@ -2572,6 +2574,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ret = io_cqring_wait_schedule(ctx, &iowq, timeout); __set_current_state(TASK_RUNNING); + WRITE_ONCE(ctx->cq_waiting, 0); + if (ret < 0) break; /*