From patchwork Thu Jan 5 11:22:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D84FEC3DA7D for ; Thu, 5 Jan 2023 11:23:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232779AbjAELXp (ORCPT ); Thu, 5 Jan 2023 06:23:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233161AbjAELXd (ORCPT ); Thu, 5 Jan 2023 06:23:33 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D705F559C5 for ; Thu, 5 Jan 2023 03:23:30 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id g10so13981296wmo.1 for ; Thu, 05 Jan 2023 03:23:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Xtm4V31HUmHjoi9kdkK6HkEdkWruS4iTPrxd+tvk8sA=; b=MfVjbD7fZpTrDIYFeERq0mEPEDiyKPYJC1ctKuXtXQRQ+qvBOfb24fhOb9X+1GaWIy sHPlOq+VcRZ5krz8a6nrZzK9+7B3pAalZhZGd2PFxvWrtzYCvFi6CC1+vFD8rF3Bw42T VIYfYSHhPyiOV0fhDQEFx4zPYgmAZs8Z235Cjy94Mr5smrTU8lXUThsOTSJAxBSyp2sA dUarXfRwaURCwyib2GMCilTQe8MS/TOQ6G60PWuuunGQPzxThrGaK20djIzWirpziakY xmiXcJzquKOdgWOzatd5RKA0JS68IxcRiD9BrG9ZigC6St7a4Z+7K3RTPanD+XjYaHsc bZWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xtm4V31HUmHjoi9kdkK6HkEdkWruS4iTPrxd+tvk8sA=; b=T0Q1HaFBpHGBZY4zmk+qB//+aUMD3GT3EHd+jmzUONkq7scC6RI5qqPn3fyxs5eLGQ kOLxqBTpn5p0WgrcZd/gejHD+oTz7wtDJx6XMMDhg/1f/LlvB2B7eC+mGEjtNIhfiWex /5u3NiUdRPVT7R56rSqXTrCZYCAqgpyHsPA78qABzju9t2emo1N832fhVumvrF826rOe 4PHcQHVWFYm+zwvjFMZVvtLZb0drJ6lacFfWJUkt7Ggb35XVjAyZzELRTydhlm5YVn1i Gl4f1BqoUkUJS6EH774cXzd+6MFViiU1y4fAX94KwIulrLFdRBbcmYb0XOb3ZiAY9xbM +PyQ== X-Gm-Message-State: AFqh2koVe36ekjMGMRIo0erR6L8ODyvR/dYzVrvMTJnqc9tBMSWWTkpb CElg5p6tDiPTZPyM73LiBkxDH3W3TMw= X-Google-Smtp-Source: AMrXdXv+tiOWK0cJbN1d1NEaEQSJAG1vy0Fmo3NKecTpKYrYipekoqPLnKVVKDhLCzTBvTP4OaHFcQ== X-Received: by 2002:a05:600c:c8a:b0:3d9:7062:e0b7 with SMTP id fj10-20020a05600c0c8a00b003d97062e0b7mr30233810wmb.33.1672917809058; Thu, 05 Jan 2023 03:23:29 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:28 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 01/10] io_uring: rearrange defer list checks Date: Thu, 5 Jan 2023 11:22:20 +0000 Message-Id: <331d63fd15ca79b35b95c82a82d9246110686392.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There should be nothing in the ->work_llist for non DEFER_TASKRUN rings, so we can skip flag checks and test the list emptiness directly. Also move it out of io_run_local_work() for inlining. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 3 --- io_uring/io_uring.h | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index fefffb02e6b0..cddfcfddcb52 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1338,9 +1338,6 @@ int io_run_local_work(struct io_ring_ctx *ctx) bool locked; int ret; - if (llist_empty(&ctx->work_llist)) - return 0; - __set_current_state(TASK_RUNNING); locked = mutex_trylock(&ctx->uring_lock); ret = __io_run_local_work(ctx, &locked); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index e9f0d41ebb99..46c0f765a77a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -274,7 +274,7 @@ static inline int io_run_task_work_ctx(struct io_ring_ctx *ctx) int ret = 0; int ret2; - if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) + if (!llist_empty(&ctx->work_llist)) ret = io_run_local_work(ctx); /* want to run this after in case more is added */ From patchwork Thu Jan 5 11:22:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A17A0C3DA7A for ; Thu, 5 Jan 2023 11:23:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232650AbjAELXl (ORCPT ); Thu, 5 Jan 2023 06:23:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233167AbjAELXd (ORCPT ); Thu, 5 Jan 2023 06:23:33 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EA75559CA for ; Thu, 5 Jan 2023 03:23:31 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id fm16-20020a05600c0c1000b003d96fb976efso1056104wmb.3 for ; Thu, 05 Jan 2023 03:23:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GscsiLJ8rccsuSYYxaQoDj9SBHinsWa28fb7CygtUTE=; b=cmavc8o0jCH0KYmL/38TcF9X89ROmwzZybx4dDkLzXT6KqjJ4XEjlp/4JjepWfxyKM FHRmG4z32g9Kyoj6qtPgjNFdq4PQwc0weqU6VqQXt4KPlW24Dbqy3Z+1uamrMs8iSdX5 gY6Xj5l1FLnMOixSYDrM5aKxRYLn9JTYNoDTxvtYDtC6Cygp12c+js1N+F8BWFtn3+IJ o0j+2/JqC8NrkqggC+d/IjErsfMBrEvhUvjtJlHiR1PGks4HQHgiu2CSpg6ARU/WUVrm LLyZ/E5UaH1QHZvQhhl0aeOWA7RLvPeHtoRdXv4S1XqmoDIEo63JGPMkRSOm+N9wAdJb AabQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GscsiLJ8rccsuSYYxaQoDj9SBHinsWa28fb7CygtUTE=; b=nClFjtlvCJ3MfLSlQU/npxeJDZ8UkBKZRgoupDDvCnXNvKbQ6/4IEDxXXBJ7j55jB8 qDs2lXCyEkjyCYo1WMjWbKfL1bWDUy88dqIycwgjor0X9nE3YIJg93163ypb4Kx/F7uT JKcC0nT/JnJRF3pHfD+bnUXY3O0wUvMDQ9njzwjBquxznR5v/3zoCD+hVglTpNV9m3Pa lcExnfTwmy/CwgVV76YJiP+enVzGIY7SgF65f4Jumtcf32eyYDjrMrsDuj/wPr7XG8Pr 9kDvQZ89nvinePAQXTVcsUrMucVay3ffrzcuAOP3G4ZLa5lBMV215opZ01m3lDkYFkoL hnIQ== X-Gm-Message-State: AFqh2krosIIj8T6XsDjoFkGO8qo8Lhr0mRRcjVqZLciVxX3iZ88zZcs7 ZPe0T9AiYPdHYfVhZDP5+5cRHcukgIM= X-Google-Smtp-Source: AMrXdXvtzts/6sKGk/2DCMr0NtjH182/PA8nGCcUY9RxEI2qDOIzD4nxczVVe+c/BLLrODFA5WbP/Q== X-Received: by 2002:a05:600c:1c21:b0:3cf:9844:7b11 with SMTP id j33-20020a05600c1c2100b003cf98447b11mr43782422wms.23.1672917809651; Thu, 05 Jan 2023 03:23:29 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:29 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 02/10] io_uring: don't iterate cq wait fast path Date: Thu, 5 Jan 2023 11:22:21 +0000 Message-Id: <1f9565726661266abaa5d921e97433c831759ecf.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Task work runners keep running until all queues tw items are exhausted. It's also rare for defer tw to queue normal tw and vise versa. Taking it into account, there is only a dim chance that further iterating the io_cqring_wait() fast path will get us anything and so we can remove the loop there. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cddfcfddcb52..ec800a8bed28 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2507,18 +2507,16 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (!io_allowed_run_tw(ctx)) return -EEXIST; - - do { - /* always run at least 1 task work to process local work */ - ret = io_run_task_work_ctx(ctx); + if (!llist_empty(&ctx->work_llist)) { + ret = io_run_local_work(ctx); if (ret < 0) return ret; - io_cqring_overflow_flush(ctx); - - /* if user messes with these they will just get an early return */ - if (__io_cqring_events_user(ctx) >= min_events) - return 0; - } while (ret > 0); + } + io_run_task_work(); + io_cqring_overflow_flush(ctx); + /* if user messes with these they will just get an early return */ + if (__io_cqring_events_user(ctx) >= min_events) + return 0; if (sig) { #ifdef CONFIG_COMPAT From patchwork Thu Jan 5 11:22:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A116BC53210 for ; Thu, 5 Jan 2023 11:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231244AbjAELXk (ORCPT ); Thu, 5 Jan 2023 06:23:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233166AbjAELXd (ORCPT ); Thu, 5 Jan 2023 06:23:33 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AED384E424 for ; Thu, 5 Jan 2023 03:23:31 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id z8-20020a05600c220800b003d33b0bda11so2276250wml.0 for ; Thu, 05 Jan 2023 03:23:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J32+a9orWjFm5kxfSVe7cvbZ2vnqIbSs9t/hFzOVmzg=; b=N4dpfa+ZpXQjakMpg5q3eicTBzPjtPsr22v3eNEqiheUaKkbgQPa5o6/PGd2X4kWar teLbS4f2KEUtrzWAz3MpEcc7d6Z2ERIo1I2r/lD/z0jgESPcDSytJ0lckSJyiWdpZToW OySJ44DE9RL82Vd/IYIaBZv+y+HurSBrqZLYidcMilpalpNtm6oyd7c8Uetg/72/ZWWZ PggqBHinvQLFEyY32kY0knPEhn8p0BB6Iug0vu9dYdKActFwJDNHEmvrVItbjaldofet +ICX2Mcr1t/gutOnPLcVY/2ZnpdNtXhqLwBo8CBCaWLbENtNfJgCnGb0G8yBjvw0V/Bl 2mDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J32+a9orWjFm5kxfSVe7cvbZ2vnqIbSs9t/hFzOVmzg=; b=K/nGKYs2H3h0IzcadyxJ2F55mR5e5u4DSa7TiIRSIM9L+QMQmHx1WMS7sV6t/Uk5fb 39N329ZjrIdfqohQJziaDVMU0ImOV4o/p6Tw3kUyv6IWucarMkKIEF/CJykblT1sQ0lu MWfA4AmJdlxXldLVdmKMY5DTLz8dp3QI0oZVm9OcMAWPK+v7eLSU7i2Xf8QHOcw3zZ0o taDSPrOQ3B4trUt7QwKsgYsdqcQGiVb0ngCis+qhs0fpXP63WSXAr2JXx2WZqxEBIJf6 3utOXymxlBL0XwZLiXNCuUDSytXEsQKi5xoFy/UnQ5SfKySyDrqJQHWbk2FcGzUo+3i+ ezGw== X-Gm-Message-State: AFqh2kqyH0YZQCyM7wTutK54pyFkdM8btGsD7/THAiQc4DNPWyJOX8rL /VzzLS2ZaaoYqpC3BcU9FujcDrwHFX8= X-Google-Smtp-Source: AMrXdXtxdrRda0VoqvpjUYhqzkYdcaBHr9wBjwfkuQk0c+Vt4NJ9HtA79XOdsWo7klgLs60E69AIrg== X-Received: by 2002:a05:600c:358f:b0:3d9:7847:96e2 with SMTP id p15-20020a05600c358f00b003d9784796e2mr26818044wmq.2.1672917810116; Thu, 05 Jan 2023 03:23:30 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:29 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 03/10] io_uring: kill io_run_task_work_ctx Date: Thu, 5 Jan 2023 11:22:22 +0000 Message-Id: <40953c65f7c88fb00cdc4d870ca5d5319fb3d7ea.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is only one user of io_run_task_work_ctx(), inline it. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 6 +++++- io_uring/io_uring.h | 20 -------------------- 2 files changed, 5 insertions(+), 21 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ec800a8bed28..bf6f9777d165 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2452,7 +2452,11 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode, int io_run_task_work_sig(struct io_ring_ctx *ctx) { - if (io_run_task_work_ctx(ctx) > 0) + if (!llist_empty(&ctx->work_llist)) { + if (io_run_local_work(ctx) > 0) + return 1; + } + if (io_run_task_work() > 0) return 1; if (task_sigpending(current)) return -EINTR; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 46c0f765a77a..8a5c3affd724 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -269,26 +269,6 @@ static inline bool io_task_work_pending(struct io_ring_ctx *ctx) return task_work_pending(current) || !wq_list_empty(&ctx->work_llist); } -static inline int io_run_task_work_ctx(struct io_ring_ctx *ctx) -{ - int ret = 0; - int ret2; - - if (!llist_empty(&ctx->work_llist)) - ret = io_run_local_work(ctx); - - /* want to run this after in case more is added */ - ret2 = io_run_task_work(); - - /* Try propagate error in favour of if tasks were run, - * but still make sure to run them if requested - */ - if (ret >= 0) - ret += ret2; - - return ret; -} - static inline int io_run_local_work_locked(struct io_ring_ctx *ctx) { bool locked; From patchwork Thu Jan 5 11:22:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 515CBC54E76 for ; Thu, 5 Jan 2023 11:23:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232560AbjAELXt (ORCPT ); Thu, 5 Jan 2023 06:23:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233195AbjAELXe (ORCPT ); Thu, 5 Jan 2023 06:23:34 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 526DB4E42B for ; Thu, 5 Jan 2023 03:23:32 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id o15so27765809wmr.4 for ; Thu, 05 Jan 2023 03:23:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PMmgDhX/8Q1PK6d4K+oWZL4fFf/Ozl6eNXYKVdbXoII=; b=AFboG8pjKdO3R3Y+V9LC9L8LAGkRoZ0cOXUcSZ4ckPTHJ/CGUPsKztRVmj2HQuq15O 86ZlbGfFwUY2dnBNYoI28QiYaNbEtYfW2FD13x4802t6FKbQz9VVi/vJsXEWxRvR3lhW eoK4lKXkzGIw8MGv5qkVZeG9hYxy85eLs4yogQFmOWukSqeYNXJXxc3pBzh+HzJ+p+B2 DMuX73i2ao43odoUB+2A+3wfV0RgpEC48fXJafS/mrs186ATTApFMEhSIo6UEhejF/wD kXYuxHeUqG46GGvvcEUjRtEidx2l+NzgFK5EEUfmTpK0IficllqdzBzLbwz/CjZhC36q f7oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PMmgDhX/8Q1PK6d4K+oWZL4fFf/Ozl6eNXYKVdbXoII=; b=faBc2eOEFIpPoaV1Nni64RIIOxyp/uY5m64NhxZ7yBnwbZPvXok5yXGFxmxhGiKvVA QSru4C89wNoTVatiTQ2ZH3Ju+GtsFVHdvEVZGIDAEwaLb3mWODrcCaaPHowjfw0s5Fz9 Q46DgAtO+CT1rk7Dg9qoYy8Ljv/mpFY8bRFwao5Ee4N/t4HnBcr1OJjW6VNn93GMUJKK YSZFVsw+xE0r60bnuk1cE+u2+szCavBpoq4FgB023sGjJJjfGZWoI78+eKJ6gDIp3KCC Ie/dfb1Xvk+kpP1JQj/kAGuho4iEX2HF4zCbcpJflgEfpprN8E+Io+8t6oKgHxVnCqjh DabQ== X-Gm-Message-State: AFqh2koVb41X/oHRkrcDta3BtKn4Ii1gHDZozaqGtGWOGxx7c6RU5PgD Nk63kKRa47jwFLPo1P2t7NIKkkHxszc= X-Google-Smtp-Source: AMrXdXs4ChaL5Ey3IfQ1XFthUDnASDSd1eMhO1chjNPsBvkjr+IJIs7LTVOeTZk8ASYf5CrUZu5nlw== X-Received: by 2002:a05:600c:4fcf:b0:3cf:360e:f37d with SMTP id o15-20020a05600c4fcf00b003cf360ef37dmr35769533wmq.22.1672917810787; Thu, 05 Jan 2023 03:23:30 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:30 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 04/10] io_uring: move defer tw task checks Date: Thu, 5 Jan 2023 11:22:23 +0000 Message-Id: <990fe0e8e70fd4d57e43625e5ce8fba584821d1a.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Most places that want to run local tw explicitly and in advance check if they are allowed to do so. Don't rely on a similar check in __io_run_local_work(), leave it as a just-in-case warning and make sure callers checks capabilities themselves. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 15 ++++++--------- io_uring/io_uring.h | 5 +++++ 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bf6f9777d165..3bb3e9889717 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1296,14 +1296,13 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool *locked) struct llist_node *node; struct llist_node fake; struct llist_node *current_final = NULL; - int ret; + int ret = 0; unsigned int loops = 1; - if (unlikely(ctx->submitter_task != current)) + if (WARN_ON_ONCE(ctx->submitter_task != current)) return -EEXIST; node = io_llist_xchg(&ctx->work_llist, &fake); - ret = 0; again: while (node != current_final) { struct llist_node *next = node->next; @@ -2511,11 +2510,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (!io_allowed_run_tw(ctx)) return -EEXIST; - if (!llist_empty(&ctx->work_llist)) { - ret = io_run_local_work(ctx); - if (ret < 0) - return ret; - } + if (!llist_empty(&ctx->work_llist)) + io_run_local_work(ctx); io_run_task_work(); io_cqring_overflow_flush(ctx); /* if user messes with these they will just get an early return */ @@ -3052,7 +3048,8 @@ static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, } } - if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) + if ((ctx->flags & IORING_SETUP_DEFER_TASKRUN) && + io_allowed_defer_tw_run(ctx)) ret |= io_run_local_work(ctx) > 0; ret |= io_cancel_defer_files(ctx, task, cancel_all); mutex_lock(&ctx->uring_lock); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 8a5c3affd724..9b7baeff5a1c 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -352,6 +352,11 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) return container_of(node, struct io_kiocb, comp_list); } +static inline bool io_allowed_defer_tw_run(struct io_ring_ctx *ctx) +{ + return likely(ctx->submitter_task == current); +} + static inline bool io_allowed_run_tw(struct io_ring_ctx *ctx) { return likely(!(ctx->flags & IORING_SETUP_DEFER_TASKRUN) || From patchwork Thu Jan 5 11:22:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77B30C53210 for ; Thu, 5 Jan 2023 11:23:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232821AbjAELXs (ORCPT ); Thu, 5 Jan 2023 06:23:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233191AbjAELXe (ORCPT ); Thu, 5 Jan 2023 06:23:34 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1290B50066 for ; Thu, 5 Jan 2023 03:23:33 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id ja17so27768098wmb.3 for ; Thu, 05 Jan 2023 03:23:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BaLwErdLYUo15ANQXvzf5YPtPlY3IX20HXZIH7F0x/M=; b=GB6lb8hQX0w752savqu2+/DkSC4K0zRuTw4rkSMM1vld6gj6SJ6/AUz9DDBg8f3ERa LgjN31Tpu4T9GFQNKWdTVD5aINn2q+6z6BxAJzoy4DWKa647S/FcSCuJSmtM3YVy3LZA U3hEdaqHyh5Yl5cwgQHTbPJTLtixdpGr1n6XztOsZH8sdcK8mLrHKM0IM0W5y9fN4WhI qhqXObMfuzkbKvEfZmjjA7zPmB1kwfA9ASzvT5UrjmKZPrzSAW2GrTwJbGdidZra4mdO KOKmVpLPfEqd2QPJ1Pq6rKFpm69tUCZVbkj1viOid6lfZVNSQTNSk1v75702yawZOuzp RWyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BaLwErdLYUo15ANQXvzf5YPtPlY3IX20HXZIH7F0x/M=; b=L5WQOthdbAT5zD9VI0DAXXRr1sIB/pRB/Q8nvufTV/HYXAu5DDS84V0/HrqrDJ8k3A z1z3cQ2uISoTaN9SScJuy2n2z9RARqH3lPn6kWrnT7wpWAJL3eI1G/SPOlFCFXKVCiR5 GnT9Oq2REPUnFkmRQ8Tuthmd4vU6zDn9jqil/9d6mVXAyUplobUyphG6zilIsEfTLrsI PePOGuzawq687C8cedmwN2qOzAikP/LP38B/tNziFP7xMMUUAO+c8At0Ti8QI1E5wVff OQV89Ordv9YCmZ8Tl230Edq5jqUxF1zCP5ZMU7zydSNU8zydOvj96OKBL2FzyTfU8TJo IEUA== X-Gm-Message-State: AFqh2kruCopS50+QOyI8/tGRiB2ZYulo/LwhYegqEU0gTDYHyl8wd931 OYYe3u0SKt/kBCfhDNjrdgELnFLcakI= X-Google-Smtp-Source: AMrXdXt1W0FCVYI6Ds8XPem5Y00OT1vi5ZKC2hWyTza8JK4aoF1wAL8MxCcL/23cdA4BUt9nkwcvZg== X-Received: by 2002:a05:600c:2252:b0:3d3:5d8b:7af with SMTP id a18-20020a05600c225200b003d35d8b07afmr39112774wmm.41.1672917811517; Thu, 05 Jan 2023 03:23:31 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:31 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 05/10] io_uring: parse check_cq out of wq waiting Date: Thu, 5 Jan 2023 11:22:24 +0000 Message-Id: <9dfcec3121013f98208dbf79368d636d74e1231a.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We already avoid flushing overflows in io_cqring_wait_schedule() but only return an error for the outer loop to handle it. Minimise it even further by moving all ->check_cq parsing there. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3bb3e9889717..067e3577ac9b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2468,21 +2468,13 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, ktime_t *timeout) { int ret; - unsigned long check_cq; + if (unlikely(READ_ONCE(ctx->check_cq))) + return 1; /* make sure we run task_work before checking for signals */ ret = io_run_task_work_sig(ctx); if (ret || io_should_wake(iowq)) return ret; - - check_cq = READ_ONCE(ctx->check_cq); - if (unlikely(check_cq)) { - /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - return 1; - if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) - return -EBADR; - } if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS)) return -ETIME; @@ -2548,13 +2540,25 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, trace_io_uring_cqring_wait(ctx, min_events); do { - if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) { - finish_wait(&ctx->cq_wait, &iowq.wq); - io_cqring_do_overflow_flush(ctx); - } + unsigned long check_cq; + prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); ret = io_cqring_wait_schedule(ctx, &iowq, &timeout); + + check_cq = READ_ONCE(ctx->check_cq); + if (unlikely(check_cq)) { + /* let the caller flush overflows, retry */ + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) { + finish_wait(&ctx->cq_wait, &iowq.wq); + io_cqring_do_overflow_flush(ctx); + } + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) { + ret = -EBADR; + break; + } + } + if (__io_cqring_events_user(ctx) >= min_events) break; cond_resched(); From patchwork Thu Jan 5 11:22:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21743C3DA7D for ; Thu, 5 Jan 2023 11:23:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232452AbjAELXw (ORCPT ); Thu, 5 Jan 2023 06:23:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233203AbjAELXe (ORCPT ); Thu, 5 Jan 2023 06:23:34 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCE9C4FD75 for ; Thu, 5 Jan 2023 03:23:33 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id z8-20020a05600c220800b003d33b0bda11so2276277wml.0 for ; Thu, 05 Jan 2023 03:23:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+pkim4tG4NDBUuRyin4QQLRlT/GqMiXafE1MXKCTtQQ=; b=Shg+/8OfjNobEeonLkO2VJyizuRofDISgUhR7IaaHYVFZV/PsaPcx5L5CXDJphaZVC 8o+XbEaje+zJhNXmF02KkzbfEdtcs3t5OJYj6up7W572ysIXG5wnpxhMBseIV4PL9FPJ FeoBudGZSRsWex7U1WiXgvaoax8Aq2pjBywAcGaHQ8OBlzdoVSrtFq5lTED62iFyZJDj IZTDT0vyS2cvrGRXEmnNldMntfRhUW20ljZv42FT10vVkLVDyG+LSWLqxsQRo7wfYXtm vYehp396ly8wQtS8GfkKJC35Rc3W6ZyCWUAVhVNHs7VmDuxBBAMHD/TPckTuF9yP0lHh TDpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+pkim4tG4NDBUuRyin4QQLRlT/GqMiXafE1MXKCTtQQ=; b=yqVrLfRWZiOx59qu0BXh9mv3Eyd0yOwAEg5M/WQ1QDioCTB9V+Z29ccde4/9ZuXXOC myTGMp+R5u1XypLmsJMUwiUeDhwbK75bPKfDR/dImi+OlOq1OwwXs++HfPTddaN09bXI Z7L7IyVdnOiyrr4x8SoFx/Dqaaz0cl4ua6DC+8PJBAiU/0kv7pvF+tpMs+gHp9seSOn2 2eSNKZbAXRBL5eLoK0kFqYd95mwj4N41kdSjrPwCkWs34lyvZm1Z5dP72Vfsf0C7wFZ0 ke0PmoAIlrbxnlHnLR5hk4OFOcoF4+wAmtoE08bDREovsbq6wkDLxctDVBc+7u4doAYT 2jgQ== X-Gm-Message-State: AFqh2kqK2j9t6Q3FlNKK6OifDblClVbIjO5BKc7Kk+JUVzabc0YKdmRz WNDOCfXFozMv4hhcGKhASorhDvy2eig= X-Google-Smtp-Source: AMrXdXsSUl5+BwFTHDvZReujZQRpgLh/F4VCbBg9XnTHZ4Rh/7AVLwWSjS7RYeHteamZnpKwoH0yyQ== X-Received: by 2002:a05:600c:220c:b0:3d2:3831:e5c4 with SMTP id z12-20020a05600c220c00b003d23831e5c4mr39546346wml.40.1672917812165; Thu, 05 Jan 2023 03:23:32 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:31 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 06/10] io_uring: mimimise io_cqring_wait_schedule Date: Thu, 5 Jan 2023 11:22:25 +0000 Message-Id: <2814fabe75e2e019e7ca43ea07daa94564349805.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_cqring_wait_schedule() is called after we started waiting on the cq wq and set the state to TASK_INTERRUPTIBLE, for that reason we have to constantly worry whether we has returned the state back to running or not. Leave only quick checks in io_cqring_wait_schedule() and move the rest including running task work to the callers. Note, we run tw in the loop after the sched checks because of the fast path in the beginning of the function. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 067e3577ac9b..b4ca238cbd63 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2467,24 +2467,19 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, struct io_wait_queue *iowq, ktime_t *timeout) { - int ret; - if (unlikely(READ_ONCE(ctx->check_cq))) return 1; - /* make sure we run task_work before checking for signals */ - ret = io_run_task_work_sig(ctx); - if (ret || io_should_wake(iowq)) - return ret; + if (unlikely(!llist_empty(&ctx->work_llist))) + return 1; + if (unlikely(test_thread_flag(TIF_NOTIFY_SIGNAL))) + return 1; + if (unlikely(task_sigpending(current))) + return -EINTR; + if (unlikely(io_should_wake(iowq))) + return 0; if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS)) return -ETIME; - - /* - * Run task_work after scheduling. If we got woken because of - * task_work being processed, run it now rather than let the caller - * do another wait loop. - */ - ret = io_run_task_work_sig(ctx); - return ret < 0 ? ret : 1; + return 0; } /* @@ -2545,6 +2540,16 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); ret = io_cqring_wait_schedule(ctx, &iowq, &timeout); + if (ret < 0) + break; + /* + * Run task_work after scheduling and before io_should_wake(). + * If we got woken because of task_work being processed, run it + * now rather than let the caller do another wait loop. + */ + io_run_task_work(); + if (!llist_empty(&ctx->work_llist)) + io_run_local_work(ctx); check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { @@ -2559,10 +2564,12 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, } } - if (__io_cqring_events_user(ctx) >= min_events) + if (io_should_wake(&iowq)) { + ret = 0; break; + } cond_resched(); - } while (ret > 0); + } while (1); finish_wait(&ctx->cq_wait, &iowq.wq); restore_saved_sigmask_unless(ret == -EINTR); From patchwork Thu Jan 5 11:22:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3544C3DA7A for ; Thu, 5 Jan 2023 11:23:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231725AbjAELXn (ORCPT ); Thu, 5 Jan 2023 06:23:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233184AbjAELXe (ORCPT ); Thu, 5 Jan 2023 06:23:34 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 792D35014E for ; Thu, 5 Jan 2023 03:23:33 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id g25-20020a7bc4d9000000b003d97c8d4941so1053647wmk.4 for ; Thu, 05 Jan 2023 03:23:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pqtjp7Gb7AWv2wE6SfRrEbU95vMm3JEyoyqrEoS+6sM=; b=pQ7Ksk/v1AzsRAICPNo666blYJutdQfZ4NqPPf2AHgFMRIjtqLqXBnibmzS1aE8Q0/ JfvA56YJTS4BceMka0awGvtluA6qLhp0vGj9oehjrhD+eZ0m8/rQOdMe7DGWSl1grT+D l5E12AzyF4lhlCR3h2w09Df89s4R5W651/CpB5TO4zHb9ngvFFL92gDtIAhGxLuUZDIO EP0KG6LKFmt4JwdNftMuT9f0DcVcTQcXo6SyWJ8KzICGocwHLzN5i6CE+xNVRlclaszu 4BMBdHHn2j6rB5GycrYHv5wOtz4rKESO/PbKwyjDhaHUxg27V9D0EV1ZPd+XgA4bieo5 LPpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pqtjp7Gb7AWv2wE6SfRrEbU95vMm3JEyoyqrEoS+6sM=; b=L9QMuFglZogj09UulzrjTg4ddK0Rw+eMoXRvrRshZ9GSvExJ7zi+AZ+BD9oaX2f/Jo 0aVsuNJQtjIHgqCYOUJDW9ZK10xIps+Jqt+FVv26iqvPbECedXbStg5KXCOIW501LNbx SfaCGMrfws8wdOfWGwy8AKWp8DPBXuaW5QStLaRbBgyiF/oksvzZ40WS+XNpIxRR3YPa ZgpbpMVpwmpPbNG3AxlFOBizvKpZ1j3xlPzd0CCUHzKs3tCGT7diWHTJTWz3YJHd8T+x QPgJPpnbuO+fyg8T0m8+xQmBcJTRoE4NeX6ew3mUGbaQbx8asfX7juYzGMHNjoH4s3GP Rqeg== X-Gm-Message-State: AFqh2kpkGHSsscdK9Tw+twmVGfW7joLBuChnAGMhR2f7VtIVcto8he3n fSTXHhTL9WqfBmWpWz0Ola3MYnPF1Gg= X-Google-Smtp-Source: AMrXdXvZ2HgHmhUYDZexuHUXQNNZ1UR0bl0iOY6dZcSMPsvLYoHX4AF0IGuI3P9XXqck4wbJGVGSiA== X-Received: by 2002:a05:600c:a51:b0:3cf:6f4d:c259 with SMTP id c17-20020a05600c0a5100b003cf6f4dc259mr35086538wmq.39.1672917812926; Thu, 05 Jan 2023 03:23:32 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:32 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 07/10] io_uring: simplify io_has_work Date: Thu, 5 Jan 2023 11:22:26 +0000 Message-Id: <26af9f73c09a56c9a035f94db56127358688f3aa.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org ->work_llist should never be non-empty for a non DEFER_TASKRUN ring, so we can safely skip checking the flag. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b4ca238cbd63..2376adce9570 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2416,8 +2416,7 @@ struct io_wait_queue { static inline bool io_has_work(struct io_ring_ctx *ctx) { return test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq) || - ((ctx->flags & IORING_SETUP_DEFER_TASKRUN) && - !llist_empty(&ctx->work_llist)); + !llist_empty(&ctx->work_llist); } static inline bool io_should_wake(struct io_wait_queue *iowq) From patchwork Thu Jan 5 11:22:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42E28C53210 for ; Thu, 5 Jan 2023 11:23:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232856AbjAELXu (ORCPT ); Thu, 5 Jan 2023 06:23:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233214AbjAELXg (ORCPT ); Thu, 5 Jan 2023 06:23:36 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4355350156 for ; Thu, 5 Jan 2023 03:23:35 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id o15so27765897wmr.4 for ; Thu, 05 Jan 2023 03:23:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wx9SshM8GFcLN9/NS/pxj++LkBKiVC0yVKFQS1M3qJU=; b=Z+oUlSp03uJeEdgGTlPcZNv5AIvWr7yU9eJ/7w7iXUuDzZbC0ln2ZJ+WGFJUCeNp5K Ff1aPOiF8Sr7m+vcJ3oPa4fz6lMZCZnF9GDVUIluHVyZE2msgoeXdQXWDTVxlltxzco7 EqarckcYLJUEXJZr4tyhJPFD4uPbD+EMonShtcsE66RSoRYn7zZ+LcBiCKbF1VeLRWv5 rse0wjE8OnbJNruxROcPJrMexw0lvIovlztZOHWQ4nAxGPJ1dLE3QRUuqiDSLy2z2Ypp UeH1G2isWfckqRvNe1wDMIwP8ny+gQm21MmPvATesa/xHqCvmVaGOkeb6jLx3atGfwA4 l9QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wx9SshM8GFcLN9/NS/pxj++LkBKiVC0yVKFQS1M3qJU=; b=fwAUOZTmJZGpmL8AhRMR/C7ZlU65Rp1Z8+JIzIlIF6luEx3uSVCs1xjzv9Ww4mUbFe QQtS/RSERUjc9pMZmvAdDl1JHoK9Bn/lQWGUuna5L4vQRRIzCU+VksQoTqdIrfrkSkZZ mjJIZ9kePjavLH7ssYBWf0/0gw5uX35EXzXt4jMZVU3RGenZsiECwY5FArHqjsm4s3My 8i/TlutRvmQ3v7RDKIjvGB8UbR9pmahToeSw90SZORggu/sQuR3+f+ylMzATnR53S6tj n32l/7Extk3pt77f929WleUwsiYvdn1W+7/40e4jzBPyC6mwNvMO1ELW3EaLTgQLGN9g 1tqQ== X-Gm-Message-State: AFqh2krPupsApClo/WlQNyVJFSZlasA2sXJi54rzkgfmFctIrl65qqXN Sr7NRuwBWQOxSkzoC8Y06UOKnqvO0Ck= X-Google-Smtp-Source: AMrXdXuGtx3Xk67t5TYP5H9ERBKjZTrm4wRiPro7N8fYWwSwvooWhqfR3glh48uWqcilhTluqHu+6A== X-Received: by 2002:a05:600c:4c21:b0:3cf:f18b:327e with SMTP id d33-20020a05600c4c2100b003cff18b327emr36864605wmp.4.1672917813662; Thu, 05 Jan 2023 03:23:33 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:33 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 08/10] io_uring: set TASK_RUNNING right after schedule Date: Thu, 5 Jan 2023 11:22:27 +0000 Message-Id: <246dddee247d89fd52023f785ed17cc34962a008.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of constantly watching that the state of the task is running before executing tw or taking locks in io_cqring_wait(), switch it back to TASK_RUNNING immediately. Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2376adce9570..54ec0106ab83 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2541,6 +2541,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ret = io_cqring_wait_schedule(ctx, &iowq, &timeout); if (ret < 0) break; + __set_current_state(TASK_RUNNING); /* * Run task_work after scheduling and before io_should_wake(). * If we got woken because of task_work being processed, run it @@ -2553,10 +2554,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) { - finish_wait(&ctx->cq_wait, &iowq.wq); + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) io_cqring_do_overflow_flush(ctx); - } if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) { ret = -EBADR; break; From patchwork Thu Jan 5 11:22:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C0D5C3DA7D for ; Thu, 5 Jan 2023 11:23:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231824AbjAELXv (ORCPT ); Thu, 5 Jan 2023 06:23:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233220AbjAELXg (ORCPT ); Thu, 5 Jan 2023 06:23:36 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9AD94E42E for ; Thu, 5 Jan 2023 03:23:35 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id k26-20020a05600c1c9a00b003d972646a7dso1050492wms.5 for ; Thu, 05 Jan 2023 03:23:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rpbSPant5cggrLBmtqBItkB+rcpfw2nGgE6wmfcc54c=; b=T/vA3qTHuQNITtmOyU78q7iBccc16GXZAEsHp6nQqP6Z9f44FOFfYV3opncVLf2i+j dI0knvUeyrSm+nvHP3D/VmN4QK/hoDEAsJQVN18Ma7Rpq+iP2zKfBAem9ZNga5bB84Cb UW89GDTHHyKbhEy7h3rHYdx3iWdv+0UPiccRm9HOgk+JWTb68dWhFcUmk4RkiM7a/k++ RU4JdKr8YKFOKuaJTQAwcEwNv4HAMRUultH0rzpgs9DwaCWK7x8ZsthsS1K3M2wuj75X yiqHWJYxEJglWbPPvBP4E7nKobSFafoaR2pkeuMBgQ9uFk1bZUSTuHd5B1KqZVNhKDng B9Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rpbSPant5cggrLBmtqBItkB+rcpfw2nGgE6wmfcc54c=; b=OrjTtYVVPkCCdWUMtChhKLDVz7zfnopxMoixqnf0CeMFeKvIeSDcc/cy5H/t9l7t+6 0/S2QP1Urq8Oru1oUgRgXd5L6Yzt+ttzLeD/OaRJ3taPbM19WXf243ySKOrvR69GmTiY qL9LZrry8dbR21xWiO9LgzrVfgfT0aAXyJlYvi5scwBaZAAnXy4Gg4BNPw9L6vREtHZH Cj/JmFmbb6+Yz2b83kj1byQc0ryOOY8JhUXwVAo6Hehv29SPjVZ5mQ5BoswKE0mUjlJq Ih1gEqoX42S1Br/Vv5KTO4osdXLcJqWnDBaVdZH7zvxgLtRFbxhYBGoX8y8Bl9IILevJ 17YA== X-Gm-Message-State: AFqh2kr1dSLVud7Z8PYBd8BNZtpow0SXy+YaRE2knYTfc6qsiWGcwIre szDJdYyUX8aEN/a8JkiRz1P/TQuTlm8= X-Google-Smtp-Source: AMrXdXuo941vlJkwTvq8xEPCTOVwhcwYpIHN5Py/29As2tgvUe6CCcKqpeHyMoN5HguKUhr3AAf79Q== X-Received: by 2002:a05:600c:13ca:b0:3d3:4427:dfbf with SMTP id e10-20020a05600c13ca00b003d34427dfbfmr35362239wmg.5.1672917814371; Thu, 05 Jan 2023 03:23:34 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:34 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 09/10] io_uring: optimise non-timeout waiting Date: Thu, 5 Jan 2023 11:22:28 +0000 Message-Id: <89f880574eceee6f4899783377ead234df7b3d04.1672916894.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Unlike the jiffy scheduling version, schedule_hrtimeout() jumps a few functions before getting into schedule() even if there is no actual timeout needed. Some tests showed that it takes up to 1% of CPU cycles. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 54ec0106ab83..420b022f6c31 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2476,7 +2476,9 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, return -EINTR; if (unlikely(io_should_wake(iowq))) return 0; - if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS)) + if (*timeout == KTIME_MAX) + schedule(); + else if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS)) return -ETIME; return 0; } From patchwork Thu Jan 5 11:22:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13089721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66CBFC3DA7A for ; Thu, 5 Jan 2023 11:23:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232866AbjAELXx (ORCPT ); Thu, 5 Jan 2023 06:23:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233227AbjAELXh (ORCPT ); Thu, 5 Jan 2023 06:23:37 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71BB45004E for ; Thu, 5 Jan 2023 03:23:36 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id k26-20020a05600c1c9a00b003d972646a7dso1050508wms.5 for ; Thu, 05 Jan 2023 03:23:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4lSEuWpHhhixo5EO/vGuzJZ2U1VkfB0a31lSuCDsCbw=; b=VPSsyOnd08u1HQAOyNWdYBB/8cBqmvLad2sn/+kZrDtPMdRxXwjScO+X3nnN1TgQiq TYsQwQKkXg0hV9pENKYa3f2F1EetdKH4TxmaDgdcQzZ+8b+ht9p+WM6dl2D/9gPTvYCo ATlZ2uYnkUmcNCwRvMr1HMpnO5aoi5G0v8O87V16Gg1PPO1UVMcQDEgBbvXiA5U34uHY fQEIAIdw1Puo72W2QCBO/jYCgkYFOfAdVBy7ztqPCiNSqY5HIyB4QjeOeETcGDZl9O8s 54f665kjzYmLdM88MvSqbkiA12fCHCMM2BPn3JLbAtXLmFPVktatOZLWfNVdhSowCDLU iXMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4lSEuWpHhhixo5EO/vGuzJZ2U1VkfB0a31lSuCDsCbw=; b=ihsI1xB/IlNO5wKh4ACteuv4upHdE/PngPez2ijFjIuib7s7hsTnigAtw6ZE/Izr2y ywQB1+mrMkgR6IwsXlLGqMpxvvDHBugVv09vvsM0euEbLE3cY2F1EY0RetM2Sh7mbw5Z DJ9qxr7Q6k2E8s8dSn0pyPcL8BgDzCJ0ef5kCGoK2djnko/Xwk8hPGOk1sG1lZAjT/qT W+2Y0v03Cup3VFblB4o0BV2YYLuBzWINLoAEuienG4pJ2xQzdhDqkLBeMlLuzgs7qH+S pWhwqFD+j4KYHLeG1vjdMsE7A3U11SlQMGOFKfQvs5HlBoUEl80jEeEqrWwt7C14+5+Y YOSw== X-Gm-Message-State: AFqh2koQgp0K7HIwdaOuoVwswS3yAi9/GgjHW6nMOaXQ0yYdmne6T7Sx bTGSBw74OaJMl6+Qwyi1WKrTE1Dklqg= X-Google-Smtp-Source: AMrXdXv/yrKJi4WuNWgt7vpJ6G+yq+HXkShYU552Mgx8qUsSkbz11DYX/L+Ap2ILA/7Y01ZCtTur5Q== X-Received: by 2002:a05:600c:4fc6:b0:3d3:4a47:52e9 with SMTP id o6-20020a05600c4fc600b003d34a4752e9mr35605624wmq.15.1672917814849; Thu, 05 Jan 2023 03:23:34 -0800 (PST) Received: from 127.com ([2620:10d:c092:600::2:5c5f]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b003c6f1732f65sm2220688wmq.38.2023.01.05.03.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Jan 2023 03:23:34 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCHSET REBASE 10/10] io_uring: keep timeout in io_wait_queue Date: Thu, 5 Jan 2023 11:22:29 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Move waiting timeout into io_wait_queue Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 420b022f6c31..a44b3b5813df 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2411,6 +2411,7 @@ struct io_wait_queue { struct io_ring_ctx *ctx; unsigned cq_tail; unsigned nr_timeouts; + ktime_t timeout; }; static inline bool io_has_work(struct io_ring_ctx *ctx) @@ -2463,8 +2464,7 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx) /* when returns >0, the caller should retry */ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq, - ktime_t *timeout) + struct io_wait_queue *iowq) { if (unlikely(READ_ONCE(ctx->check_cq))) return 1; @@ -2476,9 +2476,9 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, return -EINTR; if (unlikely(io_should_wake(iowq))) return 0; - if (*timeout == KTIME_MAX) + if (iowq->timeout == KTIME_MAX) schedule(); - else if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS)) + else if (!schedule_hrtimeout(&iowq->timeout, HRTIMER_MODE_ABS)) return -ETIME; return 0; } @@ -2493,7 +2493,6 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, { struct io_wait_queue iowq; struct io_rings *rings = ctx->rings; - ktime_t timeout = KTIME_MAX; int ret; if (!io_allowed_run_tw(ctx)) @@ -2519,20 +2518,21 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, return ret; } - if (uts) { - struct timespec64 ts; - - if (get_timespec64(&ts, uts)) - return -EFAULT; - timeout = ktime_add_ns(timespec64_to_ktime(ts), ktime_get_ns()); - } - init_waitqueue_func_entry(&iowq.wq, io_wake_function); iowq.wq.private = current; INIT_LIST_HEAD(&iowq.wq.entry); iowq.ctx = ctx; iowq.nr_timeouts = atomic_read(&ctx->cq_timeouts); iowq.cq_tail = READ_ONCE(ctx->rings->cq.head) + min_events; + iowq.timeout = KTIME_MAX; + + if (uts) { + struct timespec64 ts; + + if (get_timespec64(&ts, uts)) + return -EFAULT; + iowq.timeout = ktime_add_ns(timespec64_to_ktime(ts), ktime_get_ns()); + } trace_io_uring_cqring_wait(ctx, min_events); do { @@ -2540,7 +2540,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); - ret = io_cqring_wait_schedule(ctx, &iowq, &timeout); + ret = io_cqring_wait_schedule(ctx, &iowq); if (ret < 0) break; __set_current_state(TASK_RUNNING);