From patchwork Sat Dec 17 20:48:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13075928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 252D8C46467 for ; Sat, 17 Dec 2022 20:49:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229705AbiLQUs7 (ORCPT ); Sat, 17 Dec 2022 15:48:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229675AbiLQUs5 (ORCPT ); Sat, 17 Dec 2022 15:48:57 -0500 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0CC9BE31 for ; Sat, 17 Dec 2022 12:48:46 -0800 (PST) Received: by mail-pf1-x432.google.com with SMTP id n3so3867533pfq.10 for ; Sat, 17 Dec 2022 12:48:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gh4t/ycmHZfX06rOPtzSBBl7/QL2eq7+Ab7eP+nK1Ag=; b=eQ/Kg5fPt3APsBvHeMDc17Amcn29vRArSuDTbiPonlrNRAGfh1h9b1iK/Gns5mFOzf tTamXb5qbf0bH8xRBdxvYsZVkRnPifHBLT0Y5rAzVkw4wbezpGDqlzdni3s84bjRvluo 0dUSDRiQbadQVj6oCfmhP4Wp1zcpXwc8fYgH2Q4Zy5iowkbDkmcOsyfg9Pef2RaXhA+w YKvAqG8Y3f8p5sUVpZwoDxZcszicV1geSc0mMkoFnlk+BJHfM6XOZW6aRQLT5OUBwyjE iZ/fHZao0/uSWmnJMj1k5qOiPhTkAVbhD0xK0sqOC235lxZVzPAhnFazmUPvVWl29FZF LQbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gh4t/ycmHZfX06rOPtzSBBl7/QL2eq7+Ab7eP+nK1Ag=; b=R7SJW/rlx8lnyvO/3tXyf38gNs7YbZ9gl2e5/wHJr1caM/hpKln7gR+BKz5od93WLi I/WGnKP97HwDlG53LUB1zHa/BnD1DK7p3AFa0uk7IlEfgZQ72X2U7L6CTnvSODMvz0aT B0301siHM2qKGT6/ArNK2xId6OhZv4AIg0iQ0frOG7cY/Kp5LEej743Pl9O8FYn5B0qJ EZroY7haWGDbd3gZbd+RzHvUhdypLQpaVeKF35oX1Sj2J0dX3+CCozp2Dk+AjnAZEssJ N0oCyDytvwIQR+cm9IKIbhtChDX80GtnBDXR5cJhsCdfK8NdkHDt1NEYWFXR4uNsV9dP 7u7Q== X-Gm-Message-State: ANoB5pl7Lb+eOidevGeYVIbSEQUDG8bEmKe6b00mwdMtW61y3talXdjZ Y0GZFKufg6fSEstlJyox6qGTujoQvn6pQM/waTI= X-Google-Smtp-Source: AA0mqf6yCeUahfm828mBpPgxHZ5ixW6su+6eBlxnLc56cTn9GtE5OqzUEjJYh/IsqXsQUohCinr8OA== X-Received: by 2002:a05:6a00:1c96:b0:578:451c:84a9 with SMTP id y22-20020a056a001c9600b00578451c84a9mr6647074pfw.2.1671310125820; Sat, 17 Dec 2022 12:48:45 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id i62-20020a62c141000000b0057737e403d9sm3528761pfg.209.2022.12.17.12.48.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Dec 2022 12:48:45 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: dylany@meta.com, asml.silence@gmail.com, Jens Axboe Subject: [PATCH 1/2] io_uring: don't use TIF_NOTIFY_SIGNAL to test for availability of task_work Date: Sat, 17 Dec 2022 13:48:39 -0700 Message-Id: <20221217204840.45213-2-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221217204840.45213-1-axboe@kernel.dk> References: <20221217204840.45213-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use task_work_pending() as a better test for whether we have task_work or not, TIF_NOTIFY_SIGNAL is only valid if the any of the task_work items had been queued with TWA_SIGNAL as the notification mechanism. Hence task_work_pending() is a more reliable check. Signed-off-by: Jens Axboe --- io_uring/io_uring.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index c117e029c8dc..e9f0d41ebb99 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -266,8 +266,7 @@ static inline int io_run_task_work(void) static inline bool io_task_work_pending(struct io_ring_ctx *ctx) { - return test_thread_flag(TIF_NOTIFY_SIGNAL) || - !wq_list_empty(&ctx->work_llist); + return task_work_pending(current) || !wq_list_empty(&ctx->work_llist); } static inline int io_run_task_work_ctx(struct io_ring_ctx *ctx) From patchwork Sat Dec 17 20:48:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13075927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E82FC3DA6E for ; Sat, 17 Dec 2022 20:49:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229625AbiLQUs7 (ORCPT ); Sat, 17 Dec 2022 15:48:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229705AbiLQUs6 (ORCPT ); Sat, 17 Dec 2022 15:48:58 -0500 Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39FFFBE39 for ; Sat, 17 Dec 2022 12:48:48 -0800 (PST) Received: by mail-pf1-x434.google.com with SMTP id n3so3867550pfq.10 for ; Sat, 17 Dec 2022 12:48:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PYSsCr+Ik/F4WG/ELewOiL4Scs8dwzJEodg6CdQ0kkY=; b=c62GtoxZ/PpKdy2uVKx+bqvCscJKxeYWkORaZkG5vwfDxtIgjDxiKhOx/RZefCFhiT 6VbMs3sQgClDmg5m5kHgUYSWhINvnOD0d0mxeKDwqsbmW6KmqDpEg8ARV2qSNdXcP8w8 63hFMDwAgPy+cSa6adxKBVHgfATq4N+eRe9ltnHNe4Qb9edY8/tpbku5n9WkMdlEuNFm uFKl9Ndkz3bsS11vAlRR+yaoGYj632ibReuCF4acEul8im8eDTXYRJ44V43mN5DLRBgY ttG8a6/HFrBtQemVJA2NupI1EmQwBuxsQ7fDP0Sc/BBr5jqu9fy0KiK/5hYWMq3/rzjt 9zsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PYSsCr+Ik/F4WG/ELewOiL4Scs8dwzJEodg6CdQ0kkY=; b=z5BBUgUne/cPikEaojDYp1YP31HeVriADpFsdTdufB5jaYe5oBtAZprxM8TT2TRZlf M0fzi/HrLDaaiIecGGz6M4T6DQGwg3Iq8RCMPVKBIMThJlDQmU/3xPKEvZBbYFvhqKVI j60Ny9btHdtANUbcbBsRkhs/JmPgd1/gjal4kqYTADNYffuxUNrVasZ/f7jA60WXj9cW 8KY/ZoNyS07bbj0W47AIfpZe/LLSWo5r4CmD1SRAph75QHGxCpHJbL4svUIJUyUfYX9a jLkhn0caXtS7qzFnPXr4w3u06qzGuZTirSqh7C/f+ObTL5H6N6kKnYm2ysVce8b8+tXu 2E+g== X-Gm-Message-State: ANoB5pny41sUabBoGtqLhlEddvpEAwg8ERSfrJ/cfHthINxAvfBJP6wA uY+XoiDVGKNY0XplZGW37TCunqs0cTnuozEvZZw= X-Google-Smtp-Source: AA0mqf7WzVs1nr43bhlU6ctPpe85ptuqdZ1G8bdpfvIedEz8ueVsOcyNRfTH8nCd5UY9CUUjE9Ai2A== X-Received: by 2002:aa7:8395:0:b0:575:dfe6:5df2 with SMTP id u21-20020aa78395000000b00575dfe65df2mr7665945pfm.2.1671310127278; Sat, 17 Dec 2022 12:48:47 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id i62-20020a62c141000000b0057737e403d9sm3528761pfg.209.2022.12.17.12.48.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Dec 2022 12:48:46 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: dylany@meta.com, asml.silence@gmail.com, Jens Axboe Subject: [PATCH 2/2] io_uring: include task_work run after scheduling in wait for events Date: Sat, 17 Dec 2022 13:48:40 -0700 Message-Id: <20221217204840.45213-3-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221217204840.45213-1-axboe@kernel.dk> References: <20221217204840.45213-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org It's quite possible that we got woken up because task_work was queued, and we need to process this task_work to generate the events waited for. If we return to the wait loop without running task_work, we'll end up adding the task to the waitqueue again, only to call io_cqring_wait_schedule() again which will run the task_work. This is less efficient than it could be, as it requires adding to the cq_wait queue again. It also triggers the wakeup path for completions as cq_wait is now non-empty with the task itself, and it'll require another lock grab and deletion to remove ourselves from the waitqueue. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 16a323a9ff70..945bea3e8e5f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2481,7 +2481,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, } if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS)) return -ETIME; - return 1; + return io_run_task_work_sig(ctx); } /* @@ -2546,6 +2546,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); ret = io_cqring_wait_schedule(ctx, &iowq, timeout); + if (__io_cqring_events_user(ctx) >= min_events) + break; cond_resched(); } while (ret > 0);