From patchwork Sat Jun 25 10:52:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12895312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48792C43334 for ; Sat, 25 Jun 2022 10:53:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232635AbiFYKx2 (ORCPT ); Sat, 25 Jun 2022 06:53:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232285AbiFYKx1 (ORCPT ); Sat, 25 Jun 2022 06:53:27 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C471C193C1 for ; Sat, 25 Jun 2022 03:53:26 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id k129so1193940wme.0 for ; Sat, 25 Jun 2022 03:53:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PyXk91LZeup4OW4fOSE9VHxAndTow2OU7Kvxw5VkVlk=; b=A+Mxg+7kvJAe23+t/uT20Z3CVMvfJon8AAkYeyf/Am/eQcGcEFkGQsPhxqfBNbSIf6 5ZD80Cw+R3OHfYGsYoTUZF7essXWEjU/tg8xtKxBgqsPDzXd6gs0YgM/vHoHNGlwenmd ZpAhirawy+OZbdgs1frSQ7wHpV+p/a++eAxX59homA4n/i8DEXMkFDfqCj56AxHZ+57e m7/OTa+WdpJacVMCgRY36RB32bw8wzz/Wzw//VBcbe5Be8JBZy1JzyUXwbTrk8wlCgW+ nqC00vPF8j9hu67LkJ8lvnqiPr3Zi+u6RbVkmVrB8pqCPj/SO6qJt+yJphD6L889T5NW U/cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PyXk91LZeup4OW4fOSE9VHxAndTow2OU7Kvxw5VkVlk=; b=dY9jA0Avqbho0TVhkMNMEfCzfvdX5RNgiEOj9iWXIzlsdS0cslcEpO6zsRzlQLF4R+ 6AcYx4puwkNzPvcDqz+UHs/PfoIq28xwapw1jtZSpu6n6zq8dXHbIoQF7Sfa7C/Gq97n gHgnZFBV4sMm+KoYklajhRWVD8e2p/bC61+qmdIPdNn5x+SvGgE4JhdfA7R2YdSTlJFR oXDe/UeiVRQhdMLYNbCzcFp0ACT12cdv95tamBuzMYe8NFdFxmPtVR95Orqnlbvx6qb5 tKNFZPeO6sTXVXc+vn9Gkw1dD1RNPlxgQJi0CRsQYOEdNG4mnQ8UzHsdOP9cZeq8sjUT u/EA== X-Gm-Message-State: AJIora8UxgZ1kKexC7ZY4ChUNv8lkRKinZ/BZ8NfwQ3md1cRYzxEfM/b bkJvVLPwZgnwiUbqgio4SO7fw7v8Enmz2g== X-Google-Smtp-Source: AGRyM1s7zhxMYMR/ua6b+2kOLJgPtTnhuT30kChcppHE4l/eWq+3ekkTgH2KSz7OlCPritRw/vMODw== X-Received: by 2002:a05:600c:1c8d:b0:39c:4db1:5595 with SMTP id k13-20020a05600c1c8d00b0039c4db15595mr8580347wms.175.1656154405123; Sat, 25 Jun 2022 03:53:25 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id m17-20020a05600c3b1100b0039c5497deccsm15810144wms.1.2022.06.25.03.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Jun 2022 03:53:24 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/5] io_uring: improve io_fail_links() Date: Sat, 25 Jun 2022 11:52:58 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_fail_links() is called with ->completion_lock held and for that reason we'd want to keep it as small as we can. Instead of doing __io_req_complete_post() for each linked request under the lock, fail them in a task_work handler under ->uring_lock. Signed-off-by: Pavel Begunkov --- io_uring/timeout.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 4af074b8f6b7..2f9e56935479 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -101,32 +101,44 @@ __cold void io_flush_timeouts(struct io_ring_ctx *ctx) spin_unlock_irq(&ctx->timeout_lock); } -static void io_fail_links(struct io_kiocb *req) - __must_hold(&req->ctx->completion_lock) +static void io_req_tw_fail_links(struct io_kiocb *link, bool *locked) { - struct io_kiocb *nxt, *link = req->link; - bool ignore_cqes = req->flags & REQ_F_SKIP_LINK_CQES; - - req->link = NULL; + io_tw_lock(link->ctx, locked); while (link) { + struct io_kiocb *nxt = link->link; long res = -ECANCELED; if (link->flags & REQ_F_FAIL) res = link->cqe.res; - - nxt = link->link; link->link = NULL; + io_req_set_res(link, res, 0); + io_req_task_complete(link, locked); + link = nxt; + } +} - trace_io_uring_fail_link(req, link); +static void io_fail_links(struct io_kiocb *req) + __must_hold(&req->ctx->completion_lock) +{ + struct io_kiocb *link = req->link; + bool ignore_cqes = req->flags & REQ_F_SKIP_LINK_CQES; + + if (!link) + return; + while (link) { if (ignore_cqes) link->flags |= REQ_F_CQE_SKIP; else link->flags &= ~REQ_F_CQE_SKIP; - io_req_set_res(link, res, 0); - __io_req_complete_post(link); - link = nxt; + trace_io_uring_fail_link(req, link); + link = link->link; } + + link = req->link; + link->io_task_work.func = io_req_tw_fail_links; + io_req_task_work_add(link); + req->link = NULL; } static inline void io_remove_next_linked(struct io_kiocb *req) From patchwork Sat Jun 25 10:52:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12895313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0533CCA479 for ; Sat, 25 Jun 2022 10:53:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232651AbiFYKx3 (ORCPT ); Sat, 25 Jun 2022 06:53:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232285AbiFYKx2 (ORCPT ); Sat, 25 Jun 2022 06:53:28 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E06F817AB1 for ; Sat, 25 Jun 2022 03:53:27 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id v65-20020a1cac44000000b003a03c76fa38so1969659wme.5 for ; Sat, 25 Jun 2022 03:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r9oi2EPTtmMyncZ4TJNImdYRsQ+SPW4kA43PFdaQF1Y=; b=aX1zZ9ZkwN9GWKicd1PqB8uxoJ4dWN/NdYF10beqcZws6KJCKU21VkFCLafws0yiY7 N3/8THgMFn14+cms1MgfM+P/m8mlHcF7m4F2lsla+/PUqjxq/Y1P0Se9k/2ltU+rcYtU lER1HvmN7EztiU63lr0/0nJRH0mnvTOHhras5U7jz33HhKFIl/nYkxaK+33Th+dtHAJU sZpg24g3yvXdCmEgS/JjBNwyGtMlzENnY4Q9U+Epm9cwhLiD7MhaQw6KPAKeJQSLlL8X KpJnxx8il0dDhBrSrc6IoUGlB8mgy9TxT4UrtUDFP7kHyTdaS5ay6gNrwMhz3V3Tf4zB 1yRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r9oi2EPTtmMyncZ4TJNImdYRsQ+SPW4kA43PFdaQF1Y=; b=0txnsSlTknDvmU3XTqvDccEAUJHmNbQ7NFEaekXV+iw2NzM0Ogfy7UOUPDuZD063ce cuAeiR+PSDN4Ey30bY91YCuxT6kNfzEhjJmebLCcsRfUQFK9ZkK9RpOCCPdQkENBSIYa BImbAerb0XoFGdHV9YO10ynqd1yH+7opCLPXCxNab1wVPH/9dAaQc4LIRHp9cYHot1k0 roiqJVcv/0SJYCxdQ2hAwSM4bRAHRoLbTGpnvjz76hL4fwTxb5wQzlgYFLa+cJO+e5U9 Vt1gTkwZCLw/q5/Se1IaZWJ53dRMds+0c7q6D+1lVgJ/qwgt74JFaxbuESZwu9/HdkvK hTpQ== X-Gm-Message-State: AJIora+C0EE64QnAYSd6/38rnlNnW1K2KBRWAAs7iorfa3fA5akDOUrq No4gvjvr9qK0Grs8p+VUm3VNt/qS8EbtFA== X-Google-Smtp-Source: AGRyM1vwiNrVOtb3KSj7tf8W/Ld12DYO3r0oObjMG7MmHpmg8pxZY1LXBl9k8Kc0L1dUDhqw5XHw9A== X-Received: by 2002:a05:600c:3b96:b0:397:485a:f5c5 with SMTP id n22-20020a05600c3b9600b00397485af5c5mr3958021wms.185.1656154406172; Sat, 25 Jun 2022 03:53:26 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id m17-20020a05600c3b1100b0039c5497deccsm15810144wms.1.2022.06.25.03.53.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Jun 2022 03:53:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/5] io_uring: fuse fallback_node and normal tw node Date: Sat, 25 Jun 2022 11:52:59 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Now as both normal and fallback paths use llist, just keep one node head in struct io_task_work and kill off ->fallback_node. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 +---- io_uring/io_uring.c | 5 ++--- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 918165a20053..3ca8f363f504 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -427,10 +427,7 @@ enum { typedef void (*io_req_tw_func_t)(struct io_kiocb *req, bool *locked); struct io_task_work { - union { - struct llist_node node; - struct llist_node fallback_node; - }; + struct llist_node node; io_req_tw_func_t func; }; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 45538b3c3a76..86a0b0c6f5bf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -233,7 +233,7 @@ static __cold void io_fallback_req_func(struct work_struct *work) bool locked = false; percpu_ref_get(&ctx->refs); - llist_for_each_entry_safe(req, tmp, node, io_task_work.fallback_node) + llist_for_each_entry_safe(req, tmp, node, io_task_work.node) req->io_task_work.func(req, &locked); if (locked) { @@ -1091,13 +1091,12 @@ void io_req_task_work_add(struct io_kiocb *req) if (likely(!task_work_add(req->task, &tctx->task_work, ctx->notify_method))) return; - node = llist_del_all(&tctx->task_list); while (node) { req = container_of(node, struct io_kiocb, io_task_work.node); node = node->next; - if (llist_add(&req->io_task_work.fallback_node, + if (llist_add(&req->io_task_work.node, &req->ctx->fallback_llist)) schedule_delayed_work(&req->ctx->fallback_work, 1); } From patchwork Sat Jun 25 10:53:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12895314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31921C433EF for ; Sat, 25 Jun 2022 10:53:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232285AbiFYKxa (ORCPT ); Sat, 25 Jun 2022 06:53:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232649AbiFYKx3 (ORCPT ); Sat, 25 Jun 2022 06:53:29 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2D7B183A2 for ; Sat, 25 Jun 2022 03:53:28 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id f190so2317153wma.5 for ; Sat, 25 Jun 2022 03:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=k++j3jMCmembuoHqzW2/+fFXcAGOMbv7QAh6X6Q/plI=; b=EKBHbL2sqARHtkKWkBgE/6FuqGinJzOO+V2A69Kqggicie8T+PdfZIQEb+iHUrvjGs fOJIiVAeOg5zdMzpWgsq+dWRrqW1Vq1IOWkHjf/HoVMRFVLQ1ccV4sLY5DZyNSKY3Ros vz2i8y4WYRXSmcT0ggZzamMS4Vq4i5iUQ9i+HPYs4/bCAfNZmhYVHjTrGKro72eI5+N9 Stku7G1EAkFli7VoyfwAuJUPdafiCiWabCHR+sU9fZvFaB6Ui9HZBKjXEgGLctoS8eT0 MmQi2bNXT3AC0igd3gtC8Gu8m2eIQemm2rWk0JSAzIaBRHywT6XLbLhXB3gRE3389cG7 CBcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=k++j3jMCmembuoHqzW2/+fFXcAGOMbv7QAh6X6Q/plI=; b=0KO3+P/lNDdc7knn+1Sbi/HvYR8lLoYTx+VUyBDC7SPRwg8Th8TJNDVU7z/KCm/b9x QmhnTwts+1wJIMlrgpiHPjJaVWbIYs9QzftaGQsijmZFeVyAJDRMVHNLyRAbtBUF9gEA 1otD41da9aGl6sPYo8NXaHrKQ18a9GKIqEiLXHT2lLSjsLJ129hZsVj/acL6KRS0d9oS 5143vMmdXa6mAvKRk9CxEcH52L9+qHql7lx10vWBbE01YwTwA/0ZwMcu4qAPei1eL7us sf32Ozz2dwwsgjtGme1yoby1AuddvyqqOFIbyshvnlWblhJMWiOAL1kb3Zaxo0Mz2Wee Bn+g== X-Gm-Message-State: AJIora/WaEwSQR7/sFKR3MKISECjKKZTM48xx2bmv0TFt9Q3ImbAW+Lw M2oQwZMEc9hRlM6sVmeLJsWKDDXcw0aHCw== X-Google-Smtp-Source: AGRyM1vk17GcpyhQoAWgUglYUEvLYnf+erOVVN9e1uBT/vLmHMysnA2jiouKKnnE7JJqr0kmWkF5Gw== X-Received: by 2002:a7b:c921:0:b0:3a0:41c3:f44a with SMTP id h1-20020a7bc921000000b003a041c3f44amr2344078wml.122.1656154407088; Sat, 25 Jun 2022 03:53:27 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id m17-20020a05600c3b1100b0039c5497deccsm15810144wms.1.2022.06.25.03.53.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Jun 2022 03:53:26 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/5] io_uring: remove extra TIF_NOTIFY_SIGNAL check Date: Sat, 25 Jun 2022 11:53:00 +0100 Message-Id: <52ce41a592ad904511697f432141e5690fd4b968.1656153285.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_run_task_work() accounts for TIF_NOTIFY_SIGNAL, so no need to have an second check in io_run_task_work_sig(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 86a0b0c6f5bf..f40526426db8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2205,8 +2205,6 @@ int io_run_task_work_sig(void) { if (io_run_task_work()) return 1; - if (test_thread_flag(TIF_NOTIFY_SIGNAL)) - return -ERESTARTSYS; if (task_sigpending(current)) return -EINTR; return 0; From patchwork Sat Jun 25 10:53:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12895315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B093C433EF for ; Sat, 25 Jun 2022 10:53:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232649AbiFYKxc (ORCPT ); Sat, 25 Jun 2022 06:53:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232674AbiFYKxb (ORCPT ); Sat, 25 Jun 2022 06:53:31 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA59F1CB20 for ; Sat, 25 Jun 2022 03:53:29 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id r81-20020a1c4454000000b003a0297a61ddso3151072wma.2 for ; Sat, 25 Jun 2022 03:53:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qJrhBFpF10IM4q4cMndXnFTorXE6d6zjBVBWHT87ASc=; b=SugC85Z7XfKmBdezmux2bq55mfuFIS51zeuvd1UcTpw7HwIJG3xH3tjw6lanySrcOW TWfsGCqlWpktI1iw0SSa2GNHCUKCm+fK5Z9LqbP316qtIsID/9mg5s320NtlBlou8O5y wS+H2ke2ezKO5qF5KRpQwdTo9BmwVwJZUQ4BtdC0XQCoowVn1AWBTHJqKBzadQ6G4Ft6 Kh5C93cQ0V1DDrB3vXSON6+bKcoLmVJAiPs7XaYOp4xJbtLGoQWn2VFzETIkxD1jgiVP fzLsLJpHjp9YYzXmmazNKSiVbqe/MaxybxS/qZgj7kGwgWaA2bBaYWIDdqru4Jf/NLfY 91Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qJrhBFpF10IM4q4cMndXnFTorXE6d6zjBVBWHT87ASc=; b=MR2bYQS7yZohgjVnpeaRK1JkRfaNY9cLZRablLo6yIoXgAmVhsKjXgw7TXJbLTquVf KIRA0+ZHziFgn12YSYthTaSznp9VLm6TOAY7e2vpL+SZBiqz3utu17Ug/o2QmW71hIVH 9711Zo9sjbpW11lRZi3fkn36Jxj4iQcTwPyLxCTqyOy/L47bNoIj3xb4DesTq2ReYRsD B9fkXAzV3WbGj3w4mwRm8pQqwgLcQzSxXkO1h3ty9CzcOkuI4YdfL68s2liva9qaxNkL pRO6T2KQ4DVx6swgdIDnzQ9HUS2y3oY91Xt9JeD4NyZu7u3LljT6qlEeRR3VmIGy6G54 Irww== X-Gm-Message-State: AJIora9nryGAZIQwAfTNTIKdo+ZBty5442Mjz3vO1YoKms4zND1BWU1D qaCnwrm6ZDF9VMcos/QeJQlUS0W/LIRwfA== X-Google-Smtp-Source: AGRyM1tpHk3Z5C01kbwwBdcuzkfOEtVic7+DBDJjeCBi+/qQiVApKILc+SdeC0Sre8P1nIas0ql6dA== X-Received: by 2002:a1c:7903:0:b0:3a0:3936:b71f with SMTP id l3-20020a1c7903000000b003a03936b71fmr9007735wme.168.1656154408115; Sat, 25 Jun 2022 03:53:28 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id m17-20020a05600c3b1100b0039c5497deccsm15810144wms.1.2022.06.25.03.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Jun 2022 03:53:27 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/5] io_uring: don't check file ops of registered rings Date: Sat, 25 Jun 2022 11:53:01 +0100 Message-Id: <425cd64fd885b8e329a46c205ee811987691baaf.1656153286.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Registered rings are per definitions io_uring files, so we don't need to additionally verify them. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index f40526426db8..e1e8dcd17df3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3036,22 +3036,22 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, if (flags & IORING_ENTER_REGISTERED_RING) { struct io_uring_task *tctx = current->io_uring; - if (!tctx || fd >= IO_RINGFD_REG_MAX) + if (unlikely(!tctx || fd >= IO_RINGFD_REG_MAX)) return -EINVAL; fd = array_index_nospec(fd, IO_RINGFD_REG_MAX); f.file = tctx->registered_rings[fd]; f.flags = 0; + if (unlikely(!f.file)) + return -EBADF; } else { f = fdget(fd); + if (unlikely(!f.file)) + return -EBADF; + ret = -EOPNOTSUPP; + if (unlikely(!io_is_uring_fops(f.file))) + goto out_fput; } - if (unlikely(!f.file)) - return -EBADF; - - ret = -EOPNOTSUPP; - if (unlikely(!io_is_uring_fops(f.file))) - goto out_fput; - ret = -ENXIO; ctx = f.file->private_data; if (unlikely(!percpu_ref_tryget(&ctx->refs))) From patchwork Sat Jun 25 10:53:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12895316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36063C43334 for ; Sat, 25 Jun 2022 10:53:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232676AbiFYKxe (ORCPT ); Sat, 25 Jun 2022 06:53:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232688AbiFYKxb (ORCPT ); Sat, 25 Jun 2022 06:53:31 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08F35193C1 for ; Sat, 25 Jun 2022 03:53:31 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id n185so2583094wmn.4 for ; Sat, 25 Jun 2022 03:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ILOlYtBJHR0amL/QmGXZ22FVloEJvHsPQZNwxmR3VJs=; b=OFlFSKHjJHZEn8+fwcUIyb5CqpmVFMKoC1Z1OXrqvHVJONnD4w+Lfd6+sseQy6gczJ zvM8JvVC3LYD9y7opVl4W9LjJYBDulILSlWLM6vZLrdIg/+KLpBJAzmDbyXu3oMVTDHP k09Tt07qhW3wfJiVkii7qhmMUeV3++i8+fWEUkvfRopf8fZ+49T/CF1vxdJmIbW2LLmd NWBQ48hxBcf6fTAaRcXb3GDXQDgfnMNx6eP6UeewR6UnGzNe4Fjukw5MUpCfiLSQZP9c 11zc5yOONRP3gmhKNvHuuZP/xcCcE6caDvYNr1VHbAUiI7igrdUP2sQDaNXikNs4xANp FaLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ILOlYtBJHR0amL/QmGXZ22FVloEJvHsPQZNwxmR3VJs=; b=EMjbQ2G3gKN8UUePWs5xvqUJJEQTpAnrNNVKhkujv00Kv2LX3druYS3m4VvuxEnTKg pUEi4e2zpWzm2yKZhIJ0YyAv01aLzPGKRLAJQB5Rv/EP93dznI3VBOH6hfdPnNSBltCn GEFytsoGoCaoGD249JSprgbExcxC96VOa1DyTVNfPsRJWNQgQ8/kC8ppBkedkjvcYMOZ 7L1paMrohJUEXdL6AMqwq7xoMqYEMEQKxJgVAvU7bFKtvHblMyKcjWyyadjsheGxT0mv 5RUcUAI0NqNSH2nT62UJm4KEB08Q4keQnV9eR7oDYoLj9bxUS85snQByUXJmqwwcndyd edmA== X-Gm-Message-State: AJIora+0f1cXkLY8XHizpKz1D+qn7qRUivZ0krldDleMCGZ1w8WFILqd vzPtEQ1UtaVDADpd0TeynxhdI3T09Dn0Pw== X-Google-Smtp-Source: AGRyM1vXEihr/fGXxjMMjaYw7rIVS0HK8UgN/xIxxlt7K3/Ce1RSiz7M8Wp2+iiD1HAZtwhRCzvVUw== X-Received: by 2002:a05:600c:34d0:b0:3a0:2c07:73ac with SMTP id d16-20020a05600c34d000b003a02c0773acmr8862765wmq.85.1656154409233; Sat, 25 Jun 2022 03:53:29 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id m17-20020a05600c3b1100b0039c5497deccsm15810144wms.1.2022.06.25.03.53.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Jun 2022 03:53:28 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/5] io_uring: remove ctx->refs pinning on enter Date: Sat, 25 Jun 2022 11:53:02 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_uring_enter() takes ctx->refs, which was previously preventing racing with register quiesce. However, as register now doesn't touch the refs, we can freely kill extra ctx pinning and rely on the fact that we're holding a file reference preventing the ring from being destroyed. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e1e8dcd17df3..070ee9ec9ee7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3049,14 +3049,10 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, return -EBADF; ret = -EOPNOTSUPP; if (unlikely(!io_is_uring_fops(f.file))) - goto out_fput; + goto out; } - ret = -ENXIO; ctx = f.file->private_data; - if (unlikely(!percpu_ref_tryget(&ctx->refs))) - goto out_fput; - ret = -EBADFD; if (unlikely(ctx->flags & IORING_SETUP_R_DISABLED)) goto out; @@ -3141,10 +3137,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, &ctx->check_cq); } } - out: - percpu_ref_put(&ctx->refs); -out_fput: fdput(f); return ret; } @@ -3730,11 +3723,10 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, int ret; /* - * We're inside the ring mutex, if the ref is already dying, then - * someone else killed the ctx or is already going through - * io_uring_register(). + * We don't quiesce the refs for register anymore and so it can't be + * dying as we're holding a file ref here. */ - if (percpu_ref_is_dying(&ctx->refs)) + if (WARN_ON_ONCE(percpu_ref_is_dying(&ctx->refs))) return -ENXIO; if (ctx->restricted) {