From patchwork Mon Jan 23 14:37:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13112299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 845A0C05027 for ; Mon, 23 Jan 2023 14:42:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229894AbjAWOmM (ORCPT ); Mon, 23 Jan 2023 09:42:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230134AbjAWOmM (ORCPT ); Mon, 23 Jan 2023 09:42:12 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB39518A8F for ; Mon, 23 Jan 2023 06:42:10 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id h16so11004225wrz.12 for ; Mon, 23 Jan 2023 06:42:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Zz8mte92pKCxIeKLVprZKS17+HTmr83atw+WYLy32Vc=; b=Vj1M8TO8amcgkWABvF/oy7xptxFL/Gusqo6sLqo1bAvRHrv0imIWciRhFLjdyOiLJQ O4w+7vPstf+5Owpdx6QOL4XtCTlFvHForJW26lZxp+bU9gJ1RCdaPebsLf+QWmU2oydi ZKkTRvh8gQ2821S7YZs+MTa26K/OIcr9FZoGzOxQthtkhjF460T4l8I0QF5QkItubyY2 llmGgQIAtdFmMm+/Bv1FUFwZ4IEMUBSRDx7yIxU1+IiGcjxzwCA0Ha+xnmbFwU0vDivu rD0+zWwnBkW9cLoivAqQPOPGJYEhYe4f1Bvj1F7avvP/LdqSBkqFvjjdRSJ+ILXMmKm8 P6yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Zz8mte92pKCxIeKLVprZKS17+HTmr83atw+WYLy32Vc=; b=Qr4UTba6wstL8affXRhHvzy8EgQasfD4rDri9smh6CJRYwt+KqIOJFrkewRFCEWqAu KoeCHPOPBfHT37Yk5xwT3Rr8JMKfI+Yl8K8nVE1apixb2YLUXF6iiOoLliwMc0lw7C62 TnXENray4yH7GxWFPGAHT24tjJrXH8+adNYH1Kic2fTdQIr/uZzJzN+/e8zA1NC3aM4o 53Vr3NZRF03k5mI+5DGWeqtHeq6WQVGm3Xr7X/7CSAoTBv6ZtKTDNikO/V5H6r6hxrN2 cPP/aFz4oAmJqG70efkTgCa3StvoNrB98c/DqPSb68Gk2fKixgpLtyK5vMieL7mvyX+7 WP0g== X-Gm-Message-State: AFqh2koA1bGoTm9WLwE+LXPaswU5k2RcWEeU6E5hve/A4P1kV0GKQS3Q zgLfWIz/YqoCYx/VPfVOvN4p2kR1rTA= X-Google-Smtp-Source: AMrXdXsjodMupCNr8n5cE+7o42mWBW1XaAfpb0NgcnER5hy+ldjKyij10BQgs8oYiBty4RJFVxeyHQ== X-Received: by 2002:a5d:67d2:0:b0:2be:50a7:cfa9 with SMTP id n18-20020a5d67d2000000b002be50a7cfa9mr11576694wrw.63.1674484929331; Mon, 23 Jan 2023 06:42:09 -0800 (PST) Received: from 127.0.0.1localhost (188.30.84.186.threembb.co.uk. [188.30.84.186]) by smtp.gmail.com with ESMTPSA id d24-20020adfa358000000b00236883f2f5csm3250534wrb.94.2023.01.23.06.42.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 06:42:09 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/7] io_uring: use user visible tail in io_uring_poll() Date: Mon, 23 Jan 2023 14:37:13 +0000 Message-Id: <228ffcbf30ba98856f66ffdb9a6a60ead1dd96c0.1674484266.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We return POLLIN from io_uring_poll() depending on whether there are CQEs for the userspace, and so we should use the user visible tail pointer instead of a transient cached value. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index aef30d265a13..c42c1124ad5c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2888,7 +2888,7 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait) * pushes them to do the flush. */ - if (io_cqring_events(ctx) || io_has_work(ctx)) + if (__io_cqring_events_user(ctx) || io_has_work(ctx)) mask |= EPOLLIN | EPOLLRDNORM; return mask; From patchwork Mon Jan 23 14:37:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13112301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C62FC54E94 for ; Mon, 23 Jan 2023 14:42:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230134AbjAWOmN (ORCPT ); Mon, 23 Jan 2023 09:42:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230220AbjAWOmM (ORCPT ); Mon, 23 Jan 2023 09:42:12 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCA7B1B55F for ; Mon, 23 Jan 2023 06:42:11 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id n7so11025893wrx.5 for ; Mon, 23 Jan 2023 06:42:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dETK0oJbLNqgIFPxwRAQ4aNmg0XogJEIkBO9gXrECaY=; b=Ot4n5WWsoSpB+k3J5C/UhBjEpVwi6VwHRs/cjR/PmkNpFSRRD/Z4oQbPfIo/9R1yzr X1/YZP4dvCOLJNlJNx/T3ottHuGlRy62HyM8PthnQezojYBsqPF9KZpPLgWTw3Y4WT8Z 4rOPOGj5CvLE1E7gzk9NWxzXAprE/nLaK7K0Ity3wKmWnNAteJ4RJDarGczt1aUWvoev 2vvVZGPfzJJnLFwZlpT757F9+dxM0nD17Jl4kPceIzNPjY2T6yWxUt59sMYC8afKNcqg QAPhSI2Ic4ZOdQT3MPrXzOEtzwMZDpL923YYxb7FKzD3ABNTxyboxgAPicvF15eOaroR lSmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dETK0oJbLNqgIFPxwRAQ4aNmg0XogJEIkBO9gXrECaY=; b=IDOWGylwv/5zfWTREbV8kh1M3Rcr7jHAu4j4/jyddvWygl90fQ4bEYzfTru44GyPEY ohH3ykjVE69IQCP3kDa5LJjXSAnssDnNElyIWcyLCjcFy81m5Y59L2PIxBWBKPWcpfxw Y19leH3aMVNaK9th88wfacSL8UU0y0Q0Mg2zubYUC2OWZ2FlcwqKFNUmFcWDIT5M7HS/ MNrkzTaEB8jM6122RScZjY489ilN4tpXUbNUJmSW7IompWgkw1U3qBWqytgBRHDSOoK9 LtENM7fGv/cD8gVgdJCXdqbBk9gLYVsUlKVgSooSZ8E1rggv2wns1pvfVFP1EUzZWxMg frhQ== X-Gm-Message-State: AFqh2kpIKidUZ67E52D7QRRnrAaOiaIUDedSkUgMbLDUYqUjIfaV4WGh mJ01jm3WkfQaBrV4AM42cFhVU4H2yPs= X-Google-Smtp-Source: AMrXdXvB54vDJq3rWDuh5euvjjk5S4abQj90Z48zgLOqDeYq7gkjHY/RGJufHeAT8Z4VlKzv59Ep4A== X-Received: by 2002:a5d:6b09:0:b0:2be:110d:5d59 with SMTP id v9-20020a5d6b09000000b002be110d5d59mr20422996wrw.51.1674484930091; Mon, 23 Jan 2023 06:42:10 -0800 (PST) Received: from 127.0.0.1localhost (188.30.84.186.threembb.co.uk. [188.30.84.186]) by smtp.gmail.com with ESMTPSA id d24-20020adfa358000000b00236883f2f5csm3250534wrb.94.2023.01.23.06.42.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 06:42:09 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/7] io_uring: kill outdated comment about overflow flush Date: Mon, 23 Jan 2023 14:37:14 +0000 Message-Id: <4ce2bcbb17eac80cdf883fd1459d5ee6586e238c.1674484266.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org __io_cqring_overflow_flush() doesn't return anything anymore, remove outdate comment. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c42c1124ad5c..118b2fe254ba 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -666,7 +666,6 @@ static void io_cqring_overflow_kill(struct io_ring_ctx *ctx) } } -/* Returns true if there are no backlogged entries after the flush */ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx) { size_t cqe_size = sizeof(struct io_uring_cqe); From patchwork Mon Jan 23 14:37:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13112300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D534C54EB4 for ; Mon, 23 Jan 2023 14:42:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230220AbjAWOmO (ORCPT ); Mon, 23 Jan 2023 09:42:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231131AbjAWOmN (ORCPT ); Mon, 23 Jan 2023 09:42:13 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92649126FD for ; Mon, 23 Jan 2023 06:42:12 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id bk16so11003863wrb.11 for ; Mon, 23 Jan 2023 06:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MMvHvBXJ6HVq2ctrCftC5eJz76/irYGFUMsWT9/tXuI=; b=JkaLi9aKj86qLQhn09LsiJr+DqcdfGFZUaeoHxI5OYHoPzZ5l4IEKm4QIO0b67n99b gRal2gRnKgfgAkZseplIZvU+ec5LmGIBue/vzTErNWnEAOG1i9OUT7WyCAYxysZdk1sk axdvVVKQF8NK1W5TBEnsF8qSEosAKv1uzgx+7Apol6/6MY5wHLsdg3rSgR1Xk4Ph41If UyWKujActklFC8FkIdSvgf1rHUgjuh5UfoNciiOOOSNtdGfNux/7XjTJM0kW4t/hK+kr 3hdx+qwOwVj9j35nTszAPZWuWuV7Cej67y05Hzwbi/RjpLByd9rbhHiivWjf4ORJh44A W/vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MMvHvBXJ6HVq2ctrCftC5eJz76/irYGFUMsWT9/tXuI=; b=E6BpLkWxKtkbQmRw+krST8TXiA3jG5wAnPO92rf5KfKQtKrASqHGZw/EUYTIobF6oj 0oma2gRV1ky6h7PvYjEt2JS+WMZbZ1trOA/NJQpWQmIRnbKvnLrsPt9avRGu8HNYDySf Er8F+Nh2B8X6M76mhsoXcD5FQpAx/2SHPNYbRaeccLT0DpD0DkvfqJ4XCQE/2UColqWu rMkpYHni302A8P/YflxKBxM9LMXB9iR8PeIg8kCZUr99A8vhpdd6EupETFeKB5mawc61 h/xhPj/RHfZ23SBrqCDdzdannFuICdpi8SthjKMR2NQ2aZxlomtW7yv9EaX4ZOqX1NBY VYBg== X-Gm-Message-State: AFqh2kpvof8xl82b2/svwr+6WoR/lrn7ocWA75+bAtkaMSOtO8IxVlrb 9X3g/M8ikyx+Gjni0WNy0Pt2NFd84sY= X-Google-Smtp-Source: AMrXdXvqvrdSnu1XAVQvB7A+n2qVlNe5bS/jq4naoqsIQQyFJTTP2VUDfpcpzwsvkdk4FV0lq+Q7iw== X-Received: by 2002:a5d:6648:0:b0:2bc:805a:3e77 with SMTP id f8-20020a5d6648000000b002bc805a3e77mr17591222wrw.15.1674484930894; Mon, 23 Jan 2023 06:42:10 -0800 (PST) Received: from 127.0.0.1localhost (188.30.84.186.threembb.co.uk. [188.30.84.186]) by smtp.gmail.com with ESMTPSA id d24-20020adfa358000000b00236883f2f5csm3250534wrb.94.2023.01.23.06.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 06:42:10 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/7] io_uring: improve io_get_sqe Date: Mon, 23 Jan 2023 14:37:15 +0000 Message-Id: <9cceb11329240ea097dffef6bf0a675bca14cf42.1674484266.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Return an SQE from io_get_sqe() as a parameter and use the return value to determine if it failed or not. This lets the compiler to compile out the sqe NULL check when we know that the return SQE is valid. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 118b2fe254ba..6af11a60dc8a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2370,7 +2370,7 @@ static void io_commit_sqring(struct io_ring_ctx *ctx) * used, it's important that those reads are done through READ_ONCE() to * prevent a re-load down the line. */ -static const struct io_uring_sqe *io_get_sqe(struct io_ring_ctx *ctx) +static const bool io_get_sqe(struct io_ring_ctx *ctx, const struct io_uring_sqe **sqe) { unsigned head, mask = ctx->sq_entries - 1; unsigned sq_idx = ctx->cached_sq_head++ & mask; @@ -2388,14 +2388,15 @@ static const struct io_uring_sqe *io_get_sqe(struct io_ring_ctx *ctx) /* double index for 128-byte SQEs, twice as long */ if (ctx->flags & IORING_SETUP_SQE128) head <<= 1; - return &ctx->sq_sqes[head]; + *sqe = &ctx->sq_sqes[head]; + return true; } /* drop invalid entries */ ctx->cq_extra--; WRITE_ONCE(ctx->rings->sq_dropped, READ_ONCE(ctx->rings->sq_dropped) + 1); - return NULL; + return false; } int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) @@ -2419,8 +2420,7 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) if (unlikely(!io_alloc_req_refill(ctx))) break; req = io_alloc_req(ctx); - sqe = io_get_sqe(ctx); - if (unlikely(!sqe)) { + if (unlikely(!io_get_sqe(ctx, &sqe))) { io_req_add_to_cache(req, ctx); break; } From patchwork Mon Jan 23 14:37:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13112302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2871C38142 for ; Mon, 23 Jan 2023 14:42:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231131AbjAWOmO (ORCPT ); Mon, 23 Jan 2023 09:42:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231274AbjAWOmO (ORCPT ); Mon, 23 Jan 2023 09:42:14 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6408621A0F for ; Mon, 23 Jan 2023 06:42:13 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id r2so11015398wrv.7 for ; Mon, 23 Jan 2023 06:42:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y/gwzENu7t4nMdkMhSj5Kug78/grQEpfrseTPk5aKbs=; b=EgVyv1lhLgLuxvvizCiU1TURdoNYztMwMHfHJMD7F9smRI/4EwwOC8hlFNhG+s+xfB lNGTW61BC+DFCZwQffckOTYg2QxQUuyfK7Fq1ShmUd1FOBSDoOc5YNzgKy5dU8Cbzhoz A1/XJYQsveJ26eUU+Ll+6Ynfj2DaR27dIDWO5Cdq3fYWUkLlPKTDe0TfQm2iTpjWvBl3 ccABb+J2uxZt5vwVzthZdR5da8WuugVJCsY4zwOaHEwkkNdEAIwjcQFUAGOP21yaA8+i CR1XyluCwWfycUHNRT8xoNNTK6Axl7cc0r1wfPq73hoiINVPVJ5So6/+Mcqds9bJ9Tw+ foiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y/gwzENu7t4nMdkMhSj5Kug78/grQEpfrseTPk5aKbs=; b=zUUWKINUOLiiQ0AQEPzbk+5mnB6XcrzsGsOkoGVqMRy7x6fMwwbV6JwbbzbzoTXC3o u9MLdv8pLDhOLjlPrlnFjekbz10RbAbo/luRsb1AAX8HqZauppS0wPnQ9gGt7Pb5Ynpq S1JCmkmurVHAiv7B+y3v00OSsPKb/aUB4WSaPQSO4i5SNQM1Hs3iJHiYhCpT3jK8amVL NPmOzWQOXAmjpYHQ8A6BdxKcLGCUyB4DgqLOqe0cRu5Xz0oLKafdUOrBI0YTcgDPOipH iMP9Dpf3N16fyoL5UbyCNks95RqiRnMPIgOn16i6O1NwoMeZCS/G0VknVthndNJ9+v95 nuqg== X-Gm-Message-State: AFqh2kqaGTZp0jazyvtc3j21hd+mr4TLoLoq3eg1M9uNJDKR3Dp9ziZz WG0PYdSow2VjrTSbKO/V76TMyTRUn5k= X-Google-Smtp-Source: AMrXdXu+XmBXgUrz7ZR2LQ6Cu4wRW5Y32YTiPtT4intQdJ3rKOJruEv8Q7ZA/hJ17F+8cFKeny7Xqg== X-Received: by 2002:adf:eb8f:0:b0:2be:34fa:786b with SMTP id t15-20020adfeb8f000000b002be34fa786bmr16227834wrn.38.1674484931720; Mon, 23 Jan 2023 06:42:11 -0800 (PST) Received: from 127.0.0.1localhost (188.30.84.186.threembb.co.uk. [188.30.84.186]) by smtp.gmail.com with ESMTPSA id d24-20020adfa358000000b00236883f2f5csm3250534wrb.94.2023.01.23.06.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 06:42:11 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/7] io_uring: refactor req allocation Date: Mon, 23 Jan 2023 14:37:16 +0000 Message-Id: <8c37c2e8a3cb5e4cd6a8ae3b91371227a92708a6.1674484266.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Follow the io_get_sqe pattern returning the result via a pointer and hide request cache refill inside io_alloc_req(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 7 +++---- io_uring/io_uring.h | 19 +++++++++++-------- io_uring/notif.c | 3 +-- 3 files changed, 15 insertions(+), 14 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6af11a60dc8a..8a99791a507a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2417,9 +2417,8 @@ int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) const struct io_uring_sqe *sqe; struct io_kiocb *req; - if (unlikely(!io_alloc_req_refill(ctx))) + if (unlikely(!io_alloc_req(ctx, &req))) break; - req = io_alloc_req(ctx); if (unlikely(!io_get_sqe(ctx, &sqe))) { io_req_add_to_cache(req, ctx); break; @@ -2738,14 +2737,14 @@ static int io_eventfd_unregister(struct io_ring_ctx *ctx) static void io_req_caches_free(struct io_ring_ctx *ctx) { + struct io_kiocb *req; int nr = 0; mutex_lock(&ctx->uring_lock); io_flush_cached_locked_reqs(ctx, &ctx->submit_state); while (!io_req_cache_empty(ctx)) { - struct io_kiocb *req = io_alloc_req(ctx); - + req = io_extract_req(ctx); kmem_cache_free(req_cachep, req); nr++; } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 3ee6fc74f020..1cc6c2a8696b 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -333,16 +333,9 @@ static inline bool io_req_cache_empty(struct io_ring_ctx *ctx) return !ctx->submit_state.free_list.next; } -static inline bool io_alloc_req_refill(struct io_ring_ctx *ctx) -{ - if (unlikely(io_req_cache_empty(ctx))) - return __io_alloc_req_refill(ctx); - return true; -} - extern struct kmem_cache *req_cachep; -static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) +static inline struct io_kiocb *io_extract_req(struct io_ring_ctx *ctx) { struct io_kiocb *req; @@ -352,6 +345,16 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) return req; } +static inline bool io_alloc_req(struct io_ring_ctx *ctx, struct io_kiocb **req) +{ + if (unlikely(io_req_cache_empty(ctx))) { + if (!__io_alloc_req_refill(ctx)) + return false; + } + *req = io_extract_req(ctx); + return true; +} + static inline bool io_allowed_defer_tw_run(struct io_ring_ctx *ctx) { return likely(ctx->submitter_task == current); diff --git a/io_uring/notif.c b/io_uring/notif.c index c4bb793ebf0e..09dfd0832d19 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -68,9 +68,8 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) struct io_kiocb *notif; struct io_notif_data *nd; - if (unlikely(!io_alloc_req_refill(ctx))) + if (unlikely(!io_alloc_req(ctx, ¬if))) return NULL; - notif = io_alloc_req(ctx); notif->opcode = IORING_OP_NOP; notif->flags = 0; notif->file = NULL; From patchwork Mon Jan 23 14:37:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13112303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCCDBC54EED for ; Mon, 23 Jan 2023 14:42:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231495AbjAWOmP (ORCPT ); Mon, 23 Jan 2023 09:42:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231388AbjAWOmP (ORCPT ); Mon, 23 Jan 2023 09:42:15 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27CA322013 for ; Mon, 23 Jan 2023 06:42:14 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id y1so6525644wru.2 for ; Mon, 23 Jan 2023 06:42:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s6KX31iMQuQKOGs2Moo3zwZbCMvMmm5Kx+GJKn6/Oyk=; b=bCNuDNKWB2AxAdBBqwIboGbVhHFL8Zv/zh89b9ceGP6n4dTpmHX31eQ1WoQpv+NlSg gNpx7XdxdFFwpgv7FqZGpdCSvBMGvM+4uk3ne4aEs/W1qpVjN+FteMtm8CNKRh6c6eRv wrn8GM85wbUWKCe8/KeWeU4J0aQ0TXStwTtP8kjZ6H9DU01edJdKpr/glJZx6cPtaYNM 7G3NpVzeUglNp51NbpTbE8HkadZRwhi2/vFSsVGqPWtteYiY6vSf5YKLXiWTNnbdg7kd UZKmFLQ3rWpyoCbs7Aax8ScJkHg5EqJEZOiUVFz0FCTd+cD5ABi0GqlKVXbBZLXwkMH1 DYyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s6KX31iMQuQKOGs2Moo3zwZbCMvMmm5Kx+GJKn6/Oyk=; b=od/wov9Yv4y1iYxLB8WGYOWVrUeVKqZI0skSWpt1QowgfivqOiKjfb8D931Z/kS+CJ nqjSjxUekQvKidvhwVAeXJOmW45TzQXhkm9BZL6PR1rxKWTOc405Pfd3jYRuoXyQyEhe gYg12zqYY6TZ8aLOCdX2vwH4Fxrwo5Vbq6qpTu2HBkKCF4EaZKEQfO8HWxZIePGm7iXQ YOXCD8vQl0/ypBF7gzpWIrJOLgA1VR5wY3YpufHucIWMIRxAYVYmHcHiQYN5bNBspN2Z +R0DlBIkTtSd2QA+kSVGv59gh+CHR46eSI0E2vOG3aqVXmbrKajNqCFuvz/eAqz7TPTv LwrA== X-Gm-Message-State: AFqh2koPTQmLRvSV58voqmy0dHvClgikmQbc42nbyc55k6F7c33hVJ9v Cxn71nu046RpxdildxulHWFpbKXI6Os= X-Google-Smtp-Source: AMrXdXvoiCdlwlugFbv6NzlT/gXD4L3KrPW8XyyAg0OAA4poQhofOAyG3PISd4sK+ztORqE0tLGVzg== X-Received: by 2002:a5d:660a:0:b0:2be:5162:c8da with SMTP id n10-20020a5d660a000000b002be5162c8damr13248432wru.16.1674484932548; Mon, 23 Jan 2023 06:42:12 -0800 (PST) Received: from 127.0.0.1localhost (188.30.84.186.threembb.co.uk. [188.30.84.186]) by smtp.gmail.com with ESMTPSA id d24-20020adfa358000000b00236883f2f5csm3250534wrb.94.2023.01.23.06.42.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 06:42:12 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/7] io_uring: refactor io_put_task helpers Date: Mon, 23 Jan 2023 14:37:17 +0000 Message-Id: <3bf92ebd594769d8a5d648472a8e335f2031d542.1674484266.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a helper for putting refs from the target task context, rename __io_put_task() and add a couple of comments around. Use the remote version for __io_req_complete_post(), the local is only needed for __io_submit_flush_completions(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8a99791a507a..faada7e76f2d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -713,7 +713,8 @@ static void io_cqring_overflow_flush(struct io_ring_ctx *ctx) io_cqring_do_overflow_flush(ctx); } -static void __io_put_task(struct task_struct *task, int nr) +/* can be called by any task */ +static void io_put_task_remote(struct task_struct *task, int nr) { struct io_uring_task *tctx = task->io_uring; @@ -723,13 +724,19 @@ static void __io_put_task(struct task_struct *task, int nr) put_task_struct_many(task, nr); } +/* used by a task to put its own references */ +static void io_put_task_local(struct task_struct *task, int nr) +{ + task->io_uring->cached_refs += nr; +} + /* must to be called somewhat shortly after putting a request */ static inline void io_put_task(struct task_struct *task, int nr) { if (likely(task == current)) - task->io_uring->cached_refs += nr; + io_put_task_local(task, nr); else - __io_put_task(task, nr); + io_put_task_remote(task, nr); } void io_task_refs_refill(struct io_uring_task *tctx) @@ -982,7 +989,7 @@ static void __io_req_complete_post(struct io_kiocb *req) * we don't hold ->completion_lock. Clean them here to avoid * deadlocks. */ - io_put_task(req->task, 1); + io_put_task_remote(req->task, 1); wq_list_add_head(&req->comp_list, &ctx->locked_free_list); ctx->locked_free_nr++; } @@ -1105,7 +1112,7 @@ __cold void io_free_req(struct io_kiocb *req) io_req_put_rsrc(req); io_dismantle_req(req); - io_put_task(req->task, 1); + io_put_task_remote(req->task, 1); spin_lock(&ctx->completion_lock); wq_list_add_head(&req->comp_list, &ctx->locked_free_list); From patchwork Mon Jan 23 14:37:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13112304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5098AC54E94 for ; Mon, 23 Jan 2023 14:42:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231388AbjAWOmQ (ORCPT ); Mon, 23 Jan 2023 09:42:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231492AbjAWOmP (ORCPT ); Mon, 23 Jan 2023 09:42:15 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E87AF22014 for ; Mon, 23 Jan 2023 06:42:14 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id z5so11024910wrt.6 for ; Mon, 23 Jan 2023 06:42:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6VtTZipjq85PArcJmx2dGRRWjObu2HyoKvfr/TFfiL0=; b=KEgZ0DVEvN6Ngrn0bDPGB8CkY3p/dDsOY1LgnK1IluZ9rynV7zHlqWLgp3cMHMp+wt JJN59SCieovluRG/4Hl4wHgIh8LIigpZ7/NTXVDB6F2Fp07Zoq4jZzGga9Cs1RerZ+Q5 9h+hTdFb5Do3GsJqDG/VztaKzCJiWlqnMmi5QujPlBTPQBewBhOPXmOCCQG5HZcCc1On nwJDVc56p4tpG3a7BoMS2bI4YT44bFnyUOirDkPbxOhoHNPlU028hT1b51/u2s2vwYUw iGV/WZZ77Uxngs7Q01CbHJkfkV6IO88R20KZ7q72R801eO9Xox6uTA+dihLTyqyzgB7g KWvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6VtTZipjq85PArcJmx2dGRRWjObu2HyoKvfr/TFfiL0=; b=KlCnqLwHZBs1bK+ZLmSDG9r4E+MD2TeBd9QTiU2OS2g8T/ARc4HPLAtUTWQpXYwK88 1spfCTfHQf6dxapyArM3l+kfNhRRYPR+DFk7KbBt4zamLdkR+DtkNwuZKh8KQajp3H73 RXEarMaD0rzBvaVvuwvOnyV3bzLLJ1Ksnkkw9dygdbpoOr5psbZwyL1CM+q44zTmBTW7 uGfRTb0Bl5Nkx3Au0VA7X10Vmr68X1WqngHOmaK5ggG82+ZTmLK9ygdcEVv4PyzW/29i fMRFyVGQMbSe5Zvf/nIszS5L6a3tLJcq94z6UbDmzJjpzG9kfzAhZ4Hf+rhowLAD6Y4L giVg== X-Gm-Message-State: AFqh2kq4ij1ZS1O4y54xKuCKhVxwNl3rKDW4RnqEKJc9ZbDTIpiJF8Mp SU23E3B6DzOlq8fFYz0oZdcR5vwWyg8= X-Google-Smtp-Source: AMrXdXubM2D/BB3AOZSWpGyRU3uju+5W4twFpuJxvJdiBnd8MkGACNn6SVcYuG5DWWdz43LJEjS7Pw== X-Received: by 2002:a05:6000:689:b0:2bd:ffad:1bce with SMTP id bo9-20020a056000068900b002bdffad1bcemr24442189wrb.59.1674484933305; Mon, 23 Jan 2023 06:42:13 -0800 (PST) Received: from 127.0.0.1localhost (188.30.84.186.threembb.co.uk. [188.30.84.186]) by smtp.gmail.com with ESMTPSA id d24-20020adfa358000000b00236883f2f5csm3250534wrb.94.2023.01.23.06.42.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 06:42:13 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 6/7] io_uring: refactor tctx_task_work Date: Mon, 23 Jan 2023 14:37:18 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Merge almost identical sections of tctx_task_work(), this will make code modifications later easier and also inlines handle_tw_list(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index faada7e76f2d..586e70f686ce 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1166,7 +1166,7 @@ static unsigned int handle_tw_list(struct llist_node *node, { unsigned int count = 0; - while (node != last) { + while (node && node != last) { struct llist_node *next = node->next; struct io_kiocb *req = container_of(node, struct io_kiocb, io_task_work.node); @@ -1226,7 +1226,7 @@ void tctx_task_work(struct callback_head *cb) task_work); struct llist_node fake = {}; struct llist_node *node; - unsigned int loops = 1; + unsigned int loops = 0; unsigned int count; if (unlikely(current->flags & PF_EXITING)) { @@ -1234,15 +1234,12 @@ void tctx_task_work(struct callback_head *cb) return; } - node = io_llist_xchg(&tctx->task_list, &fake); - count = handle_tw_list(node, &ctx, &uring_locked, NULL); - node = io_llist_cmpxchg(&tctx->task_list, &fake, NULL); - while (node != &fake) { + do { loops++; node = io_llist_xchg(&tctx->task_list, &fake); count += handle_tw_list(node, &ctx, &uring_locked, &fake); node = io_llist_cmpxchg(&tctx->task_list, &fake, NULL); - } + } while (node != &fake); ctx_flush_and_put(ctx, &uring_locked); From patchwork Mon Jan 23 14:37:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13112305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE89DC05027 for ; Mon, 23 Jan 2023 14:42:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231492AbjAWOmR (ORCPT ); Mon, 23 Jan 2023 09:42:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231274AbjAWOmP (ORCPT ); Mon, 23 Jan 2023 09:42:15 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4DD121A0F for ; Mon, 23 Jan 2023 06:42:14 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id d14so7338802wrr.9 for ; Mon, 23 Jan 2023 06:42:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SCWbGwBl576a0ArFhLpktKBP44fCtoi8yPjJSutd26c=; b=Ru+NvOrZat8Q4naKOvtZZE5eIJ0xQ0ktc27fRXyv1tWCfYZEeL6BwWHbFc5XMxDK8L Fvw/NETP+ei3HtMs11TOtLptJniwvp6fv/oslYqvT8wGjWXB+Vik+s/sHyMU4LgtqDOL RGy6FZU/M9yRMeOBq2VGEgrZ2KCsn06xEusVTRSo8Ej9+dWLmAWr6w9K/a8/JASo62HT 3ctXv8rQiFFCgUM+XeEiOLqN4n10h6S3eYFLt3yui6StqDcO/9GIcXE6PymWrImJe8ut FOFdw7PN3BSEizvYg3bDzPqJ8keFFN7Ju3dykLB8YmIGR9QEIq1vzINXJoZQSQkCS1AR Tcew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SCWbGwBl576a0ArFhLpktKBP44fCtoi8yPjJSutd26c=; b=N8PPRsK7zEBKuxXEEprCpJbg7Zaot9/Sqr/dHHwk3LzTysok0MM7W3lsf1NQolMFr1 GQvjCat+NBQxXTS6C7f5YxlctqXPs3k+OsIhmM+Xp/5EcFhAeNIoDi4R4DnA19kgEa1O 5fRIVX83vM9W96h0kkCk7YZoY7qWbNxez1wBwOxDNNUu1HqQyw4mdB6FTVcfjs/WMKQO p1WFY9La42ezDpyA72833NWxtyGla661ZKGVMiN/TWmiPl8oysFVjTOuG1t/pfSOSHzI jcygCXgR00eRjF0VVsaGdkfJTdbBa+34XNoAbxr6x+5yUMNUxDzxa+71j9+9n5u6xmyK 1e2w== X-Gm-Message-State: AFqh2kpzIwdVRy78d3pRcaxsivYtSc/vGIAdg/eDufYdFHfoW9QwF69l 6fmza6GYNvazjO0FWGudPZMfYUgU8bw= X-Google-Smtp-Source: AMrXdXuvIa4UQlMVSHG3GUuF92gxi34AWjhovyqIk3PSEPm1SxEdb02kvaSX6HTAaq1kJrnL16f3/g== X-Received: by 2002:adf:f7c5:0:b0:2be:5a87:4e5 with SMTP id a5-20020adff7c5000000b002be5a8704e5mr10641777wrq.12.1674484934079; Mon, 23 Jan 2023 06:42:14 -0800 (PST) Received: from 127.0.0.1localhost (188.30.84.186.threembb.co.uk. [188.30.84.186]) by smtp.gmail.com with ESMTPSA id d24-20020adfa358000000b00236883f2f5csm3250534wrb.94.2023.01.23.06.42.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 06:42:13 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 7/7] io_uring: return normal tw run linking optimisation Date: Mon, 23 Jan 2023 14:37:19 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_submit_flush_completions() may produce new task_work items, so it's a good idea to recheck the task_work list after flushing completions. The optimisation is not new and was accidentially removed by f88262e60bb9 ("io_uring: lockless task list") Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 586e70f686ce..8c4d92e64c20 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1238,6 +1238,15 @@ void tctx_task_work(struct callback_head *cb) loops++; node = io_llist_xchg(&tctx->task_list, &fake); count += handle_tw_list(node, &ctx, &uring_locked, &fake); + + /* skip expensive cmpxchg if there are items in the list */ + if (READ_ONCE(tctx->task_list.first) != &fake) + continue; + if (uring_locked && !wq_list_empty(&ctx->submit_state.compl_reqs)) { + io_submit_flush_completions(ctx); + if (READ_ONCE(tctx->task_list.first) != &fake) + continue; + } node = io_llist_cmpxchg(&tctx->task_list, &fake, NULL); } while (node != &fake);