From patchwork Tue Jun 14 14:36:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68FA2C43334 for ; Tue, 14 Jun 2022 14:37:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236475AbiFNOho (ORCPT ); Tue, 14 Jun 2022 10:37:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238080AbiFNOhm (ORCPT ); Tue, 14 Jun 2022 10:37:42 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63B75CE22 for ; Tue, 14 Jun 2022 07:37:41 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id o8so11589460wro.3 for ; Tue, 14 Jun 2022 07:37:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ORixVfQAPkrUEdbPULkgFBU4hdADQl3LSugR9FT2k0U=; b=qzWm0g5k1+Lp3SK0gLG+mGrPvOZWCa4HtZgLT9oN2eVm1W7ir3q9AwVT29ypOow4vn Mwsk45+VnZ/2Ow9I0fimSye+9FM8HumrJp3DaHEuz3EyMlXeBWzSnSoDdIbvVBjjl0I5 hUHpNcbWdg0HunoNBsMeKNyhlQmOTd+ch3ZWj14IciCYELc8Z+D0hI1U/YmvkYPpLEVK DTgn+xdR39DK08E2Y3h7mE11jmmhxI904iGlamn6VAlR0zD75km+5HJfOPzbao3K0vDq vSYIsvZptmQjFPpPKaKdEhPIhNJOFTpVXgpEVsf/bYKv7a6kSZLu0EsPr7ritD0O416h 9AEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ORixVfQAPkrUEdbPULkgFBU4hdADQl3LSugR9FT2k0U=; b=Ya7bmeds5P9gf4uLekWTs+2nyaLD8Y8Ra5MKJzx50r+8RcolpTTpXlRk0TP1glgpHL u4ZuVEsuVjVUg1H+iWjTwZdvWMsGvNYSm1TugfSGo0jVXHnT2vj3DSOXxX5GivXUZMVo xQe6l6CPfsvFtTWPvDLx29pOP8emifnaEzaJX4yAqv/kPttntsuJW2P7oyimSyh7+YJ0 UUMmWo9gyZ5zTQZ3O5N8Y23KJO7LbCaVSC/rsC4S4kdko7UIFTE8DxN43UAW7GvIHBmi enp+Ih21KzrIPtEp/4u3gviQCG15X5cuGMznfbmz28xHvO4PRGzbJMKo/qh5w+G9dr3S LkbA== X-Gm-Message-State: AJIora8KhG23MSpz5amuaOOlEpCZ19CbLUNCzfKLS6upQXlwSpbVZzD2 4UvcZsn2EUxh99EaoCc5EwnEAPECj5t0OQ== X-Google-Smtp-Source: AGRyM1vXyzScAKkIhSa37u2dY357/CDwB46xMHCFkAKy5GxIqyd2LZgJmGzeLB3R4HTIYGTis3NpVQ== X-Received: by 2002:a05:6000:988:b0:219:cb95:79c4 with SMTP id by8-20020a056000098800b00219cb9579c4mr5299454wrb.587.1655217459589; Tue, 14 Jun 2022 07:37:39 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 01/25] io_uring: make reg buf init consistent Date: Tue, 14 Jun 2022 15:36:51 +0100 Message-Id: <1835f8612fd77ed79712f566778cad6691d41c06.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The default (i.e. empty) state of register buffer is dummy_ubuf, so set it to dummy on init instead of NULL. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index fef46972c327..fd1323482030 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -567,7 +567,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, io_buffer_unmap(ctx, &imu); break; } - ctx->user_bufs[i] = NULL; + ctx->user_bufs[i] = ctx->dummy_ubuf; needs_switch = true; } @@ -1200,14 +1200,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, size_t size; int ret, nr_pages, i; - if (!iov->iov_base) { - *pimu = ctx->dummy_ubuf; + *pimu = ctx->dummy_ubuf; + if (!iov->iov_base) return 0; - } - *pimu = NULL; ret = -ENOMEM; - pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, &nr_pages); if (IS_ERR(pages)) { From patchwork Tue Jun 14 14:36:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39983CCA47C for ; Tue, 14 Jun 2022 14:37:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231584AbiFNOhp (ORCPT ); Tue, 14 Jun 2022 10:37:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235644AbiFNOho (ORCPT ); Tue, 14 Jun 2022 10:37:44 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 972A811443 for ; Tue, 14 Jun 2022 07:37:42 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id o16so11586834wra.4 for ; Tue, 14 Jun 2022 07:37:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ToujjDMrgxjXImVoVU6hciJcYYHsx0thIW3gWEU5SEA=; b=HE3wDBPv0/eb+nDpl+5ucOakSdPeonehkWrbXi0ZxX0nK2AYMnOtB8El2y+P3Mnimp Rbbx1cXwqsyAtFN+hCBqka5TsoRi/B3OV9RmZRwRJsib46r5gb5zMO0dUhicnqMQoMZ7 99N1+wwT1IW4lQG1JR6LSHN6xE/zLrT+oheZiDNsQQm9FJVlWSmqj5yxlVY5wJQyvdn7 6mRk9DEcPLeTfhKFB3eDtJRZQNlOFmWDpOH4ibpUF+o+QRbM4wNHJypUTlF+2jjpKiVM dCWqqIP1qoSHfyCd/7KxYCecztCXwofIXlGIXeBhBnbFM9cW8UJi+GS48NqicbpLX4IT 1isw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ToujjDMrgxjXImVoVU6hciJcYYHsx0thIW3gWEU5SEA=; b=5xaBCf4g3fehVVcmWaA28IBeA3krAZK82ekTyr4X8pPbm7/ftpF0DwKgtsKxzfJy7c viR+UwaRfoE6bn01re4Jw3ysWy6LhizUz1GvgVrD6WuoxJzorzX59CuyJzNlaVteXnkb 3eJUlAJSDzX1+mePKA9aiv/UVbL3kMauEym8aEEv/DDD/EDOFC008z1BfetEvevGxrGA 6UHoVEKA7xT7xdYEWoCCr9Y+29RUvfM8JZQ7N8NsAz8+9BLThoOhmMxHHtAqKgU0c8cg OKPFJ9SaYGtD1oSJ4a1YnEIw7CrH6RCsp8hUAFNPmJzE4RuoDdtuND7XoX7Gnjxic3+/ arIQ== X-Gm-Message-State: AJIora8UNUg8JPqYP3KreMZzzxXH/9tB8PppuH5jlh4g2epaLgKvwqCZ r+TDv9QQLRySf0Eg1bvsKTzOQ/apsuGPjw== X-Google-Smtp-Source: AGRyM1vwI+6ddkJNb98+NqsWs0EuB06M3u4vhEeSxMtMjBo2qHIvUib5s1i5iqH9w2rqvh2g3P5mFQ== X-Received: by 2002:a05:6000:1812:b0:210:2eb1:4606 with SMTP id m18-20020a056000181200b002102eb14606mr5409646wrh.593.1655217460710; Tue, 14 Jun 2022 07:37:40 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 02/25] io_uring: move defer_list to slow data Date: Tue, 14 Jun 2022 15:36:52 +0100 Message-Id: <097c9f3eb3f8ab052e10160567928ec8df0f1303.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org draining is slow path, move defer_list to the end where slow data lives inside the context. Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 7c22cf35a7e2..52e91c3df8d5 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -160,7 +160,6 @@ struct io_ring_ctx { struct io_uring_sqe *sq_sqes; unsigned cached_sq_head; unsigned sq_entries; - struct list_head defer_list; /* * Fixed resources fast path, should be accessed only under @@ -272,8 +271,12 @@ struct io_ring_ctx { struct work_struct exit_work; struct list_head tctx_list; struct completion ref_comp; + + /* io-wq management, e.g. thread count */ u32 iowq_limits[2]; bool iowq_limits_set; + + struct list_head defer_list; }; }; From patchwork Tue Jun 14 14:36:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 395F2C433EF for ; Tue, 14 Jun 2022 14:37:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235644AbiFNOhr (ORCPT ); Tue, 14 Jun 2022 10:37:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237898AbiFNOhp (ORCPT ); Tue, 14 Jun 2022 10:37:45 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B68101147A for ; Tue, 14 Jun 2022 07:37:43 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id x17so11577589wrg.6 for ; Tue, 14 Jun 2022 07:37:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=43ATD8fKnmjyJdZ2R6qxl1QYnz/Icg31elC1cRBKr+w=; b=dWSJcQy861faEAMob54R9WAvsnQs2gN2exzOxO9MfDV5fZ/Q4hoyzssXere454eLQi It3xv6q0Vl0aNw9vhLM38zpvZdoS8zNKawymC44is2WpJnLoNe8r8u92erbikD7EQWaM uldltSzhgrgwkdRonSejQ0L3CMnlWLYj/KxLrCBI6QyDFZg6nL6YQGXh8ydcLTz3xQin P5SSNF5kpr6kCHFvAzLj8T8cSLkE90c1KwEI/OMDupPVMtARaHg1smfIvsitwHOW0G5h fh5tu0F53ewQynV8MIXMTZY6r6Um/PvMo0MMm0Mlb3kWdt/W2a7bSzo2sIl11ATUiD1d dYgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=43ATD8fKnmjyJdZ2R6qxl1QYnz/Icg31elC1cRBKr+w=; b=lASKtwL4vV9csIhhAYARouOZmqE+Cvv8XCE1qevjuuvlO0S/LnEWLFxPBzjujjdoHz 7s5FuSjYejTabw/SEMqPlLruOqPgow9Me9YPTBVzVD5GljayWuHcfUs0lYTDSIYGirR0 o4ZF9BFt4jCyTlfSPhAc+j6SOjpqHAJmyNiwDkqTbt7ql2cSwp0BRkYlYy+lZWlHx8Hl yL7hqkP+reLWga32PnHy9ZTt2A0Xlzi5LLhssZAU//+3Z0T7NQne2lHEgAfDSbwX4p3D WK9C7htYOQvM/7p6bAgnOP2/fxAIFb1pcKgfMUItE0HiO0di1960U3II978VGkVxtY9d fcmw== X-Gm-Message-State: AJIora9JgLr2k3m5P76iNXZDssb1u/JnavZvu9hjC1tH6i0+Sp3OR2Hn b1sptIL+47eteCu62lkN5c1Dc6QeZw95SA== X-Google-Smtp-Source: AGRyM1vNQNKrvWhUjOitvg2CNSvKDDpY7+NbodWCJh3jGKqGG0zYlC9L+5kG0fEu4kMhS+f6OJat0g== X-Received: by 2002:a5d:55ca:0:b0:211:4092:1c27 with SMTP id i10-20020a5d55ca000000b0021140921c27mr5404546wrw.108.1655217462025; Tue, 14 Jun 2022 07:37:42 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 03/25] io_uring: better caching for ctx timeout fields Date: Tue, 14 Jun 2022 15:36:53 +0100 Message-Id: <51f0bc096197ab07fbc54b975dafbd22a31a634c.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Following timeout fields access patterns, move all of them into a separate cache line inside ctx, so they don't intervene with normal completion caching, especially since timeout removals and completion are separated and the later is done via tw. It also sheds some bytes from io_ring_ctx, 1216B -> 1152B Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 52e91c3df8d5..4f52dcbbda56 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -179,8 +179,6 @@ struct io_ring_ctx { struct xarray io_bl_xa; struct list_head io_buffers_cache; - struct list_head timeout_list; - struct list_head ltimeout_list; struct list_head cq_overflow_list; struct list_head apoll_cache; struct xarray personalities; @@ -213,15 +211,11 @@ struct io_ring_ctx { struct io_ev_fd __rcu *io_ev_fd; struct wait_queue_head cq_wait; unsigned cq_extra; - atomic_t cq_timeouts; - unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; struct { spinlock_t completion_lock; - spinlock_t timeout_lock; - /* * ->iopoll_list is protected by the ctx->uring_lock for * io_uring instances that don't use IORING_SETUP_SQPOLL. @@ -253,6 +247,15 @@ struct io_ring_ctx { struct list_head io_buffers_pages; }; + /* timeouts */ + struct { + spinlock_t timeout_lock; + atomic_t cq_timeouts; + struct list_head timeout_list; + struct list_head ltimeout_list; + unsigned cq_last_tm_flush; + } ____cacheline_aligned_in_smp; + /* Keep this last, we don't need it for the fast path */ struct { #if defined(CONFIG_UNIX) From patchwork Tue Jun 14 14:36:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD234C43334 for ; Tue, 14 Jun 2022 14:37:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234697AbiFNOhr (ORCPT ); Tue, 14 Jun 2022 10:37:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238080AbiFNOhq (ORCPT ); Tue, 14 Jun 2022 10:37:46 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F7EA17593 for ; Tue, 14 Jun 2022 07:37:45 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id i81-20020a1c3b54000000b0039c76434147so117423wma.1 for ; Tue, 14 Jun 2022 07:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=D8sFDmp4NCj8bAwGZ38Oxd60P7D2GVHJ5ZCu/JRHls8=; b=JQ2rUzbbr6ynGLDV1KXga1yA3VViPP0IkuWyTwsxl1m9u2O6ZXLFBm+sdNI5E/Eyng 2SMcLPBd5kS8UHwOm4sLwQ/QxN8F2vZx5M67y8g+LmDBmuE+T6fF85rVzKgT+Rgm10J5 lKv3wuKvO2UjuOhFiRTax9MlU+loKoJmNFzK3qS9ahmcGlT7l4MsJxM1jcddRkziHx3c DuaoJwgJLSuuDKU3zYHJfAGwMJJIL2RH8KFaXB1UnMu+TEbs8KWBOxiYbVV3vGRhIdZj mMg+NAy1xFjVvJTRKaTMEMBO+mmt/+HD7FuWeGTXcB1ukJ484uMmWo9L03AOPXUHWAQM 5MhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D8sFDmp4NCj8bAwGZ38Oxd60P7D2GVHJ5ZCu/JRHls8=; b=gQb+N3WRNyB9dG+qKbuhKIoWxKrbYDxeJ8shw97z7M9V57U069APqmcN9KQmp6QE/K Y2V1oJW9lexDujHbth4ChQlUnMEOSrnvcLuAZZjvOLcAlxgYO9qR3EB4JFkZ3C+o6ICh MX3Yr18D2kvRqcV2FgDqtQfYRoPGDGhVNK419CasYA0YAYcpCfUSkKPYZOPESg4Wpqrq fcgT4lNLxmzVCdVsCQdnChikhOk1LcRLcH+gyWuAWifpl9fmSwsDUYtiTTAxAshObvwF HAyFbdIk8uzHJpgSwWT9TTx4ocquuQ3tTHZSAnLjUrU9IwaQQSOugsDaCaA3TsRN2XkW aRFw== X-Gm-Message-State: AOAM5328vlOFMULu2GtZ/bn+0KUUQ7CXLTLpXDe+gIaAXfZrRLNK+FdC IjIektou4s2ES+ZAo95ru0azi/GG+SEYTw== X-Google-Smtp-Source: ABdhPJxBzELWc5rfW1QCUhZqBMPfwwwSPUPSZGvdm656ldCCjLnZj+4UJ9sySXUJITYHq5SBYmqMPw== X-Received: by 2002:a05:600c:35d2:b0:397:84c9:6fe8 with SMTP id r18-20020a05600c35d200b0039784c96fe8mr4499873wmq.206.1655217463376; Tue, 14 Jun 2022 07:37:43 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 04/25] io_uring: refactor ctx slow data placement Date: Tue, 14 Jun 2022 15:36:54 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Shove all slow path data at the end of ctx and get rid of extra indention. Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 81 +++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 42 deletions(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 4f52dcbbda56..ca8e25992ece 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -183,7 +183,6 @@ struct io_ring_ctx { struct list_head apoll_cache; struct xarray personalities; u32 pers_next; - unsigned sq_thread_idle; } ____cacheline_aligned_in_smp; /* IRQ completion list, under ->completion_lock */ @@ -230,23 +229,6 @@ struct io_ring_ctx { struct list_head io_buffers_comp; } ____cacheline_aligned_in_smp; - struct io_restriction restrictions; - - /* slow path rsrc auxilary data, used by update/register */ - struct { - struct io_rsrc_node *rsrc_backup_node; - struct io_mapped_ubuf *dummy_ubuf; - struct io_rsrc_data *file_data; - struct io_rsrc_data *buf_data; - - struct delayed_work rsrc_put_work; - struct llist_head rsrc_put_llist; - struct list_head rsrc_ref_list; - spinlock_t rsrc_ref_lock; - - struct list_head io_buffers_pages; - }; - /* timeouts */ struct { spinlock_t timeout_lock; @@ -257,30 +239,45 @@ struct io_ring_ctx { } ____cacheline_aligned_in_smp; /* Keep this last, we don't need it for the fast path */ - struct { - #if defined(CONFIG_UNIX) - struct socket *ring_sock; - #endif - /* hashed buffered write serialization */ - struct io_wq_hash *hash_map; - - /* Only used for accounting purposes */ - struct user_struct *user; - struct mm_struct *mm_account; - - /* ctx exit and cancelation */ - struct llist_head fallback_llist; - struct delayed_work fallback_work; - struct work_struct exit_work; - struct list_head tctx_list; - struct completion ref_comp; - - /* io-wq management, e.g. thread count */ - u32 iowq_limits[2]; - bool iowq_limits_set; - - struct list_head defer_list; - }; + + struct io_restriction restrictions; + + /* slow path rsrc auxilary data, used by update/register */ + struct io_rsrc_node *rsrc_backup_node; + struct io_mapped_ubuf *dummy_ubuf; + struct io_rsrc_data *file_data; + struct io_rsrc_data *buf_data; + + struct delayed_work rsrc_put_work; + struct llist_head rsrc_put_llist; + struct list_head rsrc_ref_list; + spinlock_t rsrc_ref_lock; + + struct list_head io_buffers_pages; + + #if defined(CONFIG_UNIX) + struct socket *ring_sock; + #endif + /* hashed buffered write serialization */ + struct io_wq_hash *hash_map; + + /* Only used for accounting purposes */ + struct user_struct *user; + struct mm_struct *mm_account; + + /* ctx exit and cancelation */ + struct llist_head fallback_llist; + struct delayed_work fallback_work; + struct work_struct exit_work; + struct list_head tctx_list; + struct completion ref_comp; + + /* io-wq management, e.g. thread count */ + u32 iowq_limits[2]; + bool iowq_limits_set; + + struct list_head defer_list; + unsigned sq_thread_idle; }; enum { From patchwork Tue Jun 14 14:36:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AB67C43334 for ; Tue, 14 Jun 2022 14:37:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237898AbiFNOhs (ORCPT ); Tue, 14 Jun 2022 10:37:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237722AbiFNOhr (ORCPT ); Tue, 14 Jun 2022 10:37:47 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DE5217A9D for ; Tue, 14 Jun 2022 07:37:46 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id o37-20020a05600c512500b0039c4ba4c64dso6346536wms.2 for ; Tue, 14 Jun 2022 07:37:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XgZAW0m4+0LZD+WHW8dXUET6dvmfsAmhLr96GJWzonE=; b=C+OW6ff8rC8QaFOQt9W94DfNOMVOwIf8JAioNFBdBCJL3vyoiTNMy8d5kfQN0AP4dP ZYfjsxAYp/fgL0u63Mdzb4Tu/Feea67touwH6maVVqAZAq7bf7pKyYn4DSpBbOhkV9Lc jV+cSkF2NGptc4QwlRZij//tsGPvo87B+3AdZ6r4PPmkGSjXyUIxdycLahyByOXS9BLi nwKeIaVSGxXrFqK9wvzIrdNKptYxUDrU3WXnf1AFrTWMxE1pk+2Mg4F9rLFFXLogdljs LirbajDySVGF48wvZvMva/tCqEnzViXjk39c0JQLm3hTExkmJSeKOP5LrCjxglqQhGWJ NP2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XgZAW0m4+0LZD+WHW8dXUET6dvmfsAmhLr96GJWzonE=; b=fVc3QYc8v7tPvkUCzeSe/rkiIIwPF4/oYZQaqNbQiTYRmIrokMTJruy4lXpLKUUax2 BKRyx7vg5CIZeIQiLqDOYJdpmx11a1znRIIBu/uRHCV4sUxTicv1uzv4r195AWOhKlq/ i611E4iwjNI9DgyxRgROeB+CISaZmv8mAgeifml3NMqkF/lCccS9ByyStZlUR2Gt4jVH Y3ILB9mNgrQPw80bQnV8zbXH20EmtiUf8t2RzL9ZSkdh9kwZ4He8q2WcT4nCq8PYSGWz afyagWFU80MwT7lrobYZ1XoqfA3oGnN/EOUP3jzCBKevcR9BugaadpU9ZVF9byPKJpFH vMwg== X-Gm-Message-State: AOAM533YXMjRnb6VyF97lhu0HIOdDftS8z0cFkn/cpfysqEQaYsLKV+D H2aYogZgZS5N4dRPOXB7iNk+MOVUv36S9Q== X-Google-Smtp-Source: ABdhPJzzKVcfVvLKKIl5DMzNFmCCmHx1KRg/sG1ER/h4eH3lqsZX5MoF/3sw6/gCSMfZWpjOW1snvg== X-Received: by 2002:a05:600c:3588:b0:39c:87f1:b31 with SMTP id p8-20020a05600c358800b0039c87f10b31mr4603447wmq.4.1655217464603; Tue, 14 Jun 2022 07:37:44 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 05/25] io_uring: move small helpers to headers Date: Tue, 14 Jun 2022 15:36:55 +0100 Message-Id: <7bef8769a1f74a0af7c45f8c20d2fb2a6abc9f89.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is a bunch of inline helpers that will be useful not only to the core of io_uring, move them to headers. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 22 ---------------------- io_uring/io_uring.h | 22 ++++++++++++++++++++++ 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6a94d1682aaf..3fdb368820c9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -616,14 +616,6 @@ struct sock *io_uring_get_socket(struct file *file) } EXPORT_SYMBOL(io_uring_get_socket); -static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) -{ - if (!*locked) { - mutex_lock(&ctx->uring_lock); - *locked = true; - } -} - static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { if (!wq_list_empty(&ctx->submit_state.compl_reqs)) @@ -879,15 +871,6 @@ static void io_prep_async_link(struct io_kiocb *req) } } -static inline void io_req_add_compl_list(struct io_kiocb *req) -{ - struct io_submit_state *state = &req->ctx->submit_state; - - if (!(req->flags & REQ_F_CQE_SKIP)) - state->flush_cqes = true; - wq_list_add_tail(&req->comp_list, &state->compl_reqs); -} - void io_queue_iowq(struct io_kiocb *req, bool *dont_use) { struct io_kiocb *link = io_prep_linked_timeout(req); @@ -1293,11 +1276,6 @@ static void io_req_complete_post32(struct io_kiocb *req, u64 extra1, u64 extra2) io_cqring_ev_posted(ctx); } -static inline void io_req_complete_state(struct io_kiocb *req) -{ - req->flags |= REQ_F_COMPLETE_INLINE; -} - inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) { if (issue_flags & IO_URING_F_COMPLETE_DEFER) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 3660df80e589..26b669746d61 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -193,6 +193,28 @@ static inline bool io_run_task_work(void) return false; } +static inline void io_req_complete_state(struct io_kiocb *req) +{ + req->flags |= REQ_F_COMPLETE_INLINE; +} + +static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) +{ + if (!*locked) { + mutex_lock(&ctx->uring_lock); + *locked = true; + } +} + +static inline void io_req_add_compl_list(struct io_kiocb *req) +{ + struct io_submit_state *state = &req->ctx->submit_state; + + if (!(req->flags & REQ_F_CQE_SKIP)) + state->flush_cqes = true; + wq_list_add_tail(&req->comp_list, &state->compl_reqs); +} + int io_run_task_work_sig(void); void io_req_complete_failed(struct io_kiocb *req, s32 res); void __io_req_complete32(struct io_kiocb *req, unsigned int issue_flags, From patchwork Tue Jun 14 14:36:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C786C433EF for ; Tue, 14 Jun 2022 14:37:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241187AbiFNOhu (ORCPT ); Tue, 14 Jun 2022 10:37:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238290AbiFNOht (ORCPT ); Tue, 14 Jun 2022 10:37:49 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94F2D1147A for ; Tue, 14 Jun 2022 07:37:47 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id m39-20020a05600c3b2700b0039c511ebbacso6337874wms.3 for ; Tue, 14 Jun 2022 07:37:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pRvBVOE/hkiozWmaa4rfYGZwNs1AXWCtzZOpvsa5jls=; b=KW5wgCEIRCTTEW48+U0V70LgeRyFAPjmEisiMvWe48l36sobA9osW9ESPXDRoCgfra 54PqBo73Az5H+BrYGGVBC4gwSsTiP2Ilnp7ccq3rgHG1slYQAgQYPm2XeW1CLYQOmXfC PzeBcyCo+WRF2f9mq7EpO9NOMiSmNauQdpPDMEOm65PTBNp0cTeMUeuQCztUqk1TUHhE E22uZAqdN16n/LgrWfhUHiR8xeE3HxSzvt0yhv1lITzJTJyNiKBz7FJlmN/Dht0CcxEd BqH28LhdL8yWzGNjOtD0w4DC69Ws8O4nnjHIDl90K7rQr9n4hxk7iwK2HkK3FlQMgCx3 yAkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pRvBVOE/hkiozWmaa4rfYGZwNs1AXWCtzZOpvsa5jls=; b=ExsgCVJEhSn2n/jk6Rtx6mmSCycaM/1ZeRvHwnWHf/sg6gMs+LMRxwf8kFe2r0aJ5T QmTRnHZ6gAf7gziDMIFtwAruuJ17zz7Wxb7he4BMcabCcVi8ZPgmYNnI8UC3XLKGNGsc dzCMj8Hs6L+kRSF1iij8stl3M4AwbVmTdYWkLQHGpUHwdxZDAgblXxtu+ec2rU5siUjA 868jAkp7V05g59g8JVayS0eWKR9VMiRdyNIdu+zv2J/ALnDft6U4g+eLULU6Uoy0Un9v 0uLSAaIwfPvFB7M+J5EwiS+V6KKkKfpDUYEdk6lpXLzHVU1sUNqn+yXL53w3Yr4Z0dd7 FeZA== X-Gm-Message-State: AOAM530lrHgRK+dENrFgM/Z2m3+fTbfpRecs04964pexFyqe8R8o8+lI OMK9rMhbaPT1dIVPDU3nOM385JSHbRCnOg== X-Google-Smtp-Source: ABdhPJw+dgZRmWeOglrX0+xhlPytNFCIYlGli60nqojcZ5zl21tiCuxpJmMqjDMtwNc9QLT1KlUtHQ== X-Received: by 2002:a05:600c:3d18:b0:39c:474c:eb with SMTP id bh24-20020a05600c3d1800b0039c474c00ebmr4479104wmb.87.1655217465868; Tue, 14 Jun 2022 07:37:45 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:45 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 06/25] io_uring: explain io_wq_work::cancel_seq placement Date: Tue, 14 Jun 2022 15:36:56 +0100 Message-Id: <0a6cd1d420834f440928642df2c88a175d969cdb.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a comment on why we keep ->cancel_seq in struct io_wq_work instead of struct io_kiocb despite it needed only by io_uring but not io-wq. Signed-off-by: Pavel Begunkov --- io_uring/io-wq.h | 1 + 1 file changed, 1 insertion(+) diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index ba6eee76d028..3f54ee2a8eeb 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -155,6 +155,7 @@ struct io_wq_work_node *wq_stack_extract(struct io_wq_work_node *stack) struct io_wq_work { struct io_wq_work_node list; unsigned flags; + /* place it here instead of io_kiocb as it fills padding and saves 4B */ int cancel_seq; }; From patchwork Tue Jun 14 14:36:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 978DEC43334 for ; Tue, 14 Jun 2022 14:37:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240190AbiFNOhz (ORCPT ); Tue, 14 Jun 2022 10:37:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240402AbiFNOht (ORCPT ); Tue, 14 Jun 2022 10:37:49 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9013D17A9D for ; Tue, 14 Jun 2022 07:37:48 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id i17-20020a7bc951000000b0039c4760ec3fso1007085wml.0 for ; Tue, 14 Jun 2022 07:37:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B+58UuY4vbQY6D/bejFyUsL7Botzww29ZJtrjUE28+I=; b=nnDvlSrdFxnGTAdj11qoOPEEmztWZNbrS4SxAZtdFdSH2Tg40EJv+gR5zOT81YPANl FL2qtCeAK47tELNVYkYOEFPajLGxR7p4n3s+s5krzrINS8zW/HLqf4ZxApbAxJQnPJs1 T8CVPRZz/aNrJhdPVhjisDLRl/gmxnOJsIFh6jMjUkhTdXo+z7LfABYzoKfDAM8bk2Ge SywqfRFg7RhFY3OK+4esAvK/xqya6HqDn5ed8qKGLe95ZqQu2dhAnRlR5qyjmQ9oS2xb aT7lrq78cGCUVVc5wW7Mn973PlIhKzL9UZ4/l0rlcJ4gIR1zvaQ54y4AXJqzk7w0GUYW 4XjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B+58UuY4vbQY6D/bejFyUsL7Botzww29ZJtrjUE28+I=; b=oJ40lw1s1wrZjmxFhtgZw1nY3ZGDw4VL3JayI8m7a5U/nyJOEXpMCjRsAR4PqEPwyW DsqZwXJwheCFEFVf8RQneVw7kERwAMHwcxH6kpZKykstErz0PPZ/DtUzYJu5JWXpQKsS wVkSuitspwOxO6GpZ1oBAgHnZbKput/XMyh4yFaXm98VZv50Hm6BDiiTqCJtwz1YIqq0 lNMAoYI451KSQgqcFmczvBOEApXz+OhKHuNxPSNFPlcf5w/1EvOOV8Xka2JeQWuSUdY1 RYIIyZXFJDFggcrjUR0DO3kt4llza6E1vBtnSo1/vpLB/ukRld+yy8lacQWUg7YibKyf H45A== X-Gm-Message-State: AOAM531xS69ae9k/dDM3ORHkNqYD/SJPjCaP/hBE8pNv64ASV0F7aeO8 U3iA3W9xutvUEpaTU+wchN+FRuS9Gx2rFg== X-Google-Smtp-Source: ABdhPJyaOXK98bwRk+u3fXR4be2Ze9eqqef9M1tPBg2D3FKgfbFulbAEFlX2DuAuSARnhYKUlmnCBw== X-Received: by 2002:a05:600c:3b02:b0:397:5cfb:b849 with SMTP id m2-20020a05600c3b0200b003975cfbb849mr4442520wms.183.1655217466917; Tue, 14 Jun 2022 07:37:46 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:46 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 07/25] io_uring: inline ->registered_rings Date: Tue, 14 Jun 2022 15:36:57 +0100 Message-Id: <6f9e5a5c3bddf92c848664230c9a0ecaf846950e.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There can be only 16 registered rings, no need to allocate an array for them separately but store it in tctx. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - io_uring/tctx.c | 9 --------- io_uring/tctx.h | 3 ++- 3 files changed, 2 insertions(+), 11 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3fdb368820c9..dbf0dbc87758 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2840,7 +2840,6 @@ void __io_uring_free(struct task_struct *tsk) WARN_ON_ONCE(tctx->io_wq); WARN_ON_ONCE(tctx->cached_refs); - kfree(tctx->registered_rings); percpu_counter_destroy(&tctx->inflight); kfree(tctx); tsk->io_uring = NULL; diff --git a/io_uring/tctx.c b/io_uring/tctx.c index f3262eef55d4..6adf659687f8 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -55,16 +55,8 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, if (unlikely(!tctx)) return -ENOMEM; - tctx->registered_rings = kcalloc(IO_RINGFD_REG_MAX, - sizeof(struct file *), GFP_KERNEL); - if (unlikely(!tctx->registered_rings)) { - kfree(tctx); - return -ENOMEM; - } - ret = percpu_counter_init(&tctx->inflight, 0, GFP_KERNEL); if (unlikely(ret)) { - kfree(tctx->registered_rings); kfree(tctx); return ret; } @@ -73,7 +65,6 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, if (IS_ERR(tctx->io_wq)) { ret = PTR_ERR(tctx->io_wq); percpu_counter_destroy(&tctx->inflight); - kfree(tctx->registered_rings); kfree(tctx); return ret; } diff --git a/io_uring/tctx.h b/io_uring/tctx.h index f4964e40d07e..7684713e950f 100644 --- a/io_uring/tctx.h +++ b/io_uring/tctx.h @@ -20,8 +20,9 @@ struct io_uring_task { struct io_wq_work_list task_list; struct io_wq_work_list prio_task_list; struct callback_head task_work; - struct file **registered_rings; bool task_running; + + struct file *registered_rings[IO_RINGFD_REG_MAX]; }; struct io_tctx_node { From patchwork Tue Jun 14 14:36:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3BB8C433EF for ; Tue, 14 Jun 2022 14:37:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245307AbiFNOh4 (ORCPT ); Tue, 14 Jun 2022 10:37:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243978AbiFNOhu (ORCPT ); Tue, 14 Jun 2022 10:37:50 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1C221903F for ; Tue, 14 Jun 2022 07:37:49 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id c130-20020a1c3588000000b0039c6fd897b4so3306354wma.4 for ; Tue, 14 Jun 2022 07:37:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AUP09+kdqf76IuyEW30/qwEVk2WF201FOd1yA2k7FaI=; b=he7/fWcaV1+No50UIKHrqvLOgCFLOGwjPiihIDlhiuAR8tsIcbyw75QcMnSjaJOret mt7nqnTyxgYdaAXcpoJoDgBkUan4QBlwQZOJHXDcIWbxxRTZ1C83VlupIrDpHQ9hEjjD Yyp+WAMl2iTN3HjaVK7gwRecO5gju641K92XE7WvRjLaYuAuPqUQWlKOG7XsyEanJQcr fksF4rh+vgBiyDcv5tvH89uzqAISDHrm5QTJSOKhFg6r6Ti7j0uH7BPwD0viu4DDVw7N ixpxybdhx55GTDCUzWUywd5jFJYksLHwlfeM9tUJMoP3T/b5MrxoLZ/VQ7zhNc4XDzgX AvDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AUP09+kdqf76IuyEW30/qwEVk2WF201FOd1yA2k7FaI=; b=PxjL4Y0gXWZ5qXdQaiP17y+XPMy92FOSgd5YzMRHzOzYQ3FPahRlXwJ/5ch1OxVxpA 0WXJZut8NmE1h6XgmCJXnkvad4tUTiI5ewq5rdHU4DgI87TLU+0oZsTAXaCNPUgJYaxw rF97liOLjVTmp5OQf8zxac3dSSjxoVjtsq9dz2XYxJVITPOmsCY0HLmS/epiQmtX2Ore zpSs74IUizgWAoESLMNOZpwOUvG/OVwSp9LxxQYlc2Ncw57hDdHP7MmFVCZ4TXePmlnQ zywFwPI1gn1oh7bkZFcSIDlSgLEmX7EOzNn3vD26GDTbdXJSGBRV/6EWeUEDi9Xw0KlL 9UOQ== X-Gm-Message-State: AOAM532INU6UiyIYP4vxDh9CAhMe+l3DafCjfRQnQI10zswAnqhm7Cpe aMXerKJyoo+71SLUfYx+QCvvYGmwZ3wY+A== X-Google-Smtp-Source: ABdhPJxCCLci1oG2XWY64hbTjrdxnb7J2D1r/BmUWHzzVVTyowVZYr2dtF8xQFglQO2GQ5Ml+AvtHA== X-Received: by 2002:a05:600c:a42:b0:39c:9086:8a34 with SMTP id c2-20020a05600c0a4200b0039c90868a34mr4385970wmq.169.1655217467985; Tue, 14 Jun 2022 07:37:47 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 08/25] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Date: Tue, 14 Jun 2022 15:36:58 +0100 Message-Id: <203d37f8c8ace5c70d3890132c5a3a6cca72ba73.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_req_task_complete() enqueues requests for state completion itself, no need for REQ_F_COMPLETE_INLINE, which is only serve the purpose of not bloating the kernel. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index dbf0dbc87758..d895f70977b0 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1864,7 +1864,6 @@ inline void io_req_task_complete(struct io_kiocb *req, bool *locked) { if (*locked) { req->cqe.flags |= io_put_kbuf(req, 0); - io_req_complete_state(req); io_req_add_compl_list(req); } else { req->cqe.flags |= io_put_kbuf(req, IO_URING_F_UNLOCKED); From patchwork Tue Jun 14 14:36:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2790FCCA47A for ; Tue, 14 Jun 2022 14:38:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243940AbiFNOh5 (ORCPT ); Tue, 14 Jun 2022 10:37:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244406AbiFNOhw (ORCPT ); Tue, 14 Jun 2022 10:37:52 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3499219C for ; Tue, 14 Jun 2022 07:37:50 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id z9so4776054wmf.3 for ; Tue, 14 Jun 2022 07:37:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5N7TMgeWBdRzVUyrDSTUAGo45ogYC2r5XYPpUezh63w=; b=J94+M8NILMZkczmn1c5tz1B/8X7icYh05Y1xq6TaueAkB/j6whH4JTD8/8O2uW4ozy 5lUCUoloLcgYKGw1rrhGrRKrNgO78+DgqoiNjpYzei1FBeBhdaAQ+yhdA12EufZqx2fW jGTTHCrNXBYRBbSluZ4qHJMPAggYRX7j+8LhaZDWI8N8VBBZxjSL1+OISGclfoCBTaD5 HgCn71UzN6Mam5/8GbDyKGSY7xO95vvdmqCWyU3/yhbSTGMpxPGEi4uYz+DU3jgEREWx YeffSegZ1czRkfIbSvk+17uoue98ozshq/JObRqBjew903h/KdXe1CvJyKjqDwTE8YJL 985g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5N7TMgeWBdRzVUyrDSTUAGo45ogYC2r5XYPpUezh63w=; b=riX1t9YdbpCq7BZRnC3k8Ar2qlwBv2ZNgBtXcxQu1GaYMK6cZ53dqpKqmWG2VSyTIY JO41uHbFnuwzbQydtAkG+ap91iFr2lG+n7EvNR0XYPici5I+5sVYRxAUQm70GydSx4qT iWsrGk0YFny/KEnHZGZJqcmKlBPrp3rKCP+W9Qf1v1hnAqCFF37zSlNGxPwnWfXnJDKn h64jZH23MY8hZUvov7TT/BlR4ifAaIN8yl+8O9LTLipbWaOf14RQYN0w5Cm6ZlwEVd73 k8KYgtfk7dmgOR/9VM3AA5Um0WcFtYdVwSiOFLpGUzYDqsloHbPU2Bbl0p64iCq/1yVV uvHw== X-Gm-Message-State: AOAM533mZUF5IwwX5gwwdLdxlhJVw83Oe/qf/nJDogoEIDoVZ+Bh3b7B JiSPwf7URVDILqlLG40kIIkwiwsXNfO66A== X-Google-Smtp-Source: ABdhPJynDUvNYJbnElnh9nxo+PC5Cehb6b4LWmAFO9AedcdeOr4npCNeozZbkKw/Ov7JgZxYDZldew== X-Received: by 2002:a05:600c:1990:b0:39c:81f0:a882 with SMTP id t16-20020a05600c199000b0039c81f0a882mr4518161wmq.72.1655217469180; Tue, 14 Jun 2022 07:37:49 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 09/25] io_uring: never defer-complete multi-apoll Date: Tue, 14 Jun 2022 15:36:59 +0100 Message-Id: <9ce557af28d199cb03cd24db65fad6579a2e9c2b.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Luckily, nnobody completes multi-apoll requests outside the polling functions, but don't set IO_URING_F_COMPLETE_DEFER in any case as there is nobody who is catching REQ_F_COMPLETE_INLINE, and so will leak requests if used. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d895f70977b0..1fb93fdcfbab 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2149,7 +2149,7 @@ int io_poll_issue(struct io_kiocb *req, bool *locked) io_tw_lock(req->ctx, locked); if (unlikely(req->task->flags & PF_EXITING)) return -EFAULT; - return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER); + return io_issue_sqe(req, IO_URING_F_NONBLOCK); } struct io_wq_work *io_wq_free_work(struct io_wq_work *work) From patchwork Tue Jun 14 14:37:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C9FFC43334 for ; Tue, 14 Jun 2022 14:38:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243978AbiFNOiA (ORCPT ); Tue, 14 Jun 2022 10:38:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245372AbiFNOhx (ORCPT ); Tue, 14 Jun 2022 10:37:53 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BC8BB7F2 for ; Tue, 14 Jun 2022 07:37:52 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id w17so4075107wrg.7 for ; Tue, 14 Jun 2022 07:37:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Peng1QhMzZQpZunkpzCR1eikF0YslET37BvOxslBf8s=; b=ZAv7y97dnw3gWNiOCLFesZAV8l9UNvFpGsoEn5i3j+IT9YUBzO+dg9smXaf1lNgTC3 9BAdtmncq3vyP9HZd9+o0WlpJLYS1o0qDhQPlZ2gWu/1ygi7IeCWspIawGl1eA0X1Jbi CjqxYMZAgWAtv0HuUcDZLLNpWmkzuB4d5wNz6RhemXeHCugg8VIA4uFzS4JiKRyzEO1T V2t1klm/zbVx91BsrrB/5SHtFvkO+185EmLWficCglo8ESimnxPKclqZcKTaBkZ9i2O+ GVPXTN60NeeTZlCD2WHnKVcI61V9KV+xa3QKzVsFVuhKOEDAvpOs0ZL3B3EIE7kN7gAt O0cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Peng1QhMzZQpZunkpzCR1eikF0YslET37BvOxslBf8s=; b=JQcaPFwYF9ErXsYkUaKA3hivk9ZI8Yiwz54oIb5zXAejwv5hzOfHmoYNqTVfdKfpBe IuUAleh//xqLvJwBuak/L8rPgRy1Y9l+i8r/99tjMDfBxTVzjzOSFYEuUKMfKzRIUV4X EYlxbJVbFG2fHH9iH6SEHNYDd9rWsIaB3SIsnpFLO0DDIPGbIH0zb7vCwE+Soy2EVLH/ z1lcs7xIaSmso/C+mlFMyG8YiyNeycNVYXf34W86ufF6K5xWCjw2J+n1nq7Ag+FchrRP XQsJlC4qg+uHWMHyzFkGtMs2jJRnrbITZX8T8+ti9BDM534gY9Oct2UegXpRpTX4EP7K GiWQ== X-Gm-Message-State: AJIora9ptVed9PyalYZ0veqVoJgFzp+/zFp2Q+kS2Lipg4iYM5y3KHj+ 1f81Q8vwOhnmVX9aLn/zgct6q9aEKpV5Lw== X-Google-Smtp-Source: AGRyM1v6s+5hvouYp0MrS6SpkdOWxTcMmX+fWDWYF6wR2iE/LTSx7T7mHJYJrx0+JUWzsL+PvQvbjw== X-Received: by 2002:adf:e28a:0:b0:210:b31:722 with SMTP id v10-20020adfe28a000000b002100b310722mr5244274wri.65.1655217470330; Tue, 14 Jun 2022 07:37:50 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 10/25] io_uring: kill REQ_F_COMPLETE_INLINE Date: Tue, 14 Jun 2022 15:37:00 +0100 Message-Id: <378d3aba69ea2b6a8b14624810a551c2ae011791.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org REQ_F_COMPLETE_INLINE is only needed to delay queueing into the completion list to io_queue_sqe() as __io_req_complete() is inlined and we don't want to bloat the kernel. As now we complete in a more centralised fashion in io_issue_sqe() we can get rid of the flag and queue to the list directly. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 20 ++++++++------------ io_uring/io_uring.h | 5 ----- io_uring/io_uring_types.h | 3 --- 3 files changed, 8 insertions(+), 20 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 1fb93fdcfbab..fcee58c6c35e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1278,17 +1278,14 @@ static void io_req_complete_post32(struct io_kiocb *req, u64 extra1, u64 extra2) inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) { - if (issue_flags & IO_URING_F_COMPLETE_DEFER) - io_req_complete_state(req); - else - io_req_complete_post(req); + io_req_complete_post(req); } void __io_req_complete32(struct io_kiocb *req, unsigned int issue_flags, u64 extra1, u64 extra2) { if (issue_flags & IO_URING_F_COMPLETE_DEFER) { - io_req_complete_state(req); + io_req_add_compl_list(req); req->extra1 = extra1; req->extra2 = extra2; } else { @@ -2132,9 +2129,12 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) if (creds) revert_creds(creds); - if (ret == IOU_OK) - __io_req_complete(req, issue_flags); - else if (ret != IOU_ISSUE_SKIP_COMPLETE) + if (ret == IOU_OK) { + if (issue_flags & IO_URING_F_COMPLETE_DEFER) + io_req_add_compl_list(req); + else + io_req_complete_post(req); + } else if (ret != IOU_ISSUE_SKIP_COMPLETE) return ret; /* If the op doesn't have a file, we're not polling for it */ @@ -2299,10 +2299,6 @@ static inline void io_queue_sqe(struct io_kiocb *req) ret = io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER); - if (req->flags & REQ_F_COMPLETE_INLINE) { - io_req_add_compl_list(req); - return; - } /* * We async punt it if the file wasn't marked NOWAIT, or if the file * doesn't support non-blocking read/write attempts diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 26b669746d61..2141519e995a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -193,11 +193,6 @@ static inline bool io_run_task_work(void) return false; } -static inline void io_req_complete_state(struct io_kiocb *req) -{ - req->flags |= REQ_F_COMPLETE_INLINE; -} - static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) { if (!*locked) { diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index ca8e25992ece..3228872c199b 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -299,7 +299,6 @@ enum { REQ_F_POLLED_BIT, REQ_F_BUFFER_SELECTED_BIT, REQ_F_BUFFER_RING_BIT, - REQ_F_COMPLETE_INLINE_BIT, REQ_F_REISSUE_BIT, REQ_F_CREDS_BIT, REQ_F_REFCOUNT_BIT, @@ -353,8 +352,6 @@ enum { REQ_F_BUFFER_SELECTED = BIT(REQ_F_BUFFER_SELECTED_BIT), /* buffer selected from ring, needs commit */ REQ_F_BUFFER_RING = BIT(REQ_F_BUFFER_RING_BIT), - /* completion is deferred through io_comp_state */ - REQ_F_COMPLETE_INLINE = BIT(REQ_F_COMPLETE_INLINE_BIT), /* caller should reissue async */ REQ_F_REISSUE = BIT(REQ_F_REISSUE_BIT), /* supports async reads/writes */ From patchwork Tue Jun 14 14:37:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19608CCA47E for ; Tue, 14 Jun 2022 14:38:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235339AbiFNOh7 (ORCPT ); Tue, 14 Jun 2022 10:37:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347316AbiFNOhx (ORCPT ); Tue, 14 Jun 2022 10:37:53 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 41E08219C for ; Tue, 14 Jun 2022 07:37:53 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id v14so11585026wra.5 for ; Tue, 14 Jun 2022 07:37:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=S94zs6oiwyxg9I3NOeaPA1zyFabatRdXZPpjQjN6XFg=; b=npdYI4kZDZ8RktRge70L5dHrSMQvOgs0FLocrcrhBq4e4dCKCYyHY2KLXtTwuLraLu lXQwIjxijVsYrYukL2OY3S18Yxn2qZDGusxEsmIRJs0hwcx1qIFd816PX14/qHgs6gvh 3AjWlwVZZ86iLTIk8D0H7V5Z5k4J+j1tPnxKU/fy74TxKPyNWZLBDSIQZxBsuB0JGomB 0AwJpaTdeGAky97MbK8WhvPg6Qc9Gv0COllTCZ9sm5ZxvhVM8Ex+OdHcnN/slsjWL62C LT15xKUPeJJC13XRn0/jK57kHUsj9odGvxq4HCXPkUbLY6+AoVuMzFkT+44DpMPHsOMY zvow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=S94zs6oiwyxg9I3NOeaPA1zyFabatRdXZPpjQjN6XFg=; b=t7wL/LZBu/uVZxWpM4Ht91347hxI93Nw2devvi5/GY3Ld1PtJV1FvlK/qr7074k99i APp3gz9x2kPgG75XnRjrg9zjX0iSmOZi7iIgBSXDBYfUA2wjkp1Vdei25kGmmZEbBT4N Zcv4PkKxE7U4TXAj0eanYdKOP2rb/3Klt/ARuB9lZVdPpAxvw+/0RDecFZu/o4uptYDh 9so5ntXipH974QT4s9aDlrTH9US9jVJ8NvHuV5I8uZ8Uu3Wb1ZpOpSK/SvjklW2r9gf6 Ljw7Bn0Jkm5oDW50nE2M750pHNC10aLuCbXXJSQUTzovGWZ+3Vho5vn33VKrXMThD8x6 iCFw== X-Gm-Message-State: AJIora8krkDTe0r9BhPxpD5zKl8opjY6/lndNwrrpHGUQNbxns8Z2N36 PMooxO9dMy/4Rj0oTkd1aOydYy0Fy77VQw== X-Google-Smtp-Source: AGRyM1scPg0tfWZO5WQUoSwUiGjGGaecX4AmUgmsoJxvumw43/tprrVJm3cWbuf3LoDzgOcjQ9w1uA== X-Received: by 2002:a5d:6da3:0:b0:211:3597:62b1 with SMTP id u3-20020a5d6da3000000b00211359762b1mr5361834wrs.660.1655217471445; Tue, 14 Jun 2022 07:37:51 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:51 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 11/25] io_uring: refactor io_req_task_complete() Date: Tue, 14 Jun 2022 15:37:01 +0100 Message-Id: <60f4b51e219d1be0a390d53aae2e5a19b775ab69.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Clean up io_req_task_complete() and deduplicate io_put_kbuf() calls. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index fcee58c6c35e..0f6edf82f262 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1857,15 +1857,19 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) return ret; } -inline void io_req_task_complete(struct io_kiocb *req, bool *locked) + +void io_req_task_complete(struct io_kiocb *req, bool *locked) { - if (*locked) { - req->cqe.flags |= io_put_kbuf(req, 0); + if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { + unsigned issue_flags = *locked ? IO_URING_F_UNLOCKED : 0; + + req->cqe.flags |= io_put_kbuf(req, issue_flags); + } + + if (*locked) io_req_add_compl_list(req); - } else { - req->cqe.flags |= io_put_kbuf(req, IO_URING_F_UNLOCKED); + else io_req_complete_post(req); - } } /* From patchwork Tue Jun 14 14:37:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8025ACCA47A for ; Tue, 14 Jun 2022 14:38:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238564AbiFNOiB (ORCPT ); Tue, 14 Jun 2022 10:38:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243425AbiFNOh4 (ORCPT ); Tue, 14 Jun 2022 10:37:56 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71892B7F2 for ; Tue, 14 Jun 2022 07:37:54 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id r123-20020a1c2b81000000b0039c1439c33cso4910385wmr.5 for ; Tue, 14 Jun 2022 07:37:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fBuzqx+gzC5/J7sqLEp0sAY1HfeFLOvDJjV1YuH4kBU=; b=SwzGMVM9xEcKeBEuYMvTsD0VVKufNx4b2jRMEwDx1+x9LNRb9naC3n+2yqUljU+lEK j0mQOII0K+n03YYEDH77nbnfAB5S8/P+B3Gokt9D4iwcsdZ7HRRXsd7EhGOiHaNQBWrW n8Rgg0LuwRvXChg64edRBRRFRLeXzvwRco099pNedxD9K9/xdiYuhTXEdHeiZt6hQnrR SzJytPxuLFvPjLjIAsxKviXeaocM7M4+MVKf01ArynNPT4iU5j6K2SVFRM8TUHTExRpu OMomoQE834AEYC+YKYHOaDLkIODyEarBpObK2hu9i1SeLuRkZQQ97X7IPW9c6M40WmHV D7YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fBuzqx+gzC5/J7sqLEp0sAY1HfeFLOvDJjV1YuH4kBU=; b=6wMfM1paCJFl7UaT/UD76Yaxi4ICOCcfd9+aHmi7gMnLHQM+2awpUyCcJCEWPshbD3 hPiNnwM3iBOoJlImx12y8CjYWPFuEsvy7kqX1Nny4XAkiV+YF6HIBDbb1ibQcpD1RD3H EH5jH+RH/8fs6ed3IE0uHbT6ipNoXbKbFyzOH2HFmsRzRla6NtN6h4b/BVMIamAyaet3 AAjMttlJIrfbje9uf1FioIe/5C0Xy6xoV8B53cNcbKz3wht7mODggwkQWzhOXtw1C6h3 a9MAfk+3RUzJxvqNpKRFnM+F6aFx7QM2ch815OjboS4pm2/zHSBaPbKkAzXG3krS4eVZ Ut6A== X-Gm-Message-State: AOAM5327mq4qzpdzbTZDssbLV2gg27y/CuCdcPtBDY1fQAEhXXYITvmz 6kr5qTsGLUjN3PHm3+w5aXAPGuhxu9FITQ== X-Google-Smtp-Source: ABdhPJwfnyxEwbO16KN33XtsjrR/RbW/9b/wWqBWOC+zYPmeSh10DAyzyvFRCbfrw7ECuYGxHmelZw== X-Received: by 2002:a1c:cc07:0:b0:39c:7416:6a43 with SMTP id h7-20020a1ccc07000000b0039c74166a43mr4408303wmb.130.1655217472674; Tue, 14 Jun 2022 07:37:52 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:52 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 12/25] io_uring: don't inline io_put_kbuf Date: Tue, 14 Jun 2022 15:37:02 +0100 Message-Id: <8abee74fa6f3c6565b69a0facfaffafb3ed4bf8a.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_put_kbuf() is huge, don't bloat the kernel with inlining. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 33 +++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 38 ++++++-------------------------------- 2 files changed, 39 insertions(+), 32 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 9cdbc018fd64..6f2adb481a0d 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -81,6 +81,39 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } +unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) +{ + unsigned int cflags; + + /* + * We can add this buffer back to two lists: + * + * 1) The io_buffers_cache list. This one is protected by the + * ctx->uring_lock. If we already hold this lock, add back to this + * list as we can grab it from issue as well. + * 2) The io_buffers_comp list. This one is protected by the + * ctx->completion_lock. + * + * We migrate buffers from the comp_list to the issue cache list + * when we need one. + */ + if (req->flags & REQ_F_BUFFER_RING) { + /* no buffers to recycle for this case */ + cflags = __io_put_kbuf_list(req, NULL); + } else if (issue_flags & IO_URING_F_UNLOCKED) { + struct io_ring_ctx *ctx = req->ctx; + + spin_lock(&ctx->completion_lock); + cflags = __io_put_kbuf_list(req, &ctx->io_buffers_comp); + spin_unlock(&ctx->completion_lock); + } else { + lockdep_assert_held(&req->ctx->uring_lock); + + cflags = __io_put_kbuf_list(req, &req->ctx->io_buffers_cache); + } + return cflags; +} + static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, struct io_buffer_list *bl) { diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 80b6df2c7535..5da3d4039aed 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -47,6 +47,8 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags); int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); +unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags); + static inline bool io_do_buffer_select(struct io_kiocb *req) { if (!(req->flags & REQ_F_BUFFER_SELECT)) @@ -70,7 +72,8 @@ static inline void io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags) __io_kbuf_recycle(req, issue_flags); } -static unsigned int __io_put_kbuf(struct io_kiocb *req, struct list_head *list) +static inline unsigned int __io_put_kbuf_list(struct io_kiocb *req, + struct list_head *list) { if (req->flags & REQ_F_BUFFER_RING) { if (req->buf_list) @@ -90,44 +93,15 @@ static inline unsigned int io_put_kbuf_comp(struct io_kiocb *req) if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) return 0; - return __io_put_kbuf(req, &req->ctx->io_buffers_comp); + return __io_put_kbuf_list(req, &req->ctx->io_buffers_comp); } static inline unsigned int io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) { - unsigned int cflags; if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) return 0; - - /* - * We can add this buffer back to two lists: - * - * 1) The io_buffers_cache list. This one is protected by the - * ctx->uring_lock. If we already hold this lock, add back to this - * list as we can grab it from issue as well. - * 2) The io_buffers_comp list. This one is protected by the - * ctx->completion_lock. - * - * We migrate buffers from the comp_list to the issue cache list - * when we need one. - */ - if (req->flags & REQ_F_BUFFER_RING) { - /* no buffers to recycle for this case */ - cflags = __io_put_kbuf(req, NULL); - } else if (issue_flags & IO_URING_F_UNLOCKED) { - struct io_ring_ctx *ctx = req->ctx; - - spin_lock(&ctx->completion_lock); - cflags = __io_put_kbuf(req, &ctx->io_buffers_comp); - spin_unlock(&ctx->completion_lock); - } else { - lockdep_assert_held(&req->ctx->uring_lock); - - cflags = __io_put_kbuf(req, &req->ctx->io_buffers_cache); - } - - return cflags; + return __io_put_kbuf(req, issue_flags); } #endif From patchwork Tue Jun 14 14:37:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B458DCCA47A for ; Tue, 14 Jun 2022 14:38:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242898AbiFNOiF (ORCPT ); Tue, 14 Jun 2022 10:38:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242741AbiFNOh4 (ORCPT ); Tue, 14 Jun 2022 10:37:56 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3D19193C5 for ; Tue, 14 Jun 2022 07:37:55 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id s1so11559122wra.9 for ; Tue, 14 Jun 2022 07:37:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6eJxYgBNpK8cENMseaGM9WlOCNZFZvANvUQtryLvK5Y=; b=K5DTZNouAcLLflzs3R+LX0vttt9U1ZUHVC0hlnvTuYofknZyBI2thYo4wZoGUE/+Aj q13zyWWg3iYPySo5BrX8YKWUVF2F6EK5VHXWbi33HjNqejG/nF7Fqb+ZVeh1JLgvZgq4 voqzqAYB4wRU1dyRoXZrbog/M0nxtk6fvx8aZwVkRQ77JpOzRQAQP0NU37AvTzdxWa5F K/DkM+kWt94MwgpwAYgape+/KyyIPnxzfgwpCK2/jn6l4JPIb53uMnf6x44IqLaIvz0/ OD67ioKhNSEu2dpaqUX1ghh+tu4keJW3aTkWdYvIrK/sNBSz4X01oYdIEe3q6G7L4uJ2 u7Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6eJxYgBNpK8cENMseaGM9WlOCNZFZvANvUQtryLvK5Y=; b=4ltucL6+iERdSnGdyqp2UtO1l4XG/T2+HnxwaLwFzM83RD71t9gOSgfIsTI8099KIV aiVeSUje/9A2mt9PaBFSvv3abVP+M9vectzBJRkoGfQfIWGJYFzOLxFeJcuOLnYJZWGz ekXOjJ++TUwIuODGL2VpZUlwp+jAsajq+DzXXo+qCjltt0n5HXNUdwDH/Q7FrxGAf9TU 1EIL06vHIbivieid6848EqIbXl55D0BxvOHheJsEcgWlQ+Bsl1+EBXGpUDH5w9n5zzCT 2i1j1DL1LyEwZZR7B3ZM9/o7einzuPy+OkqoYGVHVojVKOn37i4C9r4sKF7x0rYca93g xOqg== X-Gm-Message-State: AJIora/7lvpNGC81GWkPT124YEBEYTnGVbQzZ1POs+2bNT6goBVBnFdc gxyhFbrhEK8/xcRtT8Ic1uDdVeEsSd5MXg== X-Google-Smtp-Source: AGRyM1tML264tJAB1VQTJo8R8A6tqR1itUcGYC0CAgkrmiQVWp7wPXlg4hrtl7AyA9jrZNc9M5ruzg== X-Received: by 2002:adf:d1e7:0:b0:215:2126:dede with SMTP id g7-20020adfd1e7000000b002152126dedemr5230940wrd.297.1655217473913; Tue, 14 Jun 2022 07:37:53 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:53 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 13/25] io_uring: remove check_cq checking from hot paths Date: Tue, 14 Jun 2022 15:37:03 +0100 Message-Id: <9ab7e307e77ffeb92ec788694c87beff27d55c05.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org All ctx->check_cq events are slow path, don't test every single flag one by one in the hot path, but add a common guarding if. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0f6edf82f262..e43eccf173ff 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1807,24 +1807,25 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) int ret = 0; unsigned long check_cq; + check_cq = READ_ONCE(ctx->check_cq); + if (unlikely(check_cq)) { + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + __io_cqring_overflow_flush(ctx, false); + /* + * Similarly do not spin if we have not informed the user of any + * dropped CQE. + */ + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) + return -EBADR; + } /* * Don't enter poll loop if we already have events pending. * If we do, we can potentially be spinning for commands that * already triggered a CQE (eg in error). */ - check_cq = READ_ONCE(ctx->check_cq); - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - __io_cqring_overflow_flush(ctx, false); if (io_cqring_events(ctx)) return 0; - /* - * Similarly do not spin if we have not informed the user of any - * dropped CQE. - */ - if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))) - return -EBADR; - do { /* * If a submit got punted to a workqueue, we can have the @@ -2752,12 +2753,15 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, ret = io_run_task_work_sig(); if (ret || io_should_wake(iowq)) return ret; + check_cq = READ_ONCE(ctx->check_cq); - /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - return 1; - if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))) - return -EBADR; + if (unlikely(check_cq)) { + /* let the caller flush overflows, retry */ + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + return 1; + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) + return -EBADR; + } if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS)) return -ETIME; return 1; From patchwork Tue Jun 14 14:37:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881211 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 363B5C433EF for ; Tue, 14 Jun 2022 14:38:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243356AbiFNOiD (ORCPT ); Tue, 14 Jun 2022 10:38:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244916AbiFNOh4 (ORCPT ); Tue, 14 Jun 2022 10:37:56 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DAFD7193E6 for ; Tue, 14 Jun 2022 07:37:55 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id m39-20020a05600c3b2700b0039c511ebbacso6337874wms.3 for ; Tue, 14 Jun 2022 07:37:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xtoDyDvNXPTaclwDuSrBJ25RRT45S7N/gTULdXm19LA=; b=eqahh/BIRPOKHJ6IPjJII9pfU50/U3Tn0P9DivGktoH/rehdFNOHRPat/7lRq9BZnR S6orsQX/1bxk28H9V5nNzRqU9gZ6pKDQ2LrtpUnUayolrDNfFSYrNzBCXK42KfKxW4w3 suNnlCkwOHSlbOWYI2HxvB6G1ZY22zOqvVwfFCA296KXtnFa/ZMvdZgJ+AZzVEy+IkuW Uv4ML6l0gOpzkSo/xNC5Qw1AMqu6Dinhqqpa4hevEXWalJxThTUFJ8SIH/5oDDuBBZGI MInICFTrIwaPwYHO/r55f5gAuLsfWYHtPvxtfec44HNOneIWMl5ITw1XrjdkdgD6XDbE PdPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xtoDyDvNXPTaclwDuSrBJ25RRT45S7N/gTULdXm19LA=; b=sz4C2iOZvd5WOIC+o09Md34rzxcDrAeaiX4S2CbVT1saBtJPotvc4pVj3Dngrk8aWp ASk3Dw4UloT3h4iyHPs/mjiqgPoxENKbFxUTf+BUz1CkOMSU1a53tJc1/TE4/sm7lb26 UuJaEVoTZiVj0T09hRMifgoJI7V5KrFa8H9YVBCmpY9fOah++VkolfAHnnipsO3F+82U MCIXl0tmX8jl01r+oEMZBFYCRGjU1ASL3DFTg7YuGddTKJGaKO062DghGbAnJg//oEmg 8MJkERSf7VmfzKUbj2R6DPta9WBTUfFEz5PzINHamBXqUHeEQL3E3McODM1B34LcUEtm UZVg== X-Gm-Message-State: AOAM5333vDfeYu58SA8tEOmTX5mdxI7OA67rCZ6sNo1/k/jkTNcVcFw6 ZbxTG3NC61Svd+WaKy8VCLJCSC+7eeW23Q== X-Google-Smtp-Source: ABdhPJyCYGYmdX1Urx6qpYVz1tndgiqPqRPETCMfeuMOVogo/JTeCjGD3D+/QCQVWy3AXTKzxRWgyQ== X-Received: by 2002:a7b:c389:0:b0:39c:49fe:25d3 with SMTP id s9-20020a7bc389000000b0039c49fe25d3mr4553278wmj.83.1655217475119; Tue, 14 Jun 2022 07:37:55 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:54 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Hao Xu Subject: [PATCH for-next v2 14/25] io_uring: poll: remove unnecessary req->ref set Date: Tue, 14 Jun 2022 15:37:04 +0100 Message-Id: <010576dc7ac2cbc6059958795adeaf6cef1e02a5.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu We now don't need to set req->refcount for poll requests since the reworked poll code ensures no request release race. Signed-off-by: Hao Xu Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 0df5eca93b16..73584c4e3e9b 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -683,7 +683,6 @@ int io_poll_add_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if ((flags & IORING_POLL_ADD_MULTI) && (req->flags & REQ_F_CQE_SKIP)) return -EINVAL; - io_req_set_refcount(req); req->apoll_events = poll->events = io_poll_parse_events(sqe, flags); return 0; } From patchwork Tue Jun 14 14:37:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2A43C43334 for ; Tue, 14 Jun 2022 14:38:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243425AbiFNOiD (ORCPT ); Tue, 14 Jun 2022 10:38:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238373AbiFNOiA (ORCPT ); Tue, 14 Jun 2022 10:38:00 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46E351A814 for ; Tue, 14 Jun 2022 07:37:57 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id s1so11559122wra.9 for ; Tue, 14 Jun 2022 07:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3A0DKHRjofAc/lP8/f+KM5PfTO3qjPV0CIG3FFiFDl0=; b=GUJDlnjzQ26tVQSiVdc1i3jqMuARP6L67wZ1mMRhlPTwQBWU1etyJZcyuplFUgFPn0 w2o9aNIF6HpOB1er0UNRfHO8X5sQzLhIPpXUJt5jW8m7S3G01YXSuFKm7AGVKyf942L3 XK753/W1fvk1hN6doMSLlUwwMf5t3f7d0wEBUGjYyjomHHOujAjYlCFCsYOkTMG/8ReR Z3BcH1fiepd+axSAjFNPlrD5kHVzWkFokaVlwUCor9MXj16yol+MKSHDIdPqYpyV1ByP 5XEc+uvsfbmA1yVrv3TQdspt9QgvM0B72fuOq/bmMjoZL32pk1B7d0Lqlcu8XmatC9k7 xF0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3A0DKHRjofAc/lP8/f+KM5PfTO3qjPV0CIG3FFiFDl0=; b=pV8WuCjcaJRZlbV7Ao51mw0Om+T3oNwj4zYOmOwzatENp9yZ2ZT1bltPqyQzkOmy6U hXN413YjqthniIdQxQHX665gfA+zuqq7xCeMa9+lTow0fNtysssOA5hy7HDlNCrMxlOh +xJTAYeQHlh+Y99g+pSpwUmey4x0jAVES1lLi8vE1CXb+J/Y/ehA01kc8gdg9HzbpgrP zfHer245BUtxgCu7Z81xwEVHPu7+UYFa3F3lqVHUnDl2zEb+i7yjDW/i7LKh577xDtS/ VUiKgLycNVQaY3x4g6ganVLHUNwTQsJVOVa4QsqFPcYSVqMrVjBUVEtmhvzQzsjDB4Je SvWQ== X-Gm-Message-State: AJIora+2Ggqkq9su8NvhtYEsQ6hi8O0P7kReq2GRhmzvA/4NCdgG4cEu bfhe/2J+HYuuCGxK3DvLk5d2tfAHYoDI3A== X-Google-Smtp-Source: AGRyM1sHhM5n2qM22uGHBZbzGtHiU2An7if+IHHYz6vnwQarq8GU5g0APpXjvV3gTC6fdopJ/1CAoA== X-Received: by 2002:adf:e3cf:0:b0:219:e523:905f with SMTP id k15-20020adfe3cf000000b00219e523905fmr5162399wrm.240.1655217476406; Tue, 14 Jun 2022 07:37:56 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:55 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Hao Xu Subject: [PATCH for-next v2 15/25] io_uring: switch cancel_hash to use per entry spinlock Date: Tue, 14 Jun 2022 15:37:05 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new io_hash_bucket structure so that each bucket in cancel_hash has separate spinlock. Use per entry lock for cancel_hash, this removes some completion lock invocation and remove contension between different cancel_hash entries. Signed-off-by: Hao Xu Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 14 ++++++- io_uring/cancel.h | 6 +++ io_uring/fdinfo.c | 9 +++-- io_uring/io_uring.c | 8 ++-- io_uring/io_uring_types.h | 2 +- io_uring/poll.c | 80 ++++++++++++++++++++++++--------------- 6 files changed, 79 insertions(+), 40 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 83cceb52d82d..6f2888388a40 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -93,14 +93,14 @@ int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) if (!ret) return 0; - spin_lock(&ctx->completion_lock); ret = io_poll_cancel(ctx, cd); if (ret != -ENOENT) goto out; + spin_lock(&ctx->completion_lock); if (!(cd->flags & IORING_ASYNC_CANCEL_FD)) ret = io_timeout_cancel(ctx, cd); -out: spin_unlock(&ctx->completion_lock); +out: return ret; } @@ -192,3 +192,13 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags) io_req_set_res(req, ret, 0); return IOU_OK; } + +void init_hash_table(struct io_hash_bucket *hash_table, unsigned size) +{ + unsigned int i; + + for (i = 0; i < size; i++) { + spin_lock_init(&hash_table[i].lock); + INIT_HLIST_HEAD(&hash_table[i].list); + } +} diff --git a/io_uring/cancel.h b/io_uring/cancel.h index 4f35d8696325..556a7dcf160e 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -4,3 +4,9 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags); int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd); +void init_hash_table(struct io_hash_bucket *hash_table, unsigned size); + +struct io_hash_bucket { + spinlock_t lock; + struct hlist_head list; +} ____cacheline_aligned_in_smp; diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index fcedde4b4b1e..f941c73f5502 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -13,6 +13,7 @@ #include "io_uring.h" #include "sqpoll.h" #include "fdinfo.h" +#include "cancel.h" #ifdef CONFIG_PROC_FS static __cold int io_uring_show_cred(struct seq_file *m, unsigned int id, @@ -157,17 +158,19 @@ static __cold void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, mutex_unlock(&ctx->uring_lock); seq_puts(m, "PollList:\n"); - spin_lock(&ctx->completion_lock); for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct hlist_head *list = &ctx->cancel_hash[i]; + struct io_hash_bucket *hb = &ctx->cancel_hash[i]; struct io_kiocb *req; - hlist_for_each_entry(req, list, hash_node) + spin_lock(&hb->lock); + hlist_for_each_entry(req, &hb->list, hash_node) seq_printf(m, " op=%d, task_works=%d\n", req->opcode, task_work_pending(req->task)); + spin_unlock(&hb->lock); } seq_puts(m, "CqOverflowList:\n"); + spin_lock(&ctx->completion_lock); list_for_each_entry(ocqe, &ctx->cq_overflow_list, list) { struct io_uring_cqe *cqe = &ocqe->cqe; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e43eccf173ff..4d94757e6e28 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -717,11 +717,13 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) if (hash_bits <= 0) hash_bits = 1; ctx->cancel_hash_bits = hash_bits; - ctx->cancel_hash = kmalloc((1U << hash_bits) * sizeof(struct hlist_head), - GFP_KERNEL); + ctx->cancel_hash = + kmalloc((1U << hash_bits) * sizeof(struct io_hash_bucket), + GFP_KERNEL); if (!ctx->cancel_hash) goto err; - __hash_init(ctx->cancel_hash, 1U << hash_bits); + + init_hash_table(ctx->cancel_hash, 1U << hash_bits); ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 3228872c199b..aba0f8cd6f49 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -222,7 +222,7 @@ struct io_ring_ctx { * manipulate the list, hence no extra locking is needed there. */ struct io_wq_work_list iopoll_list; - struct hlist_head *cancel_hash; + struct io_hash_bucket *cancel_hash; unsigned cancel_hash_bits; bool poll_multi_queue; diff --git a/io_uring/poll.c b/io_uring/poll.c index 73584c4e3e9b..7f6b16f687b0 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -19,6 +19,7 @@ #include "opdef.h" #include "kbuf.h" #include "poll.h" +#include "cancel.h" struct io_poll_update { struct file *file; @@ -73,10 +74,22 @@ static struct io_poll *io_poll_get_single(struct io_kiocb *req) static void io_poll_req_insert(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; - struct hlist_head *list; + u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); + struct io_hash_bucket *hb = &ctx->cancel_hash[index]; - list = &ctx->cancel_hash[hash_long(req->cqe.user_data, ctx->cancel_hash_bits)]; - hlist_add_head(&req->hash_node, list); + spin_lock(&hb->lock); + hlist_add_head(&req->hash_node, &hb->list); + spin_unlock(&hb->lock); +} + +static void io_poll_req_delete(struct io_kiocb *req, struct io_ring_ctx *ctx) +{ + u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); + spinlock_t *lock = &ctx->cancel_hash[index].lock; + + spin_lock(lock); + hash_del(&req->hash_node); + spin_unlock(lock); } static void io_init_poll_iocb(struct io_poll *poll, __poll_t events, @@ -220,8 +233,8 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) } io_poll_remove_entries(req); + io_poll_req_delete(req, ctx); spin_lock(&ctx->completion_lock); - hash_del(&req->hash_node); req->cqe.flags = 0; __io_req_complete_post(req); io_commit_cqring(ctx); @@ -231,7 +244,6 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) static void io_apoll_task_func(struct io_kiocb *req, bool *locked) { - struct io_ring_ctx *ctx = req->ctx; int ret; ret = io_poll_check_events(req, locked); @@ -239,9 +251,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) return; io_poll_remove_entries(req); - spin_lock(&ctx->completion_lock); - hash_del(&req->hash_node); - spin_unlock(&ctx->completion_lock); + io_poll_req_delete(req, req->ctx); if (!ret) io_req_task_submit(req, locked); @@ -435,9 +445,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, return 0; } - spin_lock(&ctx->completion_lock); io_poll_req_insert(req); - spin_unlock(&ctx->completion_lock); if (mask && (poll->events & EPOLLET)) { /* can't multishot if failed, just queue the event we've got */ @@ -534,32 +542,31 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, bool found = false; int i; - spin_lock(&ctx->completion_lock); for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct hlist_head *list; + struct io_hash_bucket *hb = &ctx->cancel_hash[i]; - list = &ctx->cancel_hash[i]; - hlist_for_each_entry_safe(req, tmp, list, hash_node) { + spin_lock(&hb->lock); + hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) { if (io_match_task_safe(req, tsk, cancel_all)) { hlist_del_init(&req->hash_node); io_poll_cancel_req(req); found = true; } } + spin_unlock(&hb->lock); } - spin_unlock(&ctx->completion_lock); return found; } static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd) - __must_hold(&ctx->completion_lock) { - struct hlist_head *list; struct io_kiocb *req; + u32 index = hash_long(cd->data, ctx->cancel_hash_bits); + struct io_hash_bucket *hb = &ctx->cancel_hash[index]; - list = &ctx->cancel_hash[hash_long(cd->data, ctx->cancel_hash_bits)]; - hlist_for_each_entry(req, list, hash_node) { + spin_lock(&hb->lock); + hlist_for_each_entry(req, &hb->list, hash_node) { if (cd->data != req->cqe.user_data) continue; if (poll_only && req->opcode != IORING_OP_POLL_ADD) @@ -571,21 +578,21 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, } return req; } + spin_unlock(&hb->lock); return NULL; } static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, struct io_cancel_data *cd) - __must_hold(&ctx->completion_lock) { struct io_kiocb *req; int i; for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct hlist_head *list; + struct io_hash_bucket *hb = &ctx->cancel_hash[i]; - list = &ctx->cancel_hash[i]; - hlist_for_each_entry(req, list, hash_node) { + spin_lock(&hb->lock); + hlist_for_each_entry(req, &hb->list, hash_node) { if (!(cd->flags & IORING_ASYNC_CANCEL_ANY) && req->file != cd->file) continue; @@ -594,12 +601,12 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, req->work.cancel_seq = cd->seq; return req; } + spin_unlock(&hb->lock); } return NULL; } static bool io_poll_disarm(struct io_kiocb *req) - __must_hold(&ctx->completion_lock) { if (!io_poll_get_ownership(req)) return false; @@ -609,17 +616,23 @@ static bool io_poll_disarm(struct io_kiocb *req) } int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) - __must_hold(&ctx->completion_lock) { struct io_kiocb *req; + u32 index; + spinlock_t *lock; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) req = io_poll_file_find(ctx, cd); else req = io_poll_find(ctx, false, cd); - if (!req) + if (!req) { return -ENOENT; + } else { + index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); + lock = &ctx->cancel_hash[index].lock; + } io_poll_cancel_req(req); + spin_unlock(lock); return 0; } @@ -713,18 +726,23 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) struct io_poll_update *poll_update = io_kiocb_to_cmd(req); struct io_cancel_data cd = { .data = poll_update->old_user_data, }; struct io_ring_ctx *ctx = req->ctx; + u32 index = hash_long(cd.data, ctx->cancel_hash_bits); + spinlock_t *lock = &ctx->cancel_hash[index].lock; struct io_kiocb *preq; int ret2, ret = 0; bool locked; - spin_lock(&ctx->completion_lock); preq = io_poll_find(ctx, true, &cd); - if (!preq || !io_poll_disarm(preq)) { - spin_unlock(&ctx->completion_lock); - ret = preq ? -EALREADY : -ENOENT; + if (!preq) { + ret = -ENOENT; + goto out; + } + ret2 = io_poll_disarm(preq); + spin_unlock(lock); + if (!ret2) { + ret = -EALREADY; goto out; } - spin_unlock(&ctx->completion_lock); if (poll_update->update_events || poll_update->update_user_data) { /* only mask one event flags, keep behavior flags */ From patchwork Tue Jun 14 14:37:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B33BC433EF for ; Tue, 14 Jun 2022 14:38:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244922AbiFNOiE (ORCPT ); Tue, 14 Jun 2022 10:38:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242898AbiFNOiA (ORCPT ); Tue, 14 Jun 2022 10:38:00 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AA951AF2C for ; Tue, 14 Jun 2022 07:37:58 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id i81-20020a1c3b54000000b0039c76434147so117423wma.1 for ; Tue, 14 Jun 2022 07:37:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=whmbV8SR6MxGfqcouaajDpb1UgDnXjtmeXGDkViNhtk=; b=ml6hj18Gg9XE9WmmpzC68rm4LQW3/uRnHRKUuce34X2rWc38sXbSGln2ZyQ9Qwx8qQ cxwTk+YQ3af9I4tt7b2fzh0SBZgIFe1aBkZA1Cfpbbpu2KaM6E9mfBt+xCLtr8keddf5 ptc0YXDbHzgMUJNOSThlFLFI7PXfFZOwoWORdDcfQtExkk2m/+9A7S+W7fjNpzhFt9Mu H/+f8tV+QkpcXKB2rKZAvTio5qp4EerBXMNQm9eOADUUEF5JPphi5SJtyuU+8IZ0YlY0 vGTC6ljQC0nEgPa2ErcFAZbzo41UmGXkKaooGLc/heF9dvHFNzD/csf7+BaqoaXHCqH2 UAoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=whmbV8SR6MxGfqcouaajDpb1UgDnXjtmeXGDkViNhtk=; b=myynef0bYNzQf/HMERD6SoIJ2l6fAhVCO0UFlwedNJ2QB+TzUAQzynjv6BuufDviYq 2FQywSj1pNGSM/XH2L4ekf5Tv/FLRS0DX88aikkj+5wP6ol7Ql30amFrJAFMiFYCNUsW +sDWFF/BASW09QuXpGAvy9AnN13CsXPj4Wp+sssSEYnEAJE2T68huNg0y8mdBfChm4wG p9+CsrNhxCxR+YV7kgmUus4u4ih3Cj5/QeT983fwoOzMGb7KJf4ATppa7n3eKsfs636j wmwMkpYz9tfEJqIyUGmadaE3ZRX/mhah+WmTMls+uLrkIiFKu8bUDetV1FWw6zklD265 DVlQ== X-Gm-Message-State: AOAM530UxNaUKVaW4J+0jSDKkV0TBCeF074JHMJXiWTiZaEECpbGM2WX zZJ/7xz8xrnqh/BO2Zi1Sq3j/pKC9C1ZfQ== X-Google-Smtp-Source: ABdhPJycIoHugb4DJDZedtEG1zqbS5qWjPFoeAaCWknZbbQR3ZjB71e2ycxbA0DRKsqZdO3Y6PhsNQ== X-Received: by 2002:a05:600c:2312:b0:397:7647:2ac4 with SMTP id 18-20020a05600c231200b0039776472ac4mr4383493wmo.125.1655217477679; Tue, 14 Jun 2022 07:37:57 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:57 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 16/25] io_uring: pass poll_find lock back Date: Tue, 14 Jun 2022 15:37:06 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of using implicit knowledge of what is locked or not after io_poll_find() and co returns, pass back a pointer to the locked bucket if any. If set the user must to unlock the spinlock. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 7f6b16f687b0..7fc4aafcca95 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -559,12 +559,15 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, } static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, - struct io_cancel_data *cd) + struct io_cancel_data *cd, + struct io_hash_bucket **out_bucket) { struct io_kiocb *req; u32 index = hash_long(cd->data, ctx->cancel_hash_bits); struct io_hash_bucket *hb = &ctx->cancel_hash[index]; + *out_bucket = NULL; + spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { if (cd->data != req->cqe.user_data) @@ -576,6 +579,7 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, continue; req->work.cancel_seq = cd->seq; } + *out_bucket = hb; return req; } spin_unlock(&hb->lock); @@ -583,11 +587,14 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, } static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, - struct io_cancel_data *cd) + struct io_cancel_data *cd, + struct io_hash_bucket **out_bucket) { struct io_kiocb *req; int i; + *out_bucket = NULL; + for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { struct io_hash_bucket *hb = &ctx->cancel_hash[i]; @@ -599,6 +606,7 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, if (cd->seq == req->work.cancel_seq) continue; req->work.cancel_seq = cd->seq; + *out_bucket = hb; return req; } spin_unlock(&hb->lock); @@ -617,23 +625,19 @@ static bool io_poll_disarm(struct io_kiocb *req) int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) { + struct io_hash_bucket *bucket; struct io_kiocb *req; - u32 index; - spinlock_t *lock; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd); + req = io_poll_file_find(ctx, cd, &bucket); else - req = io_poll_find(ctx, false, cd); - if (!req) { - return -ENOENT; - } else { - index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); - lock = &ctx->cancel_hash[index].lock; - } - io_poll_cancel_req(req); - spin_unlock(lock); - return 0; + req = io_poll_find(ctx, false, cd, &bucket); + + if (req) + io_poll_cancel_req(req); + if (bucket) + spin_unlock(&bucket->lock); + return req ? 0 : -ENOENT; } static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, @@ -726,19 +730,21 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) struct io_poll_update *poll_update = io_kiocb_to_cmd(req); struct io_cancel_data cd = { .data = poll_update->old_user_data, }; struct io_ring_ctx *ctx = req->ctx; - u32 index = hash_long(cd.data, ctx->cancel_hash_bits); - spinlock_t *lock = &ctx->cancel_hash[index].lock; + struct io_hash_bucket *bucket; struct io_kiocb *preq; int ret2, ret = 0; bool locked; - preq = io_poll_find(ctx, true, &cd); + preq = io_poll_find(ctx, true, &cd, &bucket); + if (preq) + ret2 = io_poll_disarm(preq); + if (bucket) + spin_unlock(&bucket->lock); + if (!preq) { ret = -ENOENT; goto out; } - ret2 = io_poll_disarm(preq); - spin_unlock(lock); if (!ret2) { ret = -EALREADY; goto out; From patchwork Tue Jun 14 14:37:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A832DC43334 for ; Tue, 14 Jun 2022 14:38:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245242AbiFNOiG (ORCPT ); Tue, 14 Jun 2022 10:38:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237013AbiFNOiC (ORCPT ); Tue, 14 Jun 2022 10:38:02 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B7201A07B for ; Tue, 14 Jun 2022 07:37:59 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id o8so11589460wro.3 for ; Tue, 14 Jun 2022 07:37:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6Mn1I3qJJuXoCLtaTkB3ey7V/NeeqbASVSbKJNj/gRM=; b=F+pm97aEKODJo6Ubmf53UxkCYgv/VcIfdqICviMGdlYJW9qM99sslm2ap4oqqOgGqp yHGARi3a9OGS1mMzO1PZXcMGQbQoEpgMNa+4eV5qz4yykNg4D5o/zChsb/26ck/oDbxa +1cnjnZyeps6eOEYk/xqOX//lSnAud3T147FINkUE+HycmQ4rR82cX5GQpJWNqLIm8lo 4f03daRuhnTv4kFvdn5O2GVV8vkKmAcvbnhfMKDDBeAi6mMtR6pMzoWQIEgD7au9nXjF JHkbE8NoYjXw+q7NmS/7dJduqvFWAAQA1wkuIwUkneOBiUTQsRON3y605P7QBpzdglw/ aBBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6Mn1I3qJJuXoCLtaTkB3ey7V/NeeqbASVSbKJNj/gRM=; b=f8Z02ja9RBJ4Jk/4DeAH+V3diFXi3BPRq3/Pux4faak9Y8kSRGK0HAk6yLTtb4zHqr nY/TasBAhbv5zrYt8CNyjcpTbMBALjiIOd63wijE+bO919c/nl8PKPOTBCY/JXBCG0vk tK9ZR6E621IP2WJFT5Gmr65Wk9sWXJ01ZNmXA6GLrU2mo4hJScA3NF5S7nRo2aH53S8S SBpoS1aGRmHZzn65p3tpBgo/QZuKTdJkk5VgchTBy90n8uFtrQPohVxAec7cQMyjqgif JJnCAW5wCoPc8uAB8lIJ6sFVt3wlzf3z/ujg4L3IOvJut0eVxDQQgwqIO9Dwo/JFrCjq 0dRQ== X-Gm-Message-State: AJIora8ArKIm/KTihumAFj3CsBK7o6UxzWhc6nZj8qt/Ism6bxMLKJPg XEDqCZ/TjrJeocSBi46lSr0+L/Q7Ik6Mpg== X-Google-Smtp-Source: AGRyM1v3rBbAIZKXDHRI2jy97uiuzYTcsASNuSuQnQ9X0rDtPGnBBUz1XxxM0WZk2e4/sLnPKtHa8w== X-Received: by 2002:a05:6000:1812:b0:210:2eb1:4606 with SMTP id m18-20020a056000181200b002102eb14606mr5410900wrh.593.1655217478812; Tue, 14 Jun 2022 07:37:58 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:58 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 17/25] io_uring: clean up io_try_cancel Date: Tue, 14 Jun 2022 15:37:07 +0100 Message-Id: <5b54a27b6b943a63dabde902ddaca7c6c812d036.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Get rid of an unnecessary extra goto in io_try_cancel() and simplify the function. Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 6f2888388a40..a253e2ad22eb 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -95,12 +95,12 @@ int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) ret = io_poll_cancel(ctx, cd); if (ret != -ENOENT) - goto out; + return ret; + spin_lock(&ctx->completion_lock); if (!(cd->flags & IORING_ASYNC_CANCEL_FD)) ret = io_timeout_cancel(ctx, cd); spin_unlock(&ctx->completion_lock); -out: return ret; } From patchwork Tue Jun 14 14:37:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39260C433EF for ; Tue, 14 Jun 2022 14:38:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237013AbiFNOiH (ORCPT ); Tue, 14 Jun 2022 10:38:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238292AbiFNOiE (ORCPT ); Tue, 14 Jun 2022 10:38:04 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 934EEB7F2 for ; Tue, 14 Jun 2022 07:38:01 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id u8so11544860wrm.13 for ; Tue, 14 Jun 2022 07:38:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rLp+y2Cb7SGsOa4Obd9vWUi02TdIW1M+K1OPPdobolA=; b=Mn8Ky7Jy4loDj5LXl+qEPXExYO+fdvaUxAMaXVRKjDtsjtekmmgSLcZ8L9Shrxu816 oUDrxnK6GhEWy1Q6691s2cbW21MmJLhispCkfYty+R++jDZ9NarZUXbdYDIynTdo9moc V4mTSB+JQLmWpOpnWFryk0pGlbop0gW1OeeGUWLvRalUivctYrd40i+PMaZ+7mfWdC33 85bamVg34HD8m04I9GcbPjGfOcHIwx90dXGGCN2xdsUTc6DzbsH464xvbT/vrriCNKC2 GQvWsFCndEDXskpXqAAzlrAOA83gmKkPaTw014+EIRslB4QDbJgbkZcICwZRVHMw2TzS HlZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rLp+y2Cb7SGsOa4Obd9vWUi02TdIW1M+K1OPPdobolA=; b=rZNfQnhgOyFyTUd0jEdk7++f+40EUTXASpvKZLFwosSGFuTqPlOo5i6kZ7ICLDOnHQ O8drfAgoxbqf2yrPfxAbbRUXyuZHMJbeld7/WlQv3qCgBbpjip+Eu4BS9KaOQK80nmlP KpfJyHvTVWFrAtVcbJbec66tUkt3AB6UFrLgcy1oFXRUBznrTmOFDqn8HLr4zFlq6VAU OictliRgcV4BiraRyNpJfh+EMK1zOYIEo2ZOrlCPkGjhXricGzKzaaZBJTTFb6fI0+Z6 IBYwwBKoYuNXESCE/ICsgQ/Xt3FJztWpaffzdRHjnI0Qu8LNs2cc2tSAEB/tKT0ymEYZ CL9w== X-Gm-Message-State: AJIora+IxBBS5MBwv8pSzB6AeIiVdvKHn8tek6mAly5zrc1fckpETY5x 4m8F/PjKd52dc5lbu+R5IJYvk4d+HR4eBQ== X-Google-Smtp-Source: AGRyM1v77VP+8Peb3biaoWVo6lzSoqPJlKccOfowZVRMskAUw+z7U1tAnzJKUz4+GJeT5q69WcbtbA== X-Received: by 2002:a5d:55ca:0:b0:211:4092:1c27 with SMTP id i10-20020a5d55ca000000b0021140921c27mr5405759wrw.108.1655217479849; Tue, 14 Jun 2022 07:37:59 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:37:59 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 18/25] io_uring: limit number hash buckets Date: Tue, 14 Jun 2022 15:37:08 +0100 Message-Id: <57a30f28ac6235d7913444ff3bee9bc5ce70f5d4.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't allocate to many hash/cancellation buckets, there might be too many, clamp it to 8 bits, or 256 * 64B = 16KB. We don't usually have too many requests, and 256 buckets should be enough, especially since we do hash search only in the cancellation path. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4d94757e6e28..2a7a5db12a0e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -710,12 +710,12 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) /* * Use 5 bits less than the max cq entries, that should give us around - * 32 entries per hash list if totally full and uniformly spread. + * 32 entries per hash list if totally full and uniformly spread, but + * don't keep too many buckets to not overconsume memory. */ - hash_bits = ilog2(p->cq_entries); - hash_bits -= 5; - if (hash_bits <= 0) - hash_bits = 1; + hash_bits = ilog2(p->cq_entries) - 5; + hash_bits = clamp(hash_bits, 1, 8); + ctx->cancel_hash_bits = hash_bits; ctx->cancel_hash = kmalloc((1U << hash_bits) * sizeof(struct io_hash_bucket), From patchwork Tue Jun 14 14:37:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC73CCCA47C for ; Tue, 14 Jun 2022 14:38:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238292AbiFNOiI (ORCPT ); Tue, 14 Jun 2022 10:38:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238373AbiFNOiE (ORCPT ); Tue, 14 Jun 2022 10:38:04 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1B2213F93 for ; Tue, 14 Jun 2022 07:38:01 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id v14so11585026wra.5 for ; Tue, 14 Jun 2022 07:38:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+JMB8B8dyekj9aNFFK4S1oUXrhmLvJwYZDZItF2JzC0=; b=o/O0wxEjafZhVmdyI5WhEC5iiOYANkX/lwYHXJLzHIYWLr8nS1ZxA0DOiSPXDJ2xFF tVV6B/CKcfOTjmNrXOyJl1Y7VNOKlgVwTvSaZGh7UgxHdG+jvGopSbZ7MWKglLErqbLU CEd7Q0q0hjOUjPfrDJlpoizzj4ZlN7kNCEDLQbgtJtObn88haEYC5zykGmfuINo1lCeG +lMUhjj0AS8v0Jqf+7pN7VeUpleyWuFBxnLJQtUb8VtK0XRtn1AN3Br8UqDTzq5krb/M j+gkNghe97N06Gz9RSI4sXjDuqSA4GxRKilWd4ro0hmh03dwuYw8ZI75QQL2qPXTkfub gWig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+JMB8B8dyekj9aNFFK4S1oUXrhmLvJwYZDZItF2JzC0=; b=FCMEXzwL2pHdij82+fO63+yvVB2rbNRI4nKnnRKX06NX+60sdc8Hjzw6vnli4bWXQU 4Wbq7iPeqULIiZ4ImjXVjVe2uaBrJp2tV//2mBAJnTJUhCVm7vR4kKyzkKNma90kAxB6 JNJxBp04+mQ1wxOlMXBlshXZ7zoUJyRJtcUkbwlkjD9vZYEtnuk9XvQLprEeAdfuuh+2 xuimV/B7/D7M8chCCv1ZG+S2KT8mU68KATPPVuuP8FURbtmwGHA8iMyxAMT7npfHGr6x IHwFO5VodGFE2LH08NoFyN49ehaClotwZc+3OXTJZKQWPzhdQeqq0ClWx8mKuwqHvdQS BA5w== X-Gm-Message-State: AJIora8dm3D14gM90Iy+ThotoRh4M5UfxWaSArP6oyZhB3f3s02lKrdu 2WJGJLFP7YWN7Zehk/gwRAgL8lvA/D7c2Q== X-Google-Smtp-Source: AGRyM1t1Bj9dvHJljkNYYB3dDa5qljmKTk1Y75gxCw9grpD0wEqvaloJkXp+bBVbHoW9P+R6TToe3A== X-Received: by 2002:adf:fd0a:0:b0:219:ea8d:c09e with SMTP id e10-20020adffd0a000000b00219ea8dc09emr5173395wrr.13.1655217481028; Tue, 14 Jun 2022 07:38:01 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.37.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:38:00 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 19/25] io_uring: clean up io_ring_ctx_alloc Date: Tue, 14 Jun 2022 15:37:09 +0100 Message-Id: <3d6a3659fcb7d777530b5cc67aac3e036a212dda.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a variable for the number of hash buckets in io_ring_ctx_alloc(), makes it more readable. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2a7a5db12a0e..15d209f334eb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -700,6 +700,8 @@ static __cold void io_fallback_req_func(struct work_struct *work) static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) { struct io_ring_ctx *ctx; + unsigned hash_buckets; + size_t hash_size; int hash_bits; ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); @@ -715,15 +717,15 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) */ hash_bits = ilog2(p->cq_entries) - 5; hash_bits = clamp(hash_bits, 1, 8); + hash_buckets = 1U << hash_bits; + hash_size = hash_buckets * sizeof(struct io_hash_bucket); ctx->cancel_hash_bits = hash_bits; - ctx->cancel_hash = - kmalloc((1U << hash_bits) * sizeof(struct io_hash_bucket), - GFP_KERNEL); + ctx->cancel_hash = kmalloc(hash_size, GFP_KERNEL); if (!ctx->cancel_hash) goto err; - init_hash_table(ctx->cancel_hash, 1U << hash_bits); + init_hash_table(ctx->cancel_hash, hash_buckets); ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) From patchwork Tue Jun 14 14:37:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BF4FC433EF for ; Tue, 14 Jun 2022 14:38:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238801AbiFNOiO (ORCPT ); Tue, 14 Jun 2022 10:38:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245372AbiFNOiG (ORCPT ); Tue, 14 Jun 2022 10:38:06 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 613761A811 for ; Tue, 14 Jun 2022 07:38:04 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id h19so8350665wrc.12 for ; Tue, 14 Jun 2022 07:38:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ROudprvWUcWfvZ4cZzMiGRRFxKziH9KPUlRbWRK/uOQ=; b=GAMbl+ZtTPaSwzVHjzrdBbNqiOQR2LQPA3kw5MsErPMNIj6kSHvlmE7GDn0dVHMNyi D/eZAegiapZhPiL8E1ZVioDz9skdr+DOpeT2CFjE3mhM7dyc3MKa/uLCfWawmW3yG7s2 2u6B+soOtm1bm4g0vz7gZ6Ho82e+t/Z1vDAfCwWT1VyjOuCmtDjPwVjkjvk/ZiJQUi33 E10cY7K6SksEwXQQsBeQi5MZi7HLlr8IAe+nMO50PbwAjMcj+Tr3I7mV3XDKm+OZTMaJ k/rIZeO6xLthOvO7dZOJpkJtMa1k86gkXjx3TOtCpBuD7LtnhaO2XnPd92cays4uqkZH BPqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ROudprvWUcWfvZ4cZzMiGRRFxKziH9KPUlRbWRK/uOQ=; b=beezAI3vuGWk9uTCihTx7SWjYaxqlR1n/0A3RJntSi0P3KN9UjJultVkJB/zYIAeP7 eG7uRz4mBpe/iyBP1WElVYNIEym0r7VflN1buflzfG5PKDqNxZO8mjO6UOb1zlFD/Eux WtBn08sBuRWvJ3c6+7xosAEUe3gmBEtzAu6lAALmYVuuk4YKvNYTUka86THjfVBSuuIy jxIn5BHX/tNjGrDZWSupwQ97CBg/H13tMCdnNg7p/itslQffze3Cd7wTvgS/j+WrLxjm aRMuIbVX9Bb5Ub7O7bFVJ1n9C5wiOY8hnipZzsORTkVJdbfIz7gKp5Ga+PkBB8KUBQjc Czzw== X-Gm-Message-State: AJIora+dnBZ4lvcKSPmZEnV3VayjtFY3PlTmPGotx6KbXxseadvQR7lZ NnHOFUxuvVZgZ613CVxwvxLE0c2f0RYH+g== X-Google-Smtp-Source: AGRyM1ufJnFvraGneiNgRnc51M/mbO6vW0TmZwNSMXYEgUiPZ9bnD4EEGWIUaZxPd5OTRUR2wE5VDw== X-Received: by 2002:adf:ed0a:0:b0:217:7f86:2e0c with SMTP id a10-20020adfed0a000000b002177f862e0cmr5316773wro.322.1655217482312; Tue, 14 Jun 2022 07:38:02 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.38.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:38:01 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 20/25] io_uring: use state completion infra for poll reqs Date: Tue, 14 Jun 2022 15:37:10 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use io_req_task_complete() for poll request completions, so it can utilise state completions and save lots of unnecessary locking. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 7fc4aafcca95..c4ce98504986 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -234,12 +234,8 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) io_poll_remove_entries(req); io_poll_req_delete(req, ctx); - spin_lock(&ctx->completion_lock); - req->cqe.flags = 0; - __io_req_complete_post(req); - io_commit_cqring(ctx); - spin_unlock(&ctx->completion_lock); - io_cqring_ev_posted(ctx); + io_req_set_res(req, req->cqe.res, 0); + io_req_task_complete(req, locked); } static void io_apoll_task_func(struct io_kiocb *req, bool *locked) From patchwork Tue Jun 14 14:37:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CF2FCCA47C for ; Tue, 14 Jun 2022 14:38:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239570AbiFNOiQ (ORCPT ); Tue, 14 Jun 2022 10:38:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237979AbiFNOiH (ORCPT ); Tue, 14 Jun 2022 10:38:07 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90C82219C for ; Tue, 14 Jun 2022 07:38:05 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id m24so11546658wrb.10 for ; Tue, 14 Jun 2022 07:38:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OlumRzI1KLIbIp9c72c5FvGFwKCu8xvw16ufWxwZEZY=; b=A+K4eUB+qwwaPdIdkHOwvSAMP1jVfJjw6StzphTSpfIkn2UkVbUsyFtNqxazqGvG0v b1KaeA/k0PFCf22vRCQFDYAcNBcWSDnxBqZBj8iz3XmuoebzwwvcqBhfpxGbA0lV8fxY mAjQ7jwzwyewBhphwvoVphcbPXTvMoK5HIxPXepnnuWsk3wmR6fBrvYtcsFMpc89yd+S 5LQipqZioInwE1YgOHojfJx6qtjj9XqpH5XXwATenFeIvJiHK9NZVD+wCK0HbXte5Lk1 RmB/0SrnGlowwLWevGRB+BCUjedXNJnAo3HpemXEsfbFjeMVsZWmM7gPZcwQ9JOqLcrt EJnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OlumRzI1KLIbIp9c72c5FvGFwKCu8xvw16ufWxwZEZY=; b=NrHUoghEhfGDWRR1ACQLn0hjV0odLBY1RunQwMT29/mnQRDR9mpIBPhgRJvVkPNP0Y KVo+SyrMIfmNc+LWFNam3mX1M4/9RKgqcAzayPYSYcbczmzhlN1PpxCJwczrepiul7VN bDJ6LcyqlJIKSM5/DXO2SGkSFrKbLLFn9sun1IpebEogdeXPF20+Yxtq/p75LgdzzXGe X6UEapGgmX623r7VdwzCok8UPB+tXtyPCRCuSKxr4j+xcO0v7mJukySMoJDp2BmnH+Ij YVLyiIiXHOVSNwVrZlYhtInTSqafBVPCW8vGbjd3cg0TWekkKCKQuAcMkAY2ozH2yFkp ud5g== X-Gm-Message-State: AJIora8JkpOffgrwuwab+mRac4ExP4X+B0fTMNZhUJfFl95qULDztkf3 NlpwlapHWvoMVJ0XP6VuYUErxQwRfjAvnQ== X-Google-Smtp-Source: AGRyM1sc2mJcRJtWDKBGJ7afWeQ9m8zvMwrhON5YORChE+jO8Ht63YyHWA9gNZzu9yTvR4SSw0BuJg== X-Received: by 2002:a5d:6d84:0:b0:217:e7af:224b with SMTP id l4-20020a5d6d84000000b00217e7af224bmr5338266wrs.512.1655217483761; Tue, 14 Jun 2022 07:38:03 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.38.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:38:03 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 21/25] io_uring: add IORING_SETUP_SINGLE_ISSUER Date: Tue, 14 Jun 2022 15:37:11 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a new IORING_SETUP_SINGLE_ISSUER flag and the userspace visible part of it, i.e. put limitations of submitters. Also, don't allow it together with IOPOLL as we're not going to put it to good use. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 5 ++++- io_uring/io_uring.c | 7 +++++-- io_uring/io_uring_types.h | 1 + io_uring/tctx.c | 27 ++++++++++++++++++++++++--- io_uring/tctx.h | 4 ++-- 5 files changed, 36 insertions(+), 8 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index a41ddb8c5e1f..a3a691340d3e 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -138,9 +138,12 @@ enum { * IORING_SQ_TASKRUN in the sq ring flags. Not valid with COOP_TASKRUN. */ #define IORING_SETUP_TASKRUN_FLAG (1U << 9) - #define IORING_SETUP_SQE128 (1U << 10) /* SQEs are 128 byte */ #define IORING_SETUP_CQE32 (1U << 11) /* CQEs are 32 byte */ +/* + * Only one task is allowed to submit requests + */ +#define IORING_SETUP_SINGLE_ISSUER (1U << 12) enum io_uring_op { IORING_OP_NOP, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 15d209f334eb..4b90439808e3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3020,6 +3020,8 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_destroy_buffers(ctx); if (ctx->sq_creds) put_cred(ctx->sq_creds); + if (ctx->submitter_task) + put_task_struct(ctx->submitter_task); /* there are no registered resources left, nobody uses it */ if (ctx->rsrc_node) @@ -3752,7 +3754,7 @@ static int io_uring_install_fd(struct io_ring_ctx *ctx, struct file *file) if (fd < 0) return fd; - ret = io_uring_add_tctx_node(ctx); + ret = __io_uring_add_tctx_node(ctx, false); if (ret) { put_unused_fd(fd); return ret; @@ -3972,7 +3974,8 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params) IORING_SETUP_CLAMP | IORING_SETUP_ATTACH_WQ | IORING_SETUP_R_DISABLED | IORING_SETUP_SUBMIT_ALL | IORING_SETUP_COOP_TASKRUN | IORING_SETUP_TASKRUN_FLAG | - IORING_SETUP_SQE128 | IORING_SETUP_CQE32)) + IORING_SETUP_SQE128 | IORING_SETUP_CQE32 | + IORING_SETUP_SINGLE_ISSUER)) return -EINVAL; return io_uring_create(entries, &p, params); diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index aba0f8cd6f49..f6d0ad25f377 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -241,6 +241,7 @@ struct io_ring_ctx { /* Keep this last, we don't need it for the fast path */ struct io_restriction restrictions; + struct task_struct *submitter_task; /* slow path rsrc auxilary data, used by update/register */ struct io_rsrc_node *rsrc_backup_node; diff --git a/io_uring/tctx.c b/io_uring/tctx.c index 6adf659687f8..012be261dc50 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -81,12 +81,32 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, return 0; } -int __io_uring_add_tctx_node(struct io_ring_ctx *ctx) +static int io_register_submitter(struct io_ring_ctx *ctx) +{ + int ret = 0; + + mutex_lock(&ctx->uring_lock); + if (!ctx->submitter_task) + ctx->submitter_task = get_task_struct(current); + else if (ctx->submitter_task != current) + ret = -EEXIST; + mutex_unlock(&ctx->uring_lock); + + return ret; +} + +int __io_uring_add_tctx_node(struct io_ring_ctx *ctx, bool submitter) { struct io_uring_task *tctx = current->io_uring; struct io_tctx_node *node; int ret; + if ((ctx->flags & IORING_SETUP_SINGLE_ISSUER) && submitter) { + ret = io_register_submitter(ctx); + if (ret) + return ret; + } + if (unlikely(!tctx)) { ret = io_uring_alloc_task_context(current, ctx); if (unlikely(ret)) @@ -120,7 +140,8 @@ int __io_uring_add_tctx_node(struct io_ring_ctx *ctx) list_add(&node->ctx_node, &ctx->tctx_list); mutex_unlock(&ctx->uring_lock); } - tctx->last = ctx; + if (submitter) + tctx->last = ctx; return 0; } @@ -228,7 +249,7 @@ int io_ringfd_register(struct io_ring_ctx *ctx, void __user *__arg, return -EINVAL; mutex_unlock(&ctx->uring_lock); - ret = io_uring_add_tctx_node(ctx); + ret = __io_uring_add_tctx_node(ctx, false); mutex_lock(&ctx->uring_lock); if (ret) return ret; diff --git a/io_uring/tctx.h b/io_uring/tctx.h index 7684713e950f..dde82ce4d8e2 100644 --- a/io_uring/tctx.h +++ b/io_uring/tctx.h @@ -34,7 +34,7 @@ struct io_tctx_node { int io_uring_alloc_task_context(struct task_struct *task, struct io_ring_ctx *ctx); void io_uring_del_tctx_node(unsigned long index); -int __io_uring_add_tctx_node(struct io_ring_ctx *ctx); +int __io_uring_add_tctx_node(struct io_ring_ctx *ctx, bool submitter); void io_uring_clean_tctx(struct io_uring_task *tctx); void io_uring_unreg_ringfd(void); @@ -52,5 +52,5 @@ static inline int io_uring_add_tctx_node(struct io_ring_ctx *ctx) if (likely(tctx && tctx->last == ctx)) return 0; - return __io_uring_add_tctx_node(ctx); + return __io_uring_add_tctx_node(ctx, true); } From patchwork Tue Jun 14 14:37:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8B8BC43334 for ; Tue, 14 Jun 2022 14:38:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349561AbiFNOiN (ORCPT ); Tue, 14 Jun 2022 10:38:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238801AbiFNOiH (ORCPT ); Tue, 14 Jun 2022 10:38:07 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5B0AB7F2 for ; Tue, 14 Jun 2022 07:38:05 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id o8so11589460wro.3 for ; Tue, 14 Jun 2022 07:38:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5V/K45g9wpqoJNn9BXBkT927aYIh71OeYAeKJeObVFk=; b=Gy5vuXnuP7saIwdahAkXjinI/Yd+KNxa/Vwmt/ZfnMfWL0on4K+XRJ8gyOlrBmvML/ H/0aCjCW53VtxPraDoIrnOyIeb+SNMtshDMJLqnr2BWNjNFqe1siqaWOaB3o6cBsLY2P 7z8fZqP+/DjHuLMu+RuomszlxuyAcsrghvp+Tq/LYBkHVzqBQiYapepY+ob1SIPzpOzt p3QDj2ePy5EFmoNpbZWul46VZHTlnnFkDgbuFepFCWkH2TKcPa+d+YGuxT8/7rReGvph jHUHBOSUZ8+w5/H3tIeQYLjumHgmrCao4DvEkzHTmLH/DPXcrPCiBaqPZcpsCtNBhZvo MV7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5V/K45g9wpqoJNn9BXBkT927aYIh71OeYAeKJeObVFk=; b=VOEu3j7EsSv+dWJCVCDzukby54YQerpDRbuM6ReC3r3g8gHXO7rlkYe6Wb6aNgixv3 /9qC3oaFbuU2txUCRgV7Zno+0XSCGp4SISMWCmjmmKiV9nrziJRDGsTeZFyf5V6GBlbT /9TRN7nygrQSycNK5bZwXxTO04BLuSrtrcWGMcFVRZCz4U+v/JumLxC79ahhf1UF+3u8 7E+mUpufSvvmqyXjs2f4qQ7sVerYI6Hkea+vVfGHoPJl3U3VwrXRf+LOfWODkJ4TFXgG uzXTq1mf2NFwpjURtr+p0B38SCEbag5oP82KrTuPoAFBmmWZTxf6Nf2kItmbGBG4xGyY BogQ== X-Gm-Message-State: AJIora/v6Y/F6FKrKyc6fzE0P2QcaH1B06wpL9DxnhBUZuAJPxIxD3f3 he8gYAX4dRMLNg/M71grw0alwZAwRaZqDw== X-Google-Smtp-Source: AGRyM1vUEPnhNRN4BsQTN+dWMQg/ky6jIyssdgWeRupGHZjbzeMCNoVh6b//qJgqks4e8Nuc06PRTQ== X-Received: by 2002:a5d:6da9:0:b0:218:5b98:41b with SMTP id u9-20020a5d6da9000000b002185b98041bmr5259986wrs.111.1655217485062; Tue, 14 Jun 2022 07:38:05 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.38.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:38:04 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 22/25] io_uring: pass hash table into poll_find Date: Tue, 14 Jun 2022 15:37:12 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org In preparation for having multiple cancellation hash tables, pass a table pointer into io_poll_find() and other poll cancel functions. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index c4ce98504986..5cc03be365e3 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -556,11 +556,12 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd, + struct io_hash_bucket hash_table[], struct io_hash_bucket **out_bucket) { struct io_kiocb *req; u32 index = hash_long(cd->data, ctx->cancel_hash_bits); - struct io_hash_bucket *hb = &ctx->cancel_hash[index]; + struct io_hash_bucket *hb = &hash_table[index]; *out_bucket = NULL; @@ -584,6 +585,7 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + struct io_hash_bucket hash_table[], struct io_hash_bucket **out_bucket) { struct io_kiocb *req; @@ -592,7 +594,7 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, *out_bucket = NULL; for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &ctx->cancel_hash[i]; + struct io_hash_bucket *hb = &hash_table[i]; spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { @@ -619,15 +621,16 @@ static bool io_poll_disarm(struct io_kiocb *req) return true; } -int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) +static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + struct io_hash_bucket hash_table[]) { struct io_hash_bucket *bucket; struct io_kiocb *req; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd, &bucket); + req = io_poll_file_find(ctx, cd, ctx->cancel_hash, &bucket); else - req = io_poll_find(ctx, false, cd, &bucket); + req = io_poll_find(ctx, false, cd, ctx->cancel_hash, &bucket); if (req) io_poll_cancel_req(req); @@ -636,6 +639,11 @@ int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) return req ? 0 : -ENOENT; } +int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) +{ + return __io_poll_cancel(ctx, cd, ctx->cancel_hash); +} + static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, unsigned int flags) { @@ -731,7 +739,7 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) int ret2, ret = 0; bool locked; - preq = io_poll_find(ctx, true, &cd, &bucket); + preq = io_poll_find(ctx, true, &cd, ctx->cancel_hash, &bucket); if (preq) ret2 = io_poll_disarm(preq); if (bucket) From patchwork Tue Jun 14 14:37:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C51BCCA47A for ; Tue, 14 Jun 2022 14:38:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349613AbiFNOiQ (ORCPT ); Tue, 14 Jun 2022 10:38:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343850AbiFNOiL (ORCPT ); Tue, 14 Jun 2022 10:38:11 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D2AC303 for ; Tue, 14 Jun 2022 07:38:08 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id x6-20020a1c7c06000000b003972dfca96cso4911823wmc.4 for ; Tue, 14 Jun 2022 07:38:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bsOgLB7d4Bz4URPvlQh5dv2DDF4WMnAM31PCgutwcLw=; b=NuSyeyulg6Y2OZ/wQmbQWmsij9CuU4jYGYR2cxpLfiKTUosFVoDN4qaJCC6KNo717F sYJZa/AY+v12GgnvBn4GOAb2Ay4zOx5/auQN9lB6FwCSulkaZmro3XvO+fXMAOFlDEsu VbAGjCOwDRiUuvM+82NWhEABb9O1L2QXMYMwp3Pe6wLhLVkTrgGwEW7VCMJTnZK1Raom dBynjR0S/yNPH2rECLkse5qeSzmenL7wr/ciUK76zOWH3drGqbUntCX7//2mk4OAzmao 9/H3Ii7oIdOQJ3gpZSTyGJO0fvbnLzmPuwN2o7Sakhx0zXhSkyXQyBIE5nteEZD/9TGM NLBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bsOgLB7d4Bz4URPvlQh5dv2DDF4WMnAM31PCgutwcLw=; b=vs07PWXE/ylujSODn8/BJnTi6ND3qcQQk4J1jOC7LcF5Rm2XlxCzWXfUQgy6riHkTe bXqualYGrNG73B9uLIbrrIT8wJYFDRoHszmyaH4CEXZ1NxlP/S4nwdNSTCkZYWz5X454 huqLV0Kg1I5Y3/TFlVWgAltVK4Sl9o+VdgqwOsyQhyl0OMYGIMF3DNje8Nx4XgX9FZkA rSOa1rTzygi9YDYq4gcSJzdaEBQq/ZEMYau/mvRx+FHYhtnOR4eWKDnxdz05Lmiab6qh c04pAEi0K8KopEn1Fm3uHYZHwV/raSByG75n734nXNVvaQedrbYqWZB6wAv2vqISe0vv TH2A== X-Gm-Message-State: AOAM533QAn+POha+/sbUUF/l1n+kR95IWr3x5uos0/Zdps73GhYSeviv j7SulowFbJ4ToQKOVRZV8cQCXJixqtkgGA== X-Google-Smtp-Source: ABdhPJw7LkJzdukd5+j5VI2QcQtBkYsZPTDlaznajNdIT54+A9rkYF9n8su+Iap9vf2m3Enem4KPiA== X-Received: by 2002:a1c:4c0d:0:b0:39c:5233:1873 with SMTP id z13-20020a1c4c0d000000b0039c52331873mr4584737wmf.28.1655217486224; Tue, 14 Jun 2022 07:38:06 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.38.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:38:05 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 23/25] io_uring: introduce a struct for hash table Date: Tue, 14 Jun 2022 15:37:13 +0100 Message-Id: <3ec530d62e0efac1e50daacaf00c31c2e4a5a48b.1655213915.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of passing around a pointer to hash buckets, add a bit of type safety and wrap it into a structure. Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 6 +++--- io_uring/cancel.h | 7 +------ io_uring/fdinfo.c | 4 ++-- io_uring/io_uring.c | 29 ++++++++++++++++------------ io_uring/io_uring_types.h | 13 +++++++++++-- io_uring/poll.c | 40 +++++++++++++++++++++------------------ 6 files changed, 56 insertions(+), 43 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index a253e2ad22eb..f28f0a7d1272 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -193,12 +193,12 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -void init_hash_table(struct io_hash_bucket *hash_table, unsigned size) +void init_hash_table(struct io_hash_table *table, unsigned size) { unsigned int i; for (i = 0; i < size; i++) { - spin_lock_init(&hash_table[i].lock); - INIT_HLIST_HEAD(&hash_table[i].list); + spin_lock_init(&table->hbs[i].lock); + INIT_HLIST_HEAD(&table->hbs[i].list); } } diff --git a/io_uring/cancel.h b/io_uring/cancel.h index 556a7dcf160e..fd4cb1a2595d 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -4,9 +4,4 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags); int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd); -void init_hash_table(struct io_hash_bucket *hash_table, unsigned size); - -struct io_hash_bucket { - spinlock_t lock; - struct hlist_head list; -} ____cacheline_aligned_in_smp; +void init_hash_table(struct io_hash_table *table, unsigned size); diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index f941c73f5502..344e7d90d557 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -158,8 +158,8 @@ static __cold void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, mutex_unlock(&ctx->uring_lock); seq_puts(m, "PollList:\n"); - for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &ctx->cancel_hash[i]; + for (i = 0; i < (1U << ctx->cancel_table.hash_bits); i++) { + struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; struct io_kiocb *req; spin_lock(&hb->lock); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4b90439808e3..4bead16e57f7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -697,11 +697,23 @@ static __cold void io_fallback_req_func(struct work_struct *work) percpu_ref_put(&ctx->refs); } +static int io_alloc_hash_table(struct io_hash_table *table, unsigned bits) +{ + unsigned hash_buckets = 1U << bits; + size_t hash_size = hash_buckets * sizeof(table->hbs[0]); + + table->hbs = kmalloc(hash_size, GFP_KERNEL); + if (!table->hbs) + return -ENOMEM; + + table->hash_bits = bits; + init_hash_table(table, hash_buckets); + return 0; +} + static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) { struct io_ring_ctx *ctx; - unsigned hash_buckets; - size_t hash_size; int hash_bits; ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); @@ -717,16 +729,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) */ hash_bits = ilog2(p->cq_entries) - 5; hash_bits = clamp(hash_bits, 1, 8); - hash_buckets = 1U << hash_bits; - hash_size = hash_buckets * sizeof(struct io_hash_bucket); - - ctx->cancel_hash_bits = hash_bits; - ctx->cancel_hash = kmalloc(hash_size, GFP_KERNEL); - if (!ctx->cancel_hash) + if (io_alloc_hash_table(&ctx->cancel_table, hash_bits)) goto err; - init_hash_table(ctx->cancel_hash, hash_buckets); - ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) goto err; @@ -767,7 +772,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) return ctx; err: kfree(ctx->dummy_ubuf); - kfree(ctx->cancel_hash); + kfree(ctx->cancel_table.hbs); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); kfree(ctx); @@ -3050,7 +3055,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_req_caches_free(ctx); if (ctx->hash_map) io_wq_put_hash(ctx->hash_map); - kfree(ctx->cancel_hash); + kfree(ctx->cancel_table.hbs); kfree(ctx->dummy_ubuf); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index f6d0ad25f377..ce2fbe6749bb 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -7,6 +7,16 @@ #include "io-wq.h" #include "filetable.h" +struct io_hash_bucket { + spinlock_t lock; + struct hlist_head list; +} ____cacheline_aligned_in_smp; + +struct io_hash_table { + struct io_hash_bucket *hbs; + unsigned hash_bits; +}; + struct io_uring { u32 head ____cacheline_aligned_in_smp; u32 tail ____cacheline_aligned_in_smp; @@ -222,8 +232,7 @@ struct io_ring_ctx { * manipulate the list, hence no extra locking is needed there. */ struct io_wq_work_list iopoll_list; - struct io_hash_bucket *cancel_hash; - unsigned cancel_hash_bits; + struct io_hash_table cancel_table; bool poll_multi_queue; struct list_head io_buffers_comp; diff --git a/io_uring/poll.c b/io_uring/poll.c index 5cc03be365e3..9c7793f5e93b 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -73,9 +73,9 @@ static struct io_poll *io_poll_get_single(struct io_kiocb *req) static void io_poll_req_insert(struct io_kiocb *req) { - struct io_ring_ctx *ctx = req->ctx; - u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); - struct io_hash_bucket *hb = &ctx->cancel_hash[index]; + struct io_hash_table *table = &req->ctx->cancel_table; + u32 index = hash_long(req->cqe.user_data, table->hash_bits); + struct io_hash_bucket *hb = &table->hbs[index]; spin_lock(&hb->lock); hlist_add_head(&req->hash_node, &hb->list); @@ -84,8 +84,9 @@ static void io_poll_req_insert(struct io_kiocb *req) static void io_poll_req_delete(struct io_kiocb *req, struct io_ring_ctx *ctx) { - u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); - spinlock_t *lock = &ctx->cancel_hash[index].lock; + struct io_hash_table *table = &req->ctx->cancel_table; + u32 index = hash_long(req->cqe.user_data, table->hash_bits); + spinlock_t *lock = &table->hbs[index].lock; spin_lock(lock); hash_del(&req->hash_node); @@ -533,13 +534,15 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, bool cancel_all) { + struct io_hash_table *table = &ctx->cancel_table; + unsigned nr_buckets = 1U << table->hash_bits; struct hlist_node *tmp; struct io_kiocb *req; bool found = false; int i; - for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &ctx->cancel_hash[i]; + for (i = 0; i < nr_buckets; i++) { + struct io_hash_bucket *hb = &table->hbs[i]; spin_lock(&hb->lock); hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) { @@ -556,12 +559,12 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd, - struct io_hash_bucket hash_table[], + struct io_hash_table *table, struct io_hash_bucket **out_bucket) { struct io_kiocb *req; - u32 index = hash_long(cd->data, ctx->cancel_hash_bits); - struct io_hash_bucket *hb = &hash_table[index]; + u32 index = hash_long(cd->data, table->hash_bits); + struct io_hash_bucket *hb = &table->hbs[index]; *out_bucket = NULL; @@ -585,16 +588,17 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, struct io_cancel_data *cd, - struct io_hash_bucket hash_table[], + struct io_hash_table *table, struct io_hash_bucket **out_bucket) { + unsigned nr_buckets = 1U << table->hash_bits; struct io_kiocb *req; int i; *out_bucket = NULL; - for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &hash_table[i]; + for (i = 0; i < nr_buckets; i++) { + struct io_hash_bucket *hb = &table->hbs[i]; spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { @@ -622,15 +626,15 @@ static bool io_poll_disarm(struct io_kiocb *req) } static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, - struct io_hash_bucket hash_table[]) + struct io_hash_table *table) { struct io_hash_bucket *bucket; struct io_kiocb *req; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd, ctx->cancel_hash, &bucket); + req = io_poll_file_find(ctx, cd, table, &bucket); else - req = io_poll_find(ctx, false, cd, ctx->cancel_hash, &bucket); + req = io_poll_find(ctx, false, cd, table, &bucket); if (req) io_poll_cancel_req(req); @@ -641,7 +645,7 @@ static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) { - return __io_poll_cancel(ctx, cd, ctx->cancel_hash); + return __io_poll_cancel(ctx, cd, &ctx->cancel_table); } static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, @@ -739,7 +743,7 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) int ret2, ret = 0; bool locked; - preq = io_poll_find(ctx, true, &cd, ctx->cancel_hash, &bucket); + preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table, &bucket); if (preq) ret2 = io_poll_disarm(preq); if (bucket) From patchwork Tue Jun 14 14:37:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCC5DCCA480 for ; Tue, 14 Jun 2022 14:38:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237979AbiFNOim (ORCPT ); Tue, 14 Jun 2022 10:38:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348963AbiFNOiM (ORCPT ); Tue, 14 Jun 2022 10:38:12 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 997002253A for ; Tue, 14 Jun 2022 07:38:09 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id k19so11560954wrd.8 for ; Tue, 14 Jun 2022 07:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FnpmHNOzLmPzZqlagM4s9zuVri1qJ9NLqujQ99u3Owo=; b=PuFylavCOt+W/9Uo29+Z271fp2DAP5jMsIFFk7EP941Hx34YHthhJAjU2Pzr2z/Ypr yVcuegLDk4kI6f3JSraG87MLjZfM3L88WQG7vdb+Z5ViqG5SfxsHcSL2hOq8CV1Q+huz K7Caz6gmtVqHOBnhUC/MftMmUmgBgU4MCbYNr9g8ofLwLcJ/M5RD7GofiZ9A2kvcDqQH hfzqjn7k7xLav2Ono4Sz0B9vJcbhDmp9RvqqTxNj2iiDbnKg/ln03aRw1hm4L3o4viHr CriZTcFbbb1Bgao6g+Uz0/0j1SwS33JLbEbAIt591gvI8epLs+8aJ0qGEbq2LM+YNukW bxEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FnpmHNOzLmPzZqlagM4s9zuVri1qJ9NLqujQ99u3Owo=; b=E+nUyfKAsY/szR0yvy2OhpPUTXQgG9+kgGeQqcYpUn5nWLGtc8fjxT4nTfLIUcbds9 r4TfTVOU/Sb6QkYViTGvaa40+ffeqi8MJl9TUeljZden9EeZPRlJt/nhzg+fFGxl6m+I CpHOucmKaaKnFsxvkqrCjgVwMTctLTWRpXB5zKf9M2SuEcHOuYAkS7jYEexxlyblnI0d wb7Ar5YZmRYASe/aUH6MN3EuE2Ej+UM7dge3N2CrAGEhUVG5IG+lEKWiBYqv1pOzyiWU ktKPQVC9D2GWP4j2o4eRkblOs8xiBKbsNIKmFhlzHw9NUamziA4j3zGLY26UKj6dY6AT PWgw== X-Gm-Message-State: AJIora9fG3mDgD9UA/naICyfWNm/8riavwYn8T73+b4cgvUVCRghqZB0 KnEAuuRz+VuqY0FndpVrPbDpYL9TTcgTPw== X-Google-Smtp-Source: AGRyM1vWMZVd2gRiS5wVVoCqpGNpUBRaY9oZatCpWctQUkmhe2zLkBb44O+s7dkXFZH7WrSOHHmqow== X-Received: by 2002:a5d:4d8f:0:b0:210:3e14:ff27 with SMTP id b15-20020a5d4d8f000000b002103e14ff27mr5080991wru.81.1655217487802; Tue, 14 Jun 2022 07:38:07 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.38.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:38:07 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 24/25] io_uring: propagate locking state to poll cancel Date: Tue, 14 Jun 2022 15:37:14 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Poll cancellation will be soon need to grab ->uring_lock inside, pass the locking state, i.e. issue_flags, inside the cancellation functions. Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 7 ++++--- io_uring/cancel.h | 3 ++- io_uring/poll.c | 3 ++- io_uring/poll.h | 3 ++- io_uring/timeout.c | 3 ++- 5 files changed, 12 insertions(+), 7 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index f28f0a7d1272..f07bfd27c98a 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -78,7 +78,8 @@ static int io_async_cancel_one(struct io_uring_task *tctx, return ret; } -int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) +int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd, + unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; int ret; @@ -93,7 +94,7 @@ int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) if (!ret) return 0; - ret = io_poll_cancel(ctx, cd); + ret = io_poll_cancel(ctx, cd, issue_flags); if (ret != -ENOENT) return ret; @@ -136,7 +137,7 @@ static int __io_async_cancel(struct io_cancel_data *cd, struct io_kiocb *req, int ret, nr = 0; do { - ret = io_try_cancel(req, cd); + ret = io_try_cancel(req, cd, issue_flags); if (ret == -ENOENT) break; if (!all) diff --git a/io_uring/cancel.h b/io_uring/cancel.h index fd4cb1a2595d..8dd259dc383e 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -3,5 +3,6 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags); -int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd); +int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd, + unsigned int issue_flags); void init_hash_table(struct io_hash_table *table, unsigned size); diff --git a/io_uring/poll.c b/io_uring/poll.c index 9c7793f5e93b..07157da1c2cb 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -643,7 +643,8 @@ static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, return req ? 0 : -ENOENT; } -int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) +int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned issue_flags) { return __io_poll_cancel(ctx, cd, &ctx->cancel_table); } diff --git a/io_uring/poll.h b/io_uring/poll.h index cc75c1567a84..fa3e19790281 100644 --- a/io_uring/poll.h +++ b/io_uring/poll.h @@ -24,7 +24,8 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags); int io_poll_remove_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags); -int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd); +int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned issue_flags); int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags); bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, bool cancel_all); diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 69cca42d6835..526fc8b2e3b6 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -262,6 +262,7 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) static void io_req_task_link_timeout(struct io_kiocb *req, bool *locked) { + unsigned issue_flags = *locked ? 0 : IO_URING_F_UNLOCKED; struct io_timeout *timeout = io_kiocb_to_cmd(req); struct io_kiocb *prev = timeout->prev; int ret = -ENOENT; @@ -273,7 +274,7 @@ static void io_req_task_link_timeout(struct io_kiocb *req, bool *locked) .data = prev->cqe.user_data, }; - ret = io_try_cancel(req, &cd); + ret = io_try_cancel(req, &cd, issue_flags); } io_req_set_res(req, ret ?: -ETIME, 0); io_req_complete_post(req); From patchwork Tue Jun 14 14:37:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5017CCA481 for ; Tue, 14 Jun 2022 14:38:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244198AbiFNOin (ORCPT ); Tue, 14 Jun 2022 10:38:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349809AbiFNOiQ (ORCPT ); Tue, 14 Jun 2022 10:38:16 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA70928E11 for ; Tue, 14 Jun 2022 07:38:10 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id a15so11589184wrh.2 for ; Tue, 14 Jun 2022 07:38:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zAujV+xdBjpkNVZ4Od0UmlONBcZnGFs9ZLKfUO5esWA=; b=XPa1ItOdPD73XsXtfp4Bh1zEldwY1+yESehoPphopw6hb3xiFgL06jMuG6FhboS+cP 8+JmkiGH7BmPcCx5COnFMl8Y0NUxokM14bTc2rmo5d1XtC+qV3ma5hjBMUzHQHAQmDKg X89Q0rQ3kKqLNY394gi7IhBjWPJp/JzLeFZWrquo5O8lFas71iPAjG5SkyEIpzRW5M9y ftPZlizsZWQLoc6UR78X7j5MS6DR7wYb3DL+Fxm3fbzwSfnkquG0tAaSKMBXfR7C+zK8 mzUq9BnppJNLlT2/CCX6ggkmMvUiO03CIsdPz5kToRXsqmzVWfETCv3U8pUsV7AqH0sj EEAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zAujV+xdBjpkNVZ4Od0UmlONBcZnGFs9ZLKfUO5esWA=; b=FCMaGdQJCRV8XH4UqHmV3Z8qd0Xq8ydIwP9zMEvnkDoHzJHudm7B9QvwDf9+sto9WL 9TSfz7tUfKCAk4zYXiBrf1+zvc7gyf8N8KXGscnKPSax4bUrLWCuLJbTet3CXYViW0F6 cMXcmQICZ9KR2bEDrngrDktHCSWq4/KoGfwiKC1ga9mf2RlLrJeSwkHrMfgkMrHe9aFE S/0wjCWNeW+k0x5gqWV75834PiJeGU6wtVgEsdX5BcpiT7Yt1op45Zl8njyGcLSGnv1q YPxQcnpk98Bidxj7K8ZvDT46gTHsX1DK+2m4fmhWgtGw3ZTDRYNxcHb4pqZu5SlathSC +F2Q== X-Gm-Message-State: AJIora+TdnUq+10ozDYH6F3j/eRtk0WClGrh69JewDPKcYtngP5TvcDi e2DFQmO33UPLHG/0SmQCOB0bibr8kGVwag== X-Google-Smtp-Source: AGRyM1t74S9qjK8pdbIuhiPqF3Nwu9YbGtBOKxKBInXj9FeGWf7mHlExN1iZzvERf8xxmc+0+l2A1Q== X-Received: by 2002:a05:6000:988:b0:219:cb95:79c4 with SMTP id by8-20020a056000098800b00219cb9579c4mr5301445wrb.587.1655217489106; Tue, 14 Jun 2022 07:38:09 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id a4-20020adff7c4000000b0021033caa332sm12353064wrq.42.2022.06.14.07.38.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 07:38:08 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 25/25] io_uring: mutex locked poll hashing Date: Tue, 14 Jun 2022 15:37:15 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Currently we do two extra spin lock/unlock pairs to add a poll/apoll request to the cancellation hash table and remove it from there. On the submission side we often already hold ->uring_lock and tw completion is likely to hold it as well. Add a second cancellation hash table protected by ->uring_lock. In concerns for latency because of a need to have the mutex locked on the completion side, use the new table only in following cases: 1) IORING_SETUP_SINGLE_ISSUER: only one task grabs uring_lock, so there is no contention and so the main tw hander will always end up grabbing it before calling into callbacks. 2) IORING_SETUP_SQPOLL: same as with single issuer, only one task is using ->uring_lock. 3) apoll: we normally grab the lock on the completion side anyway to execute the request, so it's free. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 9 +++- io_uring/io_uring_types.h | 4 ++ io_uring/poll.c | 111 ++++++++++++++++++++++++++++++-------- 3 files changed, 102 insertions(+), 22 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4bead16e57f7..1395176bc2ea 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -731,6 +731,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) hash_bits = clamp(hash_bits, 1, 8); if (io_alloc_hash_table(&ctx->cancel_table, hash_bits)) goto err; + if (io_alloc_hash_table(&ctx->cancel_table_locked, hash_bits)) + goto err; ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) @@ -773,6 +775,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) err: kfree(ctx->dummy_ubuf); kfree(ctx->cancel_table.hbs); + kfree(ctx->cancel_table_locked.hbs); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); kfree(ctx); @@ -3056,6 +3059,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) if (ctx->hash_map) io_wq_put_hash(ctx->hash_map); kfree(ctx->cancel_table.hbs); + kfree(ctx->cancel_table_locked.hbs); kfree(ctx->dummy_ubuf); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); @@ -3217,12 +3221,13 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx) __io_cqring_overflow_flush(ctx, true); xa_for_each(&ctx->personalities, index, creds) io_unregister_personality(ctx, index); + if (ctx->rings) + io_poll_remove_all(ctx, NULL, true); mutex_unlock(&ctx->uring_lock); /* failed during ring init, it couldn't have issued any requests */ if (ctx->rings) { io_kill_timeouts(ctx, NULL, true); - io_poll_remove_all(ctx, NULL, true); /* if we failed setting up the ctx, we might not have any rings */ io_iopoll_try_reap_events(ctx); } @@ -3347,7 +3352,9 @@ static __cold void io_uring_try_cancel_requests(struct io_ring_ctx *ctx, } ret |= io_cancel_defer_files(ctx, task, cancel_all); + mutex_lock(&ctx->uring_lock); ret |= io_poll_remove_all(ctx, task, cancel_all); + mutex_unlock(&ctx->uring_lock); ret |= io_kill_timeouts(ctx, task, cancel_all); if (task) ret |= io_run_task_work(); diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index ce2fbe6749bb..557b8e7719c9 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -189,6 +189,7 @@ struct io_ring_ctx { struct xarray io_bl_xa; struct list_head io_buffers_cache; + struct io_hash_table cancel_table_locked; struct list_head cq_overflow_list; struct list_head apoll_cache; struct xarray personalities; @@ -323,6 +324,7 @@ enum { /* keep async read/write and isreg together and in order */ REQ_F_SUPPORT_NOWAIT_BIT, REQ_F_ISREG_BIT, + REQ_F_HASH_LOCKED_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -388,6 +390,8 @@ enum { REQ_F_APOLL_MULTISHOT = BIT(REQ_F_APOLL_MULTISHOT_BIT), /* recvmsg special flag, clear EPOLLIN */ REQ_F_CLEAR_POLLIN = BIT(REQ_F_CLEAR_POLLIN_BIT), + /* hashed into ->cancel_hash_locked */ + REQ_F_HASH_LOCKED = BIT(REQ_F_HASH_LOCKED_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, bool *locked); diff --git a/io_uring/poll.c b/io_uring/poll.c index 07157da1c2cb..d20484c1cbb7 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -93,6 +93,26 @@ static void io_poll_req_delete(struct io_kiocb *req, struct io_ring_ctx *ctx) spin_unlock(lock); } +static void io_poll_req_insert_locked(struct io_kiocb *req) +{ + struct io_hash_table *table = &req->ctx->cancel_table_locked; + u32 index = hash_long(req->cqe.user_data, table->hash_bits); + + hlist_add_head(&req->hash_node, &table->hbs[index].list); +} + +static void io_poll_tw_hash_eject(struct io_kiocb *req, bool *locked) +{ + struct io_ring_ctx *ctx = req->ctx; + + if (req->flags & REQ_F_HASH_LOCKED) { + io_tw_lock(ctx, locked); + hash_del(&req->hash_node); + } else { + io_poll_req_delete(req, ctx); + } +} + static void io_init_poll_iocb(struct io_poll *poll, __poll_t events, wait_queue_func_t wake_func) { @@ -217,7 +237,6 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) static void io_poll_task_func(struct io_kiocb *req, bool *locked) { - struct io_ring_ctx *ctx = req->ctx; int ret; ret = io_poll_check_events(req, locked); @@ -234,7 +253,8 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) } io_poll_remove_entries(req); - io_poll_req_delete(req, ctx); + io_poll_tw_hash_eject(req, locked); + io_req_set_res(req, req->cqe.res, 0); io_req_task_complete(req, locked); } @@ -248,7 +268,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) return; io_poll_remove_entries(req); - io_poll_req_delete(req, req->ctx); + io_poll_tw_hash_eject(req, locked); if (!ret) io_req_task_submit(req, locked); @@ -442,7 +462,10 @@ static int __io_arm_poll_handler(struct io_kiocb *req, return 0; } - io_poll_req_insert(req); + if (req->flags & REQ_F_HASH_LOCKED) + io_poll_req_insert_locked(req); + else + io_poll_req_insert(req); if (mask && (poll->events & EPOLLET)) { /* can't multishot if failed, just queue the event we've got */ @@ -480,6 +503,15 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) __poll_t mask = POLLPRI | POLLERR | EPOLLET; int ret; + /* + * apoll requests already grab the mutex to complete in the tw handler, + * so removal from the mutex-backed hash is free, use it by default. + */ + if (issue_flags & IO_URING_F_UNLOCKED) + req->flags &= ~REQ_F_HASH_LOCKED; + else + req->flags |= REQ_F_HASH_LOCKED; + if (!def->pollin && !def->pollout) return IO_APOLL_ABORTED; if (!file_can_poll(req->file)) @@ -528,13 +560,10 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) return IO_APOLL_OK; } -/* - * Returns true if we found and killed one or more poll requests - */ -__cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, - bool cancel_all) +static __cold bool io_poll_remove_all_table(struct task_struct *tsk, + struct io_hash_table *table, + bool cancel_all) { - struct io_hash_table *table = &ctx->cancel_table; unsigned nr_buckets = 1U << table->hash_bits; struct hlist_node *tmp; struct io_kiocb *req; @@ -557,6 +586,17 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, return found; } +/* + * Returns true if we found and killed one or more poll requests + */ +__cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, + bool cancel_all) + __must_hold(&ctx->uring_lock) +{ + return io_poll_remove_all_table(tsk, &ctx->cancel_table, cancel_all) | + io_poll_remove_all_table(tsk, &ctx->cancel_table_locked, cancel_all); +} + static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd, struct io_hash_table *table, @@ -616,13 +656,15 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, return NULL; } -static bool io_poll_disarm(struct io_kiocb *req) +static int io_poll_disarm(struct io_kiocb *req) { + if (!req) + return -ENOENT; if (!io_poll_get_ownership(req)) - return false; + return -EALREADY; io_poll_remove_entries(req); hash_del(&req->hash_node); - return true; + return 0; } static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, @@ -646,7 +688,16 @@ static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, unsigned issue_flags) { - return __io_poll_cancel(ctx, cd, &ctx->cancel_table); + int ret; + + ret = __io_poll_cancel(ctx, cd, &ctx->cancel_table); + if (ret != -ENOENT) + return ret; + + io_ring_submit_lock(ctx, issue_flags); + ret = __io_poll_cancel(ctx, cd, &ctx->cancel_table_locked); + io_ring_submit_unlock(ctx, issue_flags); + return ret; } static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, @@ -721,6 +772,16 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) ipt.pt._qproc = io_poll_queue_proc; + /* + * If sqpoll or single issuer, there is no contention for ->uring_lock + * and we'll end up holding it in tw handlers anyway. + */ + if (!(issue_flags & IO_URING_F_UNLOCKED) && + (req->ctx->flags & (IORING_SETUP_SQPOLL | IORING_SETUP_SINGLE_ISSUER))) + req->flags |= REQ_F_HASH_LOCKED; + else + req->flags &= ~REQ_F_HASH_LOCKED; + ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); if (ipt.error) { return ipt.error; @@ -745,20 +806,28 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) bool locked; preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table, &bucket); - if (preq) - ret2 = io_poll_disarm(preq); + ret2 = io_poll_disarm(preq); if (bucket) spin_unlock(&bucket->lock); - - if (!preq) { - ret = -ENOENT; + if (!ret2) + goto found; + if (ret2 != -ENOENT) { + ret = ret2; goto out; } - if (!ret2) { - ret = -EALREADY; + + io_ring_submit_lock(ctx, issue_flags); + preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table_locked, &bucket); + ret2 = io_poll_disarm(preq); + if (bucket) + spin_unlock(&bucket->lock); + io_ring_submit_unlock(ctx, issue_flags); + if (ret2) { + ret = ret2; goto out; } +found: if (poll_update->update_events || poll_update->update_user_data) { /* only mask one event flags, keep behavior flags */ if (poll_update->update_events) {