From patchwork Wed Jun 15 16:33:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ACD7C433EF for ; Wed, 15 Jun 2022 16:34:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348606AbiFOQeb (ORCPT ); Wed, 15 Jun 2022 12:34:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356543AbiFOQe3 (ORCPT ); Wed, 15 Jun 2022 12:34:29 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 996312D1E5 for ; Wed, 15 Jun 2022 09:34:27 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id x6-20020a1c7c06000000b003972dfca96cso1425376wmc.4 for ; Wed, 15 Jun 2022 09:34:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ORixVfQAPkrUEdbPULkgFBU4hdADQl3LSugR9FT2k0U=; b=HvpG75mBDhy9kjQBf/6pqYSgY/ysh09pVsz0Y6iP4GPtBuzNrTBG37+8XrYPb2XbA5 mGWMHaE94TP/CgDI4yn6bQMmWU6XRqagyZ1MSpVv1F+6rqh/KOKwxdPnRzc/yxzkmMTk 7mVY+fkyA5FalFYIqGjoY/GgRHNQxdIvaZ1jWI3ndUanBleEhlY2vtHZKtbCSNdCdgJQ pEaX/EC6Mi//iH7qHjYxiLuOb2kaJgT3IPyVUFVYndZeyCGu9Hgnm37Mw6KqHBPfe1y2 Z5h6a2GGDnjjNouBgdXpsGHYN2yTDcLorHUmmL9Sx7DUA0BUhGPJvV6WznpT1zt4BR31 VmHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ORixVfQAPkrUEdbPULkgFBU4hdADQl3LSugR9FT2k0U=; b=Z+Brpcm8A0cX95cIS1HdPRT/9XuVdeXuKzaVzInpPamg8T8kKPoa5A3RS3P/XhIXF1 nuZg/Rpr/MalA1km2/LBqpVmL0K+DBRNBb738FO7cVs9kAISHGoD4bFy+okeXH3TWQdv Tt4dz1JD+yZZ2oW/PuBQ3XcSEpRJv5526KPf1/XXgnxuqI0+cWjWsMZP8aajU6mw5Vrx TRuKHmea1DhYg1NfZlkoO2GhwyNH0WiPEuCUXA3suj9lEHagD1ItkKn5ZENi52njvpvT iEQnu1R8j6xbwXwr3PEmv6UNxE6+T21caSRyABzSQVuA7VQe9czU/Mv/GzVYU5g7WKp2 JmKw== X-Gm-Message-State: AOAM532tEzcnkLxh6U2/HGflPwk0dYqV+kgvnEytHn4aZ9fsS7lMbck7 d78P50FK+f40U6WEUdLoETjmu729dKv3Ew== X-Google-Smtp-Source: ABdhPJw2Gv3GCHZh8JyBXaUGEHqNnKBFcA/nW8dZrw+2zGgYbMB6Xv0M7ZrQn/uSUmQ+uZt4H6GEiw== X-Received: by 2002:a05:600c:4e04:b0:39c:66bc:46d2 with SMTP id b4-20020a05600c4e0400b0039c66bc46d2mr10964450wmq.71.1655310865774; Wed, 15 Jun 2022 09:34:25 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 01/10] io_uring: make reg buf init consistent Date: Wed, 15 Jun 2022 17:33:47 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The default (i.e. empty) state of register buffer is dummy_ubuf, so set it to dummy on init instead of NULL. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index fef46972c327..fd1323482030 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -567,7 +567,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, io_buffer_unmap(ctx, &imu); break; } - ctx->user_bufs[i] = NULL; + ctx->user_bufs[i] = ctx->dummy_ubuf; needs_switch = true; } @@ -1200,14 +1200,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, size_t size; int ret, nr_pages, i; - if (!iov->iov_base) { - *pimu = ctx->dummy_ubuf; + *pimu = ctx->dummy_ubuf; + if (!iov->iov_base) return 0; - } - *pimu = NULL; ret = -ENOMEM; - pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, &nr_pages); if (IS_ERR(pages)) { From patchwork Wed Jun 15 16:33:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7301CCA473 for ; Wed, 15 Jun 2022 16:34:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356571AbiFOQec (ORCPT ); Wed, 15 Jun 2022 12:34:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348775AbiFOQeb (ORCPT ); Wed, 15 Jun 2022 12:34:31 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A841634677 for ; Wed, 15 Jun 2022 09:34:28 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id e5so6632280wma.0 for ; Wed, 15 Jun 2022 09:34:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1Xnp5doseyV/R5Fgg6sqTDUrOugFuNWb6JYgnUVAxHY=; b=P5beFsz52CoABlwAXpZUOTj+mEMHr6jUQg0i438ocRpWNeKodb1bp1ILLXpvd+2Pc+ HphMU+lXjsTcXQDJT7tMPkZVznt4WHaQIzTJ4D1X6UNSdEAtTwXWTWZjq1qAqmvGOK5z wmJsFWri7uZ3u8LJLtziqhRvrKTF/CAoKuVqcrUb4z4VLE5OP2b2oFiKmuwc664+0PO2 mqQodDrJjQ0HP/BM/n3yXkxlPJNk8mNY5X/IG7O/UUDzLPLK4JIKYlvHSFHhTx0dwUWr DtA2q6dPmehZeDtKwI4BiWY0VqdoxQG+MZ78fCpno3tZpOV6t9ES031bENWRZqB0RHrW XQUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1Xnp5doseyV/R5Fgg6sqTDUrOugFuNWb6JYgnUVAxHY=; b=wEAXQX33IJcyci9g+OlURFmhJmmIcBLVfoFxu/Pb3ckWF1fjLLEmiNdW8meDJG3Iv0 a/+fODkqQOs+h3Wm3iTivjJfx3ftiaS0Br6cyld1J7RevVKEnB1BKSmPh4jJGWa8v+M+ mmcV74GADFJ7B9JsoPgAt18ks78QotowrUe7ur1LHyFbYkWhduz9iCjpjFFFDVycRz7Y 2P/hAgjWitM28WiZVQ09Pt3yAq9cZH7LITsxtohcpWF+JqSemwHxIDDyFMue5qaDJyn1 2kNbyFPghFcQhIf8E8hBZs0AgXXMfnt86Svn06C3nKQsMVBlTdOAraiVM7uUhIXkjdBg i/HA== X-Gm-Message-State: AJIora/XMGS/OWCZuZclSCEnPCIYjJ4YKbYSB8jm6eM1ALTxmlMozJn1 jSX7gehhKJ5eh8lbD/EaFo+k+P5SLgGq1Q== X-Google-Smtp-Source: AGRyM1s9K/IiwuFqHODQq4gUI6XyQ+VHgDjJuTyYQ4bmV0NMvuQ9pZ/yy1blVO41l5HrjUbl+LejJA== X-Received: by 2002:a05:600c:4e50:b0:39c:8ca8:faff with SMTP id e16-20020a05600c4e5000b0039c8ca8faffmr282411wmq.67.1655310866927; Wed, 15 Jun 2022 09:34:26 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:26 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 02/10] io_uring: move defer_list to slow data Date: Wed, 15 Jun 2022 17:33:48 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org draining is slow path, move defer_list to the end where slow data lives inside the context. Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 147e1e597530..bff73107f0f3 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -160,7 +160,6 @@ struct io_ring_ctx { struct io_uring_sqe *sq_sqes; unsigned cached_sq_head; unsigned sq_entries; - struct list_head defer_list; /* * Fixed resources fast path, should be accessed only under @@ -272,8 +271,12 @@ struct io_ring_ctx { struct work_struct exit_work; struct list_head tctx_list; struct completion ref_comp; + + /* io-wq management, e.g. thread count */ u32 iowq_limits[2]; bool iowq_limits_set; + + struct list_head defer_list; }; }; From patchwork Wed Jun 15 16:33:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01EC7CCA47E for ; Wed, 15 Jun 2022 16:34:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356290AbiFOQec (ORCPT ); Wed, 15 Jun 2022 12:34:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356571AbiFOQeb (ORCPT ); Wed, 15 Jun 2022 12:34:31 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A70E337BFE for ; Wed, 15 Jun 2022 09:34:29 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id s1so16098995wra.9 for ; Wed, 15 Jun 2022 09:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i2M2vqRQ9RAoU3bnpdZxapmtH2oq59EBP0ozhlRRLD0=; b=n+D5Z7h0EBWZocZu3NTaQ99ROfPBOfhlspVUjwaQPpXbQ1fi9NJYO97FzzzPAaPcZ/ 2Pjr9jXfVywH8k1Y/r7Stav4Z6ZcCgl9S5F3r1X7T+bM1uhtWE8Q1BoItHgnp02ShyH7 +EuWQulc2YzHdCBKrnT/Y5j3VJbEQOSo4x2XjBHLH58MBQvL1CaM+7euUhcoCmPstHnm USTt/Ak9Ji+PYgGY41c8NRaAc8j17Lje+O37O+9J2D5003ep0t9bk45V1VsY+YVQ/exl MbtwYRY/zgFfYafiaT3t5HMjME53uDIjYVmu5GkOUaaM270iz7ry2L5zeZtcvw4DLQ/Y Gq2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i2M2vqRQ9RAoU3bnpdZxapmtH2oq59EBP0ozhlRRLD0=; b=fqpyCJKUKctK3tXbXvm7lgjWGuZQjrqiErPLGZ+ldvbp+4pHa2FOROWir3g/XBD1WE TI53kvbz5jYJ7f/lHWBbeZsn603sWHPTy4zAUDSGljYaowUtj96Ji/oiYaTyU6DIn5R9 +Qlm1AvfIH1J/Hk2LDMxspTsbPNFI/KD5gfsQnOvLQiFJ4ymseDv5cUI7hNzKpBPvE1X nc4ifR0y2Lo4xL+Shr+oW8Jli/Ov3CrDvRqjNYuOXeEtd7Rm8bFlCzIh6kkqta1JSne9 SfFNOmEiMYKqBFBGFJW3mAJxSseg4nyZ0bsdDkg4szk710BGcyYjlXVIy66SrPL67B0z a9/w== X-Gm-Message-State: AJIora8W6AzG9AVgG5nps+yEloZNt4kcm2hV08YAZWOeEKNgdxNNfVv6 vOI9bn96oj9/AjM7ok2xQeiGw4GiGtumFw== X-Google-Smtp-Source: AGRyM1tWHHKe4uGZMAppyDKdJ5WD3gwlXi15FPRbwJS/Z1Oxs9Se0GZiVFkBjYT/QPxmfmujIsgtCQ== X-Received: by 2002:a5d:47cf:0:b0:213:bbe1:ba4d with SMTP id o15-20020a5d47cf000000b00213bbe1ba4dmr620714wrc.173.1655310867975; Wed, 15 Jun 2022 09:34:27 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:27 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields Date: Wed, 15 Jun 2022 17:33:49 +0100 Message-Id: <4b163793072840de53b3cb66e0c2995e7226ff78.1655310733.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Following timeout fields access patterns, move all of them into a separate cache line inside ctx, so they don't intervene with normal completion caching, especially since timeout removals and completion are separated and the later is done via tw. It also sheds some bytes from io_ring_ctx, 1216B -> 1152B Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index bff73107f0f3..e050c3c4a7ac 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -179,8 +179,6 @@ struct io_ring_ctx { struct xarray io_bl_xa; struct list_head io_buffers_cache; - struct list_head timeout_list; - struct list_head ltimeout_list; struct list_head cq_overflow_list; struct list_head apoll_cache; struct xarray personalities; @@ -213,15 +211,11 @@ struct io_ring_ctx { struct io_ev_fd __rcu *io_ev_fd; struct wait_queue_head cq_wait; unsigned cq_extra; - atomic_t cq_timeouts; - unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; struct { spinlock_t completion_lock; - spinlock_t timeout_lock; - /* * ->iopoll_list is protected by the ctx->uring_lock for * io_uring instances that don't use IORING_SETUP_SQPOLL. @@ -253,6 +247,15 @@ struct io_ring_ctx { struct list_head io_buffers_pages; }; + /* timeouts */ + struct { + spinlock_t timeout_lock; + atomic_t cq_timeouts; + struct list_head timeout_list; + struct list_head ltimeout_list; + unsigned cq_last_tm_flush; + } ____cacheline_aligned_in_smp; + /* Keep this last, we don't need it for the fast path */ struct { #if defined(CONFIG_UNIX) From patchwork Wed Jun 15 16:33:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 019F9C43334 for ; Wed, 15 Jun 2022 16:34:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356543AbiFOQed (ORCPT ); Wed, 15 Jun 2022 12:34:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356579AbiFOQeb (ORCPT ); Wed, 15 Jun 2022 12:34:31 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F058E41312 for ; Wed, 15 Jun 2022 09:34:30 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id m125-20020a1ca383000000b0039c63fe5f64so1441088wme.0 for ; Wed, 15 Jun 2022 09:34:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iYS0pPd4DSDus0EjpouGi5yssHH1VdtV2aAPsGUOfVo=; b=QTbYI2codfBkD1HZWYIgrgt4xzLeqK5dr+GNtZGhQOrs8eUaep0cO8hSxRpYAWBquk 3CUlJFFCeOpl9NYlgIWREYtyV1caGJv9FOExMqMyaBvdHhSfmYv0Vt8jRoLrOTSide6s 8R3JDrIOTGUDmb+GkToO2YhtJPfv/8iTXf96H339qjNowUXIeZXrQTD+TlQKId2Fdapa pwSqZfO3rV5yaAUffxQyKNAIs7vSq+/DagXhrIzu4D88ayhA81zs8QNSYeWP7T1CyZ13 /p+sV3YR/fzQ1Sq6L8QSzmKKao+ZTBcUkrF3/O5Owis0ilEybYJRd5ElyBIP6bxok+2X m35w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iYS0pPd4DSDus0EjpouGi5yssHH1VdtV2aAPsGUOfVo=; b=qLYJGekLEFa8qBSOWytkpw3zdW8xi4tJzA7mdI3OQRphHVXGmIF2cdiiCQulKxDtWs S7mHjTiOFhsMZuefmeZN04IxUkF5q+EyLNsVtEenVOR3Jx8rzH3neIN2PXqRaMUsa7GM okDqGppGKHG9dS3lap92+6y/YrRXAsWG7Hh/UGvoao5Pz3uVCfIc4tmA0L46VZpmR6ZZ G+roUeVZBqF7z5zLvYCslVyRe6d6os3LfR6F/el4KbO/FSBSiyFC08fVUoMZaQuWzwEF QmaA3rl2+OO55qu4ovxp+hL3YDKPnaCuJTUV6UrqFBmp8r3MR9E0XcDlpEMBCg/SeJLA Bw8g== X-Gm-Message-State: AJIora9YLasxlugMzv4f+OIh8+gB9GjYb2rqyL3tgjzXxn3sbRBF2SX+ GkREf+7ZDiz7qru5CiyZ+1c1vrvdeNpIBQ== X-Google-Smtp-Source: AGRyM1vb6l/gcuSp4LJtWvR9iz8Z6zjj/oXJS/VzxyFFRqt/hzgIZYnhXdi3mLBrSWkPcukoIELJeQ== X-Received: by 2002:a7b:cb88:0:b0:39d:16a7:dac7 with SMTP id m8-20020a7bcb88000000b0039d16a7dac7mr305674wmi.128.1655310868963; Wed, 15 Jun 2022 09:34:28 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:28 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 04/10] io_uring: refactor ctx slow data placement Date: Wed, 15 Jun 2022 17:33:50 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Shove all slow path data at the end of ctx and get rid of extra indention. Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 81 +++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 42 deletions(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index e050c3c4a7ac..94f76256721a 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -183,7 +183,6 @@ struct io_ring_ctx { struct list_head apoll_cache; struct xarray personalities; u32 pers_next; - unsigned sq_thread_idle; } ____cacheline_aligned_in_smp; /* IRQ completion list, under ->completion_lock */ @@ -230,23 +229,6 @@ struct io_ring_ctx { struct list_head io_buffers_comp; } ____cacheline_aligned_in_smp; - struct io_restriction restrictions; - - /* slow path rsrc auxilary data, used by update/register */ - struct { - struct io_rsrc_node *rsrc_backup_node; - struct io_mapped_ubuf *dummy_ubuf; - struct io_rsrc_data *file_data; - struct io_rsrc_data *buf_data; - - struct delayed_work rsrc_put_work; - struct llist_head rsrc_put_llist; - struct list_head rsrc_ref_list; - spinlock_t rsrc_ref_lock; - - struct list_head io_buffers_pages; - }; - /* timeouts */ struct { spinlock_t timeout_lock; @@ -257,30 +239,45 @@ struct io_ring_ctx { } ____cacheline_aligned_in_smp; /* Keep this last, we don't need it for the fast path */ - struct { - #if defined(CONFIG_UNIX) - struct socket *ring_sock; - #endif - /* hashed buffered write serialization */ - struct io_wq_hash *hash_map; - - /* Only used for accounting purposes */ - struct user_struct *user; - struct mm_struct *mm_account; - - /* ctx exit and cancelation */ - struct llist_head fallback_llist; - struct delayed_work fallback_work; - struct work_struct exit_work; - struct list_head tctx_list; - struct completion ref_comp; - - /* io-wq management, e.g. thread count */ - u32 iowq_limits[2]; - bool iowq_limits_set; - - struct list_head defer_list; - }; + + struct io_restriction restrictions; + + /* slow path rsrc auxilary data, used by update/register */ + struct io_rsrc_node *rsrc_backup_node; + struct io_mapped_ubuf *dummy_ubuf; + struct io_rsrc_data *file_data; + struct io_rsrc_data *buf_data; + + struct delayed_work rsrc_put_work; + struct llist_head rsrc_put_llist; + struct list_head rsrc_ref_list; + spinlock_t rsrc_ref_lock; + + struct list_head io_buffers_pages; + + #if defined(CONFIG_UNIX) + struct socket *ring_sock; + #endif + /* hashed buffered write serialization */ + struct io_wq_hash *hash_map; + + /* Only used for accounting purposes */ + struct user_struct *user; + struct mm_struct *mm_account; + + /* ctx exit and cancelation */ + struct llist_head fallback_llist; + struct delayed_work fallback_work; + struct work_struct exit_work; + struct list_head tctx_list; + struct completion ref_comp; + + /* io-wq management, e.g. thread count */ + u32 iowq_limits[2]; + bool iowq_limits_set; + + struct list_head defer_list; + unsigned sq_thread_idle; }; enum { From patchwork Wed Jun 15 16:33:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A3D0C43334 for ; Wed, 15 Jun 2022 16:34:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356649AbiFOQee (ORCPT ); Wed, 15 Jun 2022 12:34:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356588AbiFOQec (ORCPT ); Wed, 15 Jun 2022 12:34:32 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2F752BB2D for ; Wed, 15 Jun 2022 09:34:31 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id c21so16145238wrb.1 for ; Wed, 15 Jun 2022 09:34:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+J4OGcbFiS//ULvT34KSE6gEhtpT20A8WEIx4b+yT04=; b=l7AN13RpL6JM0hOdb6HJr3+YQOOE1GsLte58b88jPKXvMDRsy23f5R1r+ZNBxNul8v U6v7vgqdy8meoUszDCleaKPbZFwJysUMOvk6vX3w1hLvFWoUdx5rq4wVka0XlmNUz6CD IYVRyf9pQQhtQ1vQjLUYjyGfA4HWRpNdEj4rgHwqgwHn3BsYPjplRZIupQm+pPtxPd/Y V/cDX1tdiFNZOzMtv94CgGijMcPmjdw3devTZB+GIRleN0VlFcYQhxZeZkC4wjWPpU59 sVze4vIbQjvapSbhDQ0za/NMThTf7pUngf/XVDDv3cSYCbQ3dgc6G0zuQp7XkwqAYGuJ zWqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+J4OGcbFiS//ULvT34KSE6gEhtpT20A8WEIx4b+yT04=; b=vjdFXcvVuX1s+hOuYhbeNqxfVoPhOBZ6yQJWQMIKM9wKm2dvJmn4zozEmivP/gAx5x 2bTvUYfuetDda45fOo8fF5Qr6wYWxkzvbKeUZ+wDL8w1D2/TKewZJS4Y7VosKyqJRS6Q eA0TooYDwQZ/8C0ElTxuDKPeR0DFP8xXSpPlE3G1aq+f7/aWRxLTE0/WiGe6Rbv2CRkw SzjzLUO61tTawH1uaYHsk4q8hvPbdNqeHsSQfmV4w7SNk5zbQLZo22fg/i1KnS1LbsdP r9+drZrXtAudDOa8v0I9wDx788KOPzengoTcX1S8vdtKVf0TThJgTeySGDMHB0ZIY6AV IveQ== X-Gm-Message-State: AJIora+F8Y+S/615ZCYUf3Nc27AtpWr+XFvzw3C0GW2+9w8mJ678vW2d qGPTox7RMmuJ+Tkn5IrEqRFcTBwzSBDONA== X-Google-Smtp-Source: AGRyM1sztX+WWvSWqXjczdUpoi/gYlh4meIou0npIjrjtfVqk3dTYT3NdybJKisOnWxHZIfWsD0FYA== X-Received: by 2002:a5d:4151:0:b0:210:316c:7bed with SMTP id c17-20020a5d4151000000b00210316c7bedmr591170wrq.369.1655310870020; Wed, 15 Jun 2022 09:34:30 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:29 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 05/10] io_uring: move small helpers to headers Date: Wed, 15 Jun 2022 17:33:51 +0100 Message-Id: <22df99c83723e44cba7e945e8519e64e3642c064.1655310733.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is a bunch of inline helpers that will be useful not only to the core of io_uring, move them to headers. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 17 ----------------- io_uring/io_uring.h | 22 ++++++++++++++++++++++ 2 files changed, 22 insertions(+), 17 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 78aee292d109..38b53011e0e9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -615,14 +615,6 @@ struct sock *io_uring_get_socket(struct file *file) } EXPORT_SYMBOL(io_uring_get_socket); -static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) -{ - if (!*locked) { - mutex_lock(&ctx->uring_lock); - *locked = true; - } -} - static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { if (!wq_list_empty(&ctx->submit_state.compl_reqs)) @@ -878,15 +870,6 @@ static void io_prep_async_link(struct io_kiocb *req) } } -static inline void io_req_add_compl_list(struct io_kiocb *req) -{ - struct io_submit_state *state = &req->ctx->submit_state; - - if (!(req->flags & REQ_F_CQE_SKIP)) - state->flush_cqes = true; - wq_list_add_tail(&req->comp_list, &state->compl_reqs); -} - void io_queue_iowq(struct io_kiocb *req, bool *dont_use) { struct io_kiocb *link = io_prep_linked_timeout(req); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ee718f7d54d3..e30e639c2822 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -217,6 +217,28 @@ static inline bool io_run_task_work(void) return false; } +static inline void io_req_complete_state(struct io_kiocb *req) +{ + req->flags |= REQ_F_COMPLETE_INLINE; +} + +static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) +{ + if (!*locked) { + mutex_lock(&ctx->uring_lock); + *locked = true; + } +} + +static inline void io_req_add_compl_list(struct io_kiocb *req) +{ + struct io_submit_state *state = &req->ctx->submit_state; + + if (!(req->flags & REQ_F_CQE_SKIP)) + state->flush_cqes = true; + wq_list_add_tail(&req->comp_list, &state->compl_reqs); +} + int io_run_task_work_sig(void); void io_req_complete_failed(struct io_kiocb *req, s32 res); void __io_req_complete(struct io_kiocb *req, unsigned issue_flags); From patchwork Wed Jun 15 16:33:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8204EC433EF for ; Wed, 15 Jun 2022 16:34:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356579AbiFOQej (ORCPT ); Wed, 15 Jun 2022 12:34:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356594AbiFOQed (ORCPT ); Wed, 15 Jun 2022 12:34:33 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D771E2AE20 for ; Wed, 15 Jun 2022 09:34:32 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id o8so16133081wro.3 for ; Wed, 15 Jun 2022 09:34:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pRvBVOE/hkiozWmaa4rfYGZwNs1AXWCtzZOpvsa5jls=; b=mkDEbLx048bx1/e07voGxuV2tza+hsmZn3tB/pZ7yUvMmJ5JTjM5AZUnqcZAV62b8E SjXarMmiZvZusXC/1TqdCYOm6hPhoJy2aIsVwycCog0OabeOSmovt+YD6M/sV1Yt3ZTZ XhHoqrqLqRu/MNAHCKFAshbSTNIXS1zqb3YT4entdC96rY8t2VDsCNuHF5DntbErwryW yUwdvG0Pnpe+m2fm96oQzTajGUkoSP8n0fae4+4saLd3rihEuHLghqgq5fHpr5PsdTbz Z4BqwPcaZBS0fne6aRMJ8J9kY4z5BTN9+7cAuG9S4qcSRmOZJh5uJq09JKHF4s3lpERI kXzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pRvBVOE/hkiozWmaa4rfYGZwNs1AXWCtzZOpvsa5jls=; b=2Pk9aVeGwHtYqM4PTrIVV7dPUnonh+GJRdOyDdpK8OiWiLhydQ1fzGzl2HX78kse9N 629OLV9yQxuFJrCI2f8U79f7XJy5fO8dIJb5SxxhY2m4R3f+3PoJirJNmV7d5hpOP/9+ lFS2YqEhQgB9K10/ojrevZJIdNpw8TbRwOMhQJ3MJI7Gjuwtz7EMIPeOSV63kcW/sApI 9TE4K10MxoiruQU9Y9MyLJb7QpMzGComFEjYiz7jKVOyXXb/R0AohR/YMTAaM73XR9r0 P+WZY6xNiGW/tHaJ53tnVplQYMmyC6LqKKP3LxSIYEbtSzfRgmcsqvB9xUSvSFnfyIMV q3cA== X-Gm-Message-State: AJIora9bRYchEkEhA30SFPSpatomEltvXITd3Tyshg1YIoD/sZHwV1eP YLTH5ApLCmmeeLaqjRB09C7AFFeqNPqaSw== X-Google-Smtp-Source: AGRyM1vCbCshGB2T1No40yygfprda9pdTH4mYbYGjAZKAO5r34dfkNhbNT1IkhzaYIfGzSbTohUNcw== X-Received: by 2002:a05:6000:168c:b0:218:4523:c975 with SMTP id y12-20020a056000168c00b002184523c975mr625119wrd.23.1655310871203; Wed, 15 Jun 2022 09:34:31 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:30 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement Date: Wed, 15 Jun 2022 17:33:52 +0100 Message-Id: <988e87eec9dc700b5dae933df3aefef303502f6c.1655310733.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a comment on why we keep ->cancel_seq in struct io_wq_work instead of struct io_kiocb despite it needed only by io_uring but not io-wq. Signed-off-by: Pavel Begunkov --- io_uring/io-wq.h | 1 + 1 file changed, 1 insertion(+) diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index ba6eee76d028..3f54ee2a8eeb 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -155,6 +155,7 @@ struct io_wq_work_node *wq_stack_extract(struct io_wq_work_node *stack) struct io_wq_work { struct io_wq_work_node list; unsigned flags; + /* place it here instead of io_kiocb as it fills padding and saves 4B */ int cancel_seq; }; From patchwork Wed Jun 15 16:33:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0C30C433EF for ; Wed, 15 Jun 2022 16:34:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356588AbiFOQel (ORCPT ); Wed, 15 Jun 2022 12:34:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356664AbiFOQej (ORCPT ); Wed, 15 Jun 2022 12:34:39 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE5CD35277 for ; Wed, 15 Jun 2022 09:34:33 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id m24so16082309wrb.10 for ; Wed, 15 Jun 2022 09:34:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YslBqT2nf9mKwXwSapr90WAkEwuBkksdeZNpVq3Qt0E=; b=Gulur6szQmdJcVyVzv3xbacBYgFHdyp3hQ2GLusEwaLr5c2/nCbnfYq5fOrnsxhuVg Bm03PmKGW+7qiU5oIhIWIdIzWekEnRPRDvW/DU1VViREnrhBOxjz6jgKDjDe1PJT8CzU yybFdjpJ1VmqoSoS3HpOa4ilFPfEft3jRMwhzMFOOVT4Myp6lWMxaLT4gojLEredLkfd YaUfX+pILEGJThV3px0RYLd2LF3mkKOkqTsz5SRtccdwQpHhMC522NjTp3l/YeM08sDM iiTIW7spzLWj+kKuYjoqWSnGWYLR8DHu7S2tw8P0OQkxptwT0pbVJR8eGGSsLWk7WA4q PWMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YslBqT2nf9mKwXwSapr90WAkEwuBkksdeZNpVq3Qt0E=; b=ErdPbbB06dpi9C+8nD2eHBAokbP3IWVxIEVYduq+CFJ9WTyC53K9OoDT1d8XJ/tCNv Q598Id9oInbw21Om8tZNV7MZyLom0ArYu+8MIafClWhhc6Vsyp5Ju22NeMidjYy7LoS9 /0fbRZwZdqyyRETyfNbzOtVMKuWzGeVtorewetcDAQk26YvMPswjSu0NS3h0CwmnZ9NZ Hy5GriBTzuH8NN1BefFMh36Rw4KX54FI1rsJAM/EMvpHveYW1Cnnm+K8st/F0Ibtv6Ne 4jjQ8eOnL6fmRfqoVD5xPVlJlvhW5uJ4XMYZvqd0zqnuuPFtfQWVaLqP/fupHyC4Wq44 1bvA== X-Gm-Message-State: AJIora+XAgskoKIcPJB7EwnfLHB2VbQUy1Kq63MOXX/vT2ddRqmGaclI JlWlzylRET6pr1/JSbdBSfPufZ7hX1BOww== X-Google-Smtp-Source: AGRyM1vHmTjCN5WUHLg+JcHjHEzv3EEqmxBYOxod3YDyI7iEHHEjNTFFGzS2cq43wbW0FpkiLCD1ZA== X-Received: by 2002:a5d:4b81:0:b0:216:592a:9775 with SMTP id b1-20020a5d4b81000000b00216592a9775mr645523wrt.350.1655310872202; Wed, 15 Jun 2022 09:34:32 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:31 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 07/10] io_uring: inline ->registered_rings Date: Wed, 15 Jun 2022 17:33:53 +0100 Message-Id: <495f0b953c87994dd9e13de2134019054fa5830d.1655310733.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There can be only 16 registered rings, no need to allocate an array for them separately but store it in tctx. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - io_uring/tctx.c | 9 --------- io_uring/tctx.h | 3 ++- 3 files changed, 2 insertions(+), 11 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 38b53011e0e9..dec288b5f5cd 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2744,7 +2744,6 @@ void __io_uring_free(struct task_struct *tsk) WARN_ON_ONCE(tctx->io_wq); WARN_ON_ONCE(tctx->cached_refs); - kfree(tctx->registered_rings); percpu_counter_destroy(&tctx->inflight); kfree(tctx); tsk->io_uring = NULL; diff --git a/io_uring/tctx.c b/io_uring/tctx.c index f3262eef55d4..6adf659687f8 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -55,16 +55,8 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, if (unlikely(!tctx)) return -ENOMEM; - tctx->registered_rings = kcalloc(IO_RINGFD_REG_MAX, - sizeof(struct file *), GFP_KERNEL); - if (unlikely(!tctx->registered_rings)) { - kfree(tctx); - return -ENOMEM; - } - ret = percpu_counter_init(&tctx->inflight, 0, GFP_KERNEL); if (unlikely(ret)) { - kfree(tctx->registered_rings); kfree(tctx); return ret; } @@ -73,7 +65,6 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, if (IS_ERR(tctx->io_wq)) { ret = PTR_ERR(tctx->io_wq); percpu_counter_destroy(&tctx->inflight); - kfree(tctx->registered_rings); kfree(tctx); return ret; } diff --git a/io_uring/tctx.h b/io_uring/tctx.h index f4964e40d07e..7684713e950f 100644 --- a/io_uring/tctx.h +++ b/io_uring/tctx.h @@ -20,8 +20,9 @@ struct io_uring_task { struct io_wq_work_list task_list; struct io_wq_work_list prio_task_list; struct callback_head task_work; - struct file **registered_rings; bool task_running; + + struct file *registered_rings[IO_RINGFD_REG_MAX]; }; struct io_tctx_node { From patchwork Wed Jun 15 16:33:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0E49CCA47F for ; Wed, 15 Jun 2022 16:34:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356594AbiFOQen (ORCPT ); Wed, 15 Jun 2022 12:34:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356695AbiFOQej (ORCPT ); Wed, 15 Jun 2022 12:34:39 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E58C137BFE for ; Wed, 15 Jun 2022 09:34:34 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id l126-20020a1c2584000000b0039c1a10507fso1428523wml.1 for ; Wed, 15 Jun 2022 09:34:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WRCMIxAC7zXamLszka2v8x0ZOcaQKYObTE2e1CKL58E=; b=KSHjexX/E0H6+oaS3KCHx1+KUqe1carPzWDvJaWvqTlitpwXXNIB3dvMSpyUFqMi4v dLrUy6pEqdES1Ri5VeWt0H3KXIVrMOakFHkTOIr0VU2oLhntzcz+mhYAJNh5kxx+++An UfHuOn6JqVrtq125xfNAdr5LBcLHAIxYneTbEOiKGkqah9Uoi4Y0Uqazvct4mFXnVP2N UBclJOx+2UpGWvfjV7nVZ1uoKUJYHAkazJzuNObtxFvz9xe0WfYlb0A8E0ENlTdVZ+jf y2hZBlQfnj44icVCCYgvu3v7ofGs7v6qWsBKO+yHG5Q3RzHpU1MQtfR0psXKlZIir1Qj 2sxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WRCMIxAC7zXamLszka2v8x0ZOcaQKYObTE2e1CKL58E=; b=TZFBlB9yudaY6RwpznuOgGrd9HT46VbIiUr+qEO6KNSHcezfzQn8TdyyZtqRY7Hkug V7zrdZlknZCVJemMp02nQwMsrSQpcO8vIBmQ/4bXfOAb5rx8vJnBeOE2SEItcawWsufT 5JnXPLf5O8Rsb1zSWqygiFfZ1EHuz691IQXHqqN0JCzjHyifcIjS7z5+hMW214+Zs9JA p+nRpvJBmcslrCejDuex9tSlaxxWFdt5Yky+F5p88NaK1sSWXZWi2V7CYe7HHK3UIAVl c31cM79KKdPkfgPhdMAJCmAJDUYThh2T7yv99TAr656SJm44jBQyh4+MMRiJB/Ssg/+9 Hjqg== X-Gm-Message-State: AJIora96/bG3TLtzqr5mQ1C4pa5CnXfRRf/93Fy2Q/Og1VNS2MHk/WB9 /dTXhWkZwz9BDsXaRhW/AvD/AgFiJ31mhw== X-Google-Smtp-Source: AGRyM1vHnpGGDgndq/NeNux0Vc/75aMLn8C6vCszMHjvYkHnnfKI5w/v1D1wYcTPx51Mr3AUpW1wGg== X-Received: by 2002:a05:600c:a4c:b0:39c:6517:1136 with SMTP id c12-20020a05600c0a4c00b0039c65171136mr335694wmq.12.1655310873252; Wed, 15 Jun 2022 09:34:33 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:32 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll Date: Wed, 15 Jun 2022 17:33:54 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Luckily, nnobody completes multi-apoll requests outside the polling functions, but don't set IO_URING_F_COMPLETE_DEFER in any case as there is nobody who is catching REQ_F_COMPLETE_INLINE, and so will leak requests if used. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index dec288b5f5cd..68ce8666bd32 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2054,7 +2054,7 @@ int io_poll_issue(struct io_kiocb *req, bool *locked) io_tw_lock(req->ctx, locked); if (unlikely(req->task->flags & PF_EXITING)) return -EFAULT; - return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER); + return io_issue_sqe(req, IO_URING_F_NONBLOCK); } struct io_wq_work *io_wq_free_work(struct io_wq_work *work) From patchwork Wed Jun 15 16:33:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D976CCA473 for ; Wed, 15 Jun 2022 16:34:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356746AbiFOQen (ORCPT ); Wed, 15 Jun 2022 12:34:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356726AbiFOQek (ORCPT ); Wed, 15 Jun 2022 12:34:40 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 272942BB1B for ; Wed, 15 Jun 2022 09:34:36 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id w17so8603008wrg.7 for ; Wed, 15 Jun 2022 09:34:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qNT+ZB+G8mcMlHFOU8EhXjkZoMhG/NfZq2/GkAr9hA4=; b=S9YDcuwNx7lbUBbjX5Sce4keLjMcx6Pd7bOpIZh/x0nyr7uk8blJ7OEqdlM1PU4+y4 XsAG/KxW4gMpTNBs94y98udt1KmUQwbQoMoUNWC+lFT32CAw3kfp2U8V8RV9Icu9qwjX 8lXAFvg3jMdWgmubGbdymAw7zp7KUKVRZcrrsmxmu15cetE0F9v1xtG+Fp5qP6W77S4c 3puFi+kA95O4K335etGyDa8/F0DYeGUVUhUfQ7fYoup6okihYwygNjyTgGaeHe+rkoUQ V/77+RFL5B50BgKy5fmd2z+x7/FnLdx/iClL3ngXHNJ42FHuwCVLuQASJfSprrSBzt+q bWfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qNT+ZB+G8mcMlHFOU8EhXjkZoMhG/NfZq2/GkAr9hA4=; b=Lgav6nIV3tHPHs+1OUVDT+gytiN/FKlWYUQ17LNYAmkFW13H91+2WPOWqUdyoz9Lrg fKZNS46rYGdEHuT8R8PqxhEjyV1Sfi08srv6Qvreyo688BExhNIt5nzrrhNYC4hKVg5Y K/pyIWVNZ7gTSG06K3F5hw5eP1ujYJPYur+W1uDMxDPu9vtWI051LdJM3Z6IJaVqT30n GESHFh1SmxpS30TbvBkZL4nRoJ3SGYV77hw2P/kn+hUOpUvy8kN/z7B64wHncXAw4Ksk SfeYiz/4E657otBA0Ll7qbM8efi+kl7vhQ+gb3KushueeQpJ/3KFuDbjGwsVxvSjtep+ pKng== X-Gm-Message-State: AJIora/lo1n7esiRdiTjkMYdl7/1z93yPld6U0tnuhiyBMxgck1JwKPJ YPXCnlbIQj1JsOjVI/BNadLkmgVl0viU/Q== X-Google-Smtp-Source: AGRyM1sfkxLkTv7ns+f7+T3UxwwuTzvU0QdHR5QolUBCWD7STtfeDWQNz8HagmUJqiMpv1d9og9ejg== X-Received: by 2002:adf:eb90:0:b0:21a:b9e:c3fe with SMTP id t16-20020adfeb90000000b0021a0b9ec3femr602806wrn.429.1655310874410; Wed, 15 Jun 2022 09:34:34 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:33 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths Date: Wed, 15 Jun 2022 17:33:55 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org All ctx->check_cq events are slow path, don't test every single flag one by one in the hot path, but add a common guarding if. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 68ce8666bd32..f47de2906549 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1714,24 +1714,25 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) int ret = 0; unsigned long check_cq; + check_cq = READ_ONCE(ctx->check_cq); + if (unlikely(check_cq)) { + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + __io_cqring_overflow_flush(ctx, false); + /* + * Similarly do not spin if we have not informed the user of any + * dropped CQE. + */ + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) + return -EBADR; + } /* * Don't enter poll loop if we already have events pending. * If we do, we can potentially be spinning for commands that * already triggered a CQE (eg in error). */ - check_cq = READ_ONCE(ctx->check_cq); - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - __io_cqring_overflow_flush(ctx, false); if (io_cqring_events(ctx)) return 0; - /* - * Similarly do not spin if we have not informed the user of any - * dropped CQE. - */ - if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))) - return -EBADR; - do { /* * If a submit got punted to a workqueue, we can have the @@ -2657,12 +2658,15 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, ret = io_run_task_work_sig(); if (ret || io_should_wake(iowq)) return ret; + check_cq = READ_ONCE(ctx->check_cq); - /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - return 1; - if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))) - return -EBADR; + if (unlikely(check_cq)) { + /* let the caller flush overflows, retry */ + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + return 1; + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) + return -EBADR; + } if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS)) return -ETIME; return 1; From patchwork Wed Jun 15 16:33:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12882749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70F0AC43334 for ; Wed, 15 Jun 2022 16:34:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356664AbiFOQem (ORCPT ); Wed, 15 Jun 2022 12:34:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356728AbiFOQek (ORCPT ); Wed, 15 Jun 2022 12:34:40 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F6FA3631F for ; Wed, 15 Jun 2022 09:34:37 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id j5-20020a05600c1c0500b0039c5dbbfa48so1452327wms.5 for ; Wed, 15 Jun 2022 09:34:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jRmOa9uYxv7BEB0AnrrsA833tIw8T2Q7R1ToAIQEAg4=; b=UzccXbnFCfvTbHzD6bd0Wf7zZIehd+jh3R6cdVmC4IbMBky+NhjQ/PshtDPjMyS4jA ZDlrAzOn1ZEixhaj31pI7czeuvskUS6/ZSsEKTh4VXftoMciYWh6BOqk5fncCMIXQ6Oq oUvjd1jwOV7nn7dlyynsBbYRoOVaqtbz7eedDGjx6Iy4NeTzCyRtU4h3tES2X0n9H6Ra 1mTvIJTLqCoHPaK3FPqqbBms+viqWF0XSr/8C8QMTobfwGcDmRxGfSBakN3MzMSrKQFq vukRaagZt/aE0hltm82iUEiEMbRnftVCac2Dgl4P46TUarhg5tK/qjtp+xssxPpZtYLQ aZxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jRmOa9uYxv7BEB0AnrrsA833tIw8T2Q7R1ToAIQEAg4=; b=A51O/u4sd86IabsKvS+84B0nBnUpeFM1V9YUEAkKmAU146AQPc9O7SCS+3fYPjht/y WXFGXK7EoVJAiexPQDITgIVm5LxfB3oCgjp/cq3I2RN12X5GCy/Y1ionGbQ9R4lYZgG7 HXinlpJacdB5jCU7IbYXCzbSQNuoR47hmTKm10HFRR1OjyNbOUIhwZRXz4TwKIWlYoMj hpgfvmpBKMtBGHKf6BmfI3FKIaXt34DUq2LFB3COG44H268I6Uo7DjzK7pC5qpDZk6se rnVuvqUmNRF3Ih0PyxbMq3mJSuR0+SKvUGRNqIPF9h+cAuV/fG2cNA5CRpz7HKRrd2HF jDzg== X-Gm-Message-State: AJIora8Kughm5vn6i4J+htd2P9wOn318Q9j188mGPINd11f+8CZ5iJns slGWLSLS5sNTFzDSj2aMXb7FOLIPHXJoWg== X-Google-Smtp-Source: AGRyM1uHRM5TaEoz+TIsqtLnEz/r9XrIRT2LuRgcSKT/I78kx/6GaXuH8+ExsN2fa9vL39dnsIw+pw== X-Received: by 2002:a1c:4484:0:b0:39c:4597:1f74 with SMTP id r126-20020a1c4484000000b0039c45971f74mr284050wma.13.1655310875498; Wed, 15 Jun 2022 09:34:35 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a056000038200b0020ff3a2a925sm17894953wrf.63.2022.06.15.09.34.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 09:34:35 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Date: Wed, 15 Jun 2022 17:33:56 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_req_task_complete() enqueues requests for state completion itself, no need for REQ_F_COMPLETE_INLINE, which is only serve the purpose of not bloating the kernel. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index f47de2906549..ce3302a62112 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1769,7 +1769,6 @@ inline void io_req_task_complete(struct io_kiocb *req, bool *locked) { if (*locked) { req->cqe.flags |= io_put_kbuf(req, 0); - req->flags |= REQ_F_COMPLETE_INLINE; io_req_add_compl_list(req); } else { req->cqe.flags |= io_put_kbuf(req, IO_URING_F_UNLOCKED);