From patchwork Thu Apr 13 14:28:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210338 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECC9FC77B77 for ; Thu, 13 Apr 2023 14:28:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231152AbjDMO26 (ORCPT ); Thu, 13 Apr 2023 10:28:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231203AbjDMO25 (ORCPT ); Thu, 13 Apr 2023 10:28:57 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70B049740 for ; Thu, 13 Apr 2023 07:28:56 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id e16so1100000wra.6 for ; Thu, 13 Apr 2023 07:28:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396135; x=1683988135; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nDHHjHbDhUeYee8/gBhjE43x0PfM6JmFgWG0GA+SmFM=; b=V1zHw5gi4IzQQzwksT2aM0B6DaftIE6zn4lVJPamARwOhykbst1jgSWMMo9s4gCjKx U14POX2ARYt4Y5B46gpnnkEr3OkUUmYiFH5Aoeu2EhkT1GMoOexci/Wrxfb9R3y4UhZc ZzW28QhHxsdcneDY+DC7WesW4RSj14e9e5NqnAJZpRytavWkUjMY8wm8nLucMxHL+n6S LVbVASMce66hL7SNZsKKyaqvgRxpfqkg4/S43Y2HqHUpwi9+R/INMiWBJ6z2asUkyrQC wUicTCSp8QqMZJqfTvVKF5hHoDCyFat+Tr6MQW2PkgErGbDmZdTc23yJT3bs5ucqLG3U zboQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396135; x=1683988135; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nDHHjHbDhUeYee8/gBhjE43x0PfM6JmFgWG0GA+SmFM=; b=AmkfaFQhbl7r/LplHFmXK6y/gD5iSMTXHChRTF7hZgVGPo7PjzopCnPz65m92lO7To LdM75uo0fxu7hRQA5tJzLEtdlxlfc5lwT2g8nr9q4P6mjdONdAa6M7cdY5qCY9beldWA M76pTlnd0QVcMcqdWaAZSzXAz6jpmojPUPLEHDczUWqvtTdM0yNrxJ2P9ojib+Ghg5ub 35qXMxvDKhd5JV+wlbAxZGOBuV4Z3vWWrrcr1ROLHNgbCiZgi1yUhGqxP0p0T8KKJVyc W8w7aXr0h7/KPQ9EYc+KuqalLn3jwjteNqyLv6O+qIeyifugPv4YlhHWLH6G1aX7wzXW Nu9w== X-Gm-Message-State: AAQBX9f+C1iqDmsh5Sjh9P2G+AbYzRZUAXBD8ONNDcHE3inlgqsEHqSu z1nWmK0wug+Paop4gyFM48IDhqC++iU= X-Google-Smtp-Source: AKy350Y3isKABC9FPT6KXyr1zvnwFPttY6l0BiXRTejl4UrKxSgZZsADcVnPWP6OlgRbD5mQcaRpkA== X-Received: by 2002:adf:eed0:0:b0:2d7:9206:488d with SMTP id a16-20020adfeed0000000b002d79206488dmr1668793wrp.36.1681396134758; Thu, 13 Apr 2023 07:28:54 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:54 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 01/10] io_uring/rsrc: use nospec'ed indexes Date: Thu, 13 Apr 2023 15:28:05 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We use array_index_nospec() for registered buffer indexes, but don't use it while poking into rsrc tags, fix that. Fixes: 634d00df5e1cf ("io_uring: add full-fledged dynamic buffers support") Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 11058e20bdcc..3c1538b8c8f4 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -517,7 +517,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, } ctx->user_bufs[i] = imu; - *io_get_tag_slot(ctx->buf_data, offset) = tag; + *io_get_tag_slot(ctx->buf_data, i) = tag; } if (needs_switch) From patchwork Thu Apr 13 14:28:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBB9BC77B79 for ; Thu, 13 Apr 2023 14:29:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231203AbjDMO27 (ORCPT ); Thu, 13 Apr 2023 10:28:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229947AbjDMO26 (ORCPT ); Thu, 13 Apr 2023 10:28:58 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4848644B6 for ; Thu, 13 Apr 2023 07:28:57 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id e16so1100052wra.6 for ; Thu, 13 Apr 2023 07:28:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396135; x=1683988135; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g6WVIjemP9BBMRHiCNa00K2kQtNxwSekoN4VHF7m2Es=; b=p9hEWXXAdSxrQH8/jjLISmB2eV26h3z+JomPdu3uoK2kHs0v2AmAmnBokLcMSqav9w 9bzTycCJOOtUbvSlERI1KDdygYOo/dEwl5nuUVjmf/Z/RgjLN/z2X3RHOylPDUDF4G3S aehwLTgSkDzyMQOiZ4OmZMqCegnLtgiDqE3aG1aqDfzo02P8KfGLpGbwCWeBWwVBJzEM /ZkfYxP3BfXyZRenGi4B2/UG4WaxmryKxujuEbpVj0eYlz1mAdTjyfz+klK2KXgsTJZo P/3DeOPFpTzIOVCRuLWLcQGFCz5jxVNcf5c5QzXvN54vIM1L4r7r686o3ltDo+J5G12t D7jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396135; x=1683988135; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g6WVIjemP9BBMRHiCNa00K2kQtNxwSekoN4VHF7m2Es=; b=Nmllot3F/x72DbkjY+Ez4tcHgZUmgCPpa8lK8dr6sbVkevB+OEYbXdyQJhR2furCu6 4Wkr2D4/szG0vAJwyiGtgb9SeOBRIeCjsuStkgA48+zj5Frn+qTWJLCSiiJ0b9xgJ9zL /Ozwq0C20nqpmpu2aZHpwaZG4/QvWpYzwdYZnAY8aOW/pR69wd/2wrJtc29cpQIZD5by 8puYOKc0+N+oCUDDWAbjeBXJLdUFrLGvwa3+BLIp3fu0MAWygmVJe+N7zbwRoc0otjD4 CPTdfKoGg1L0x7qvOG7anxUI2r1Pbhdv83N+3hgj5qw9wUr+IhOraNEjiC3chwW8HsqZ v2qg== X-Gm-Message-State: AAQBX9e3fBKyyHBvu3YvnGqHKij6hr2HzET5j611ScLa7pZxeBv1g2JN uFJ7GmJ927Lbqu+TZtxQ1ghzjkysX0w= X-Google-Smtp-Source: AKy350bavVRt61KZOPXKH7++x1Oi6GQGRn5ghWzS6l/dOvnev3HjZ6qGpjR9bOa93sRz3JTirGKtSw== X-Received: by 2002:a5d:6d4c:0:b0:2f2:a03f:6066 with SMTP id k12-20020a5d6d4c000000b002f2a03f6066mr1684408wri.56.1681396135520; Thu, 13 Apr 2023 07:28:55 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:55 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 02/10] io_uring/rsrc: remove io_rsrc_node::done Date: Thu, 13 Apr 2023 15:28:06 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Kill io_rsrc_node::node and check refs instead, it's set when the nodes refcount hits zero, and it won't change afterwards. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 5 +---- io_uring/rsrc.h | 1 - 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 3c1538b8c8f4..5fc9d10743e0 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -191,14 +191,12 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; - node->done = true; while (!list_empty(&ctx->rsrc_ref_list)) { node = list_first_entry(&ctx->rsrc_ref_list, struct io_rsrc_node, node); /* recycle ref nodes in order */ - if (!node->done) + if (node->refs) break; - list_del(&node->node); __io_rsrc_put_work(node); } @@ -222,7 +220,6 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->item_list); - ref_node->done = false; ref_node->inline_items = 0; return ref_node; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 17dfe180208f..88adcb0b7963 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -48,7 +48,6 @@ struct io_rsrc_node { struct list_head node; struct llist_node llist; int refs; - bool done; /* * Keeps a list of struct io_rsrc_put to be completed. Each entry From patchwork Thu Apr 13 14:28:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95B1BC77B78 for ; Thu, 13 Apr 2023 14:29:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229951AbjDMO3A (ORCPT ); Thu, 13 Apr 2023 10:29:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231364AbjDMO27 (ORCPT ); Thu, 13 Apr 2023 10:28:59 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEE381BEA for ; Thu, 13 Apr 2023 07:28:57 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id w24so4776205wra.10 for ; Thu, 13 Apr 2023 07:28:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396136; x=1683988136; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nZ6XYc0QFzHpE95+bN9/BCuZ1pRcCn/sTnnsdjAOM5A=; b=pB1/rGfDoSs/7iMDT5URWO3rI+s/iGF9Ma478V5Z0f1lI7v29GnXl1MMnV6CDLaBuG xah4xIp/3iGMSA7avZuDlKwT57bhEzsW4UjV/+rJZMWKZhtpBQNTG+gMgngv5LV9l+XU PpdHuynEYZEvLrYfmPtTCNYZNg4dEDLKT3USqE+S2sQkfnutuz7D10EYJAZ7hKpNBScl soWVii1ROsa4BpTLH/M7qPOGv9eqW01MoxXIDf+ze57nnQj5xnPaewCKB3B/WKtCF9tn nOdQitfUr0YaLwdbtNqBYzXLn5ariC/aD1Dconn8HbbRH3htcwN6adVj+0Ki7Ge+Y+Oz 7Elg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396136; x=1683988136; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nZ6XYc0QFzHpE95+bN9/BCuZ1pRcCn/sTnnsdjAOM5A=; b=A58ygA4WJqWiSnYd7mn2ywnsd8ZcziV9R3ACEqnunjZGxre8gY7uvXxCcCWPZYlcFg FhtkWQEFSpUfs2TdVUC+2LVp26W9fybXS96QwWpQ1OfDkqMi+Yua0i+aaOpy+R2yXyMi FrjqnDHZVSONg5z0SiqvMxdpWDYeK6+4my0fXAuWCBjoA5IbALK3l+tNkiI/kqp8RHVi wTxd4vQe5zWx+NR5j3No9ABsALoAgyIbs8r1LOQN+xvjxrolgYxORwnSu0hhH//W3X1l /xJbibD9axcubcNTu/4I44efwh6FayJ/4K+QHpnDBuFPJNIiOgn82YOFGNmqcWf1eRhW Vn5g== X-Gm-Message-State: AAQBX9ec9e9ycqyna6FY1z8J+is0Cyf10JvFYciWlb/9nxdboDNnLjws ZK58FViPcqcDj6oDcBNaz4xMvJTw8q8= X-Google-Smtp-Source: AKy350ZR/uVJUllFbPqE2MZw7MO57hgEblUSEFa9CdHOGUN8h0aW7kPu72mcggpigp3gOBgzblMx5Q== X-Received: by 2002:a05:6000:1292:b0:2d2:5971:68b0 with SMTP id f18-20020a056000129200b002d2597168b0mr1525076wrx.22.1681396136192; Thu, 13 Apr 2023 07:28:56 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:55 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 03/10] io_uring/rsrc: refactor io_rsrc_ref_quiesce Date: Thu, 13 Apr 2023 15:28:07 +0100 Message-Id: <65bc876271fb16bf550a53a4c76c91aacd94e52e.1681395792.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Refactor io_rsrc_ref_quiesce() by moving the first mutex_unlock(), so we don't have to have a second mutex_unlock() further in the loop. It prepares us to the next patch. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 5fc9d10743e0..d7e7528f7159 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -272,8 +272,8 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, return 0; data->quiesce = true; - mutex_unlock(&ctx->uring_lock); do { + mutex_unlock(&ctx->uring_lock); ret = io_run_task_work_sig(ctx); if (ret < 0) { mutex_lock(&ctx->uring_lock); @@ -285,18 +285,10 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, } break; } - ret = wait_for_completion_interruptible(&data->done); - if (!ret) { - mutex_lock(&ctx->uring_lock); - if (!data->refs) - break; - /* - * it has been revived by another thread while - * we were unlocked - */ - mutex_unlock(&ctx->uring_lock); - } - } while (1); + wait_for_completion_interruptible(&data->done); + mutex_lock(&ctx->uring_lock); + ret = 0; + } while (data->refs); data->quiesce = false; return ret; From patchwork Thu Apr 13 14:28:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C36EAC77B61 for ; Thu, 13 Apr 2023 14:29:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230194AbjDMO3D (ORCPT ); Thu, 13 Apr 2023 10:29:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229947AbjDMO3A (ORCPT ); Thu, 13 Apr 2023 10:29:00 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FFFC44B6 for ; Thu, 13 Apr 2023 07:28:58 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id v10so2063953wmn.5 for ; Thu, 13 Apr 2023 07:28:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396137; x=1683988137; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PTOKH/T+nrgZmNzA+urhMPeBoAc4bSAgEXD1T3vQklA=; b=j8gC9J5fvxqDr97HoS+gN2Kwv5MHIQWfai8zwwHUKfOUjVcG8VNOXs0cZnEdgC9ysF Kv4sP1go8bFIBQWPpiwSaMid9T2kwuWGTrl87pjJhadzOz99OUHabWcXwsxyC4/M02EE w73UKsQaIP4fp6YRCYBmEVKUfzx4/XxwoNoHCS/i+3mEqCjoVFAInnRUGYxrc0E/NIO9 jHLW0sKYyiSWdaWk4SdVYZ9IunwBjYQl3TvXc54DvEUXVy4e1DF9Mg8TqNdSuEkls7Wq G6y6bBMBB/cEglnlKMeR2ByIgz5Z+pf2A6/qO2YWc6gyjd78wui2QelmXaL+LHxK0vlg 3g7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396137; x=1683988137; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PTOKH/T+nrgZmNzA+urhMPeBoAc4bSAgEXD1T3vQklA=; b=T6PV2/UnEs+8WLJk85KbcwUM6Bjz2iiydTkPcTLFld6f9ijl5Qi8STWMJJemztBw2Z qK9pFFUV94JRao4fkQgPuakkxvyH3tgtHnwTvvlOCo3WHvk9Z+zbtvvTEGycbgEhZTpJ MUWN3dwnmievBi0ZoBPsqOLhAuUOg69RdueOzvBryrzqs3UHTGAe2dTJJ+yajWiDZSNi rplkjArL+cnq5G22F1VZBE/KjXfMn9FFeZQiRgOczd4rPsDvDrpUS+cvfiDddJHURpcy 08dSmZs5IzBgt+5ix1fB3ToLc+tx+8nnUyehDu/b9Bmpdbshu+26SuhoNgFtm3tfJZfG 29ow== X-Gm-Message-State: AAQBX9fTftMbswIb6TD/TnlHD/ERseZGCS2ycbYfl3iIa1Ne5t2dAfJu 2E3akkWJCHRte7xZR7/yEQ8EcokubB8= X-Google-Smtp-Source: AKy350YktSVyOlm4y40Ga40+DMlMuE1hPBMUuxJ2UggT6gBLxyTPUhrdCDY2fW93j6p0Ge8g09OXbw== X-Received: by 2002:a05:600c:28d:b0:3ed:ce50:435a with SMTP id 13-20020a05600c028d00b003edce50435amr2039162wmk.10.1681396136914; Thu, 13 Apr 2023 07:28:56 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:56 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 04/10] io_uring/rsrc: use wq for quiescing Date: Thu, 13 Apr 2023 15:28:08 +0100 Message-Id: <1d0dbc74b3b4fd67c8f01819e680c5e0da252956.1681395792.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Replace completions with waitqueues for rsrc data quiesce, the main wakeup condition is when data refs hit zero. Note that data refs are only changes under ->uring_lock, so we prepare before mutex_unlock() reacquire it after taking the lock back. This change will be needed in the next patch. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 1 + io_uring/rsrc.c | 18 ++++++++++++------ io_uring/rsrc.h | 1 - 4 files changed, 14 insertions(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 40cab420b1bd..5c9645319770 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -333,6 +333,7 @@ struct io_ring_ctx { /* protected by ->uring_lock */ struct list_head rsrc_ref_list; struct io_alloc_cache rsrc_node_cache; + struct wait_queue_head rsrc_quiesce_wq; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9083a8466ebf..3c1c8c788b7b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -321,6 +321,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) mutex_init(&ctx->uring_lock); init_waitqueue_head(&ctx->cq_wait); init_waitqueue_head(&ctx->poll_wq); + init_waitqueue_head(&ctx->rsrc_quiesce_wq); spin_lock_init(&ctx->completion_lock); spin_lock_init(&ctx->timeout_lock); INIT_WQ_LIST(&ctx->iopoll_list); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d7e7528f7159..f9ce4076c73d 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -158,6 +158,7 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; + struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; if (ref_node->inline_items) @@ -171,13 +172,13 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) io_rsrc_node_destroy(rsrc_data->ctx, ref_node); if (io_put_rsrc_data_ref(rsrc_data)) - complete(&rsrc_data->done); + wake_up_all(&ctx->rsrc_quiesce_wq); } void io_wait_rsrc_data(struct io_rsrc_data *data) { - if (data && !io_put_rsrc_data_ref(data)) - wait_for_completion(&data->done); + if (data) + WARN_ON_ONCE(!io_put_rsrc_data_ref(data)); } void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) @@ -257,6 +258,7 @@ int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, struct io_ring_ctx *ctx) { + DEFINE_WAIT(we); int ret; /* As we may drop ->uring_lock, other task may have started quiesce */ @@ -273,7 +275,9 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, data->quiesce = true; do { + prepare_to_wait(&ctx->rsrc_quiesce_wq, &we, TASK_INTERRUPTIBLE); mutex_unlock(&ctx->uring_lock); + ret = io_run_task_work_sig(ctx); if (ret < 0) { mutex_lock(&ctx->uring_lock); @@ -285,12 +289,15 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, } break; } - wait_for_completion_interruptible(&data->done); + + schedule(); + __set_current_state(TASK_RUNNING); mutex_lock(&ctx->uring_lock); ret = 0; } while (data->refs); - data->quiesce = false; + finish_wait(&ctx->rsrc_quiesce_wq, &we); + data->quiesce = false; return ret; } @@ -366,7 +373,6 @@ __cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, goto fail; } } - init_completion(&data->done); *pdata = data; return 0; fail: diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 88adcb0b7963..d93ba4e9742a 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -35,7 +35,6 @@ struct io_rsrc_data { u64 **tags; unsigned int nr; rsrc_put_fn *do_put; - struct completion done; int refs; bool quiesce; }; From patchwork Thu Apr 13 14:28:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D63CEC77B6E for ; Thu, 13 Apr 2023 14:29:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231584AbjDMO3E (ORCPT ); Thu, 13 Apr 2023 10:29:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231472AbjDMO3A (ORCPT ); Thu, 13 Apr 2023 10:29:00 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87056A5DE for ; Thu, 13 Apr 2023 07:28:59 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id s12so5902448wrb.1 for ; Thu, 13 Apr 2023 07:28:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396138; x=1683988138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tkCeEcQcb1IucFaxfE9F99eQAeingWPXd5XsekzkgXA=; b=Tg6Hvvk1KarNAxnPOOPs4Ez029pXZTvWVY77j9a8knpFBgXfSunLeBB5n5XbpxWl0s QFldZO5Sz9Q07trMGOLJlBUJC8qvmMdNHFrH9cX+4kjG2bwumBwvgVgGkcjKmomef96A saAJnNH0qfDVMp2o5VCeFaO8uqkEsSm3kJj1wBfDuTrg1Yw+SrkRqj6WaFMkO0HLULQ6 YFxCvZe9JLrQEDFmvPhKaJJ6sFWEE/ZWNWS7HZHtarjJrtU8Rb5S7igx3r1JMbhFdV4S AkL12upRu6ZcaucT0jdqDfnq0ZczAbq9cUXS1JgpOhMBYllFPS489dsJeUVOY/8WECOG MsHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396138; x=1683988138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tkCeEcQcb1IucFaxfE9F99eQAeingWPXd5XsekzkgXA=; b=fYCggoGMyOcnDaq7oyGecRkn2McKUjxs3t3lk/lnr5bSm7EvmwgwYlx3q2g8E1nr71 hd8cweCzINjlU8ixuSl28WDBTgYmua2U4WGEB3wHqWXNBCFrlHGG+RvZv5EIAHmbgnys NWoDvv1L1GPZym4jf7vd+W4qWtFwERIa4WyFNpoHUZq/PvmO7/SUbNB9LpBJzOPB3l7w U4bIk8xeNPszZC1DJtBGON4lIP24yE9Fy+KfDR0r2nbAHF49Y4qQ1TVV9EeqPsw4uiXZ V6BYuJU5Y1869KMtSMhHMR5B0MAKivIZcHWtz2J3d2VFF3zsmVm/JK9MNlI5Yih1ivE+ 09ZQ== X-Gm-Message-State: AAQBX9ewdgnm2vl6XDoMzx2Kxy6jFLJ+SfXKGn6EUFjY/OPAoJ+S5tvx cH0H/l7yNtVE5bvR7kiLRRYvGx+s/Gk= X-Google-Smtp-Source: AKy350a8MFYJbJMyMUyUyVnoeSI0Gdh0aZPnNKDIDNcTDAsavzt+sEY8RBC1huGFNFieEgI4FBNtew== X-Received: by 2002:a5d:5290:0:b0:2cf:efc7:19ad with SMTP id c16-20020a5d5290000000b002cfefc719admr1425979wrv.53.1681396137877; Thu, 13 Apr 2023 07:28:57 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:57 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 05/10] io_uring/rsrc: fix DEFER_TASKRUN rsrc quiesce Date: Thu, 13 Apr 2023 15:28:09 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org For io_rsrc_ref_quiesce() to progress it should execute all task_work items, including deferred ones. However, currently nobody would wake us, and so let's set ctx->cq_wait_nr, so io_req_local_work_add() would wake us up. Fixes: c0e0d6ba25f18 ("io_uring: add IORING_SETUP_DEFER_TASKRUN") Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index f9ce4076c73d..e634ef384724 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -273,6 +273,11 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, if (io_put_rsrc_data_ref(data)) return 0; + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + atomic_set(&ctx->cq_wait_nr, 1); + smp_mb(); + } + data->quiesce = true; do { prepare_to_wait(&ctx->rsrc_quiesce_wq, &we, TASK_INTERRUPTIBLE); @@ -298,6 +303,10 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, finish_wait(&ctx->rsrc_quiesce_wq, &we); data->quiesce = false; + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + atomic_set(&ctx->cq_wait_nr, 0); + smp_mb(); + } return ret; } From patchwork Thu Apr 13 14:28:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 885A6C77B73 for ; Thu, 13 Apr 2023 14:29:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231524AbjDMO3F (ORCPT ); Thu, 13 Apr 2023 10:29:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229821AbjDMO3D (ORCPT ); Thu, 13 Apr 2023 10:29:03 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5464CAD2A for ; Thu, 13 Apr 2023 07:29:00 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id s12so5902502wrb.1 for ; Thu, 13 Apr 2023 07:29:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396138; x=1683988138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XNhLZukWHJN4z1mvPxjsyOxj4OpV4VMzlRf/y/OXoMM=; b=QNXWvBtWP97MB0Q1XQ3yJZqBPAtwU+wmsYjq+vP5Ri78oeAKCEtHAofZkKSRhYtkHE Hesfo9J3Vc/W3v5WljhP2rmLV7pch9RWge6ju5dpinctaMhMlq/grqtR0H4ORwZldhYf Hfq0rl0Zd7ZwVtDQ/8aUgUjHvM9S+15BP+O4zp3b1AJ+x8aCTpxutnfWLldRSpUagu2O p836krlgtT3SvghKh0iOwAwXyZcTRVXCTc0zShMWhqJUgvp8k67AXcx2Zjq7Dgi3s1uz Ulxa4qtcXBBicAUVvge0mVqn96ygLPtEgQy9mR5KVayLzsZGzKDWiLTcOECeuv1Az2y7 HHWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396138; x=1683988138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XNhLZukWHJN4z1mvPxjsyOxj4OpV4VMzlRf/y/OXoMM=; b=iI7N9EcBLI6u5DAuuYiOnQVlmvN8iO9N+g/bWpYHAO0wU2c+gPtUdAJR81BMf3X3A7 5jFOX+VPoCTzDXy4oyyMXrXqGgWuEo1krVW+yPAWjd4MkHD8y2vZrh4JcJd6SY0UWhqN 6cPilNv8LTQFMdj0zSXIZazRG5PQ6eo/u5iY2J2JxUSuVXOy5hM2oR1Yg3VlgvcyZjO+ bgEGytM81kPp8fEqHzM3C9zDn1CGGly61di75GvR82u9rUiHOsb9yarsxZ3GX7ZnBuZg RnhFGuP39Gd3dP2r9NeBOxj4WFHI7SgjVYLlMbPJjK9PIervggmjHWQJYtFd6EZ2QMLj 29Qg== X-Gm-Message-State: AAQBX9fgLnLKYFIgGKBhofO2mFX5Ce1snK4ltZz4qxvRl5TdfXKgeoZk sfQY26lM6RKgTjsiMPp3NncL5GJgeKg= X-Google-Smtp-Source: AKy350ZcC+rMcwKs5CF9/ujbjPKB21IVHgT8mePOHEVqjqG8rRKw8iR/Fct5bDMm0D0KyLGxt5K9cw== X-Received: by 2002:a05:6000:128f:b0:2f0:69c4:3900 with SMTP id f15-20020a056000128f00b002f069c43900mr1532275wrx.54.1681396138708; Thu, 13 Apr 2023 07:28:58 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:58 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 06/10] io_uring/rsrc: remove rsrc_data refs Date: Thu, 13 Apr 2023 15:28:10 +0100 Message-Id: <8e33fd143d83e11af3e386aea28eb6d6c6a1be10.1681395792.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of waiting for rsrc_data->refs to be downed to zero, check whether there are rsrc nodes queued for completion, that's easier then maintaining references. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 4 ++-- io_uring/rsrc.c | 32 ++++++++------------------------ io_uring/rsrc.h | 2 -- 4 files changed, 11 insertions(+), 28 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5c9645319770..1b2a20a42413 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -334,6 +334,7 @@ struct io_ring_ctx { struct list_head rsrc_ref_list; struct io_alloc_cache rsrc_node_cache; struct wait_queue_head rsrc_quiesce_wq; + unsigned rsrc_quiesce; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3c1c8c788b7b..3d43df8f1e4e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2831,8 +2831,8 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); /* __io_rsrc_put_work() may need uring_lock to progress, wait w/o it */ - io_wait_rsrc_data(ctx->buf_data); - io_wait_rsrc_data(ctx->file_data); + if (WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list))) + return; mutex_lock(&ctx->uring_lock); if (ctx->buf_data) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index e634ef384724..5415a18844e0 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -31,11 +31,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, #define IORING_MAX_FIXED_FILES (1U << 20) #define IORING_MAX_REG_BUFFERS (1U << 14) -static inline bool io_put_rsrc_data_ref(struct io_rsrc_data *rsrc_data) -{ - return !--rsrc_data->refs; -} - int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -158,7 +153,6 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; if (ref_node->inline_items) @@ -171,14 +165,6 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) } io_rsrc_node_destroy(rsrc_data->ctx, ref_node); - if (io_put_rsrc_data_ref(rsrc_data)) - wake_up_all(&ctx->rsrc_quiesce_wq); -} - -void io_wait_rsrc_data(struct io_rsrc_data *data) -{ - if (data) - WARN_ON_ONCE(!io_put_rsrc_data_ref(data)); } void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) @@ -201,6 +187,8 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) list_del(&node->node); __io_rsrc_put_work(node); } + if (list_empty(&ctx->rsrc_ref_list) && unlikely(ctx->rsrc_quiesce)) + wake_up_all(&ctx->rsrc_quiesce_wq); } struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) @@ -235,7 +223,6 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, if (WARN_ON_ONCE(!backup)) return; - data_to_kill->refs++; node->rsrc_data = data_to_kill; list_add_tail(&node->node, &ctx->rsrc_ref_list); /* put master ref */ @@ -269,8 +256,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, return ret; io_rsrc_node_switch(ctx, data); - /* kill initial ref */ - if (io_put_rsrc_data_ref(data)) + if (list_empty(&ctx->rsrc_ref_list)) return 0; if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { @@ -278,6 +264,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, smp_mb(); } + ctx->rsrc_quiesce++; data->quiesce = true; do { prepare_to_wait(&ctx->rsrc_quiesce_wq, &we, TASK_INTERRUPTIBLE); @@ -286,12 +273,8 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, ret = io_run_task_work_sig(ctx); if (ret < 0) { mutex_lock(&ctx->uring_lock); - if (!data->refs) { + if (list_empty(&ctx->rsrc_ref_list)) ret = 0; - } else { - /* restore the master reference */ - data->refs++; - } break; } @@ -299,10 +282,12 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, __set_current_state(TASK_RUNNING); mutex_lock(&ctx->uring_lock); ret = 0; - } while (data->refs); + } while (!list_empty(&ctx->rsrc_ref_list)); finish_wait(&ctx->rsrc_quiesce_wq, &we); data->quiesce = false; + ctx->rsrc_quiesce--; + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { atomic_set(&ctx->cq_wait_nr, 0); smp_mb(); @@ -371,7 +356,6 @@ __cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, data->nr = nr; data->ctx = ctx; data->do_put = do_put; - data->refs = 1; if (utags) { ret = -EFAULT; for (i = 0; i < nr; i++) { diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index d93ba4e9742a..5dd2fcb28069 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -35,7 +35,6 @@ struct io_rsrc_data { u64 **tags; unsigned int nr; rsrc_put_fn *do_put; - int refs; bool quiesce; }; @@ -69,7 +68,6 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); -void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); From patchwork Thu Apr 13 14:28:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57AC1C77B61 for ; Thu, 13 Apr 2023 14:29:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231598AbjDMO3G (ORCPT ); Thu, 13 Apr 2023 10:29:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231534AbjDMO3E (ORCPT ); Thu, 13 Apr 2023 10:29:04 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AFF2A5FE for ; Thu, 13 Apr 2023 07:29:01 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id v14-20020a05600c470e00b003f06520825fso13758428wmo.0 for ; Thu, 13 Apr 2023 07:29:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396139; x=1683988139; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cLniZvPHknjJJS9uxbnHH/hJ2WCLueA+3HT9X9D00zQ=; b=s3HeHlyZscqOuVHDisW3K+R6v5Urk3fnFMZs8ePtVXyu/aKTDA4ZLKDXnV2U1AH6a+ 6AuJ2jHfARioE+r06U/USnsmDTQyNmjvQ7kUM+CS+zlAdBHsfOD8thWIORr0Lfpxcdz6 RRjlj8LY8F7JDLs8bCP/IJ5Aq9/ea7Y3rMSO4K0oFGxHVghLw3umwRYbAUiVMFwULmLC nC1qq5WgZLvJdkwf19bKRyIXxYy7Qf5jVc7JkPkFX7M9np8Ju9Kv5WTTYpduwD74LCS9 iplWUe8RHL7MNAjRAM2tQb8aH7ijTdhlDvcVxTdbQ1Zq8EiI+lx9d6DD88SXPRqHWARh T0oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396139; x=1683988139; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cLniZvPHknjJJS9uxbnHH/hJ2WCLueA+3HT9X9D00zQ=; b=bKJlbeaRueNkN7hKWDKKAkAB/buYEgEYjsPHUvF2CwrLuqfCIddcdI4BodLwhXo2A1 ysXt6W7wmdpihhWeHZzuup0S4UoOBNH1mezGXshm+NCqyViOJXkmLgnwzauPYwW+aQlW 1ys5Y57ZIr+X89F0HcV3mVHXAU8DulzUSv+j/cTiXZKVN1mmboGd2zBdwinGpo/Fd237 0m0k5b4A3qHO+nrjr4y/CTnDqP9ose0toKWgbOSNpdVgIym40kUv/dxGq1QjMLUFTkWO FFobCbwy1rzRfVuqvOh4Ml2zujU98dBiI91HvYRwkEA5UrCKawMgETn5uaZYb7P0hABA c+0Q== X-Gm-Message-State: AAQBX9dU/8ZrddM3Qi1xDGx2H8xcVjKPhJQDnitC6IClKjDxmwmFd4v/ Uqu0Yqd+6qUSMrXIXCX+gJ/KvQbJR/E= X-Google-Smtp-Source: AKy350YxdDgCOgdYcRRwxSJt+8SduCXwrSkwaxFyKnP+E4UCxZPXskdvfElr62mN3Zx1ewqqfr6i9Q== X-Received: by 2002:a7b:cc93:0:b0:3ed:9212:b4fe with SMTP id p19-20020a7bcc93000000b003ed9212b4femr2097446wma.0.1681396139389; Thu, 13 Apr 2023 07:28:59 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:59 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 07/10] io_uring/rsrc: inline switch_start fast path Date: Thu, 13 Apr 2023 15:28:11 +0100 Message-Id: <9619c1717a0e01f22c5fce2f1ba2735f804da0f2.1681395792.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Inline the part of io_rsrc_node_switch_start() that checks whether the cache is empty or not, as most of the times it will have some number of entries in there. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 12 +++++------- io_uring/rsrc.h | 9 ++++++++- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 5415a18844e0..bfa0b382c6c6 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -230,15 +230,13 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, ctx->rsrc_node = backup; } -int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) +int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx) { - if (io_alloc_cache_empty(&ctx->rsrc_node_cache)) { - struct io_rsrc_node *node = kzalloc(sizeof(*node), GFP_KERNEL); + struct io_rsrc_node *node = kzalloc(sizeof(*node), GFP_KERNEL); - if (!node) - return -ENOMEM; - io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache); - } + if (!node) + return -ENOMEM; + io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 5dd2fcb28069..732496afed4c 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -69,7 +69,7 @@ void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); -int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); +int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); @@ -111,6 +111,13 @@ static inline int io_scm_file_account(struct io_ring_ctx *ctx, return __io_scm_file_account(ctx, file); } +static inline int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) +{ + if (unlikely(io_alloc_cache_empty(&ctx->rsrc_node_cache))) + return __io_rsrc_node_switch_start(ctx); + return 0; +} + int io_register_files_update(struct io_ring_ctx *ctx, void __user *arg, unsigned nr_args); int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, From patchwork Thu Apr 13 14:28:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E04FC77B77 for ; Thu, 13 Apr 2023 14:29:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231532AbjDMO3G (ORCPT ); Thu, 13 Apr 2023 10:29:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231538AbjDMO3E (ORCPT ); Thu, 13 Apr 2023 10:29:04 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C00A9AF38 for ; Thu, 13 Apr 2023 07:29:01 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id v20-20020a05600c471400b003ed8826253aso3879524wmo.0 for ; Thu, 13 Apr 2023 07:29:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396140; x=1683988140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FtT5o+ZLyLrRVDGXI3KV7UlkM+1pYjjuoTQeH9xE/mo=; b=E3/QhCRtDUsjv1eIFOXwoPLbIRoeCvJUDW9rB0ssmohjfg5DT4cmQlnlX4m4HiGuxK E9+g4RvH0ESxxG9U03tHzQj7IXmNRKyAddHhksnqPpcP6X/tVnQSYrdmCzB0Xu6kCpwP LtxPI+iBMh95UtmuVoZepCwyO57Nxyw/KqNrPz4NU2QPCkeCVezwQi4VICD6T8AlMW/o 79ExQ4wYAnnjWOl++FsDd3ZBHpXBQthfIAN4b9T4fA1qJYkINIO6mAVASYyi/7LmchS2 EEKqjfPCy7xtXOOuejxNnZT+YYVCMiWY7Gv5tf97plI4T4OwwBTGoTgGSshP2fR3WSXc YRZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396140; x=1683988140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FtT5o+ZLyLrRVDGXI3KV7UlkM+1pYjjuoTQeH9xE/mo=; b=LhhOrFQpm/QFkq+mRsCZRu0v/jWsI2Ja2GHSyHmtox1B9GkkDNXdFuYzgi6+0UG1PQ J2CNJzjunRztKYWejiOaPa6gsHzhWa1LPoP7ynYYuq9YzrkM1DwOXD7Cs7K1t9zr8iIj HiKxnYgmXggZC2tsCNYzqIXMt9Fq1FRdKCPqYthSPuPDISObRVTryDQYZfvvcr+4BEIj xmHkVWeClukF6aG5ud19TDvEdeOZVW60FXrhCFBlhyQK+83vwq/ccvtNdbhEQqQC9oPz K7QPTEc2iv7T379Beenu1GhX7nPrtqnv8xqjVz7xxcFExBY4StdiBDh6guOG+zAnR9BL WeqQ== X-Gm-Message-State: AAQBX9fP7G6jxPARUyyIIl27qu6UtICGKPsvX+NZAnfXjcKLTMJMUNh/ vgdcCGGXWXdgHz1FdDxKQWGrg33niW0= X-Google-Smtp-Source: AKy350ZlhrzeKUi1dlVPjA8siGRY47Cu9Em6kCwEY0RtBONycQj0ddVfphV8Rpc0HpvgYlgpoOUPZQ== X-Received: by 2002:a7b:c8d4:0:b0:3ef:6aa1:9284 with SMTP id f20-20020a7bc8d4000000b003ef6aa19284mr1918330wml.29.1681396140089; Thu, 13 Apr 2023 07:29:00 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.28.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:28:59 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 08/10] io_uring/rsrc: clean up __io_sqe_buffers_update() Date: Thu, 13 Apr 2023 15:28:12 +0100 Message-Id: <77936d9ed23755588810c5eafcea7e1c3b90e3cd.1681395792.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Inline offset variable, so we don't use it without subjecting it to array_index_nospec() first. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index bfa0b382c6c6..38f0c9ce67a7 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -469,7 +469,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, for (done = 0; done < nr_args; done++) { struct io_mapped_ubuf *imu; - int offset = up->offset + done; u64 tag = 0; err = io_copy_iov(ctx, &iov, iovs, done); @@ -490,7 +489,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, if (err) break; - i = array_index_nospec(offset, ctx->nr_user_bufs); + i = array_index_nospec(up->offset + done, ctx->nr_user_bufs); if (ctx->user_bufs[i] != ctx->dummy_ubuf) { err = io_queue_rsrc_removal(ctx->buf_data, i, ctx->rsrc_node, ctx->user_bufs[i]); From patchwork Thu Apr 13 14:28:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1476AC77B6E for ; Thu, 13 Apr 2023 14:29:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231571AbjDMO3H (ORCPT ); Thu, 13 Apr 2023 10:29:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231558AbjDMO3E (ORCPT ); Thu, 13 Apr 2023 10:29:04 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54F7AAD09 for ; Thu, 13 Apr 2023 07:29:02 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id i8-20020a05600c354800b003ee93d2c914so9938112wmq.2 for ; Thu, 13 Apr 2023 07:29:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396140; x=1683988140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XJDeBFWflXiYXndOhi/AhPK+oyloSSkv8OIb38IyP1c=; b=DGEOyitcGDuQFn0SMe4jrtK2G7ZPztpYYYIIqEoGNnj6uFwd+krlBSyuf/XPAJLIxC PglSBhr+8dDu0qfFGdJIUMlfLIn7TaxRNb//OEMJrFeQtBqhLTG1AwN6FcPpMYmg9kYZ taibiXPLWyqhiXMMrQRQQbEeiPjDmxAgXz8U9xd1tL70FUUQ5dV2BLxBaeOh6zE8gT2v 3vBkXm8UDZBw+DyyQ0Fy3BLy0jKRrv4Tsw7O38vuIVJfqyMbdTMelkVR1Yyt3vclB2hk tzFqZQuMwjFu4h0UfPEKe1FZlGb5s1jPXymEYsO/icUMNp9tJk2/la87eLDm6lYZ7dtD 2QKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396140; x=1683988140; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XJDeBFWflXiYXndOhi/AhPK+oyloSSkv8OIb38IyP1c=; b=cr5TtpS/AxRCNVkYEVPhpD1zTa1MGHy5u7nNPdJVE3Gd9EuumS/JxVu3CD9C7nAUTS hLpsFtU+VNhoGdCmFpAjUdxlPd9jzeGgij1pq/wKPoeF27KabIJ4Yk5/HBSBQ88s5Ika JfzGGgvAdpoMWgcLdQI7WUtVJA1sLgwADuvXHcJkDtAQQQiENGPq4Fvn32H/BC/mzQuS 0EyFoZ2n46psSURfZCxPq9dBQtbZYEJWtnnlZeBijNh/8b4uli7QQ3MFeYS/mrM1yHB7 QMSvUj5LzYB61+S10hpGdw3JX7ydMecpEiots5rEqTdtHYuOLOLnJnh9yfzRrUgUHqEz kT8g== X-Gm-Message-State: AAQBX9cpeyLJw/l+0sNKAd3PMghmjJLe6fWIRvMoGxLXj3lrFZpwDUzF nlX8V1C4ZfsXgG6cy+YdOQUUh1w8/tA= X-Google-Smtp-Source: AKy350bEZmI6Cb3vL6lx1k6FVubv/GiO9tAzRiEfFkNVvTPT9iKHWCvY7jjTDp0UmB6pJe74f05gQg== X-Received: by 2002:a05:600c:1e11:b0:3f0:4275:395f with SMTP id ay17-20020a05600c1e1100b003f04275395fmr1559707wmb.13.1681396140773; Thu, 13 Apr 2023 07:29:00 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.29.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:29:00 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 09/10] io_uring/rsrc: simplify single file node switching Date: Thu, 13 Apr 2023 15:28:13 +0100 Message-Id: <37cfb46f46160f81dced79f646e97db608994574.1681395792.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org At maximum io_install_fixed_file() removes only one file, so no need to keep needs_switch state and we can call io_rsrc_node_switch() right after removal. Signed-off-by: Pavel Begunkov --- io_uring/filetable.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/io_uring/filetable.c b/io_uring/filetable.c index b80614e7d605..6255fa255ae2 100644 --- a/io_uring/filetable.c +++ b/io_uring/filetable.c @@ -64,7 +64,6 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, u32 slot_index) __must_hold(&req->ctx->uring_lock) { - bool needs_switch = false; struct io_fixed_file *file_slot; int ret; @@ -83,16 +82,17 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, ret = io_rsrc_node_switch_start(ctx); if (ret) - goto err; + return ret; old_file = (struct file *)(file_slot->file_ptr & FFS_MASK); ret = io_queue_rsrc_removal(ctx->file_data, slot_index, ctx->rsrc_node, old_file); if (ret) - goto err; + return ret; + file_slot->file_ptr = 0; io_file_bitmap_clear(&ctx->file_table, slot_index); - needs_switch = true; + io_rsrc_node_switch(ctx, ctx->file_data); } ret = io_scm_file_account(ctx, file); @@ -101,9 +101,6 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, io_fixed_file_set(file_slot, file); io_file_bitmap_set(&ctx->file_table, slot_index); } -err: - if (needs_switch) - io_rsrc_node_switch(ctx, ctx->file_data); return ret; } From patchwork Thu Apr 13 14:28:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13210347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B399EC77B71 for ; Thu, 13 Apr 2023 14:29:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231558AbjDMO3I (ORCPT ); Thu, 13 Apr 2023 10:29:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229638AbjDMO3F (ORCPT ); Thu, 13 Apr 2023 10:29:05 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42B27AD0D for ; Thu, 13 Apr 2023 07:29:03 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id v27so5165171wra.13 for ; Thu, 13 Apr 2023 07:29:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681396141; x=1683988141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cO/jYwlVM1vE8zfdvxGQpMVWtu2D+phnbnHn/ydmrWg=; b=c64n4E4fAWwI4LZq/DJDEoIr8CDR7nbf1kTAhRssaFyWqwBp/0hB0NhKcXaUqM7y5M 5MIC4dR/1XgTaduU1bJtxb59mNNQ0cXh/uOiyXa6+pPtzJOnpkseCS2pCiiGKMlWugOj dAr/pOxs/8R6m/rvtwf4cTnIB33kVN39wEkd1i9Iq9NaHAvbZD/T/ONuRpSDmnwcYlqh MNWRv/Gn6oEkmD7L5BvNF0TGf57H2ckwGPaRcbR/WLd3pH8QPxN/OETkqsGKl4+rQPOc 9WJGEfV3P+11khY8Uxn0NkQu1Ga4y++8NKnATRA5/6vsNJY1QJItqEvt97e3Rm99j4A1 2GuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681396141; x=1683988141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cO/jYwlVM1vE8zfdvxGQpMVWtu2D+phnbnHn/ydmrWg=; b=W6Tt29xSMkGbhKDrNlfrGJOXbqj6Q70YvoJRD+0WRfwmj7KzA8euTkLLDv3crvhGoM 3YPuxSNqJbZiKr3ULkFbW270fZj2/ArVQgKflD4jSeO6h66SQAk3l+F1mANx5KSjDINA 0isK8bfR+gYfIcZO+ym/FiBbyv8VkqSaqQfU3fTm0fsWve7d9DVIOORkNIVxOxtkbGeK PFUoLxU6IOYkZ+uynipxyH+/O/BAs0xHOpeKdSiNbQPozw6nyOq8k56Szb285FaJ4Puf H41MuxokyMjvPpQmGtGSSjiiC09rQfG42VOOXc2ENgkHS2tMEs0eU/84hWBQfc+x8D1S Qoeg== X-Gm-Message-State: AAQBX9ewv8bAo786xgLVyroKgHsgi8eMPi/7LCoHWNhTIomBuk20Bs51 urYHg1+Ct67C5gqPPTUb0/+8+U5VOp0= X-Google-Smtp-Source: AKy350ZhuYJv2SRXihnttjllD0R4oXl7xlfIS6vzIA4DfYx8byzclqgOx81QsSdMWzMzqAhouSgoag== X-Received: by 2002:adf:f04d:0:b0:2f0:2dca:f914 with SMTP id t13-20020adff04d000000b002f02dcaf914mr1868737wro.52.1681396141393; Thu, 13 Apr 2023 07:29:01 -0700 (PDT) Received: from 127.0.0.1localhost (94.196.97.186.threembb.co.uk. [94.196.97.186]) by smtp.gmail.com with ESMTPSA id z14-20020adff1ce000000b002f28de9f73bsm1387391wro.55.2023.04.13.07.29.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:29:01 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 10/10] io_uring/rsrc: refactor io_queue_rsrc_removal Date: Thu, 13 Apr 2023 15:28:14 +0100 Message-Id: <36bd708ee25c0e2e7992dc19b17db166eea9ac40.1681395792.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We can queue up a rsrc into a list in io_queue_rsrc_removal() while allocating io_rsrc_put and so simplify the function. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 38f0c9ce67a7..db58a51d19da 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -685,7 +685,6 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, { u64 *tag_slot = io_get_tag_slot(data, idx); struct io_rsrc_put *prsrc; - bool inline_item = true; if (!node->inline_items) { prsrc = &node->item; @@ -694,14 +693,12 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); if (!prsrc) return -ENOMEM; - inline_item = false; + list_add(&prsrc->list, &node->item_list); } prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - if (!inline_item) - list_add(&prsrc->list, &node->item_list); return 0; }