From patchwork Thu Mar 30 14:53:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEA4BC761A6 for ; Thu, 30 Mar 2023 14:54:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232527AbjC3Oyg (ORCPT ); Thu, 30 Mar 2023 10:54:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232517AbjC3Oyf (ORCPT ); Thu, 30 Mar 2023 10:54:35 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47C37C0; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id q19so16311543wrc.5; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=og4wgHhZR2GI1KYFHJJde+4r5/spcbG2U9hgu2p3spE=; b=hRTo+Lt0jHxCwNBNqYPtSs/Fu3DRJIFkKl1hfQPK34dp4JAhtXHHi5Fzn0q/FXzjdO yWceZAoczbvHCjKwpfse9IW+0x6CFa5vppjOUpUnUE/UdW8LEscWvNv0fw/S/DsqiJri hZiRsl2bTtE/B4+R7sohDYRgKD+qjohZ/PqsdyZHV6Rd4qar5WpYCOJXtLHJUY+0L/B+ XrwsAJNeY8MyxkC1lrfH78DOiXq6t/CtrUF2qb3YjeExRNeWmk5gaOBpFf5JS8kihd7X +LMLFIquPGdR8/txghJ/ZQfP80sozyEY0RuNHdszUpycfk7gzeDTBIrSxILoYMi/wM9D RSVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=og4wgHhZR2GI1KYFHJJde+4r5/spcbG2U9hgu2p3spE=; b=j8/ltYu7Zifqfwwx7pcdixAAmKhKZqWekWCRfYQ3BfDSQyPZM98InL4VVwaRe+/n3H a0kDXq7y9MJr1rEaMhvZJY+7+SUO14FB6gsFzL+53C/YvTi3YZ9xmWWfF0F/ThYcpd2H /l8qqyv22WQo69nhPebkWHd1S88Poe3+Fvt1nGtLOJ4W9hRDJn57oELlyx4dd0+hBDYf tolfv1M9ECGAeWASj4GevwuUkC71IMeDo94+LBh5tcqB289dbFG62rQjBeQvXaCTEiMN 1f75XkkXlZWH+wNvjRIQsbWg6VlNt5pXb0lOcR5XT3PRJkqnFN8yCMtA190w0okHshXu G8lg== X-Gm-Message-State: AAQBX9forATOKaNK5FaXrsLManEuQuCIDtodYcc7CEfn3cuw0jZuQYo9 hJcnBEIDYoIpTfS6aWZ6nIbYHGzGBic= X-Google-Smtp-Source: AKy350YWivha0UWh8ExkJ5k3sHfMYKTaMkckXbx+VrNrZ5j4r+l/rDRvtAaAX9Tu+hnOdcYu6VzGrA== X-Received: by 2002:adf:dd10:0:b0:2d8:e6ba:99e8 with SMTP id a16-20020adfdd10000000b002d8e6ba99e8mr17987366wrm.33.1680188072431; Thu, 30 Mar 2023 07:54:32 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:32 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 01/11] io_uring/rsrc: use non-pcpu refcounts for nodes Date: Thu, 30 Mar 2023 15:53:19 +0100 Message-Id: <243504691227d611f27a0f642ec8982db6ef4ead.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org One problem with the current rsrc infra is that often updates will generates lots of rsrc nodes, each carry pcpu refs. That takes quite a lot of memory, especially if there is a stall, and takes lots of CPU cycles. Only pcpu allocations takes >50 of CPU with a naive benchmark updating files in a loop. Replace pcpu refs with normal refcounting. There is already a hot path avoiding atomics / refs, but following patches will further improve it. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 15 +++++---------- io_uring/rsrc.h | 6 ++++-- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 7a43aed8e395..f2da9e251e3f 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -155,7 +155,7 @@ void io_rsrc_refs_refill(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { ctx->rsrc_cached_refs += IO_RSRC_REF_BATCH; - percpu_ref_get_many(&ctx->rsrc_node->refs, IO_RSRC_REF_BATCH); + refcount_add(IO_RSRC_REF_BATCH, &ctx->rsrc_node->refs); } static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) @@ -220,13 +220,11 @@ void io_wait_rsrc_data(struct io_rsrc_data *data) void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) { - percpu_ref_exit(&ref_node->refs); kfree(ref_node); } -static __cold void io_rsrc_node_ref_zero(struct percpu_ref *ref) +__cold void io_rsrc_node_ref_zero(struct io_rsrc_node *node) { - struct io_rsrc_node *node = container_of(ref, struct io_rsrc_node, refs); struct io_ring_ctx *ctx = node->rsrc_data->ctx; unsigned long flags; bool first_add = false; @@ -269,11 +267,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) if (!ref_node) return NULL; - if (percpu_ref_init(&ref_node->refs, io_rsrc_node_ref_zero, - 0, GFP_KERNEL)) { - kfree(ref_node); - return NULL; - } + refcount_set(&ref_node->refs, 1); INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->rsrc_list); ref_node->done = false; @@ -298,7 +292,8 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, spin_unlock_irq(&ctx->rsrc_ref_lock); atomic_inc(&data_to_kill->refs); - percpu_ref_kill(&rsrc_node->refs); + /* put master ref */ + io_rsrc_put_node(rsrc_node, 1); ctx->rsrc_node = NULL; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index f27f4975217d..1467b31843bc 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -37,7 +37,7 @@ struct io_rsrc_data { }; struct io_rsrc_node { - struct percpu_ref refs; + refcount_t refs; struct list_head node; struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; @@ -54,6 +54,7 @@ struct io_mapped_ubuf { }; void io_rsrc_put_tw(struct callback_head *cb); +void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_rsrc_refs_refill(struct io_ring_ctx *ctx); void io_wait_rsrc_data(struct io_rsrc_data *data); @@ -109,7 +110,8 @@ int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) { - percpu_ref_put_many(&node->refs, nr); + if (refcount_sub_and_test(nr, &node->refs)) + io_rsrc_node_ref_zero(node); } static inline void io_req_put_rsrc(struct io_kiocb *req) From patchwork Thu Mar 30 14:53:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194308 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99590C6FD1D for ; Thu, 30 Mar 2023 14:54:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232537AbjC3Oyh (ORCPT ); Thu, 30 Mar 2023 10:54:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232508AbjC3Oyg (ORCPT ); Thu, 30 Mar 2023 10:54:36 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F043186; Thu, 30 Mar 2023 07:54:35 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id v1so19392294wrv.1; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QZd7Q0w7/z/xq1l37RPu+7/1znEwBdl9Vpp14WIchi8=; b=oF/E6TAr4lrfZC3fmGeL29xj5v2LilnNvXGMYN+gdNTEYSTobcTwHL6LMKkTnvTilM guWV31Ph4qQ12vMVFlMB9ziYdjPcyyFyUsGiPQlTbFQTPIuT/h/y7KoNi5T3NvGWxZXS btEX8m6FwtdOP1vj2Xk0DudDnm0Z4lJxtRd75g9EBhQIW8PmRDGrJ5M+2yv3yevg0Z0Y z3EOIPlf1W67j7MJZTP7tTv5DGVjSaCwuRN4D+6umji+8us0pcx4vt1FPWIsSwP2/BBK 2Qj0AS1PinRa6wnfLaukPvvitAD67UH7whoWpVytPV9clOKYcEZr+Qw4xNomkHHYSsSK t4hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QZd7Q0w7/z/xq1l37RPu+7/1znEwBdl9Vpp14WIchi8=; b=riWvFsM7E8fx0B2T40iZEvXFzW74SBenDk3odP9lvs2gxuzZvqUPPyzL4GQu9XISMO JhQ/lq/7zA+HPmJa751y/iHlkPgwGSPmZJQrfx5e58AlZouDBw1oSmLqHhBPru5X8yCs 2LJNdVwzC/NBGE1xsEii0Q3mq8QOJP+ejeQKm/nJtNrnwYTwU7/r3mKXy5lP3cOHHAN8 gNDU1LkBKQYe/TPig90n6G2SZf0ULitcWltKqUTYnfJditkxYPPxl8Iysd72v9pNZzEl tlhF6TLvUDoof8nM2C3kLGTc5z6ngzp0XX3f/zC+irM0e6tt/p53oaU3YxcicgGrKuC4 9t7w== X-Gm-Message-State: AAQBX9chM8VXRmM6OVk5RdOJroJc5yuh3e3tNb0LKXmJclXucaTbTyaz w2AhoamhpYp79ueTocEUWBAPXDlCPrw= X-Google-Smtp-Source: AKy350Y/uNss0dH9UJEjHVr3ULZNinKqCUYLvQ+llFtcI5K/Q44twm7N8/1LPksPqh49OQZR8Pn+Gw== X-Received: by 2002:a05:6000:12c3:b0:2ce:9f35:b645 with SMTP id l3-20020a05600012c300b002ce9f35b645mr19081107wrx.20.1680188074358; Thu, 30 Mar 2023 07:54:34 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:34 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 02/11] io_uring/rsrc: keep cached refs per node Date: Thu, 30 Mar 2023 15:53:20 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We cache refs of the current node (i.e. ctx->rsrc_node) in ctx->rsrc_cached_refs. We'll be moving away from atomics, so move the cached refs in struct io_rsrc_node for now. It's a prep patch and shouldn't change anything in practise. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 - io_uring/rsrc.c | 15 +++++++++------ io_uring/rsrc.h | 16 +++++++++------- 3 files changed, 18 insertions(+), 14 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 561fa421c453..a0a5b5964d3a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -240,7 +240,6 @@ struct io_ring_ctx { * uring_lock, and updated through io_uring_register(2) */ struct io_rsrc_node *rsrc_node; - int rsrc_cached_refs; atomic_t cancel_seq; struct io_file_table file_table; unsigned nr_user_files; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index f2da9e251e3f..1e7c960737fd 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -36,9 +36,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, void io_rsrc_refs_drop(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { - if (ctx->rsrc_cached_refs) { - io_rsrc_put_node(ctx->rsrc_node, ctx->rsrc_cached_refs); - ctx->rsrc_cached_refs = 0; + struct io_rsrc_node *node = ctx->rsrc_node; + + if (node && node->cached_refs) { + io_rsrc_put_node(node, node->cached_refs); + node->cached_refs = 0; } } @@ -151,11 +153,11 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -void io_rsrc_refs_refill(struct io_ring_ctx *ctx) +void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node) __must_hold(&ctx->uring_lock) { - ctx->rsrc_cached_refs += IO_RSRC_REF_BATCH; - refcount_add(IO_RSRC_REF_BATCH, &ctx->rsrc_node->refs); + node->cached_refs += IO_RSRC_REF_BATCH; + refcount_add(IO_RSRC_REF_BATCH, &node->refs); } static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) @@ -300,6 +302,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, if (!ctx->rsrc_node) { ctx->rsrc_node = ctx->rsrc_backup_node; ctx->rsrc_backup_node = NULL; + ctx->rsrc_node->cached_refs = 0; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 1467b31843bc..950535e2b9f4 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -43,6 +43,7 @@ struct io_rsrc_node { struct io_rsrc_data *rsrc_data; struct llist_node llist; bool done; + int cached_refs; }; struct io_mapped_ubuf { @@ -56,7 +57,7 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); -void io_rsrc_refs_refill(struct io_ring_ctx *ctx); +void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node); void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); void io_rsrc_refs_drop(struct io_ring_ctx *ctx); @@ -128,17 +129,18 @@ static inline void io_req_put_rsrc_locked(struct io_kiocb *req, if (node) { if (node == ctx->rsrc_node) - ctx->rsrc_cached_refs++; + node->cached_refs++; else io_rsrc_put_node(node, 1); } } -static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx) +static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, + struct io_rsrc_node *node) { - ctx->rsrc_cached_refs--; - if (unlikely(ctx->rsrc_cached_refs < 0)) - io_rsrc_refs_refill(ctx); + node->cached_refs--; + if (unlikely(node->cached_refs < 0)) + io_rsrc_refs_refill(ctx, node); } static inline void io_req_set_rsrc_node(struct io_kiocb *req, @@ -151,7 +153,7 @@ static inline void io_req_set_rsrc_node(struct io_kiocb *req, lockdep_assert_held(&ctx->uring_lock); req->rsrc_node = ctx->rsrc_node; - io_charge_rsrc_node(ctx); + io_charge_rsrc_node(ctx, ctx->rsrc_node); io_ring_submit_unlock(ctx, issue_flags); } } From patchwork Thu Mar 30 14:53:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C296C761A6 for ; Thu, 30 Mar 2023 14:54:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232559AbjC3Oyt (ORCPT ); Thu, 30 Mar 2023 10:54:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232539AbjC3Oyi (ORCPT ); Thu, 30 Mar 2023 10:54:38 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A01BA5D8; Thu, 30 Mar 2023 07:54:37 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id m6-20020a05600c3b0600b003ee6e324b19so11974552wms.1; Thu, 30 Mar 2023 07:54:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WWAczytuFz9lGZfCu/Rbuh88s55sTzs57HoEPIGs0GU=; b=ZPQgYFuWcz1SvJblTrxSJFLx5OAkM/KL/BYFd0RSfa/H2AiXu8SDOQ5j5v3iJprcop Jx9t48yuY9un4JFMIBZ0Tzqb/jVGOO6oTYOOE87IYBLtZrXfvYaxbD9Fa+ErOFN9ROHZ uEMnKIRmPXJoSZRA4y1uHnQeIGVXv7ebHVolEBgDmcM91gxa9mS4vxV+dLlkfDHiGEUq hrmrqXUi5RXmuK+b/ctz8UsjGEg9kZ6N18KJ9wFAb3LF5mxKTohvByLBmHIrGkRq3Oq7 BDvCISHhLESl1mCXuFCwFrNwydXnxpXpXJZ5vvnRAl9xhNkSijs3wYq93FgJUOh+MifF ecaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WWAczytuFz9lGZfCu/Rbuh88s55sTzs57HoEPIGs0GU=; b=zBUikyicM/zeIOYvvcF0FJG1rvDA7Xk2osdcEii+KEYCigA6MaeqrJ55SP3+R/sHuS JRgte4vX/mIgB4QkfZaoVwlpX2BhH7z9srWx9J0bIuoD7lkDZ/ItccSEkCA/twjw96hy Nx2nCOvchcjaoA2Bi2X6GRITHdNd9DsXk8UuhDXQmOaVJ+LdbMhpA5U26Dbev3I2PA0F Eo3na+FQgsM52MBQsQ2nRqtwzBrXMpsaJjLmlKhAoixN/dHhdBK5K01Y0FdFM2ELZeH4 LDC57UooIJ3DvKlFzYtJj0zTs5rvpFFNuaaZJW2p11lg+OePSEok2g6YQOjOyOOBboiW 9PvQ== X-Gm-Message-State: AO0yUKXOSbJ1vZxlm6P6SCWtO+UsJeOQpk0t69Gm6YqaoZyy0h7MdHwA Rausc7zmykOLp+pRRrpM2z0YpAu/DBo= X-Google-Smtp-Source: AK7set+zedwXs+fLdMumlxG3sBZ9gDN2RVPlR6v3s6LBi48heZcC1nvjcMspRQH/8cUFA6geqyC44w== X-Received: by 2002:a1c:7702:0:b0:3ed:4f7d:f6ee with SMTP id t2-20020a1c7702000000b003ed4f7df6eemr18359942wmi.14.1680188075498; Thu, 30 Mar 2023 07:54:35 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:35 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 03/11] io_uring: don't put nodes under spinlocks Date: Thu, 30 Mar 2023 15:53:21 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_req_put_rsrc() doesn't need any locking, so move it out of a spinlock section in __io_req_complete_post() and adjust helpers. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 7 +++++-- io_uring/rsrc.h | 6 +++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 536940675c67..c944833099a6 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -970,6 +970,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 static void __io_req_complete_post(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; + struct io_rsrc_node *rsrc_node = NULL; io_cq_lock(ctx); if (!(req->flags & REQ_F_CQE_SKIP)) @@ -990,7 +991,7 @@ static void __io_req_complete_post(struct io_kiocb *req) } io_put_kbuf_comp(req); io_dismantle_req(req); - io_req_put_rsrc(req); + rsrc_node = req->rsrc_node; /* * Selected buffer deallocation in io_clean_op() assumes that * we don't hold ->completion_lock. Clean them here to avoid @@ -1001,6 +1002,8 @@ static void __io_req_complete_post(struct io_kiocb *req) ctx->locked_free_nr++; } io_cq_unlock_post(ctx); + + io_put_rsrc_node(rsrc_node); } void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1117,7 +1120,7 @@ __cold void io_free_req(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; - io_req_put_rsrc(req); + io_put_rsrc_node(req->rsrc_node); io_dismantle_req(req); io_put_task_remote(req->task, 1); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 950535e2b9f4..8164777279ba 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -115,10 +115,10 @@ static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) io_rsrc_node_ref_zero(node); } -static inline void io_req_put_rsrc(struct io_kiocb *req) +static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - if (req->rsrc_node) - io_rsrc_put_node(req->rsrc_node, 1); + if (node) + io_rsrc_put_node(node, 1); } static inline void io_req_put_rsrc_locked(struct io_kiocb *req, From patchwork Thu Mar 30 14:53:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 527FDC761AF for ; Thu, 30 Mar 2023 14:54:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232539AbjC3Oyv (ORCPT ); Thu, 30 Mar 2023 10:54:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232517AbjC3Oyl (ORCPT ); Thu, 30 Mar 2023 10:54:41 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43F5E469E; Thu, 30 Mar 2023 07:54:39 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id l12so19362276wrm.10; Thu, 30 Mar 2023 07:54:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eKfMO97Puxbxg9b5V2zEzriSKSgWDMsPXMYWqPT9qnU=; b=NY7Rm6tRHFZ76SKQjHQdrDnj9n2jWzPmsbsXRr5sKH2eH3gRHQlQaR4Xoy5egkkK3c EGB/hzWp1i/3AzduUWZ0JfogkFiG/lN3KpUiFOWNOj0HuAABld2CwxHAYYR91+cifYln lLkPPpPcxJcB/9hItQ8keDXeLXji7HIwIybnBFNggrZtCIS5IwaZYLyw+ykua7UJmNwc jumPip94qNG8lZOkxUUzU8YTEBlEIkRpRgMXlhTiDkvZUBuZ4Vy/AsMWi4vfnLGJn7+Y of2tdDBRpfALUs8LZOZoKBAM22ckJDR5ThRkc3GrCWEjvxS0GdqOQzIw72A6U6d2B1cS xdQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eKfMO97Puxbxg9b5V2zEzriSKSgWDMsPXMYWqPT9qnU=; b=rnOztK7g0qXOxoferby0TlKXPJE0sR8f3Vf9pWOMbMgoaM7GAZOt+m5j2vRU+ImoeH Rgl0wKkBN7/HHitMojaD1+MbZ9QXNaDvRBaaEsTYETz0vmpynM3glevcuFRAWAqFxB4a NaesG+542U+xeRDtIYY3pBSttkQNW+WegNEL2kfloxkzkBiP533cPTtMdKN9DGE9+dck QQ0mPuUHRiV7vDU3gIhDqSKQ+G87Lm5FPyLIDNVQht36M15bb9PEFKSo2HpO64x7Y0j4 5qSwOMBsl25mTIEbscG9MEEYJZGqT5hI37a6y+fvo3Ty0PwzZCyMPNAzRV8XENJRk4Uq bYjA== X-Gm-Message-State: AAQBX9ecNOoehB6Rh6UIDnDjU+HRWCUqGXI8TyLDIF/0sy51yFCz4qib tmFSvpk0OrpqLIFEn5ztbxevWUxComM= X-Google-Smtp-Source: AKy350bQ2GkJl2HDcIMYZj5AAqTY05jsUSoyx44ho7iOR7A3DUp83IYa9NPnl7OD3yU2vmXXWOB3+g== X-Received: by 2002:a5d:500a:0:b0:2cb:29eb:a35e with SMTP id e10-20020a5d500a000000b002cb29eba35emr5127532wrt.11.1680188077486; Thu, 30 Mar 2023 07:54:37 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:37 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 04/11] io_uring: io_free_req() via tw Date: Thu, 30 Mar 2023 15:53:22 +0100 Message-Id: <6b0c640860a6250abf883840b160ccd50fd5f311.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_free_req() is not often used but nevertheless problematic as there is no way to know the current context, it may be used from the submission path or even by an irq handler. Push it to a fresh context using task_work. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c944833099a6..52a88da65f57 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1116,7 +1116,7 @@ static inline void io_dismantle_req(struct io_kiocb *req) io_put_file(req->file); } -__cold void io_free_req(struct io_kiocb *req) +__cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) { struct io_ring_ctx *ctx = req->ctx; @@ -1130,6 +1130,12 @@ __cold void io_free_req(struct io_kiocb *req) spin_unlock(&ctx->completion_lock); } +__cold void io_free_req(struct io_kiocb *req) +{ + req->io_task_work.func = io_free_req_tw; + io_req_task_work_add(req); +} + static void __io_req_find_next_prep(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; From patchwork Thu Mar 30 14:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68F91C6FD1D for ; Thu, 30 Mar 2023 14:54:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232543AbjC3Oy5 (ORCPT ); Thu, 30 Mar 2023 10:54:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232566AbjC3Oyt (ORCPT ); Thu, 30 Mar 2023 10:54:49 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C665FA273; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id j24so19424969wrd.0; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0i8xnepg2BvrVcYo9lxg1ps1ZVOTUo6D6qHGZbwYEt0=; b=cl2ADzpkTY/LFYdmyRhsDqzayDOvTYOGD2jnCQGmZx/Kgyw4F9oopSiAn7OUyNZTJ7 USRzQoUySTxFUnfOZROxaZQku9kXxtLAoJZiF4XFTdiCm7k9xkhfV0Wa/thfKM3o86UG ee1EQqQSWJNTdPcBxtCxl+yfeTZeG0OXhETx/dSmZd80H3IaFb/Cy7TfFvCj14JBY4OK hdcoQ4dllmyuqg86bTKJ3D5DfneriZ03NKkt3XaCbtt0GmgFV7LtZXrkPzSI/W0DSrPK whCN5fxCyTMxC6FKcmgsxf0ejVKAnbEl3S6PNgARe6JSPpaQbSpE5Uaj/BjAxRPjLjAK jxxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0i8xnepg2BvrVcYo9lxg1ps1ZVOTUo6D6qHGZbwYEt0=; b=V8IPxSGx3e/yb8N2rupvqa5G53NttYJ1XY19vZePRNWgY8hdcOZDg46Y3DeRhrvDd8 e7p8HG3/hmSokIwAIag3N5tOAUIKmeSFlxYVB8oP30fjOaZAu8AxSAfoXxSSAZO0my71 fqDMBubaLQ02cGIZ2ASt0tqTQYEmrgn/CAHGvSca6K4j9RLAZIu/wpKQcTCywlt5qgsJ yr2iNtGjl3uWGCh5Pkr8IuXtrfEGw5H3vqmkS5dTTQYvB8PJlKRfYtEqj+9CvlboO35O 8OrdzCzFQzKs4D3o5CykDHMD4OjFBS3huWuxudF1tRyPA7UCSkDfoeh75c0gvIAjUuc/ GfgA== X-Gm-Message-State: AAQBX9dC1EKf7DzNjW9lwD+W33oUQ6Ys07FVpcsYN7nvWmSvOTAh/JdS 8xpT/wgmUnYO+8t0xbF1xYq+GU99qRo= X-Google-Smtp-Source: AKy350aOWs6f6OFlrpRR+jpIV9FKwKggOI1fZUva9Raq1y/g/EqWO+plAudB1rPo0xC99aBoaTz7DQ== X-Received: by 2002:adf:f185:0:b0:2e4:b4f8:896c with SMTP id h5-20020adff185000000b002e4b4f8896cmr2517301wro.48.1680188080053; Thu, 30 Mar 2023 07:54:40 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 05/11] io_uring/rsrc: protect node refs with uring_lock Date: Thu, 30 Mar 2023 15:53:23 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Currently, for nodes we have an atomic counter and some cached (non-atomic) refs protected by uring_lock. Let's put all ref manipulations under uring_lock and get rid of the atomic part. It's free as in all cases we care about we already hold the lock. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 18 ++++++++++++------ io_uring/rsrc.c | 30 ++++-------------------------- io_uring/rsrc.h | 29 +++++------------------------ 3 files changed, 21 insertions(+), 56 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 52a88da65f57..e55c48d91209 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -967,7 +967,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 return true; } -static void __io_req_complete_post(struct io_kiocb *req) +static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *rsrc_node = NULL; @@ -1003,7 +1003,11 @@ static void __io_req_complete_post(struct io_kiocb *req) } io_cq_unlock_post(ctx); - io_put_rsrc_node(rsrc_node); + if (rsrc_node) { + io_ring_submit_lock(ctx, issue_flags); + io_put_rsrc_node(rsrc_node); + io_ring_submit_unlock(ctx, issue_flags); + } } void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1013,12 +1017,12 @@ void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) io_req_task_work_add(req); } else if (!(issue_flags & IO_URING_F_UNLOCKED) || !(req->ctx->flags & IORING_SETUP_IOPOLL)) { - __io_req_complete_post(req); + __io_req_complete_post(req, issue_flags); } else { struct io_ring_ctx *ctx = req->ctx; mutex_lock(&ctx->uring_lock); - __io_req_complete_post(req); + __io_req_complete_post(req, issue_flags & ~IO_URING_F_UNLOCKED); mutex_unlock(&ctx->uring_lock); } } @@ -1120,7 +1124,10 @@ __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) { struct io_ring_ctx *ctx = req->ctx; - io_put_rsrc_node(req->rsrc_node); + if (req->rsrc_node) { + io_tw_lock(ctx, ts); + io_put_rsrc_node(req->rsrc_node); + } io_dismantle_req(req); io_put_task_remote(req->task, 1); @@ -2791,7 +2798,6 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); - io_rsrc_refs_drop(ctx); /* __io_rsrc_put_work() may need uring_lock to progress, wait w/o it */ io_wait_rsrc_data(ctx->buf_data); io_wait_rsrc_data(ctx->file_data); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 1e7c960737fd..1237fc77c250 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -27,23 +27,10 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage); -#define IO_RSRC_REF_BATCH 100 - /* only define max */ #define IORING_MAX_FIXED_FILES (1U << 20) #define IORING_MAX_REG_BUFFERS (1U << 14) -void io_rsrc_refs_drop(struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) -{ - struct io_rsrc_node *node = ctx->rsrc_node; - - if (node && node->cached_refs) { - io_rsrc_put_node(node, node->cached_refs); - node->cached_refs = 0; - } -} - int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -153,13 +140,6 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node) - __must_hold(&ctx->uring_lock) -{ - node->cached_refs += IO_RSRC_REF_BATCH; - refcount_add(IO_RSRC_REF_BATCH, &node->refs); -} - static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; @@ -225,7 +205,8 @@ void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) kfree(ref_node); } -__cold void io_rsrc_node_ref_zero(struct io_rsrc_node *node) +void io_rsrc_node_ref_zero(struct io_rsrc_node *node) + __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; unsigned long flags; @@ -269,7 +250,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) if (!ref_node) return NULL; - refcount_set(&ref_node->refs, 1); + ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->rsrc_list); ref_node->done = false; @@ -283,8 +264,6 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, WARN_ON_ONCE(!ctx->rsrc_backup_node); WARN_ON_ONCE(data_to_kill && !ctx->rsrc_node); - io_rsrc_refs_drop(ctx); - if (data_to_kill) { struct io_rsrc_node *rsrc_node = ctx->rsrc_node; @@ -295,14 +274,13 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, atomic_inc(&data_to_kill->refs); /* put master ref */ - io_rsrc_put_node(rsrc_node, 1); + io_put_rsrc_node(rsrc_node); ctx->rsrc_node = NULL; } if (!ctx->rsrc_node) { ctx->rsrc_node = ctx->rsrc_backup_node; ctx->rsrc_backup_node = NULL; - ctx->rsrc_node->cached_refs = 0; } } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 8164777279ba..fd3bdd30a993 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -37,13 +37,12 @@ struct io_rsrc_data { }; struct io_rsrc_node { - refcount_t refs; + int refs; struct list_head node; struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; struct llist_node llist; bool done; - int cached_refs; }; struct io_mapped_ubuf { @@ -57,10 +56,8 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); -void io_rsrc_refs_refill(struct io_ring_ctx *ctx, struct io_rsrc_node *node); void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); -void io_rsrc_refs_drop(struct io_ring_ctx *ctx); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); @@ -109,38 +106,22 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); -static inline void io_rsrc_put_node(struct io_rsrc_node *node, int nr) -{ - if (refcount_sub_and_test(nr, &node->refs)) - io_rsrc_node_ref_zero(node); -} - static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - if (node) - io_rsrc_put_node(node, 1); + if (node && !--node->refs) + io_rsrc_node_ref_zero(node); } static inline void io_req_put_rsrc_locked(struct io_kiocb *req, struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) { - struct io_rsrc_node *node = req->rsrc_node; - - if (node) { - if (node == ctx->rsrc_node) - node->cached_refs++; - else - io_rsrc_put_node(node, 1); - } + io_put_rsrc_node(req->rsrc_node); } static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { - node->cached_refs--; - if (unlikely(node->cached_refs < 0)) - io_rsrc_refs_refill(ctx, node); + node->refs++; } static inline void io_req_set_rsrc_node(struct io_kiocb *req, From patchwork Thu Mar 30 14:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0866C6FD1D for ; Thu, 30 Mar 2023 14:56:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232630AbjC3O4v (ORCPT ); Thu, 30 Mar 2023 10:56:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232558AbjC3Oyt (ORCPT ); Thu, 30 Mar 2023 10:54:49 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C73B0BB9C; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id v1so19392652wrv.1; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MBkviYTvQX5do4saoK2KG74OjTaM9gN+C198bVZ0XcA=; b=WSAv013ms1/pzIyLkw4lGdHYzlytBUueBGnxq6JYDswxuiARTBXPhRIzJsx4xnCirS BrpknhFxgFsE0H934q/yLcThuaSk/4aeQzNY0hAhrsb2ypDepJWq5GY+/wzLnL1ClZm0 0xFdYpHaUR4wliz/kwKKHYVIG1rSwxeqLZvMqALYupXguyvhu7mG6JzhaKKWMhqndtPV 2PjRLKYSW7SUYwbQYioG+TZykNnTM1BEnf3KXpDF5iCULV/gnj/NZRIljXHOGXYM3J2q Mh8OX2Yzg1pQOLdanyhPELplfN7+8S6v48lq4AjvHAN5dP75Fgw5ObTHUStaekbadlOS RZ+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MBkviYTvQX5do4saoK2KG74OjTaM9gN+C198bVZ0XcA=; b=e5hA1kSikkRTU+dQFFvvmAOO2Dzk11V2EJXhHSx27SgODyqsE9VAJ9UQFUuNM5/ONc 5EPn51WvvgA6mLlbeaiCiNkf5ygubBMZK3fE1l+jL2Bj08R8BC79pyBg2X4olKY4ldz1 JOVv/YOZ7qhwpG4niat+OhW5a08RympMJGQny06na+Odxm9hpWFdg+SgxNrSV2VM2+TV LCgULuqQUGsyxVDCv5xt9KMKLijdWSzrODT0kGG+iTrSRwP/DuSVaUrVRMGfCisF8L0C z1w9TPy3R3dFkorIlTizsmWXv9IdV4EvAbDmLiQG9jKAx1P0ubWrROS8MrcR/spWJ54+ oMwA== X-Gm-Message-State: AAQBX9elmaqalEb+XrZ9VAL83ekD989AqtMwEOsUMCKQjttCzco3WMLQ W8kDLJv6trRTp7IIQlS/K5zSYbsgFBA= X-Google-Smtp-Source: AKy350bxHT4LZAk9AhO30h16NHSllIGRMDFqNeCz8YcyZDaG+5ho6qaiJlPfNfkg+sTT5t0PFqMJLg== X-Received: by 2002:a5d:404c:0:b0:2e5:1ee3:df77 with SMTP id w12-20020a5d404c000000b002e51ee3df77mr1142541wrp.46.1680188081160; Thu, 30 Mar 2023 07:54:41 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 06/11] io_uring/rsrc: kill rsrc_ref_lock Date: Thu, 30 Mar 2023 15:53:24 +0100 Message-Id: <80e76722fb1317d5315528b8b57de559ca5aea33.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We use ->rsrc_ref_lock spinlock to protect ->rsrc_ref_list in io_rsrc_node_ref_zero(). Now we removed pcpu refcounting, which means io_rsrc_node_ref_zero() is not executed from the irq context as an RCU callback anymore, and we also put it under ->uring_lock. io_rsrc_node_switch(), which queues up nodes into the list, is also protected by ->uring_lock, so we can safely get rid of ->rsrc_ref_lock. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 +- io_uring/io_uring.c | 1 - io_uring/rsrc.c | 5 ----- 3 files changed, 1 insertion(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index a0a5b5964d3a..9492889f00c0 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -333,8 +333,8 @@ struct io_ring_ctx { struct delayed_work rsrc_put_work; struct callback_head rsrc_put_tw; struct llist_head rsrc_put_llist; + /* protected by ->uring_lock */ struct list_head rsrc_ref_list; - spinlock_t rsrc_ref_lock; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e55c48d91209..e94780c0a024 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -325,7 +325,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->defer_list); INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); - spin_lock_init(&ctx->rsrc_ref_lock); INIT_LIST_HEAD(&ctx->rsrc_ref_list); INIT_DELAYED_WORK(&ctx->rsrc_put_work, io_rsrc_put_work); init_task_work(&ctx->rsrc_put_tw, io_rsrc_put_tw); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 1237fc77c250..e122b6e5f9c5 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -209,11 +209,9 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; - unsigned long flags; bool first_add = false; unsigned long delay = HZ; - spin_lock_irqsave(&ctx->rsrc_ref_lock, flags); node->done = true; /* if we are mid-quiesce then do not delay */ @@ -229,7 +227,6 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) list_del(&node->node); first_add |= llist_add(&node->llist, &ctx->rsrc_put_llist); } - spin_unlock_irqrestore(&ctx->rsrc_ref_lock, flags); if (!first_add) return; @@ -268,9 +265,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, struct io_rsrc_node *rsrc_node = ctx->rsrc_node; rsrc_node->rsrc_data = data_to_kill; - spin_lock_irq(&ctx->rsrc_ref_lock); list_add_tail(&rsrc_node->node, &ctx->rsrc_ref_list); - spin_unlock_irq(&ctx->rsrc_ref_lock); atomic_inc(&data_to_kill->refs); /* put master ref */ From patchwork Thu Mar 30 14:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97565C77B6D for ; Thu, 30 Mar 2023 14:54:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232517AbjC3Oy4 (ORCPT ); Thu, 30 Mar 2023 10:54:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232431AbjC3Oyu (ORCPT ); Thu, 30 Mar 2023 10:54:50 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3054DBDDC; Thu, 30 Mar 2023 07:54:43 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id r11so19373800wrr.12; Thu, 30 Mar 2023 07:54:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mFbJzcJB80Re1Q+sgkx/0FejbOmENveKGMdPIGHHusQ=; b=fAaanC13ZIcJPOqlgTHTTR8OQ622D5DOKgBDq8AivQ6OZol52ff75ivXCbEWtrKlC5 58sKNg8ei9TLxFzneeQcX+weXD5pVTkx5Una8QbUQ0N0zBSz4/9zQLBi2nTYeg17d/07 BDhvnc5atjth6fl/hQ5ySvlGOQ4T/zeoOqc1+3smISDFKVYbnru8vA3kiUons8RW7x/9 9BBOsInbG1nNwAtXQR4YCUNxAWpzV6a4ttCV3W5sudKKKy/FayqZQN9Pt/AsjOb5bby8 VnWKU1P6pGUsEfhtkSI0HXYIv9yMTfVnkL7D6zLr5YFZHIhgMR7qyMLLX2+qBWF9JQkJ jaZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mFbJzcJB80Re1Q+sgkx/0FejbOmENveKGMdPIGHHusQ=; b=7pdXaNG7xaXlQoYQxmzQdHSSeilQEnirizGCc5mSCujOHfKjUwniSq1Pf/gjuNOS5L GDW63J88pzh0TvN2etLkbFWD5qU52vkrANFpkX2kJLrkjH7PmMKNQaGxIl02ccwgpeaS KDsXSsAUc2P+L0vDfG1RKUFNK5VB/pXiDDvXFNNB9+PYCv1Kdi2iKo360wWzU8uscMhl NVT3gV/h3hsxSTzpywLrx5+6U/CyjURJ1nTLsq0Imrdteaqed5I2vdJt+6rv3Rs/53Av FtODsPjjD1qNaFahqIzWIYiI7L8al3AJrH5SKisVQbUGF4baF3qrfJhSEkfL+UlirgmN MTkQ== X-Gm-Message-State: AAQBX9e/w8p5SKdUhVIRKpspyG4DExUS/oCHWPpPxUm3gQs13yMuz3Lo uGlkt6hsqkpGzexBLc1pO3wzXX5gXiY= X-Google-Smtp-Source: AKy350ZeatGaowOqM6eQoPi2XZ7zNijxYQWjvrwuXid7H5G1nzF7H9I6wArvCfjDSVxDFk7TAqclEQ== X-Received: by 2002:a5d:53c8:0:b0:2dd:2a04:b73f with SMTP id a8-20020a5d53c8000000b002dd2a04b73fmr15772746wrw.49.1680188082258; Thu, 30 Mar 2023 07:54:42 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 07/11] io_uring/rsrc: rename rsrc_list Date: Thu, 30 Mar 2023 15:53:25 +0100 Message-Id: <00684614f29297882e2c03456351532cf027e167.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We have too many "rsrc" around which makes the name of struct io_rsrc_node::rsrc_list confusing. The field is responsible for keeping a list of files or buffers, so call it item_list and add comments around. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 6 +++--- io_uring/rsrc.h | 8 +++++++- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index e122b6e5f9c5..10006fb169d2 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -146,7 +146,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; - list_for_each_entry_safe(prsrc, tmp, &ref_node->rsrc_list, list) { + list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { list_del(&prsrc->list); if (prsrc->tag) { @@ -249,7 +249,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); - INIT_LIST_HEAD(&ref_node->rsrc_list); + INIT_LIST_HEAD(&ref_node->item_list); ref_node->done = false; return ref_node; } @@ -737,7 +737,7 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - list_add(&prsrc->list, &node->rsrc_list); + list_add(&prsrc->list, &node->item_list); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index fd3bdd30a993..c68846de031f 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -39,10 +39,16 @@ struct io_rsrc_data { struct io_rsrc_node { int refs; struct list_head node; - struct list_head rsrc_list; struct io_rsrc_data *rsrc_data; struct llist_node llist; bool done; + + /* + * Keeps a list of struct io_rsrc_put to be completed. Each entry + * represents one rsrc (e.g. file or buffer), but all of them should've + * came from the same table and so are of the same type. + */ + struct list_head item_list; }; struct io_mapped_ubuf { From patchwork Thu Mar 30 14:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECABAC77B60 for ; Thu, 30 Mar 2023 14:54:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232561AbjC3Oy5 (ORCPT ); Thu, 30 Mar 2023 10:54:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232598AbjC3Oyv (ORCPT ); Thu, 30 Mar 2023 10:54:51 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE967CA33; Thu, 30 Mar 2023 07:54:45 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id v1so19392779wrv.1; Thu, 30 Mar 2023 07:54:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188083; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h5UqEuOCJJfrll2Bn9RfMOzzWqUksKYu3hj36bqt65g=; b=ZV3O1gZvzbcSV9N01nRjxEXlT1bE5bVrzuzGPkbJEMK3lyPmu8mkDZhDpCBqhkg0cR zdmioPHKXXP7kioj47w4vHRX96j5rdpk16up0OUTOvY2RCZjdHkaNzEWWIJF6ARNg82i YAw6u1a14YutSOwyhHtBqZSgX5fxgNcY31F1AZTVVPObhcJrRAzpw37Dd2s4JL05NlDM KQiSUc2kTw/LAP69o2xXedgkAQzeYVDRC4BfCcGi5+n8bmzrTT4OI+Xf/nZp1Rw6RCXh YrMKkwiuOU7BCd/MHcqtn1iT6Yq7hGSHRF0clJJhQ47vxc5rwx7aVSJMqEFlanWQg0JX r+YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188083; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h5UqEuOCJJfrll2Bn9RfMOzzWqUksKYu3hj36bqt65g=; b=rDVHcY8pTewPfeYwCp0x4IY7n+AXmSK2LfiqBiZmA4R7IOVjE+IjUp2JXnLP+F1ocD LiRFjZnxHt9ZOcw6qGgZyt4RoHgQzwlOIY79AbrW6HFu5gmPumIG5qrwUhyZXJSqKkPc Q40On2B6Ry6ytikQkMQy8zglir2EJxZVdDj/MQEy73hB+Thg8USPFUzgRUrIkvb4Ecup gY0xBhsXLmUdArcNi8vVQu8lVRuWxdZV0krI03a+Ms2f4O4Bi4lIJuZM4C7ZX5JZBtYz NFAlgpQ9qxSnxli8Un26aoTT+y9Y/558qrgN2GUVePf+WkPIJtczYuPHSflySEQIWnD0 +xiA== X-Gm-Message-State: AAQBX9dIkC2bNbGZkZoq5bOyyp7etffrO5rR+/FyQ5xa0rvGu0SG2vQW IKYjf8H0Bu9wims8vUJ6eHtMsABP7mQ= X-Google-Smtp-Source: AKy350ZWndoACgthbhEEaxmyTm76K0EShpk6E2/GgEhGBrd0yM05B/swKX0ll08af/vfgf+V8SiRgQ== X-Received: by 2002:adf:e552:0:b0:2d6:8d2d:5a7c with SMTP id z18-20020adfe552000000b002d68d2d5a7cmr18407231wrm.57.1680188083411; Thu, 30 Mar 2023 07:54:43 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 08/11] io_uring/rsrc: optimise io_rsrc_put allocation Date: Thu, 30 Mar 2023 15:53:26 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Every io_rsrc_node keeps a list of items to put, and all entries are kmalloc()'ed. However, it's quite often to queue up only one entry per node, so let's add an inline entry there to avoid extra allocations. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 51 ++++++++++++++++++++++++++++++++----------------- io_uring/rsrc.h | 2 ++ 2 files changed, 36 insertions(+), 17 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 10006fb169d2..95e71300bb35 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -140,26 +140,34 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } +static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, + struct io_rsrc_put *prsrc) +{ + struct io_ring_ctx *ctx = rsrc_data->ctx; + + if (prsrc->tag) { + if (ctx->flags & IORING_SETUP_IOPOLL) { + mutex_lock(&ctx->uring_lock); + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + mutex_unlock(&ctx->uring_lock); + } else { + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); + } + } + rsrc_data->do_put(ctx, prsrc); +} + static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - struct io_ring_ctx *ctx = rsrc_data->ctx; struct io_rsrc_put *prsrc, *tmp; + if (ref_node->inline_items) + io_rsrc_put_work_one(rsrc_data, &ref_node->item); + list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { list_del(&prsrc->list); - - if (prsrc->tag) { - if (ctx->flags & IORING_SETUP_IOPOLL) { - mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - mutex_unlock(&ctx->uring_lock); - } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - } - } - - rsrc_data->do_put(ctx, prsrc); + io_rsrc_put_work_one(rsrc_data, prsrc); kfree(prsrc); } @@ -251,6 +259,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(void) INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->item_list); ref_node->done = false; + ref_node->inline_items = 0; return ref_node; } @@ -729,15 +738,23 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, { u64 *tag_slot = io_get_tag_slot(data, idx); struct io_rsrc_put *prsrc; + bool inline_item = true; - prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); - if (!prsrc) - return -ENOMEM; + if (!node->inline_items) { + prsrc = &node->item; + node->inline_items++; + } else { + prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); + if (!prsrc) + return -ENOMEM; + inline_item = false; + } prsrc->tag = *tag_slot; *tag_slot = 0; prsrc->rsrc = rsrc; - list_add(&prsrc->list, &node->item_list); + if (!inline_item) + list_add(&prsrc->list, &node->item_list); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c68846de031f..17293ab90f64 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -49,6 +49,8 @@ struct io_rsrc_node { * came from the same table and so are of the same type. */ struct list_head item_list; + struct io_rsrc_put item; + int inline_items; }; struct io_mapped_ubuf { From patchwork Thu Mar 30 14:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65C14C6FD1D for ; Thu, 30 Mar 2023 14:55:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232628AbjC3OzM (ORCPT ); Thu, 30 Mar 2023 10:55:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232611AbjC3Oyw (ORCPT ); Thu, 30 Mar 2023 10:54:52 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEC9ACC06; Thu, 30 Mar 2023 07:54:45 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id q19so16312196wrc.5; Thu, 30 Mar 2023 07:54:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TsovVhZknI6qeDE8kTU6QPdUJtFC7EMxs3/MNa3hVfA=; b=kfwYRcon7F1ORzR/RlIjWpwfgQKmtyIOPTNAWE95Z/LyNWD0toDNGfhLyrWbWRxQXb EZt3Tr2IBWMTLdJpnV4WJ0Bl2sgYaZq2nvo9N8/yB3lRkojyRpIii51pcNtLKGcS51M9 Xkh9fnMl5JKbqOEA+CA6TYVa6SjK5R2E57t0IOgkE8WPgFmMBt8EYZlkQzBho2D7nzHF 1/G1J+H/W93aiCzFTM9hcH7YOoB/KF7ELA7O+LVOn8iwZv0m0ztL1/b3SkrjQql8lyK9 3CNR/OuOZJOioEqSVeaNuvxAgridGwr9zeUt6qX5rncc6zQVeRooNNHnpoDJ33iWH19L Xrzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TsovVhZknI6qeDE8kTU6QPdUJtFC7EMxs3/MNa3hVfA=; b=fy+UFcNTj/gT4mDvXZ+I8aqWUTsfhiqsZ1FbN9NjWa8tSyUYg3V9b0W9nhRZgCBdiq Ceuyd4uAh2Y5HWHqWCQTc37TgyfLzDaOqylh+lqAo05zR8vcbe8OTr2P9M07fgZlSPRA RGnKSRY0K4/GP5UxglSg7I2qo+HwdNFEwvZqnD7CnIQnAIik5rTP2osIc98+wdzGsV2+ 9+Nqj+ZmuJwa/d01K8B17rq/oJLbyvGgT/ksZJeT6L5N/6XLhxKUbyi9Fy7fmYUKBBrY JLKVdKZsaphVEcgsnWolENqSECtMLubCTTJbeCamJyS0pD9ml7YZP1Zfxa0c6jQq/U6q dikA== X-Gm-Message-State: AAQBX9fCAlGAmJ01M/RlP8vOT9es2ay+MtHRwZT6J1BinXpf3rd5RWGi NZ33AQqNSxKYFtf1BXp8KOJlYsrRB7A= X-Google-Smtp-Source: AKy350bZLin5UOSFoyvcKHM1YioDYl+10F72AKeB16kOXfNqx4LVzfg1EkauemOq09DDu7fIB4AIPg== X-Received: by 2002:a5d:404a:0:b0:2d1:947:318b with SMTP id w10-20020a5d404a000000b002d10947318bmr18567845wrp.13.1680188084304; Thu, 30 Mar 2023 07:54:44 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 09/11] io_uring/rsrc: don't offload node free Date: Thu, 30 Mar 2023 15:53:27 +0100 Message-Id: <8083bdb49f57a968104137f1f256af8a5d46da64.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org struct delayed_work rsrc_put_work was previously used to offload node freeing because io_rsrc_node_ref_zero() was previously called by RCU in the IRQ context. Now, as percpu refcounting is gone, we can do it eagerly at the spot without pushing it to a worker. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 -- io_uring/io_uring.c | 6 ---- io_uring/rsrc.c | 59 +++------------------------------- 3 files changed, 4 insertions(+), 64 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9492889f00c0..47496059e13a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -330,9 +330,6 @@ struct io_ring_ctx { struct io_rsrc_data *file_data; struct io_rsrc_data *buf_data; - struct delayed_work rsrc_put_work; - struct callback_head rsrc_put_tw; - struct llist_head rsrc_put_llist; /* protected by ->uring_lock */ struct list_head rsrc_ref_list; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e94780c0a024..8c3886a4ca96 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -326,9 +326,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); INIT_LIST_HEAD(&ctx->rsrc_ref_list); - INIT_DELAYED_WORK(&ctx->rsrc_put_work, io_rsrc_put_work); - init_task_work(&ctx->rsrc_put_tw, io_rsrc_put_tw); - init_llist_head(&ctx->rsrc_put_llist); init_llist_head(&ctx->work_llist); INIT_LIST_HEAD(&ctx->tctx_list); ctx->submit_state.free_list.next = NULL; @@ -2822,11 +2819,8 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_rsrc_node_destroy(ctx->rsrc_node); if (ctx->rsrc_backup_node) io_rsrc_node_destroy(ctx->rsrc_backup_node); - flush_delayed_work(&ctx->rsrc_put_work); - flush_delayed_work(&ctx->fallback_work); WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); - WARN_ON_ONCE(!llist_empty(&ctx->rsrc_put_llist)); #if defined(CONFIG_UNIX) if (ctx->ring_sock) { diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 95e71300bb35..0f4e245dee1b 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -145,15 +145,8 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, { struct io_ring_ctx *ctx = rsrc_data->ctx; - if (prsrc->tag) { - if (ctx->flags & IORING_SETUP_IOPOLL) { - mutex_lock(&ctx->uring_lock); - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - mutex_unlock(&ctx->uring_lock); - } else { - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - } - } + if (prsrc->tag) + io_post_aux_cqe(ctx, prsrc->tag, 0, 0); rsrc_data->do_put(ctx, prsrc); } @@ -176,32 +169,6 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) complete(&rsrc_data->done); } -void io_rsrc_put_work(struct work_struct *work) -{ - struct io_ring_ctx *ctx; - struct llist_node *node; - - ctx = container_of(work, struct io_ring_ctx, rsrc_put_work.work); - node = llist_del_all(&ctx->rsrc_put_llist); - - while (node) { - struct io_rsrc_node *ref_node; - struct llist_node *next = node->next; - - ref_node = llist_entry(node, struct io_rsrc_node, llist); - __io_rsrc_put_work(ref_node); - node = next; - } -} - -void io_rsrc_put_tw(struct callback_head *cb) -{ - struct io_ring_ctx *ctx = container_of(cb, struct io_ring_ctx, - rsrc_put_tw); - - io_rsrc_put_work(&ctx->rsrc_put_work.work); -} - void io_wait_rsrc_data(struct io_rsrc_data *data) { if (data && !atomic_dec_and_test(&data->refs)) @@ -217,34 +184,18 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) __must_hold(&node->rsrc_data->ctx->uring_lock) { struct io_ring_ctx *ctx = node->rsrc_data->ctx; - bool first_add = false; - unsigned long delay = HZ; node->done = true; - - /* if we are mid-quiesce then do not delay */ - if (node->rsrc_data->quiesce) - delay = 0; - while (!list_empty(&ctx->rsrc_ref_list)) { node = list_first_entry(&ctx->rsrc_ref_list, struct io_rsrc_node, node); /* recycle ref nodes in order */ if (!node->done) break; - list_del(&node->node); - first_add |= llist_add(&node->llist, &ctx->rsrc_put_llist); - } - if (!first_add) - return; - - if (ctx->submitter_task) { - if (!task_work_add(ctx->submitter_task, &ctx->rsrc_put_tw, - ctx->notify_method)) - return; + list_del(&node->node); + __io_rsrc_put_work(node); } - mod_delayed_work(system_wq, &ctx->rsrc_put_work, delay); } static struct io_rsrc_node *io_rsrc_node_alloc(void) @@ -320,13 +271,11 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, if (ret < 0) { atomic_inc(&data->refs); /* wait for all works potentially completing data->done */ - flush_delayed_work(&ctx->rsrc_put_work); reinit_completion(&data->done); mutex_lock(&ctx->uring_lock); break; } - flush_delayed_work(&ctx->rsrc_put_work); ret = wait_for_completion_interruptible(&data->done); if (!ret) { mutex_lock(&ctx->uring_lock); From patchwork Thu Mar 30 14:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A51B2C761A6 for ; Thu, 30 Mar 2023 14:55:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232648AbjC3OzR (ORCPT ); Thu, 30 Mar 2023 10:55:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232647AbjC3Oyz (ORCPT ); Thu, 30 Mar 2023 10:54:55 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85756CC0B; Thu, 30 Mar 2023 07:54:48 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id y14so19396356wrq.4; Thu, 30 Mar 2023 07:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5L1thsz1ijtI+bii8vpLvzOuK72ES0dN5LpyxAEJB5c=; b=S8QHHi5z9LQOXa3uaWYMZ73OR1vDlNr3ze+rGxV67KXFNl+cuHXiqn5xbDswvDcMbt kdzkDFjmn2YYLGAt2MvCosiHyv0RaOltvUfqrEHfPO9pcC7c67zsfsQ6p9B6xrCr9TPw XLAxYueWYIYjma7qkjufO78bsMj3LhAjKmV32/78VXV0hIIfm7uZ/bjraokrdb+rnPuk CEV7/JHzJ3Dp5s5nGyMrzE3gJfrWX6m2go530EO+HSUcIN7nrLwni/bqJRAzqo9nLpQU 66tK2GGCYaq/M/oAqOdbQ6UD0qAM5CT9N8nS/Ic26HZIIJXGZMJ3MCzFNgfZdgq/92Gs Ss6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5L1thsz1ijtI+bii8vpLvzOuK72ES0dN5LpyxAEJB5c=; b=6kKQu9BAYX881ImrUsv0BvDOP7sxNGeEIOIXFIeXEE6lFPnun9Zd27RWuVkTM22FFG q+enWFsyAQpQRxuKMRJlkyOtFwoTRJhQx9uMfojsjRdXqCxLHShOH0vms6YXH5uhxXNh FAPKNAhvD3wmF4C67HKn6PUYxlU/ivOF93jx23VcVMK1+RDJopyCVa38t/u+B0kjIqqW VowGOzliUSRg60tDcLVIBL8Le4xzzN3vaYTrcB6EZ/KnM0XA8cvdwTgVNRvLul4sGF/l 0VPoZoa6D5YpHt84leF4l/rA1LvbMEJCC7IPybIbcBXyYz/24DgYvCHE1qFNGlWXGa1u 5K6g== X-Gm-Message-State: AAQBX9e7/reayQqiCBKACW2L+p+LLRgtrtrwGPjLwMjO/dZk6mHiG5oZ WIeT3BROepx3xkh9lrrOHcceo6xC8mA= X-Google-Smtp-Source: AKy350Y1t92VxNpQ0Ksc8TqGupI9EyiT/KYdiGXqDEGmSG2onayZh3ULzQ1M9OWwuGS9B0o1I6Q7fQ== X-Received: by 2002:a5d:5601:0:b0:2d2:ac99:a72 with SMTP id l1-20020a5d5601000000b002d2ac990a72mr20884439wrv.46.1680188086134; Thu, 30 Mar 2023 07:54:46 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:45 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 10/11] io_uring/rsrc: cache struct io_rsrc_node Date: Thu, 30 Mar 2023 15:53:28 +0100 Message-Id: <7f5eb1b89e8dcf93739607c79bbf7aec1784cbbe.1680187408.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add allocation cache for struct io_rsrc_node, it's always allocated and put under ->uring_lock, so it doesn't need any extra synchronisation around caches. Signed-off-by: Pavel Begunkov Reviewed-by: Gabriel Krisman Bertazi --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 11 +++++++++-- io_uring/rsrc.c | 23 +++++++++++++++-------- io_uring/rsrc.h | 5 ++++- 4 files changed, 29 insertions(+), 11 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 47496059e13a..5d772e36e7fc 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -332,6 +332,7 @@ struct io_ring_ctx { /* protected by ->uring_lock */ struct list_head rsrc_ref_list; + struct io_alloc_cache rsrc_node_cache; struct list_head io_buffers_pages; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8c3886a4ca96..beedaf403284 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -310,6 +310,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); + io_alloc_cache_init(&ctx->rsrc_node_cache, sizeof(struct io_rsrc_node)); io_alloc_cache_init(&ctx->apoll_cache, sizeof(struct async_poll)); io_alloc_cache_init(&ctx->netmsg_cache, sizeof(struct io_async_msghdr)); init_completion(&ctx->ref_comp); @@ -2791,6 +2792,11 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) mutex_unlock(&ctx->uring_lock); } +void io_rsrc_node_cache_free(struct io_cache_entry *entry) +{ + kfree(container_of(entry, struct io_rsrc_node, cache)); +} + static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); @@ -2816,9 +2822,9 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) /* there are no registered resources left, nobody uses it */ if (ctx->rsrc_node) - io_rsrc_node_destroy(ctx->rsrc_node); + io_rsrc_node_destroy(ctx, ctx->rsrc_node); if (ctx->rsrc_backup_node) - io_rsrc_node_destroy(ctx->rsrc_backup_node); + io_rsrc_node_destroy(ctx, ctx->rsrc_backup_node); WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); @@ -2830,6 +2836,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) #endif WARN_ON_ONCE(!list_empty(&ctx->ltimeout_list)); + io_alloc_cache_free(&ctx->rsrc_node_cache, io_rsrc_node_cache_free); if (ctx->mm_account) { mmdrop(ctx->mm_account); ctx->mm_account = NULL; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 0f4e245dee1b..345631091d80 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -164,7 +164,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) kfree(prsrc); } - io_rsrc_node_destroy(ref_node); + io_rsrc_node_destroy(rsrc_data->ctx, ref_node); if (atomic_dec_and_test(&rsrc_data->refs)) complete(&rsrc_data->done); } @@ -175,9 +175,10 @@ void io_wait_rsrc_data(struct io_rsrc_data *data) wait_for_completion(&data->done); } -void io_rsrc_node_destroy(struct io_rsrc_node *ref_node) +void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { - kfree(ref_node); + if (!io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache)) + kfree(node); } void io_rsrc_node_ref_zero(struct io_rsrc_node *node) @@ -198,13 +199,19 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) } } -static struct io_rsrc_node *io_rsrc_node_alloc(void) +static struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) { struct io_rsrc_node *ref_node; + struct io_cache_entry *entry; - ref_node = kzalloc(sizeof(*ref_node), GFP_KERNEL); - if (!ref_node) - return NULL; + entry = io_alloc_cache_get(&ctx->rsrc_node_cache); + if (entry) { + ref_node = container_of(entry, struct io_rsrc_node, cache); + } else { + ref_node = kzalloc(sizeof(*ref_node), GFP_KERNEL); + if (!ref_node) + return NULL; + } ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); @@ -243,7 +250,7 @@ int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) { if (ctx->rsrc_backup_node) return 0; - ctx->rsrc_backup_node = io_rsrc_node_alloc(); + ctx->rsrc_backup_node = io_rsrc_node_alloc(ctx); return ctx->rsrc_backup_node ? 0 : -ENOMEM; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 17293ab90f64..d1555eaae81a 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -4,6 +4,8 @@ #include +#include "alloc_cache.h" + #define IO_RSRC_TAG_TABLE_SHIFT (PAGE_SHIFT - 3) #define IO_RSRC_TAG_TABLE_MAX (1U << IO_RSRC_TAG_TABLE_SHIFT) #define IO_RSRC_TAG_TABLE_MASK (IO_RSRC_TAG_TABLE_MAX - 1) @@ -37,6 +39,7 @@ struct io_rsrc_data { }; struct io_rsrc_node { + struct io_cache_entry cache; int refs; struct list_head node; struct io_rsrc_data *rsrc_data; @@ -65,7 +68,7 @@ void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_wait_rsrc_data(struct io_rsrc_data *data); -void io_rsrc_node_destroy(struct io_rsrc_node *ref_node); +void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); From patchwork Thu Mar 30 14:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13194316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD5AAC6FD1D for ; Thu, 30 Mar 2023 14:55:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232695AbjC3OzX (ORCPT ); Thu, 30 Mar 2023 10:55:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232587AbjC3OzA (ORCPT ); Thu, 30 Mar 2023 10:55:00 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4056FCC28; Thu, 30 Mar 2023 07:54:50 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id h17so19368928wrt.8; Thu, 30 Mar 2023 07:54:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680188088; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Cs6esp50Jp8jC0Q9hhTRpI4nFHoCs+WBE2+fqKZb6bE=; b=lZAb3+CF9ww4EB25PAj0+FL6j458WyQZqxNaCRdNWNkCqzpefmh+epQzj26e3OrL8n TawKc4VhwlrLo35FK5XVNnUUfAZzzv4mIacNOrsSiYkn478r4ECEfAjWEG9hH5lnNh7X BkdLNArI1Xv6UnoeTi6296UPd++TUI5Ui/T93an238DKnTulXzRnQ9eMD/A9zK5xxU0y BV7BSq2FjbBKojJdqQ5mCUbaWV+np5V2VdduZBuDPk9HLHiqt4djiYChWGDy8BcrZKU2 yE/Bq3prxOLlWMFVPZv/ZEH2jnZprFNE0fyHk08xrt3HwTz72C5Ep0TuaR1HuCM8VntC 89VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680188088; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cs6esp50Jp8jC0Q9hhTRpI4nFHoCs+WBE2+fqKZb6bE=; b=ZcaqVxvXKLPbyQsDCZJK4zwNVk0roXyeFmTiJlDmtM4z6ackJp+1v9pz9YLG8RMhIG IK4mn75pz7jlSZ2frc+k20z0iYd5W3jHGWxEula/R0uC1GeednuhwCph13dm/X7z5qaH L+qGhO0N/Q3bY5Fp1Df6obk1CqXb79fQ2oi2HYj527liwBuQWhn2mNchMJiMbI//4nNy SVdb1txe0bljhZ4GhjaXEdpoU9bymdid+yUitCUeNvVDdqzJ8C7VRbS1QbZVxkw5aZod 6WY9CzkMM0jhDh7y63n/fSjTzEm8w8Mqxsg+uIVF5x04exBUBRyNh+ZxwDWESUuxw7bh ks3g== X-Gm-Message-State: AAQBX9dCAVnBmuJ8/4lw6WeQhc0BjZayrlTUEW/y/2UlUE/Pu28Im8W1 GmHK2YMPr21SAzooVdKN+oYDLA7uLGY= X-Google-Smtp-Source: AKy350apMKT24Tk1edTb7V3CDWaTS7QzV3PIphwMloAC3NGDCOIeI+eUl3G7kPsD/NcJ9F+a5GpFbQ== X-Received: by 2002:adf:f549:0:b0:2e5:ab4:22f0 with SMTP id j9-20020adff549000000b002e50ab422f0mr1582562wrp.60.1680188088366; Thu, 30 Mar 2023 07:54:48 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-231-234.dab.02.net. [82.132.231.234]) by smtp.gmail.com with ESMTPSA id d7-20020adffbc7000000b002d5a8d8442asm28727962wrs.37.2023.03.30.07.54.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Mar 2023 07:54:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH 11/11] io_uring/rsrc: add lockdep sanity checks Date: Thu, 30 Mar 2023 15:53:29 +0100 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We should hold ->uring_lock while putting nodes with io_put_rsrc_node(), add a lockdep check for that. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 4 ++-- io_uring/rsrc.c | 2 +- io_uring/rsrc.h | 6 ++++-- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index beedaf403284..a781b7243b97 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1002,7 +1002,7 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) if (rsrc_node) { io_ring_submit_lock(ctx, issue_flags); - io_put_rsrc_node(rsrc_node); + io_put_rsrc_node(ctx, rsrc_node); io_ring_submit_unlock(ctx, issue_flags); } } @@ -1123,7 +1123,7 @@ __cold void io_free_req_tw(struct io_kiocb *req, struct io_tw_state *ts) if (req->rsrc_node) { io_tw_lock(ctx, ts); - io_put_rsrc_node(req->rsrc_node); + io_put_rsrc_node(ctx, req->rsrc_node); } io_dismantle_req(req); io_put_task_remote(req->task, 1); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 345631091d80..d4bca5e18434 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -236,7 +236,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, atomic_inc(&data_to_kill->refs); /* put master ref */ - io_put_rsrc_node(rsrc_node); + io_put_rsrc_node(ctx, rsrc_node); ctx->rsrc_node = NULL; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index d1555eaae81a..99f2df4eafa1 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -117,8 +117,10 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); -static inline void io_put_rsrc_node(struct io_rsrc_node *node) +static inline void io_put_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { + lockdep_assert_held(&ctx->uring_lock); + if (node && !--node->refs) io_rsrc_node_ref_zero(node); } @@ -126,7 +128,7 @@ static inline void io_put_rsrc_node(struct io_rsrc_node *node) static inline void io_req_put_rsrc_locked(struct io_kiocb *req, struct io_ring_ctx *ctx) { - io_put_rsrc_node(req->rsrc_node); + io_put_rsrc_node(ctx, req->rsrc_node); } static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx,