From patchwork Tue Apr 18 13:06:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB724C6FD18 for ; Tue, 18 Apr 2023 13:07:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231557AbjDRNHX (ORCPT ); Tue, 18 Apr 2023 09:07:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231477AbjDRNHV (ORCPT ); Tue, 18 Apr 2023 09:07:21 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F62E1BD for ; Tue, 18 Apr 2023 06:07:10 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id xd13so39089792ejb.4 for ; Tue, 18 Apr 2023 06:07:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823229; x=1684415229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QH21MposTdKYhotLK06DVrC1pAm1KGaKinaxkkkAJyI=; b=A+71X2cXw6yQEvWEBJNfiiD7XPSBU/MTlKVzMLphV/uPFpdhMbDXOOsnsceDcItx80 Lrf3f1taxP6zBSJlURfJ+mRJwA6hgtj5imxI7XMMa3j89nCv9RQ9Gi47ijjfPrg6B3rf wZku68YhcufQy8shIOq/Nnle4XFqsLI6R2nXgEj7UEPm3kU1P0BbQxcQBwdxcRDQLm+L cV0mGZ00J3+SSsB2zfuiGnFMIXEhfMhQPgDVXXLMpH26DuJiDla2AzdR1Yu6zBlG3RWB 1U1SYeVzwQ7a6FvajpDTBaPVpqtshGP0X7/azeLjvqDW1VGe/QmqOKdjt+ZLzb1Qqlpz DJ1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823229; x=1684415229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QH21MposTdKYhotLK06DVrC1pAm1KGaKinaxkkkAJyI=; b=hpihzendgk5CALs8cjjmtHjBtz/PYgl0L/QRoObv365A/75zOJZWQdBvzaX6/WrnXo d+QIhpVdPEwChnyVg4DW9Jx13t8kc4jD33K/MneVbm5jN7opCBgf22kcoNZvPOS8Pjyo OuQXW+YYmBbsZlgbRkTGZWjqxbSH+WwoyXsb+6jqLJIOkR93eidEVKI4avp4RspuaSC3 /XZRLuOwLfOr3tqlL9VCHmcZXzFp7+ZxPokJ330chzo0k4jbdrxWcs7rfz/4DYGS8b1x oV+5DUWdfx5zN/bT/leKrWxw66Ktx2Mr27fafZHDOCcmhDA17bAQwlp1zOuNROWToMxM p18A== X-Gm-Message-State: AAQBX9fMAFxz7//qyGdl0NGwn+ya3mbZMFz/EU7kRu6PDf7iassoQFbL tUnB3xBjpjIApetzFe3UkowxhGY189g= X-Google-Smtp-Source: AKy350Y2EssRdTM7R3tN7QGICFI8Pg4p/nxdHX1truAS5aEVZjgLMS6j8hTIGV2eYJ5mnIiLvnXQhw== X-Received: by 2002:a17:906:4708:b0:94f:764e:e311 with SMTP id y8-20020a170906470800b0094f764ee311mr6533134ejq.16.1681823228818; Tue, 18 Apr 2023 06:07:08 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:08 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/8] io_uring/rsrc: remove unused io_rsrc_node::llist Date: Tue, 18 Apr 2023 14:06:34 +0100 Message-Id: <8e7d764c3f947489fde88d0927c3060d2e1bb599.1681822823.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org ->llist was needed for rsrc node destruction offload, which is removed now. Get rid of the unused field. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.h | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 732496afed4c..525905a30a55 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -44,7 +44,6 @@ struct io_rsrc_node { struct io_rsrc_data *rsrc_data; }; struct list_head node; - struct llist_node llist; int refs; /* From patchwork Tue Apr 18 13:06:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA03EC77B75 for ; Tue, 18 Apr 2023 13:07:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231477AbjDRNHY (ORCPT ); Tue, 18 Apr 2023 09:07:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231496AbjDRNHW (ORCPT ); Tue, 18 Apr 2023 09:07:22 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E61610F3 for ; Tue, 18 Apr 2023 06:07:10 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id dx24so28989662ejb.11 for ; Tue, 18 Apr 2023 06:07:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823229; x=1684415229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P0MsgxYB2Iq02trpPOl6BLs1BfzN5XDlIYa0nDp5ATM=; b=kph1pFazWK+8vqScP2gDYKDJPWdAdUWsK+5iHZ+Trk3HMVDMPbLYYQdbifizChPzVC 1EdTBkEvXnUS/mNI+vqH9zup0/y/164WGVcQAyIlCswhs44SSMqV0R6EQlqlHNiUzsJF rKHU79jtliBCa5CkvY8vmKxDH8rjJij97+jxsQJP1+0HNbuTw+YhYVP3toLipjqSFXJE IAkL7AijqODjfu3BxV7C1oduGRk8c5rnP2wM1N9EEcLkyTxve9iaw/wVnr6CuZS+ayje m5rxZkbOQkA+NnJq+8+/gc7QhpOu7gmrj207CdEx1iFLtPsLKlwzs7to+Vp7Z6JTagdg WafQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823229; x=1684415229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P0MsgxYB2Iq02trpPOl6BLs1BfzN5XDlIYa0nDp5ATM=; b=EFE393JCb1EBRqqU8YBjMjrx0NrH940MP2CX5vl2wGBrv22vf1TU+7HXazNByBl/QX AOChJs6/8nA1kPZLTQeNk/LsfwZfXxjmuLkwDt/4IFQFgRf95NyiWKrRBJnau/rY0S1A +e/isBKPRcOPbApoee6qlfwI2RmFVlOZjO5FcL/ES6BqkyQko0A4BPOV+CNeEUiLaRGm BmB+qlz5FHUoKgJn0hRo8x/M4xMGnPPfGaOuVRhJYma7XJjfwcjnvMJI8GZCsKlNYnSP x7C/Rcbzot/C59/qtshFEn3wU5UR5nyr4UpH7dd7HFJUg+0d1v/eCl+hFQ+v4i15Gt97 VArA== X-Gm-Message-State: AAQBX9fEsWRx/twXxr13sHHwdRUDV/LWNlmQBhjXo2TFO9dwaNa0oDzf 4n8qzPu/4dwKMGFfG36WQ1+xaHDis/o= X-Google-Smtp-Source: AKy350b9004eSvx1/6n78uTSPUUQdlIjLrOuTCc+ePnJxG5NpBihmVLwRxSEdu37uRfPrc78HVKlNg== X-Received: by 2002:a17:906:4f02:b0:94e:f9b:66e7 with SMTP id t2-20020a1709064f0200b0094e0f9b66e7mr10514308eju.13.1681823229197; Tue, 18 Apr 2023 06:07:09 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:09 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/8] io_uring/rsrc: infer node from ctx on io_queue_rsrc_removal Date: Tue, 18 Apr 2023 14:06:35 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org For io_queue_rsrc_removal() we should always use the current active rsrc node, don't pass it directly but let the function grab it from the context. Signed-off-by: Pavel Begunkov --- io_uring/filetable.c | 5 ++--- io_uring/rsrc.c | 9 +++++---- io_uring/rsrc.h | 3 +-- 3 files changed, 8 insertions(+), 9 deletions(-) diff --git a/io_uring/filetable.c b/io_uring/filetable.c index 6255fa255ae2..367a44a6c8c5 100644 --- a/io_uring/filetable.c +++ b/io_uring/filetable.c @@ -85,8 +85,7 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, return ret; old_file = (struct file *)(file_slot->file_ptr & FFS_MASK); - ret = io_queue_rsrc_removal(ctx->file_data, slot_index, - ctx->rsrc_node, old_file); + ret = io_queue_rsrc_removal(ctx->file_data, slot_index, old_file); if (ret) return ret; @@ -163,7 +162,7 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset) return -EBADF; file = (struct file *)(file_slot->file_ptr & FFS_MASK); - ret = io_queue_rsrc_removal(ctx->file_data, offset, ctx->rsrc_node, file); + ret = io_queue_rsrc_removal(ctx->file_data, offset, file); if (ret) return ret; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index db58a51d19da..3be483de613e 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -409,7 +409,7 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, if (file_slot->file_ptr) { file = (struct file *)(file_slot->file_ptr & FFS_MASK); - err = io_queue_rsrc_removal(data, i, ctx->rsrc_node, file); + err = io_queue_rsrc_removal(data, i, file); if (err) break; file_slot->file_ptr = 0; @@ -492,7 +492,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, i = array_index_nospec(up->offset + done, ctx->nr_user_bufs); if (ctx->user_bufs[i] != ctx->dummy_ubuf) { err = io_queue_rsrc_removal(ctx->buf_data, i, - ctx->rsrc_node, ctx->user_bufs[i]); + ctx->user_bufs[i]); if (unlikely(err)) { io_buffer_unmap(ctx, &imu); break; @@ -680,9 +680,10 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, - struct io_rsrc_node *node, void *rsrc) +int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc) { + struct io_ring_ctx *ctx = data->ctx; + struct io_rsrc_node *node = ctx->rsrc_node; u64 *tag_slot = io_get_tag_slot(data, idx); struct io_rsrc_put *prsrc; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 525905a30a55..8ed3e6a65cf6 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -70,8 +70,7 @@ void io_rsrc_put_work(struct work_struct *work); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); -int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, - struct io_rsrc_node *node, void *rsrc); +int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc); void io_rsrc_node_switch(struct io_ring_ctx *ctx, struct io_rsrc_data *data_to_kill); From patchwork Tue Apr 18 13:06:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C43C6C77B75 for ; Tue, 18 Apr 2023 13:07:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231847AbjDRNH1 (ORCPT ); Tue, 18 Apr 2023 09:07:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231464AbjDRNHX (ORCPT ); Tue, 18 Apr 2023 09:07:23 -0400 Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B89A3A8C for ; Tue, 18 Apr 2023 06:07:11 -0700 (PDT) Received: by mail-ej1-x62c.google.com with SMTP id c9so34168637ejz.1 for ; Tue, 18 Apr 2023 06:07:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823230; x=1684415230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SHGOcSbO4CFktsigyjeHOrQ9+I9wddk/Bik1I4oZ3is=; b=COD635/B7XacYaQFFVX1xojIdiqVy8Mm1TTUtDUOd3drzYR++Zioxb3jQgZ5yi2sDt aCS5rNqSa13q0w5BhLyFMkkPCHQaYc5r0mBny6W5xVXwRZOJTKQAjDHzZN4qu4WRbH15 yKvv46LUh30m6dEvQvFOpM/QdCltGQlMOjYpsFEqOZEiXhMO+vBnJ6jYZKJ3yZ70FNPm 1WLiCjpgOsko4eR6X2yshof+jqZyM5dwKpS9/wqJ1y9XT9vsBtlM9ezpC9NsjRCvBMb6 j2XwOEU0DI5UwzURote2OdrKVjLfBLboiu6YTrB8IioLmsEym3h/LjPpb3WU3NEPslZa I+6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823230; x=1684415230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SHGOcSbO4CFktsigyjeHOrQ9+I9wddk/Bik1I4oZ3is=; b=CDwY7kfnhQNG4dMeYd2+dBIR/+lgZ6DsV7pZavryvoFm0jjbNeRjgRQrlZ6msfeEwX A/BYWwueWgq1Ebk5jwLv2WOtSykXGJZWfK3eVuoYH7EeQQj0BAPsbJiFCzN7V4GsGK8D Nbe8Z4sOeT4DnuIiOVxZLlsHnF7dC6GXDlaowfElxNiTuJjyYtLbitkA7/jpZBjBlUZX OOZUiIOdBJsnNWfhOKQbSv8BsBammqCZ3bjsszB4WiqTaq5cV2Gi9PsEuhpe0iQoaqWi O9vlqW4XTV1RpLH2hADmAoiEeGgdCQeAi21npQKC8o3KL2mUeEIdqT96ihr30ND/BjHq 4qoA== X-Gm-Message-State: AAQBX9c4VkP/1IIrj36VBBrKQvKgOheAvGefos9pG0ulPUNKcp1fqpVK 0hnjsejWYHgb+UPgMBt3TZmbQ7B9Zq0= X-Google-Smtp-Source: AKy350aZ24dyAUcs+DhUZyUjIco1rcW3NFUS17SYrE0FosZX5k8Hss/OmZG6KVNqT4Ana7yppI3dAA== X-Received: by 2002:a17:906:3a82:b0:8a0:7158:15dc with SMTP id y2-20020a1709063a8200b008a0715815dcmr8709761ejd.74.1681823229859; Tue, 18 Apr 2023 06:07:09 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:09 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/8] io_uring/rsrc: merge nodes and io_rsrc_put Date: Tue, 18 Apr 2023 14:06:36 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org struct io_rsrc_node carries a number of resources represented by struct io_rsrc_put. That was handy before for sync overhead ammortisation, but all complexity is gone and nodes are simple and lightweight. Let's allocate a separate node for each resource. Nodes and io_rsrc_put and not much different in size, and former are cached, so node allocation should work better. That also removes some overhead for nested iteration in io_rsrc_node_ref_zero() / __io_rsrc_put_work(). Another reason for the patch is that it greatly reduces complexity by moving io_rsrc_node_switch[_start]() inside io_queue_rsrc_removal(), so users don't have to care about it. Signed-off-by: Pavel Begunkov --- io_uring/filetable.c | 9 ----- io_uring/rsrc.c | 91 +++++++++++--------------------------------- io_uring/rsrc.h | 22 +---------- 3 files changed, 25 insertions(+), 97 deletions(-) diff --git a/io_uring/filetable.c b/io_uring/filetable.c index 367a44a6c8c5..0f6fa791a47d 100644 --- a/io_uring/filetable.c +++ b/io_uring/filetable.c @@ -80,10 +80,6 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, if (file_slot->file_ptr) { struct file *old_file; - ret = io_rsrc_node_switch_start(ctx); - if (ret) - return ret; - old_file = (struct file *)(file_slot->file_ptr & FFS_MASK); ret = io_queue_rsrc_removal(ctx->file_data, slot_index, old_file); if (ret) @@ -91,7 +87,6 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, file_slot->file_ptr = 0; io_file_bitmap_clear(&ctx->file_table, slot_index); - io_rsrc_node_switch(ctx, ctx->file_data); } ret = io_scm_file_account(ctx, file); @@ -152,9 +147,6 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset) return -ENXIO; if (offset >= ctx->nr_user_files) return -EINVAL; - ret = io_rsrc_node_switch_start(ctx); - if (ret) - return ret; offset = array_index_nospec(offset, ctx->nr_user_files); file_slot = io_fixed_file_slot(&ctx->file_table, offset); @@ -168,7 +160,6 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset) file_slot->file_ptr = 0; io_file_bitmap_clear(&ctx->file_table, offset); - io_rsrc_node_switch(ctx, ctx->file_data); return 0; } diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 3be483de613e..a54a222a20b8 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -153,17 +153,10 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - struct io_rsrc_put *prsrc, *tmp; - if (ref_node->inline_items) + if (likely(ref_node->inline_items)) io_rsrc_put_work_one(rsrc_data, &ref_node->item); - list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) { - list_del(&prsrc->list); - io_rsrc_put_work_one(rsrc_data, prsrc); - kfree(prsrc); - } - io_rsrc_node_destroy(rsrc_data->ctx, ref_node); } @@ -206,53 +199,29 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) } ref_node->rsrc_data = NULL; - ref_node->refs = 1; - INIT_LIST_HEAD(&ref_node->node); - INIT_LIST_HEAD(&ref_node->item_list); ref_node->inline_items = 0; + ref_node->refs = 1; return ref_node; } -void io_rsrc_node_switch(struct io_ring_ctx *ctx, - struct io_rsrc_data *data_to_kill) - __must_hold(&ctx->uring_lock) -{ - struct io_rsrc_node *node = ctx->rsrc_node; - struct io_rsrc_node *backup = io_rsrc_node_alloc(ctx); - - if (WARN_ON_ONCE(!backup)) - return; - - node->rsrc_data = data_to_kill; - list_add_tail(&node->node, &ctx->rsrc_ref_list); - /* put master ref */ - io_put_rsrc_node(ctx, node); - ctx->rsrc_node = backup; -} - -int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx) -{ - struct io_rsrc_node *node = kzalloc(sizeof(*node), GFP_KERNEL); - - if (!node) - return -ENOMEM; - io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache); - return 0; -} - __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, struct io_ring_ctx *ctx) { + struct io_rsrc_node *backup; DEFINE_WAIT(we); int ret; - /* As we may drop ->uring_lock, other task may have started quiesce */ + /* As We may drop ->uring_lock, other task may have started quiesce */ if (data->quiesce) return -ENXIO; - ret = io_rsrc_node_switch_start(ctx); - if (ret) - return ret; - io_rsrc_node_switch(ctx, data); + + backup = io_rsrc_node_alloc(ctx); + if (!backup) + return -ENOMEM; + ctx->rsrc_node->rsrc_data = data; + list_add_tail(&ctx->rsrc_node->node, &ctx->rsrc_ref_list); + io_put_rsrc_node(ctx, ctx->rsrc_node); + ctx->rsrc_node = backup; if (list_empty(&ctx->rsrc_ref_list)) return 0; @@ -382,7 +351,6 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, struct file *file; int fd, i, err = 0; unsigned int done; - bool needs_switch = false; if (!ctx->file_data) return -ENXIO; @@ -414,7 +382,6 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, break; file_slot->file_ptr = 0; io_file_bitmap_clear(&ctx->file_table, i); - needs_switch = true; } if (fd != -1) { file = fget(fd); @@ -445,9 +412,6 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, io_file_bitmap_set(&ctx->file_table, i); } } - - if (needs_switch) - io_rsrc_node_switch(ctx, data); return done ? done : err; } @@ -458,7 +422,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, u64 __user *tags = u64_to_user_ptr(up->tags); struct iovec iov, __user *iovs = u64_to_user_ptr(up->data); struct page *last_hpage = NULL; - bool needs_switch = false; __u32 done; int i, err; @@ -498,15 +461,11 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, break; } ctx->user_bufs[i] = ctx->dummy_ubuf; - needs_switch = true; } ctx->user_bufs[i] = imu; *io_get_tag_slot(ctx->buf_data, i) = tag; } - - if (needs_switch) - io_rsrc_node_switch(ctx, ctx->buf_data); return done ? done : err; } @@ -515,15 +474,11 @@ static int __io_register_rsrc_update(struct io_ring_ctx *ctx, unsigned type, unsigned nr_args) { __u32 tmp; - int err; lockdep_assert_held(&ctx->uring_lock); if (check_add_overflow(up->offset, nr_args, &tmp)) return -EOVERFLOW; - err = io_rsrc_node_switch_start(ctx); - if (err) - return err; switch (type) { case IORING_RSRC_FILE: @@ -685,21 +640,21 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc) struct io_ring_ctx *ctx = data->ctx; struct io_rsrc_node *node = ctx->rsrc_node; u64 *tag_slot = io_get_tag_slot(data, idx); - struct io_rsrc_put *prsrc; - if (!node->inline_items) { - prsrc = &node->item; - node->inline_items++; - } else { - prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL); - if (!prsrc) - return -ENOMEM; - list_add(&prsrc->list, &node->item_list); + ctx->rsrc_node = io_rsrc_node_alloc(ctx); + if (unlikely(!ctx->rsrc_node)) { + ctx->rsrc_node = node; + return -ENOMEM; } - prsrc->tag = *tag_slot; + node->item.rsrc = rsrc; + node->item.tag = *tag_slot; + node->inline_items = 1; *tag_slot = 0; - prsrc->rsrc = rsrc; + + node->rsrc_data = data; + list_add_tail(&node->node, &ctx->rsrc_ref_list); + io_put_rsrc_node(ctx, node); return 0; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 8ed3e6a65cf6..bad7103f5033 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -18,7 +18,6 @@ enum { }; struct io_rsrc_put { - struct list_head list; u64 tag; union { void *rsrc; @@ -43,17 +42,10 @@ struct io_rsrc_node { struct io_cache_entry cache; struct io_rsrc_data *rsrc_data; }; - struct list_head node; int refs; - - /* - * Keeps a list of struct io_rsrc_put to be completed. Each entry - * represents one rsrc (e.g. file or buffer), but all of them should've - * came from the same table and so are of the same type. - */ - struct list_head item_list; - struct io_rsrc_put item; int inline_items; + struct list_head node; + struct io_rsrc_put item; }; struct io_mapped_ubuf { @@ -68,11 +60,8 @@ void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); void io_rsrc_put_work(struct work_struct *work); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); -int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc); -void io_rsrc_node_switch(struct io_ring_ctx *ctx, - struct io_rsrc_data *data_to_kill); int io_import_fixed(int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, @@ -109,13 +98,6 @@ static inline int io_scm_file_account(struct io_ring_ctx *ctx, return __io_scm_file_account(ctx, file); } -static inline int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) -{ - if (unlikely(io_alloc_cache_empty(&ctx->rsrc_node_cache))) - return __io_rsrc_node_switch_start(ctx); - return 0; -} - int io_register_files_update(struct io_ring_ctx *ctx, void __user *arg, unsigned nr_args); int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, From patchwork Tue Apr 18 13:06:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9E92C77B7D for ; Tue, 18 Apr 2023 13:07:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231637AbjDRNH0 (ORCPT ); Tue, 18 Apr 2023 09:07:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230411AbjDRNHX (ORCPT ); Tue, 18 Apr 2023 09:07:23 -0400 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AAB240EC for ; Tue, 18 Apr 2023 06:07:12 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id q23so63535119ejz.3 for ; Tue, 18 Apr 2023 06:07:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823230; x=1684415230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3s+aLd2VlnZ+D8NqSLeKDFBL4Pvzk/Q27FV4OleDAMY=; b=o7FaeMEc8/vveBiXyG4QC8am904TSKfSdwqYebxsookncx8UHViXCYpls1+seE+hLA 0OQnyvXA3G9H5ARaLl9jE3B+Z1r7OLR5hDFlUEZtiJ4GmEl0Hb/0jaVDes/8/zD/E4hj F/76+7kXL+9cVLdubvGeXJ8SGihzcqWj6siRnZHcPO44CjIvrtCeJpoVbAQUkxXucoIv 7dWGG45BU5lFbZCKIXW7B2tq7hL7CbD/R0BswNGCtdZzCOiGbdQAf3yV+rBUqPtbBh1l p+CnDDjrXXk85L+Pd0Q+TdTIePJ/R/c2xsMboRRc6Xystidm2HZX2t75uxPk+y7Ji3Qu Cu1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823230; x=1684415230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3s+aLd2VlnZ+D8NqSLeKDFBL4Pvzk/Q27FV4OleDAMY=; b=HgEpEYevhL+FkkdLJ0eRB4f7YgHW1iaVf6ul9DTt5gC5+N0zdQd69HOmTPNvup3xUM WkJh6Ihm25T10+ZF3keLpf2lkLmP2eBgS8+1gFI3D9BmR46C5SZqxwuWt9g7z6//GzAA m83lmGnG7F9d1kuW5gV9y9wWOfpB+hMpNgzuCT+cIlsi1mnwE4Rry0MO6iuZpzXA9KYg zTiUUT2zcp9mnevZ0Efpxb6DY3B1awcWkYefDX4ZvWuHtjgr19FmW6I59BjluekLAMgZ 63okGxobN+LOUFBjU44x9KoOyrFY16ePgwOMFtyHBeHiiwW6bx+2lFwRxBN7L6C9CGIt /log== X-Gm-Message-State: AAQBX9fui1aI+WRxjSy8pCl1Talihs0lKr5HNRKmevmP8YvE7ZgYnsVw 0lmGKYF+Zf16KnBKeuqUWua7HbrKnTE= X-Google-Smtp-Source: AKy350azwT1ucW6TCwK7zBlCX97gXXS5g3ytpGRwNbjWAvToftlkmG4Z2bCNNfB5FS7MPdFVf9RbDw== X-Received: by 2002:a17:906:e247:b0:947:d875:68f9 with SMTP id gq7-20020a170906e24700b00947d87568f9mr10520682ejb.0.1681823230257; Tue, 18 Apr 2023 06:07:10 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:10 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/8] io_uring/rsrc: add empty flag in rsrc_node Date: Tue, 18 Apr 2023 14:06:37 +0100 Message-Id: <75d384c9d2252e12af73b9cf8a44e1699106aeb1.1681822823.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Unless a node was flushed by io_rsrc_ref_quiesce(), it'll carry a resource. Replace ->inline_items with an empty flag, which is initialised to false and only raised in io_rsrc_ref_quiesce(). Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 6 +++--- io_uring/rsrc.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index a54a222a20b8..127bd602131e 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -154,7 +154,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) { struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - if (likely(ref_node->inline_items)) + if (likely(!ref_node->empty)) io_rsrc_put_work_one(rsrc_data, &ref_node->item); io_rsrc_node_destroy(rsrc_data->ctx, ref_node); @@ -199,7 +199,7 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) } ref_node->rsrc_data = NULL; - ref_node->inline_items = 0; + ref_node->empty = 0; ref_node->refs = 1; return ref_node; } @@ -218,6 +218,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, backup = io_rsrc_node_alloc(ctx); if (!backup) return -ENOMEM; + ctx->rsrc_node->empty = true; ctx->rsrc_node->rsrc_data = data; list_add_tail(&ctx->rsrc_node->node, &ctx->rsrc_ref_list); io_put_rsrc_node(ctx, ctx->rsrc_node); @@ -649,7 +650,6 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc) node->item.rsrc = rsrc; node->item.tag = *tag_slot; - node->inline_items = 1; *tag_slot = 0; node->rsrc_data = data; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index bad7103f5033..f3fe455c6c71 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -43,7 +43,7 @@ struct io_rsrc_node { struct io_rsrc_data *rsrc_data; }; int refs; - int inline_items; + bool empty; struct list_head node; struct io_rsrc_put item; }; From patchwork Tue Apr 18 13:06:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77CF4C77B78 for ; Tue, 18 Apr 2023 13:07:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231496AbjDRNHZ (ORCPT ); Tue, 18 Apr 2023 09:07:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230311AbjDRNHX (ORCPT ); Tue, 18 Apr 2023 09:07:23 -0400 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4662A44AE for ; Tue, 18 Apr 2023 06:07:12 -0700 (PDT) Received: by mail-ej1-x635.google.com with SMTP id q23so63535151ejz.3 for ; Tue, 18 Apr 2023 06:07:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823230; x=1684415230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=10fQcA/svKRuqZd5Nyyvu/RW0+SRmiK1DY7rrVIZsO4=; b=X0qGmNBggWzQ4LQMFozaQ5KqUsvi5MLSa6kqGYWI719qx58UPWEu07u/73mT9ZmyhF 25PfouX39T2tVzkcprYNsR+oPLUrTYgKkfeyRd2GxLmOlcJJmzyKfhIj1ItxItGfz/Rb uqWJ/5VW26U6OlhbHjCC1E7z9xU6oI7F7IxMFpLu6Outqw7I5ZFa0H7RPMFwjkSIlvQP LfX2ta0wHTjKmeQzWUNiMAwdaULw55LMbms3YQwXLH3alcOdN2kV/vKlhlt/3uvZVmvt ou0wc/230XEwxDdoQWKMcgH5oyOgMxP/TV/9oVv8culqYw877gdHKEug5LWKC/d25GuR lUeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823230; x=1684415230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=10fQcA/svKRuqZd5Nyyvu/RW0+SRmiK1DY7rrVIZsO4=; b=g1pVEkp7GxsmFBHzp4Q2FbQXTSAvu8fijldIaCNfaxPL1tw1m645gyVVQIzz5fZdP4 b3DUfFZ5MX1Tb+2CxtnqY0vdGXYRz7yUp/vTWrU/gOoaYmnnaB9ZYXg5uzecLKA4H6x/ VOdMjoGC/t3RyvkzpX8Vf5Q5FECwtHaFPx10be0Fkcp+UgybTcQsu88w1T9S6Ft3CSsZ P+oCaYyfRoJR9ORzhPScy84t5lKgz2zpn820zCl8kSEutMuec0De9oRnxhz+lBJuHwo2 Vm/AjGNHxYPkv6WP1DRXN3SBw4rva/CDyHRz19l630+JFqUb1B4Xh1bru+QlQUWLLyMc ZIcQ== X-Gm-Message-State: AAQBX9eoQ+fYJk2iuiLdOt8+Pu6dirAUIDjMDSRnG3xHQvYStucZOnIH NN8/hTwFQ5ocNioZSywL/dLeNwAjr7Y= X-Google-Smtp-Source: AKy350YLierP5/0sJYx1t2kL/hDADM7B1fEfuOuzuYjBv8fJK/yTaGYbLwXMAz5znXPyQxhNUYWLPQ== X-Received: by 2002:a17:906:d047:b0:94e:116:8581 with SMTP id bo7-20020a170906d04700b0094e01168581mr11179134ejb.5.1681823230632; Tue, 18 Apr 2023 06:07:10 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:10 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/8] io_uring/rsrc: inline io_rsrc_put_work() Date: Tue, 18 Apr 2023 14:06:38 +0100 Message-Id: <1b36dd46766ced39a9b160767babfa2fce07b8f8.1681822823.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_rsrc_put_work() is simple enough to be open coded into its only caller. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 19 ++++++------------- io_uring/rsrc.h | 1 - 2 files changed, 6 insertions(+), 14 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 127bd602131e..d1167b0643b7 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -140,8 +140,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, - struct io_rsrc_put *prsrc) +static void io_rsrc_put_work(struct io_rsrc_data *rsrc_data, + struct io_rsrc_put *prsrc) { struct io_ring_ctx *ctx = rsrc_data->ctx; @@ -150,16 +150,6 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data, rsrc_data->do_put(ctx, prsrc); } -static void __io_rsrc_put_work(struct io_rsrc_node *ref_node) -{ - struct io_rsrc_data *rsrc_data = ref_node->rsrc_data; - - if (likely(!ref_node->empty)) - io_rsrc_put_work_one(rsrc_data, &ref_node->item); - - io_rsrc_node_destroy(rsrc_data->ctx, ref_node); -} - void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { if (!io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache)) @@ -178,7 +168,10 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) if (node->refs) break; list_del(&node->node); - __io_rsrc_put_work(node); + + if (likely(!node->empty)) + io_rsrc_put_work(node->rsrc_data, &node->item); + io_rsrc_node_destroy(ctx, node); } if (list_empty(&ctx->rsrc_ref_list) && unlikely(ctx->rsrc_quiesce)) wake_up_all(&ctx->rsrc_quiesce_wq); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index f3fe455c6c71..232079363f6a 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -58,7 +58,6 @@ struct io_mapped_ubuf { void io_rsrc_put_tw(struct callback_head *cb); void io_rsrc_node_ref_zero(struct io_rsrc_node *node); -void io_rsrc_put_work(struct work_struct *work); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc); From patchwork Tue Apr 18 13:06:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73E27C6FD18 for ; Tue, 18 Apr 2023 13:07:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230070AbjDRNH0 (ORCPT ); Tue, 18 Apr 2023 09:07:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230489AbjDRNHX (ORCPT ); Tue, 18 Apr 2023 09:07:23 -0400 Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 131B37EE6 for ; Tue, 18 Apr 2023 06:07:13 -0700 (PDT) Received: by mail-ej1-x62c.google.com with SMTP id xi5so72986954ejb.13 for ; Tue, 18 Apr 2023 06:07:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823231; x=1684415231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1l66WSVLYWxzDyQULvNJbJOAPLJVOshaBZARMBezD5U=; b=lD0GwjUvXBLIt2El6dB/S7zZaGnIhc9kw7a65nlGdvi7CzjP1CJnMuy/SUHXRtB9tq BEe5j6r3Y9c5Ctn66Tt12WMa04n2fXjxlE/6CM4qWrRd/3+ioo3bXVcyekjn67GdQdHv gPf2s7SKuy7vDkRNsDvVo1At8VRqHGgy2vBdGsLYKtGRAJCYPM6w/C/UsVdAio2ZNZxQ N1KwavPkqwq8C3LVOOq0ZP+OvCAsr5z7msMtm4u3UxT2Ab16OLqYJTe2rtXWQNecHbXH oUApQGMae8QwemUa0OJ44kNHVj4bdgQ33iZVysAAzLrH0GbUs89aHJOqL5WW6sFE4xw5 6CaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823231; x=1684415231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1l66WSVLYWxzDyQULvNJbJOAPLJVOshaBZARMBezD5U=; b=MooxIwExzMIh7KVR89wgUzUhzAozyRaBeiyqnZxQyv2v0OMVn8rqWtHU8qlbm7RxpH D+2UzcJVDSCteM8WY80BbdeZVSIl226gc7BwjmxzWzqfKKVvsStc39Hgh/cR4oV0Cl7s SLevhq/amFEuOOk7s8Cvw9RDoHJX5rNsIX0FWdLXwNkjZKXCuhJikD7ycjx42m5O5lNc glDn/SQthYm3iuQo5I4be4swxaf1KqEp6ZAExRBuFm3pvjbKA3JZSIO55oqKk6H1Ozad WnSADyRm50GdiTfWRe8UQSrZixXHL3akjp1SIQ6kRl+iW8rS4czo17FQ7dJ8po2/pjNS 6G7w== X-Gm-Message-State: AAQBX9d5noXZMX8LFUE7/gHzfUyNptqazMUe4TMeIP3AhjbCRipz2eRB TrRLo6j1UFkJKgfiT0SxAHg0zhn68Gk= X-Google-Smtp-Source: AKy350Y1vdawcu4ZjOb/xHjTJywvhlcZ5gRN/6z1sy1aB8Q/6QaCp8ZuVbZMKU5gq6bdbvgZTmL2dg== X-Received: by 2002:a17:906:9153:b0:94e:e9c1:1932 with SMTP id y19-20020a170906915300b0094ee9c11932mr10655996ejw.43.1681823231188; Tue, 18 Apr 2023 06:07:11 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:10 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 6/8] io_uring/rsrc: pass node to io_rsrc_put_work() Date: Tue, 18 Apr 2023 14:06:39 +0100 Message-Id: <791e8edd28d78797240b74d34e99facbaad62f3b.1681822823.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of passing rsrc_data and a resource to io_rsrc_put_work() just forward node, that's all the function needs. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d1167b0643b7..9378691d49f5 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -140,14 +140,14 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot = NULL; } -static void io_rsrc_put_work(struct io_rsrc_data *rsrc_data, - struct io_rsrc_put *prsrc) +static void io_rsrc_put_work(struct io_rsrc_node *node) { - struct io_ring_ctx *ctx = rsrc_data->ctx; + struct io_rsrc_data *data = node->rsrc_data; + struct io_rsrc_put *prsrc = &node->item; if (prsrc->tag) - io_post_aux_cqe(ctx, prsrc->tag, 0, 0); - rsrc_data->do_put(ctx, prsrc); + io_post_aux_cqe(data->ctx, prsrc->tag, 0, 0); + data->do_put(data->ctx, prsrc); } void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) @@ -170,7 +170,7 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) list_del(&node->node); if (likely(!node->empty)) - io_rsrc_put_work(node->rsrc_data, &node->item); + io_rsrc_put_work(node); io_rsrc_node_destroy(ctx, node); } if (list_empty(&ctx->rsrc_ref_list) && unlikely(ctx->rsrc_quiesce)) From patchwork Tue Apr 18 13:06:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02280C77B78 for ; Tue, 18 Apr 2023 13:07:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230411AbjDRNH2 (ORCPT ); Tue, 18 Apr 2023 09:07:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231545AbjDRNHX (ORCPT ); Tue, 18 Apr 2023 09:07:23 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 424EB86BA for ; Tue, 18 Apr 2023 06:07:13 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id ud9so72809831ejc.7 for ; Tue, 18 Apr 2023 06:07:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823231; x=1684415231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KE9dqChOXKEnacrEqCkS5RH4b0y+phq2HjIj9BZdycA=; b=GdEQUdjhdWzRDdfsIqJGsdC0kNd8uz1doofnL4SNF/zNB9jtlaW8+7POmZAG9zsFr+ w5F964GvMSvPTECHAnwN4QEttp33ryuJ7k1JPJzOBlmMHoPWEJqmtDh0/JLyTGLuc3vb F1XxHB8nkvddIzKm9iSOFJPqnlDiQiEfVJKs4rf00zRByCXNPbyAlt50/QdT1hjKiiXp KU2xqINi6fEGt1owGsRfpazGxvcAt0g24WanGYpHHyCdF+dzE+/5TTxGjbmsrSQpmHV4 lvmjYtOJ/wMrnn8JtbgXkd8aoN2or5P3xQzfmT5P9u/dAU2Bg4lnLLJR+6QH/pNpA+wd UJLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823231; x=1684415231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KE9dqChOXKEnacrEqCkS5RH4b0y+phq2HjIj9BZdycA=; b=CAGBvUkxIkQ8SiyxKxaScjEV5jB9dMx7xTo3jz9Pgrr/4KH5U0pNOKLSZrJdMx09u5 FkbY5rqzv7VBxexpYyw4syXvdwQOUiuOTv7A7HW2dFk9FFmS1Ym4JyXxBXdFFPfzuCEw BCyGdK9FKQTEsiFrDNSjErTOyh3iFA/IBaRt5RgLEtBaxo6iqZsvonlK0Or/XWgpFGM9 45UaSYu5zcqBgfhIzgQEoUC7R3TH3BpRciPyfHUvHtUWE0iL8amCvF47k1QHFcpr8+sG 4oEyEvaTJVx8KiUzD78Z7UxDbbPq7xLJC15C+JHjvFGR5f9vYWix+1QL6E8XbpLE7wID dUfA== X-Gm-Message-State: AAQBX9eatd2dtwYVYEbQiOj69SRelXXbaBrSFCeQ1cyRBi6howeygNiS FZ0yXoXMDUPqOkvZDF+6TwAFCohMI8w= X-Google-Smtp-Source: AKy350abMe9TdjUVSYtx5BDjQJxi/RvgO7clXbhO9P4PDxx8CfUQZbzy4qNZgLnCFIUFiHJ6zERYNQ== X-Received: by 2002:a17:906:a0d0:b0:94f:1a11:df56 with SMTP id bh16-20020a170906a0d000b0094f1a11df56mr9457830ejb.35.1681823231629; Tue, 18 Apr 2023 06:07:11 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:11 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 7/8] io_uring/rsrc: devirtualise rsrc put callbacks Date: Tue, 18 Apr 2023 14:06:40 +0100 Message-Id: <02ca727bf8e5f7f820c2f404e95ae88c8f472930.1681822823.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We only have two rsrc types, buffers and files, replace virtual callbacks for putting resources down with a switch..case. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 25 +++++++++++++++++++------ io_uring/rsrc.h | 2 +- 2 files changed, 20 insertions(+), 7 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 9378691d49f5..62988b3aa927 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -23,6 +23,8 @@ struct io_rsrc_update { u32 offset; }; +static void io_rsrc_buf_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc); +static void io_rsrc_file_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc); static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, struct io_mapped_ubuf **pimu, struct page **last_hpage); @@ -147,7 +149,18 @@ static void io_rsrc_put_work(struct io_rsrc_node *node) if (prsrc->tag) io_post_aux_cqe(data->ctx, prsrc->tag, 0, 0); - data->do_put(data->ctx, prsrc); + + switch (data->rsrc_type) { + case IORING_RSRC_FILE: + io_rsrc_file_put(data->ctx, prsrc); + break; + case IORING_RSRC_BUFFER: + io_rsrc_buf_put(data->ctx, prsrc); + break; + default: + WARN_ON_ONCE(1); + break; + } } void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) @@ -297,8 +310,8 @@ static __cold void **io_alloc_page_table(size_t size) return table; } -__cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, - rsrc_put_fn *do_put, u64 __user *utags, +__cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, int type, + u64 __user *utags, unsigned nr, struct io_rsrc_data **pdata) { struct io_rsrc_data *data; @@ -316,7 +329,7 @@ __cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, data->nr = nr; data->ctx = ctx; - data->do_put = do_put; + data->rsrc_type = type; if (utags) { ret = -EFAULT; for (i = 0; i < nr; i++) { @@ -849,7 +862,7 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, return -EMFILE; if (nr_args > rlimit(RLIMIT_NOFILE)) return -EMFILE; - ret = io_rsrc_data_alloc(ctx, io_rsrc_file_put, tags, nr_args, + ret = io_rsrc_data_alloc(ctx, IORING_RSRC_FILE, tags, nr_args, &ctx->file_data); if (ret) return ret; @@ -1184,7 +1197,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, return -EBUSY; if (!nr_args || nr_args > IORING_MAX_REG_BUFFERS) return -EINVAL; - ret = io_rsrc_data_alloc(ctx, io_rsrc_buf_put, tags, nr_args, &data); + ret = io_rsrc_data_alloc(ctx, IORING_RSRC_BUFFER, tags, nr_args, &data); if (ret) return ret; ret = io_buffers_map_alloc(ctx, nr_args); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 232079363f6a..5d0733c4c08d 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -33,7 +33,7 @@ struct io_rsrc_data { u64 **tags; unsigned int nr; - rsrc_put_fn *do_put; + u16 rsrc_type; bool quiesce; }; From patchwork Tue Apr 18 13:06:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13215696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE690C77B76 for ; Tue, 18 Apr 2023 13:07:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231639AbjDRNH3 (ORCPT ); Tue, 18 Apr 2023 09:07:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231578AbjDRNHX (ORCPT ); Tue, 18 Apr 2023 09:07:23 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3014746A0 for ; Tue, 18 Apr 2023 06:07:14 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id dx24so28990194ejb.11 for ; Tue, 18 Apr 2023 06:07:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681823232; x=1684415232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4vBbSPgsInYAGzgfWYoncvwz5lmk4919w53t/DC+ol4=; b=TvwqcbxB4Qe4yt369SXa2wmA9bvXPD2lEf3N/Wc83Go6ih66MWzqtKxTLUvuUZ6z58 j82UYIClQqcBsxpaREYEMmNt4RcmlpmeZN1r6yzvshdeGPobkIFxSkRHXwS4DsiCFUK9 mECNf5QDJ7oba/Ts4SXZ2doinCBhFWjBVDPVvud7QdX8izI2hf41QlMmrgnRsSpJBBQJ r8cVh0/EhyZmnvHT34fS6TbGAzgOcu0cBgOQi1diOVdWnAgUN8bTiwbav6LJs7hWDm7e f6bBpm18h0CQcjbnIf06LxUYjK1Z05pMU87wwSoxvWQgNY3iYMo8DjBSYQ9D9hvFU4BO +V6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681823232; x=1684415232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4vBbSPgsInYAGzgfWYoncvwz5lmk4919w53t/DC+ol4=; b=dNIF4BZbgNL+bYx7HUT30A0+wSYy7WERNmcLy9Kx90i7pbQWBC/DwQiPe2ESlUrY42 2FutA8Lx+psbu/hkqSW4idRl1/2/wuJB5cEu0n07IAuArsV6c9CH6HUvF4pvyhNHefIf S7b2PL4WUKfeoISfNiQg14HLmjC69IHHKLMF+1BMP0BcpRDRucf3cvcZHu8xcoWMOilM xxssAlXIWY7znHfivVcAd8+Vca5nbkh+7igLKvKxmO7CbozpgUFVSuD8tM+Djbv5iVYF VNkhUUJ5/6IRlqen50mK8f1zr2nYoE3HhpQV8IQnPXfJQlTXwmCic7aNvsq7C4xT4hhu B6Ng== X-Gm-Message-State: AAQBX9fe+TETT3LDsQrQ1MZ7LEYtbyz1dW0HihoZm0pz3V/lCuGOyM1s 8SHrj8fep7oWm2TI13yXGzUq2hFwRrw= X-Google-Smtp-Source: AKy350Y0AIoFPe1Gf7XuOUZwKpXXhRupnRXoc3p3HYPuYRcM9asyIlqjaOZS+Y+Uwq5JgL2tqvgVhg== X-Received: by 2002:a17:906:11d2:b0:932:e9c7:c32 with SMTP id o18-20020a17090611d200b00932e9c70c32mr12582013eja.59.1681823232336; Tue, 18 Apr 2023 06:07:12 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:cfa6]) by smtp.gmail.com with ESMTPSA id o26-20020a1709061d5a00b0094e44899367sm7924919ejh.101.2023.04.18.06.07.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 06:07:12 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 8/8] io_uring/rsrc: disassociate nodes and rsrc_data Date: Tue, 18 Apr 2023 14:06:41 +0100 Message-Id: <4f259abe9cd4eea6a3b4ed83508635218acd3c3f.1681822823.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Make rsrc nodes independent from rsrd_data, for that we keep ctx and rsrc type in nodes. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 20 +++++++++----------- io_uring/rsrc.h | 3 ++- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 62988b3aa927..20dcc7668cb0 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -144,18 +144,17 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo static void io_rsrc_put_work(struct io_rsrc_node *node) { - struct io_rsrc_data *data = node->rsrc_data; struct io_rsrc_put *prsrc = &node->item; if (prsrc->tag) - io_post_aux_cqe(data->ctx, prsrc->tag, 0, 0); + io_post_aux_cqe(node->ctx, prsrc->tag, 0, 0); - switch (data->rsrc_type) { + switch (node->type) { case IORING_RSRC_FILE: - io_rsrc_file_put(data->ctx, prsrc); + io_rsrc_file_put(node->ctx, prsrc); break; case IORING_RSRC_BUFFER: - io_rsrc_buf_put(data->ctx, prsrc); + io_rsrc_buf_put(node->ctx, prsrc); break; default: WARN_ON_ONCE(1); @@ -170,9 +169,9 @@ void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) } void io_rsrc_node_ref_zero(struct io_rsrc_node *node) - __must_hold(&node->rsrc_data->ctx->uring_lock) + __must_hold(&node->ctx->uring_lock) { - struct io_ring_ctx *ctx = node->rsrc_data->ctx; + struct io_ring_ctx *ctx = node->ctx; while (!list_empty(&ctx->rsrc_ref_list)) { node = list_first_entry(&ctx->rsrc_ref_list, @@ -204,7 +203,7 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) return NULL; } - ref_node->rsrc_data = NULL; + ref_node->ctx = ctx; ref_node->empty = 0; ref_node->refs = 1; return ref_node; @@ -225,7 +224,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, if (!backup) return -ENOMEM; ctx->rsrc_node->empty = true; - ctx->rsrc_node->rsrc_data = data; + ctx->rsrc_node->type = -1; list_add_tail(&ctx->rsrc_node->node, &ctx->rsrc_ref_list); io_put_rsrc_node(ctx, ctx->rsrc_node); ctx->rsrc_node = backup; @@ -655,10 +654,9 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc) } node->item.rsrc = rsrc; + node->type = data->rsrc_type; node->item.tag = *tag_slot; *tag_slot = 0; - - node->rsrc_data = data; list_add_tail(&node->node, &ctx->rsrc_ref_list); io_put_rsrc_node(ctx, node); return 0; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 5d0733c4c08d..0a8a95e9b99e 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -40,10 +40,11 @@ struct io_rsrc_data { struct io_rsrc_node { union { struct io_cache_entry cache; - struct io_rsrc_data *rsrc_data; + struct io_ring_ctx *ctx; }; int refs; bool empty; + u16 type; struct list_head node; struct io_rsrc_put item; };