From patchwork Tue Apr 11 11:06:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207392 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40459C76196 for ; Tue, 11 Apr 2023 11:13:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229498AbjDKLNL (ORCPT ); Tue, 11 Apr 2023 07:13:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230040AbjDKLNB (ORCPT ); Tue, 11 Apr 2023 07:13:01 -0400 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 588E644B8 for ; Tue, 11 Apr 2023 04:12:49 -0700 (PDT) Received: by mail-ej1-x635.google.com with SMTP id jg21so19002222ejc.2 for ; Tue, 11 Apr 2023 04:12:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gejjt1nqo8/4I5cAJJVj1DToUQqE66kHLtUoIqIHgAE=; b=UTJn4il5mJD6WuNXor0U/3PDhjoTWwYzPeKMf7oIH1p47kFxCbIRUVKsXsDRSAYOub pQwjfng8SkzKwq6dbkkj2fR9zJDnj+PH/GoUvZE+cW/T/swdX2RXpWAKfKv5QwfVQ909 KSXmiWpC3JNJcud4IBcUjKLiXaTXZKAJ8wLrDj+B1pwbYxTgI275CoguxIfvLzO/diIu 3oAPotixw891I6NTIScR1GzUu0jQ9tRs92jH7KKRogGQjQWghtpGjbxhsnK2K8LqkAE8 VFq9GEkmqj5cZuf4/ET9aGbhRssT+uFEoSw/vzeBZlEfeRdUZ7Od05RWY0YkJBqKD6g8 derg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gejjt1nqo8/4I5cAJJVj1DToUQqE66kHLtUoIqIHgAE=; b=fHGxtKvVx4mm/rqpHsOh1xhqSFou98PoG+LOdRvweC5CfkatrCI6OuRawQ+A1WC5jF ND6EAy6pj0vpABj8DA1Ct1R2Et3CgOC7itBnoT9LiGCyo/L5dnnS8CREr0LkQyQtAptE k9RDVFVemxOM1Ql1/dgrp5cZMKxb2ScDuB8Op74EiqGnWtcmUqEmXq+TV90ZfKPQpOS0 OP6oEYREq3cgPJOLBm6w5Hz0wBuaQ36otRHE7nLVEZuNUNvJaLe8qhfv0yovFIpNqXh2 w0kj2xmxMaSff5DAb1ih2/bhSPbes/TOW/Lv1PHQEK7KpUqKi3LXdHeA2AForZKXWOC8 lIFQ== X-Gm-Message-State: AAQBX9e6hUKikUkAd+LvctfrN239ZNjt/b8qBH+9pDA4PoLZUAanw+Ai WvRYSyXz2XwrjnkgJRXFetYdn6kSzBo= X-Google-Smtp-Source: AKy350Zoa42IZ7/ivlb3mbHKAZaB4VacMsw0Rh9aY2LbS7sL+xPQHMEiaCpodbVWhH4PPuGV5/ypRg== X-Received: by 2002:a17:907:376:b0:94b:958c:8827 with SMTP id rs22-20020a170907037600b0094b958c8827mr4456657ejb.56.1681211567438; Tue, 11 Apr 2023 04:12:47 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 1/8] io_uring: shut io_prep_async_work warning Date: Tue, 11 Apr 2023 12:06:01 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_uring/io_uring.c:432 io_prep_async_work() error: we previously assumed 'req->file' could be null (see line 425). Even though it's a false positive as there will not be REQ_F_ISREG set without a file, let's add a simple check to make the kernel test robot happy. We don't care about performance here, but assumingly it'll be optimised out by the compiler. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9bbf58297a0e..b171c26d331d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -425,7 +425,7 @@ static void io_prep_async_work(struct io_kiocb *req) if (req->file && !io_req_ffs_set(req)) req->flags |= io_file_get_flags(req->file) << REQ_F_SUPPORT_NOWAIT_BIT; - if (req->flags & REQ_F_ISREG) { + if (req->file && (req->flags & REQ_F_ISREG)) { bool should_hash = def->hash_reg_file; /* don't serialize this request if the fs doesn't need it */ From patchwork Tue Apr 11 11:06:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3CC2C77B72 for ; Tue, 11 Apr 2023 11:13:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229766AbjDKLNM (ORCPT ); Tue, 11 Apr 2023 07:13:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229804AbjDKLNC (ORCPT ); Tue, 11 Apr 2023 07:13:02 -0400 Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16D034690 for ; Tue, 11 Apr 2023 04:12:50 -0700 (PDT) Received: by mail-ej1-x634.google.com with SMTP id qb20so18904693ejc.6 for ; Tue, 11 Apr 2023 04:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211568; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WXJHUznMJRl0elsxMAshdE0Ny+bVYXLkGUH4jQ9Xv5U=; b=Hu/eadbwd8PP23pDn6D1rR8KV4b88J7oMCe1zQuo8GhqAdxrRBSr+7oGP0eRXPFz2T Zcv/lB9ERDZvxiyKtqlDdlltN5n2CQSh7f9YhJT+ZJqf2m+h8o4RVANU8xVd8EiN5n8M cp/kIckjIuXLzCAwhPvN4Gb0n2hLy0OgFbMHTMvogv9wsphUZP9UVseaamHAqe+dnzBy HzIxcSHr0718f8GYshgyfVeysh6RCvC92HDXmOrzzWo6UHq9Dd7jUwixlYsmBdzDXMJ0 CCj/fOeSKqI3nS2HqGqmwonCIknvjnHXqPV/fkG2Ru609HdaAFeDRLJHZZdBbUsrn5rn nvkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211568; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WXJHUznMJRl0elsxMAshdE0Ny+bVYXLkGUH4jQ9Xv5U=; b=myw/t6rgB2bG75/GElvuLI3uwYnyQ7obVRNC8Cu9TcXUQJ3SJ7jIAbz9fjIBj4M27F LbwMz3EhyaXDxrwohw0ifhmE63NmKQV+w/mXQvBIGso3mfu/nM7/I3qUgbWhGRAPYYOo R4FStTcmMMri4uEVQKDZAjzmRqdPpbH8DxL16F5FCIDuiaqFTYULIUSL4HwJDYbmfP9c XpysjuaoKJsCAibbk4t2G11aES3n4W3EyHOD2tC5LJszLP9m5Zdw+izJ2ScizMeC7cbA eKvC5/Y68yTC9q0AuZr7V0rPqbf1Ub4bIB28AVCNPRoLQur4eyMjzzP9huuCeVRI8WM8 xCTA== X-Gm-Message-State: AAQBX9fChVfsFxblMyS7g0deVojad0mUKunTFQk8tLg5oxr8xZUwn378 b9RvwKUWulANN+O2KOAw9lmPwMnfJrI= X-Google-Smtp-Source: AKy350Zpj2tayHwdGr48hY0tDhV3uFYECLIsqGt3RDp4qoXHZrBsSDRXmbOa00hVCrSypfI9GCVVrQ== X-Received: by 2002:a17:906:76c8:b0:93b:5f2:36c with SMTP id q8-20020a17090676c800b0093b05f2036cmr9226298ejn.61.1681211568240; Tue, 11 Apr 2023 04:12:48 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/8] io_uring/kbuf: remove extra ->buf_ring null check Date: Tue, 11 Apr 2023 12:06:02 +0100 Message-Id: <9a632bbf749d9d911e605255652ce08d18e7d2c6.1681210788.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The kernel test robot complains about __io_remove_buffers(). io_uring/kbuf.c:221 __io_remove_buffers() warn: variable dereferenced before check 'bl->buf_ring' (see line 219) That check is not needed as ->buf_ring will always be set, so we can remove it and so silence the warning. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 79c25459e8de..0905c1761fba 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -218,14 +218,12 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, if (bl->is_mapped) { i = bl->buf_ring->tail - bl->head; if (bl->is_mmap) { - if (bl->buf_ring) { - struct page *page; - - page = virt_to_head_page(bl->buf_ring); - if (put_page_testzero(page)) - free_compound_page(page); - bl->buf_ring = NULL; - } + struct page *page; + + page = virt_to_head_page(bl->buf_ring); + if (put_page_testzero(page)) + free_compound_page(page); + bl->buf_ring = NULL; bl->is_mmap = 0; } else if (bl->buf_nr_pages) { int j; From patchwork Tue Apr 11 11:06:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41071C76196 for ; Tue, 11 Apr 2023 11:13:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229679AbjDKLNP (ORCPT ); Tue, 11 Apr 2023 07:13:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230058AbjDKLNE (ORCPT ); Tue, 11 Apr 2023 07:13:04 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B511346B1 for ; Tue, 11 Apr 2023 04:12:50 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id j17so9682892ejs.5 for ; Tue, 11 Apr 2023 04:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vk+L5iCXvmHnZt4vc6rvDG8PS1OW/kzTiKfkRlYy0ZY=; b=HlfwovvjkIP5woFiUF4XO8MoGOOVrUhZIRgJvrf7Z8sQmiKunOBVPMpZZyAgBO70mH 156U4jxmU0/MsEnrfGl6k/fhtCj2ZMqWMnoBc9Qq3U2PGVND6Xe/trWVXTBXkIwkrJxX WwJW0gtqa/dDJHzcxxXVnyNUfHqGlxQtqPpXTAmNMVRqp7uPC9YLVhfXErZ50qbYb9AK RxPxRxy5ucxsiP04TFKIsCqiFwCOMFw/4ULNFygGmL7CG8jDdyFJuAKxpE7ftoRYQTEX Iy6GEUs03HWRi2aowyAAeJ9vUsYF0izQFPZ6QXyM/Gfk6keanl8gYTR+qvz+6itGGVyq 0zgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vk+L5iCXvmHnZt4vc6rvDG8PS1OW/kzTiKfkRlYy0ZY=; b=36bYkL9uLmOON4B4OirsJJhe8j6+VzBDhCAIcGpJ4UJuxkxxobKWVgyaicEm6PZaKL 6NiEPEXTWVp0cv243jGAKZeGW9UeifVyOdzFvbskfYhEaTOWa0Hp6Zq31mITpsETkcTi yrsWnXoTZ2SkaaSm6fJArFRzY+Y7yDL95+M+JnWwuRxzMbQoIQjBZ7OcxAOVZMRLNirM LMumxWKNVMrsO/Qnj7iJ4PbA6C0VK3SMK4iRk6+kfu8sf1HlA55+LPTQb+wUAxdBmvQg bBV+3d4wVDYRPHyd2GIWGzrtTMtcoydSHeD+wBBc/QIfKCau+9sRQFwJhx46Zr5FAXkj /kig== X-Gm-Message-State: AAQBX9eP34mSgR0m++FMkdfrrN+mmWt9R62YilzotPuyTWjBmjBjA1ws dhw2w2EeAirAT6G5GxegvRh88Y8Rfx4= X-Google-Smtp-Source: AKy350YwjVmarvrgfIDX7q+nnMGZ+t7/ZlrEB8KXNFn89LagCt/dP1dK6wrH275mWwv3Fp4hgH8KUA== X-Received: by 2002:a17:906:d055:b0:94a:4d06:3de3 with SMTP id bo21-20020a170906d05500b0094a4d063de3mr2092795ejb.72.1681211568880; Tue, 11 Apr 2023 04:12:48 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 3/8] io_uring: add irq lockdep checks Date: Tue, 11 Apr 2023 12:06:03 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We don't post CQEs from the IRQ context, add a check catching that. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ef449e43d493..25515d69d205 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -94,6 +94,8 @@ bool io_match_task_safe(struct io_kiocb *head, struct task_struct *task, #define io_lockdep_assert_cq_locked(ctx) \ do { \ + lockdep_assert(in_task()); \ + \ if (ctx->flags & IORING_SETUP_IOPOLL) { \ lockdep_assert_held(&ctx->uring_lock); \ } else if (!ctx->task_complete) { \ From patchwork Tue Apr 11 11:06:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207394 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC077C77B74 for ; Tue, 11 Apr 2023 11:13:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230122AbjDKLNO (ORCPT ); Tue, 11 Apr 2023 07:13:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230056AbjDKLNE (ORCPT ); Tue, 11 Apr 2023 07:13:04 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 624D349CF for ; Tue, 11 Apr 2023 04:12:51 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id dm2so19441318ejc.8 for ; Tue, 11 Apr 2023 04:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yxiL8PVEsvMZ46+33d/I7dT556QvqnSwSgDcmN6Pf6s=; b=Vu0v7qdFDVMjkEdRqRZBJruZUKVCqwOaZpIVV0bhmEMBxBYzFeIWj+7/rSftv1SYL7 aKgSkng5UyJJYN1uEEKGlgEPm0fy4nkarLh3s/auE7UsDE6osOWlzILya6+aCyTw4+7b UB3L18aQVuZMvgNwCT98FA3U6waEmCD5T59dvecQtgVjLBbMI0AMCUZAvUmANv8sw4yM cdVFxeOIFPNWm0g/USx10ISxslfG18F5E0Nw2MaXRfpFxXQl+A2r4/4y3Q037ZbpTlUM QbIqhs7dlZ4r9pC924tJjqFmu+iVDskK7f3f1kSUHV/oSkqOEbT1Iz8dubCt7pHGaURc 7Jaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yxiL8PVEsvMZ46+33d/I7dT556QvqnSwSgDcmN6Pf6s=; b=cHVGmHj5CrCTC/dUTnx0To7ut0jemVBnVUBsvfG51UNS75Ps1lVaSqeHelBQt4Cs+g PDPLtduUbiC336fEaneZb5U0HeIa1WycUhZSu0i3uhTOmu/tmIC7Ytq9YGVs2Y4bOssy hgMZKkAJCKmcn+icmXSb2lIjFTGAbRxRKgHi1q8k1tgZx8JplbwiBWoppobNvSveGFMe H+uxz1o/W2clEl85FYLVrVnIHV2VduhwetWPchcvLZoWoi6egZ/0qlphGeeaP4fdTeMd FNNsI4A/II4GUVeD1O4uDhaQpUM8WH0aLLcvjgTTiVzSDYrUH/wvZVVqxxGwnzrTWNVl ArFw== X-Gm-Message-State: AAQBX9cLMFuOjFDXp3O//29sPlAk91MvvSfP0weaa8HIBBRoVlIss1W4 uEzwJ/g5j5drW9RmLC02pP0iqRPtj0M= X-Google-Smtp-Source: AKy350bKMhBq5/80+9DZD3MpscV99MvhRalK7nRfZxNbjqHIQ6r+zzGt+QF/IASdPkOTml83FGBJsw== X-Received: by 2002:a17:907:742:b0:930:60ba:d4b with SMTP id xc2-20020a170907074200b0093060ba0d4bmr13853807ejb.37.1681211569215; Tue, 11 Apr 2023 04:12:49 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 4/8] io_uring/rsrc: add lockdep checks Date: Tue, 11 Apr 2023 12:06:04 +0100 Message-Id: <961bbe6e433ec9bc0375127f23468b37b729df99.1681210788.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a lockdep chek to make sure that file and buffer updates hold ->uring_lock. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 603a783a0383..24e4e2109549 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -534,6 +534,8 @@ static int __io_register_rsrc_update(struct io_ring_ctx *ctx, unsigned type, __u32 tmp; int err; + lockdep_assert_held(&ctx->uring_lock); + if (check_add_overflow(up->offset, nr_args, &tmp)) return -EOVERFLOW; err = io_rsrc_node_switch_start(ctx); From patchwork Tue Apr 11 11:06:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B81C9C77B6F for ; Tue, 11 Apr 2023 11:13:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230085AbjDKLNR (ORCPT ); Tue, 11 Apr 2023 07:13:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230075AbjDKLNE (ORCPT ); Tue, 11 Apr 2023 07:13:04 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F78449E2 for ; Tue, 11 Apr 2023 04:12:51 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id gb34so19514933ejc.12 for ; Tue, 11 Apr 2023 04:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kXgohyCesDlyu8oDu5W3HIHKYO+1vXlCQPIxUEcV8ek=; b=kUBM1qRhCLGRVuhBh61FeTuTBI5/oPG7h/BPPrtY3vxArHb00u1FnSZQZxeVCJxXG1 2mpZ09/FlKiFEXcvWsaqlNF12tBvShNVTCGICuEbyfOKMzTpk6POjg/hK4iwDhB8jdNF nPcbnTpLOMUmqRvlVmKeIWCwLbOL5XqzH2dLS61i3dul7HEsIhh7n/znj0jUJRglS3c0 4VV5r/vRcVIN3ZvZYM33QI804Zqc3nl71qC3/v0CETlnf3u8LjNj7N8CQG84JhYC0Vve Ug67O0BdkF7+SuXvdXJ3Ld8JaVWB6+NV5DqmFhxS6gspXypXkIhCHEIKM8oW55MfuW+d pbYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211569; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kXgohyCesDlyu8oDu5W3HIHKYO+1vXlCQPIxUEcV8ek=; b=0c4KcIWltsqegyVc54vmqMRyz8Lt2SI52GoP3WFASxBVlgY93YW1f17LAuBbTdFwzz AVC+wwHV73B2RV9nbSlsi+wLOZIQWr/m2aFqbNurd2+zUzclFitAG5tHnEYoP5l7/tWp nl1qEd6DsOZVmORTHVnyu4hBu27K3v3mnQrg+gOgl6wygCXb2eqSgsdFt2o3ew1FKg7z VtNkKWvdBX1aRtt9zWGRI0JjeMhrdvaurJIAbRg8g57xyWhi8cAtxjJAy8mRrg/ih34k 3v/dnFuokh1mX3pdXm7FyAoQgwkuPJYAONzpfKKxfro9pFJnlFvUaplaS5exnqRVAMoD bmEw== X-Gm-Message-State: AAQBX9eXkdO65HxftnuFuvb3DwuSA67LnBhFJA/GQOiKYsVmMnLhYvvc 1as/z13JIprAY6ZYJdKdsUwH7sIzMvs= X-Google-Smtp-Source: AKy350a2LrCR42SrWYOUm6byTTDIxC6aHp6dnyxGYWZWTh1AY4y8OehqU47lYu/x8nEnpftQE7c3KQ== X-Received: by 2002:a17:906:3710:b0:94a:62e7:70e1 with SMTP id d16-20020a170906371000b0094a62e770e1mr6619267ejc.68.1681211569615; Tue, 11 Apr 2023 04:12:49 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 5/8] io_uring/rsrc: consolidate node caching Date: Tue, 11 Apr 2023 12:06:05 +0100 Message-Id: <6d5410e51ccd29be7a716be045b51d6b371baef6.1681210788.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We store one pre-allocated rsrc node in ->rsrc_backup_node, merge it with ->rsrc_node_cache. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 - io_uring/alloc_cache.h | 5 +++++ io_uring/io_uring.c | 2 -- io_uring/rsrc.c | 20 +++++++++++--------- 4 files changed, 16 insertions(+), 12 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index fa621a508a01..40cab420b1bd 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -326,7 +326,6 @@ struct io_ring_ctx { struct io_restriction restrictions; /* slow path rsrc auxilary data, used by update/register */ - struct io_rsrc_node *rsrc_backup_node; struct io_mapped_ubuf *dummy_ubuf; struct io_rsrc_data *file_data; struct io_rsrc_data *buf_data; diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 851a527afb5e..241245cb54a6 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -23,6 +23,11 @@ static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, return false; } +static inline bool io_alloc_cache_empty(struct io_alloc_cache *cache) +{ + return !cache->list.next; +} + static inline struct io_cache_entry *io_alloc_cache_get(struct io_alloc_cache *cache) { if (cache->list.next) { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b171c26d331d..075bae8a2bb1 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2852,8 +2852,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) /* there are no registered resources left, nobody uses it */ if (ctx->rsrc_node) io_rsrc_node_destroy(ctx, ctx->rsrc_node); - if (ctx->rsrc_backup_node) - io_rsrc_node_destroy(ctx, ctx->rsrc_backup_node); WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 24e4e2109549..73f9e10d9bf0 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -230,7 +230,7 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, struct io_rsrc_data *data_to_kill) __must_hold(&ctx->uring_lock) { - WARN_ON_ONCE(!ctx->rsrc_backup_node); + WARN_ON_ONCE(io_alloc_cache_empty(&ctx->rsrc_node_cache)); WARN_ON_ONCE(data_to_kill && !ctx->rsrc_node); if (data_to_kill) { @@ -245,18 +245,20 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, ctx->rsrc_node = NULL; } - if (!ctx->rsrc_node) { - ctx->rsrc_node = ctx->rsrc_backup_node; - ctx->rsrc_backup_node = NULL; - } + if (!ctx->rsrc_node) + ctx->rsrc_node = io_rsrc_node_alloc(ctx); } int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) { - if (ctx->rsrc_backup_node) - return 0; - ctx->rsrc_backup_node = io_rsrc_node_alloc(ctx); - return ctx->rsrc_backup_node ? 0 : -ENOMEM; + if (io_alloc_cache_empty(&ctx->rsrc_node_cache)) { + struct io_rsrc_node *node = kzalloc(sizeof(*node), GFP_KERNEL); + + if (!node) + return -ENOMEM; + io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache); + } + return 0; } __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, From patchwork Tue Apr 11 11:06:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4320C77B72 for ; Tue, 11 Apr 2023 11:13:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230136AbjDKLNQ (ORCPT ); Tue, 11 Apr 2023 07:13:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230082AbjDKLNE (ORCPT ); Tue, 11 Apr 2023 07:13:04 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CAA3B49E9 for ; Tue, 11 Apr 2023 04:12:51 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id q23so9873161ejz.3 for ; Tue, 11 Apr 2023 04:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b8pcSBrPZA4+KOGPAVzPd/1ZLXVHdZRqUya+mQQiFKg=; b=TQd3SyFcmcVoIk4NmthibvfpDp699QsT+7U2cQ7naTnXbjGiM/Pkkjqq+FfGPdP2F0 UpfL2IgmbPlU8m+o8u3XulPTHmSUtMYCWe0Sc7Dq8hZFjUZQrCWILv9EV34BjNgKcBFO ezcYXR5QNEE+DJtajZ8Q2KnG1BYUReS4H0Xsa1FfETLhJyWtE0QFWnOATZzuxhqWvDNG vKCxdfR8z7V2uZHfm+IsZ5bMt/wsrJJWla60KtGavLeryxp1y8IA7OGVsgrlDSuZGnMz OBd5CIwZC/8gNb+fmxxK3OqDSbue4HxhTtx7tZ8ekdVzLHa0Q/mSG2xflo2D+/u1cjp/ Gl/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b8pcSBrPZA4+KOGPAVzPd/1ZLXVHdZRqUya+mQQiFKg=; b=wukHAIS7GCkmNZxJ1L0XC+xKLpxy9hgg3VechZZ+rQaX9POKcrQ66nN+wSdHVWQIAp 9/ajYFCQg0/KidgoDSAADqaUv3v6oUwOl4VHYpdDX6o79wLd9pRoJQUWUxd59upLOr86 CyNi1ZcgEYTwgXX2vI7t8wo7/1MPHlzkxxcId2vGZ9a/CVyw/Dpp+nn+hvkR0GESqmPZ p8/RMtFUtBNOulya7hirumqb2mdy9Ddh1DiDRvmZcuIW8gugImP/kQ3lFhu9SBuB/35X bXswM3QFPQx4I0T2AdbQu9P5e369V6cQQRJyG9iyq1jlUgIOHh1nPaXhkTiP4YWjQnnk odKA== X-Gm-Message-State: AAQBX9cQbtRSuvX5TERhAlR/2uEvCDnNihXjz6DXBjiYV9OHWFeDb5Km U89KEImFAyx6y45YS0sV+CN7Ax9Nhyc= X-Google-Smtp-Source: AKy350aWfjy48ALIIlUCN+3H1wujd6rxoVtKmhAGt+u6PTWSRXV3yy2GyCTy8xSOn6mVtEZkaU1rqA== X-Received: by 2002:a17:906:c206:b0:948:112e:c49a with SMTP id d6-20020a170906c20600b00948112ec49amr11484625ejz.24.1681211570020; Tue, 11 Apr 2023 04:12:50 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 6/8] io_uring/rsrc: zero node's rsrc data on alloc Date: Tue, 11 Apr 2023 12:06:06 +0100 Message-Id: <09bd03cedc8da8a7974c5e6e4bf0489fd16593ab.1681210788.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org struct io_rsrc_node::rsrc_data field is initialised on rsrc removal and shouldn't be used before that, still let's play safe and zero the field on alloc. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 1 + 1 file changed, 1 insertion(+) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 73f9e10d9bf0..329cc3851dfd 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -218,6 +218,7 @@ static struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) return NULL; } + ref_node->rsrc_data = NULL; ref_node->refs = 1; INIT_LIST_HEAD(&ref_node->node); INIT_LIST_HEAD(&ref_node->item_list); From patchwork Tue Apr 11 11:06:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A348FC76196 for ; Tue, 11 Apr 2023 11:13:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230151AbjDKLNS (ORCPT ); Tue, 11 Apr 2023 07:13:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230094AbjDKLNK (ORCPT ); Tue, 11 Apr 2023 07:13:10 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DC7F4697 for ; Tue, 11 Apr 2023 04:12:52 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id sg7so30750513ejc.9 for ; Tue, 11 Apr 2023 04:12:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v2nyzNZvTXnEQkp2ImwUNfiYpTnsqjdM4NaswRR4XZc=; b=MYXKmDT72HV8Zlx5fpvcLGvvDVyka6XhWvPUBb/LYBDBiBA3VUXEnCQzl1id0YXh0k 1ed1Y6//sCrdZy8wUFLy8RAx0zUVKR5XDyHSIJ5PeTQxb42/qEDnLw1sx8NfIc1SXUZ8 HFoSMpiLVsG81nLfEXW9iMTSPPAWWfzgPyfZAHqwt2QRjz7PdM6FIlLOuGXaMA5Mp4oc hm9ERTvFx5uwyDHIvIgDLYP13EdqJJzqVoxJTy2NvEreX3z3FXkIljG25rlV1rgiX+T5 +uChAv55KOvySvJTeRQEgKlaPHRWqRRkmZvodOXmMH+/efKAO5kMqGFzpRwr9M6W74C2 YJyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v2nyzNZvTXnEQkp2ImwUNfiYpTnsqjdM4NaswRR4XZc=; b=jI4Ec5PqdXcwPjU55pDlEv9ZEq0HFeM0FmdTXYa6sSNtr2P+FZ9ieldBhq22ZWobCn dXDf/o1xvSc9X1psE0X8d68hcoVkVKxwdJ7MKquDtRpxoY7TeTQl8bqt9dszrM+MMgS1 JWvxAlwbU6ASJvQJHc4wftkIMEdMQCflakCpdKoJ/d14d/cF/3H0k5BWOwwRZIDV2xB0 RQzVdjvzRSXYtbSIjnGs08YjuUFP+IAfjxZ7TZfLnFTqQTvt89Ciochw61ApEizaO9sK k7mwsaja3prXfqTerv6dvSzkSHKBekooW2gFqCl1jyDDQImFFI0R9y2x8PJZ65UiGPH0 ZDnA== X-Gm-Message-State: AAQBX9e080rPQuwx6NP3BBD651Yb+1b2H9z+pGBMCkN9qHx4Rl02w5dC u1uOA/JD3xqA4oZ90EORoOqNFQTG9bg= X-Google-Smtp-Source: AKy350ZjmHBG/7FeCp1miwYTojDewuMJ/KvcpmG5mBagmeLyBpyQdn1tabn9ZHiLAXozZ4MIN4hgGw== X-Received: by 2002:a17:907:a40a:b0:93b:d1ee:5f41 with SMTP id sg10-20020a170907a40a00b0093bd1ee5f41mr7580025ejc.31.1681211570484; Tue, 11 Apr 2023 04:12:50 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 7/8] io_uring/rsrc: refactor io_rsrc_node_switch Date: Tue, 11 Apr 2023 12:06:07 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We use io_rsrc_node_switch() coupled with io_rsrc_node_switch_start() for a bunch of cases including initialising ctx->rsrc_node, i.e. by passing NULL instead of rsrc_data. Leave it to only deal with actual node changing. For that, first remove it from io_uring_create() and add a function allocating the first node. Then also remove all calls to io_rsrc_node_switch() from files/buffers register as we already have a node installed and it does essentially nothing. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 5 ++--- io_uring/rsrc.c | 36 +++++++++++------------------------- io_uring/rsrc.h | 7 +++++++ 3 files changed, 20 insertions(+), 28 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 075bae8a2bb1..9083a8466ebf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3881,11 +3881,10 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, ret = io_sq_offload_create(ctx, p); if (ret) goto err; - /* always set a rsrc node */ - ret = io_rsrc_node_switch_start(ctx); + + ret = io_rsrc_init(ctx); if (ret) goto err; - io_rsrc_node_switch(ctx, NULL); memset(&p->sq_off, 0, sizeof(p->sq_off)); p->sq_off.head = offsetof(struct io_rings, sq.head); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 329cc3851dfd..f2c660ffea74 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -204,7 +204,7 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node) } } -static struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) +struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) { struct io_rsrc_node *ref_node; struct io_cache_entry *entry; @@ -231,23 +231,18 @@ void io_rsrc_node_switch(struct io_ring_ctx *ctx, struct io_rsrc_data *data_to_kill) __must_hold(&ctx->uring_lock) { - WARN_ON_ONCE(io_alloc_cache_empty(&ctx->rsrc_node_cache)); - WARN_ON_ONCE(data_to_kill && !ctx->rsrc_node); + struct io_rsrc_node *node = ctx->rsrc_node; + struct io_rsrc_node *backup = io_rsrc_node_alloc(ctx); - if (data_to_kill) { - struct io_rsrc_node *rsrc_node = ctx->rsrc_node; - - rsrc_node->rsrc_data = data_to_kill; - list_add_tail(&rsrc_node->node, &ctx->rsrc_ref_list); - - data_to_kill->refs++; - /* put master ref */ - io_put_rsrc_node(ctx, rsrc_node); - ctx->rsrc_node = NULL; - } + if (WARN_ON_ONCE(!backup)) + return; - if (!ctx->rsrc_node) - ctx->rsrc_node = io_rsrc_node_alloc(ctx); + data_to_kill->refs++; + node->rsrc_data = data_to_kill; + list_add_tail(&node->node, &ctx->rsrc_ref_list); + /* put master ref */ + io_put_rsrc_node(ctx, node); + ctx->rsrc_node = backup; } int io_rsrc_node_switch_start(struct io_ring_ctx *ctx) @@ -921,9 +916,6 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, return -EMFILE; if (nr_args > rlimit(RLIMIT_NOFILE)) return -EMFILE; - ret = io_rsrc_node_switch_start(ctx); - if (ret) - return ret; ret = io_rsrc_data_alloc(ctx, io_rsrc_file_put, tags, nr_args, &ctx->file_data); if (ret) @@ -978,7 +970,6 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, /* default it to the whole table */ io_file_table_set_alloc_range(ctx, 0, ctx->nr_user_files); - io_rsrc_node_switch(ctx, NULL); return 0; fail: __io_sqe_files_unregister(ctx); @@ -1260,9 +1251,6 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, return -EBUSY; if (!nr_args || nr_args > IORING_MAX_REG_BUFFERS) return -EINVAL; - ret = io_rsrc_node_switch_start(ctx); - if (ret) - return ret; ret = io_rsrc_data_alloc(ctx, io_rsrc_buf_put, tags, nr_args, &data); if (ret) return ret; @@ -1300,8 +1288,6 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, ctx->buf_data = data; if (ret) __io_sqe_buffers_unregister(ctx); - else - io_rsrc_node_switch(ctx, NULL); return ret; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 8729f2fee256..17dfe180208f 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -74,6 +74,7 @@ void io_rsrc_put_work(struct work_struct *work); void io_wait_rsrc_data(struct io_rsrc_data *data); void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); int io_rsrc_node_switch_start(struct io_ring_ctx *ctx); +struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, struct io_rsrc_node *node, void *rsrc); void io_rsrc_node_switch(struct io_ring_ctx *ctx, @@ -164,6 +165,12 @@ static inline u64 *io_get_tag_slot(struct io_rsrc_data *data, unsigned int idx) return &data->tags[table_idx][off]; } +static inline int io_rsrc_init(struct io_ring_ctx *ctx) +{ + ctx->rsrc_node = io_rsrc_node_alloc(ctx); + return ctx->rsrc_node ? 0 : -ENOMEM; +} + int io_files_update(struct io_kiocb *req, unsigned int issue_flags); int io_files_update_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); From patchwork Tue Apr 11 11:06:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13207399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FD78C7619A for ; Tue, 11 Apr 2023 11:13:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230101AbjDKLNT (ORCPT ); Tue, 11 Apr 2023 07:13:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230095AbjDKLNK (ORCPT ); Tue, 11 Apr 2023 07:13:10 -0400 Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2B9C4C25 for ; Tue, 11 Apr 2023 04:12:52 -0700 (PDT) Received: by mail-ej1-x634.google.com with SMTP id ga37so19679055ejc.0 for ; Tue, 11 Apr 2023 04:12:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681211571; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dkTrgzgZfs7fwzj4aO6pWf6R6g23XUkgLev3HUA9yrU=; b=OkfWp+tsaH+944pWjpg+uC7cd+5oaShOnKYZX8Cw5QXix7kjB4XWtsRgkipeE4ebwP QVgvnj1ObauX6UCXYrT1Dgieh+LTuhrgrKA06r10UsfMmm9fPj1wmpkxom4MA7XFwKWJ VCrfAQe0r1aww8vaxbwuGKjA0QatPY5ojtn+zx/+7XkLfnp5QBpcOzCge1753spy59BM T6agBFGWMT+mRCFzKXl3IvJoCnQ979zJ+UUxhUFotgApfww33fqhF2D9bzzCDiwjAi62 BB8AjpFUT/nuLzshWa2UIBsNT8LsY2YxdsjbG2rJ1WA11jWJLtSa6ptf6Q+NOECiwByM MJQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681211571; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dkTrgzgZfs7fwzj4aO6pWf6R6g23XUkgLev3HUA9yrU=; b=nRY/SAsTMBrzss9Xil3tLyznQavcOu5YujugOkU2irakMgy6XykcKOlSIVc8lVjemv 0UsvwWrMbl4Cc/WQTn7JGydclzJJ7PUOiPwLp6kPeIyttFriIAJys8K3notwlW9Ynvmz ZZr4rFv1bIFhHa9JVgNfvj2SK+2ncWhbshva9HDPUxXYmxLGO2aLvLtqht0PNeOXg0Yk ea3v28ASF+RXvT/Cf6DXXfVunqAWjSBJTcO6eE6rTi18NyZ0oun+F4hRIsfkfgI3wWTA bI17obsU81TgsJSl9hU8MBeVFh0dn199fRSPphU9DnRGzvylc0zNtWTxx+CH7ZKLcJVU jPxw== X-Gm-Message-State: AAQBX9dAliy9jQL5+Xz0TKFb77jRlkiKyovSkmE7zyNZpilDr1f1mHXl RfG1rx24yaHC9MgbU0w6yhZSsfViuNs= X-Google-Smtp-Source: AKy350YTbXQsFMtAR71uD37DjlL3tuojweNC+1NlSjn4lQ3IAzO7yTLHYcZtfmAA6qPqkNhL+3tpVw== X-Received: by 2002:a17:907:9714:b0:932:853c:c958 with SMTP id jg20-20020a170907971400b00932853cc958mr11899154ejc.25.1681211570877; Tue, 11 Apr 2023 04:12:50 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:ddc3]) by smtp.gmail.com with ESMTPSA id ww7-20020a170907084700b00947a40ded80sm6006787ejb.104.2023.04.11.04.12.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Apr 2023 04:12:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 8/8] io_uring/rsrc: extract SCM file put helper Date: Tue, 11 Apr 2023 12:06:08 +0100 Message-Id: <58cc7bffc2ee96bec8c2b89274a51febcbfa5556.1681210788.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org SCM file accounting is a slow path and is only used for UNIX files. Extract a helper out of io_rsrc_file_put() that does the SCM unaccounting. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index f2c660ffea74..11058e20bdcc 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -832,20 +832,14 @@ int __io_scm_file_account(struct io_ring_ctx *ctx, struct file *file) return 0; } -static void io_rsrc_file_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc) +static __cold void io_rsrc_file_scm_put(struct io_ring_ctx *ctx, struct file *file) { - struct file *file = prsrc->file; #if defined(CONFIG_UNIX) struct sock *sock = ctx->ring_sock->sk; struct sk_buff_head list, *head = &sock->sk_receive_queue; struct sk_buff *skb; int i; - if (!io_file_need_scm(file)) { - fput(file); - return; - } - __skb_queue_head_init(&list); /* @@ -895,11 +889,19 @@ static void io_rsrc_file_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc) __skb_queue_tail(head, skb); spin_unlock_irq(&head->lock); } -#else - fput(file); #endif } +static void io_rsrc_file_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc) +{ + struct file *file = prsrc->file; + + if (likely(!io_file_need_scm(file))) + fput(file); + else + io_rsrc_file_scm_put(ctx, file); +} + int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, unsigned nr_args, u64 __user *tags) {