From patchwork Wed Feb 5 11:36:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960879 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA4EB22A7EF for ; Wed, 5 Feb 2025 11:36:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755412; cv=none; b=ay3lY46Yg/HaEpxf8kh5CdcY+En5HjZFs4rb+ihFLiK0CF/J0EcOghMu6c0N/iEFm5IA4BEjxEAkQ0FF857pqOCkgzzzlpnUHEU3WzFbi9uSRT4i5ZyuS5DOykL/73kL0dgPkQRwv1svcPWcRuPEBw4ymKhRfcUYR3KVsb+zjKM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755412; c=relaxed/simple; bh=2ePhBV5dU0vBGOGo6Yf2+eDlMzr7q4dCMoUf81Lbt0c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rSzisF0ojcguURer1YTo1u5/Gr3EI71p7DJJMfvsrikibgzt/x2iIGNUWDzvLvGXlDfZdSxiswexGSDNIY4+dU/PPZYDBL9otDJ5AR5NvmsUdkZiL1Uwi9iH7vipNhpeIXWUy4QSNRKCX5kcH3se0Yh2o5mx0MvDuphglGo5syg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=j9gj5YfG; arc=none smtp.client-ip=209.85.128.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="j9gj5YfG" Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-436a39e4891so45782625e9.1 for ; Wed, 05 Feb 2025 03:36:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755408; x=1739360208; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RVgW6hEFQQ7YI5zAH9fN6CiNeQYki6D9SMOXqmnbQ1Q=; b=j9gj5YfGlo/teMn1Yy3P12tZ6m0Ne7RKfOZfnK780xUHG/BvGE/9WQK+uppru+orwn Y/qr/UAJp9kvOcpCpsVbhCXcSzCU2stuh86o135dm5PrakrimnjzTFxT40FcXaf89vU0 2tsJZ2VdDtXm0hpWCgj1tl+mU1Yj1yd0kP8k9QYJ8NlbaOFp+fCNaRe8O7BkYDQaAjd2 2rSoRe/D/vKEtTcr1aWD8Sm2MAMnJHV6t/9+RymxjglArOJ5A3zQa/sa+T013Q+u3/iM SRxU++PoXSE347YgFCEeNHIpD/48w7oQIzH2N4/Z6wkeBJSYCbPlUal8UYCr9o7pjEB/ kj3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755408; x=1739360208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RVgW6hEFQQ7YI5zAH9fN6CiNeQYki6D9SMOXqmnbQ1Q=; b=iSetgdAY9BJM45jTQenO7V/UnNjCueVzUBEp+eVCkdxUo5ty7I2bujpyi7VwsCs/uw mBT4QOyxHIG8+yTXRbgKyC/R6kNQo9jssXZI9sBZgwAp6J2saliiNqCplAiE2Zj2hRz1 mB6bjeyY9JVT4aDVUn07QOLAMOfbmInnVch2aObQtiZWj50PIEx24dtdX4gspkdXfCkF /SCp+otpYN9gOg8PWPNwIOkr8pgpvkg/MaWm550FIN3aT5vfPtSHzLSRHSWANU02qrcj WmFbzpR+ypDb2hNJQGRCBTwbe0dcadfv8watbKB9EzYozmmnO2Zhjf94a1Xo+YUXrN/m UGqw== X-Gm-Message-State: AOJu0Yz8281+8JIX/uu68JGIBTunftaX/V6xPO80GjtxfXhg5fB84shC ESUtZl9DnjDoieYQutK0kZKkgoKgwDV+qeXGKw2M/LPRjwFsVco3f0ZQjQ== X-Gm-Gg: ASbGncucmNXDUXdDYQpks/mPHZiREVJKOminPLsx/ksYiGNa6AKeUejN59XbPwFKg5C DW3oH9M9gB0w5qS/mbqZFpQIPFXsnEc8m8hiDxxm9SKe1cLYlsX7W0H7QI9QPt3Fguq8HuDDej0 yr83sHjT34f8hJ/q0CR3A6TFiCGzdUT6WtV0vhIuOLh4ubU+rBRxZN4IvAxALR5mkRWtqWoyjTj GOclNOBrKQLZCRhPTMOhm4xSHdG5kIKlLI9bBNt84ydjmSDOJfvlN4CUrCFqgMuVHp6Bdk1l0/l nTZvPNFDTR5vidj51YAEERpdaPs= X-Google-Smtp-Source: AGHT+IFXLbRIO3pJ86xQMpxZIiVon+0LI5KjFWQVFFWQkZiPc3Bb4a0idfHEBsMAhxVMuKVLEC62/w== X-Received: by 2002:a5d:5f56:0:b0:38a:88bc:ace1 with SMTP id ffacd0b85a97d-38db4929f66mr1767154f8f.34.1738755407121; Wed, 05 Feb 2025 03:36:47 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:46 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 1/8] io_uring/kbuf: remove legacy kbuf bulk allocation Date: Wed, 5 Feb 2025 11:36:42 +0000 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Legacy provided buffers are slow and discouraged in favour of the ring variant. Remove the bulk allocation to keep it simpler as we don't care about performance. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 30 +++++------------------------- 1 file changed, 5 insertions(+), 25 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 04bf493eecae0..0bed40f6fe3a5 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -494,12 +494,9 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe return 0; } -#define IO_BUFFER_ALLOC_BATCH 64 - static int io_refill_buffer_cache(struct io_ring_ctx *ctx) { - struct io_buffer *bufs[IO_BUFFER_ALLOC_BATCH]; - int allocated; + struct io_buffer *buf; /* * Completions that don't happen inline (eg not under uring_lock) will @@ -517,27 +514,10 @@ static int io_refill_buffer_cache(struct io_ring_ctx *ctx) spin_unlock(&ctx->completion_lock); } - /* - * No free buffers and no completion entries either. Allocate a new - * batch of buffer entries and add those to our freelist. - */ - - allocated = kmem_cache_alloc_bulk(io_buf_cachep, GFP_KERNEL_ACCOUNT, - ARRAY_SIZE(bufs), (void **) bufs); - if (unlikely(!allocated)) { - /* - * Bulk alloc is all-or-nothing. If we fail to get a batch, - * retry single alloc to be on the safe side. - */ - bufs[0] = kmem_cache_alloc(io_buf_cachep, GFP_KERNEL); - if (!bufs[0]) - return -ENOMEM; - allocated = 1; - } - - while (allocated) - list_add_tail(&bufs[--allocated]->list, &ctx->io_buffers_cache); - + buf = kmem_cache_alloc(io_buf_cachep, GFP_KERNEL); + if (!buf) + return -ENOMEM; + list_add_tail(&buf->list, &ctx->io_buffers_cache); return 0; } From patchwork Wed Feb 5 11:36:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960880 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21AB222A4F2 for ; Wed, 5 Feb 2025 11:36:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755412; cv=none; b=SzxxdHbSbU7pagVtcW/Y4FLfF7xQnvpBdt0aszSLlhkagnAVXY36lUQQvFAeCwp95k9hWSTZX1iaaZZy6+HQQ+WYzohrk2ii74tBUlUFoozbRYF71Y+xkGCrV9jbzRYDQYqm9caUrhgp2i2fOhdbLWG/dJUrTTlKQUKFToKZqzI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755412; c=relaxed/simple; bh=R8B2bAOqbdY9BKKM8Zmeep2GB7REgWnCUGhbUhJNYQ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Tnp4ot0Rm4thj59A0+zilaCLPYllpq/epkF8i3e4tfR8mCadhDZBTtgoc8EbwIXypOr2l2r+ad+H4y+Wl84sxfhy2EcFeLEnmk9nmF6qJ0Abx+Ur4HcR+KJjVd3tFc8JUkSuyo8XDMIWemy8Mx/3DKX0LOE4Aetg3fErlgB5T9E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ze3tnAA0; arc=none smtp.client-ip=209.85.221.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ze3tnAA0" Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-38db0146117so811533f8f.3 for ; Wed, 05 Feb 2025 03:36:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755409; x=1739360209; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KdQMMM+vu69NjwQNUGDxHQUFtinRBBO5UjK5C9xPV8E=; b=Ze3tnAA0K00Llx6r7ILGHobBY55jUNF9HefIciZ0SaB7E8TsSXvgkirXNa28QcGaBt VL/V910flj/04kXfIbvfcrp3/5FKPF3mJouq0EcNTj8+bKQzSzZMeFLaQF1m7kUUO5Qf QXhAml1t/HwbdEjtJILwDOFcWcVhjxgTosGDllOmw/W2tmj7qw/zQ4Zy15ZUxlqotqaZ I0rmFCbuNGYCP187ff6nbwlWjkp+UJpsXJ1IOIjP3swwSVSrX8XbzcAQLRqCSrTKJ5qH 0E6aTBrTELTEyTXyvRLJdnuVTA+KRdf7XYP40uhJAC07YRnN4w8aQ9jx6S+wJkI0kGwP s2tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755409; x=1739360209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KdQMMM+vu69NjwQNUGDxHQUFtinRBBO5UjK5C9xPV8E=; b=bvHg+EH1buSxeZLpXJq/ha0HsdcSw6XCeFY4hbGWEbBYOdmDr+nzqG8QojXi++gZN+ 8HavFgVYYB8UVQAiwddiC5+30WlOG34NHsF2awneUbmikKx1+ShVYsr2D6vKWNOGOHzd oX1vQXVeRi7t13Pn3ZdwmotTq0N9XPI4ztpqAb/wQEMraeYez3/rvVntquAq1XtqJ9v3 HAa+xgUaPbxkTjVP3FMjy7Nf3p+DD9UxXHfnICYOU1W6zrCECo9c7wpPAr0bI890i4LZ B8mE0pY9SWIrMNXC8OPfPBzynx0wEU3XqiX97oLWqOnuiNe4LS1dgRzdhz+uQiNYqm2y BCZg== X-Gm-Message-State: AOJu0Yz116BByrwRmx1I3KrI5ZWQKKtrelLm3BK0n4sCNfbbfdGa893Y DatgDtxxDjH4nHEEr2p56IPHlLlrhfHioc7VsK5zPQenPti9Ur6n2kPdJQ== X-Gm-Gg: ASbGnct155pQDc8iSG3tPtBx5vc01uJjgfZPObtwzRCkiG2Jlm9xo5lnImvjyMkCTOP VrZWwfOvO7UqnfIbd97Qzucl3+bXcZCblf0gQoMGFHjoR0WE4Dp9e7sPcgIOu/LMZv1xYj4wooP ZTzQkf2u7CQsxZXGbRaDsgRs2r7VqdJXdbO/v1tQFMNeD10emdaFGpkm4TSnCSKdC8HJwWNbYWX mzs7bOJt2lKVHRT4F1EiKZ3aIJrhp4GApGIM6hXNVxp27VMG6LhWppjEATM6w1S/yZA7G+LLeVp qYhNEN6jGgV9HqruazJjAAa7dXg= X-Google-Smtp-Source: AGHT+IFaSD1ST6tOUeoAkdO+dC3bmWlV/NtvNiZ5zKonnHn6PHHWg8YZ1RkBkpDtS/1ylSSOeq+xQw== X-Received: by 2002:a5d:64ec:0:b0:38d:adcd:a05e with SMTP id ffacd0b85a97d-38db48fda85mr1723558f8f.39.1738755408357; Wed, 05 Feb 2025 03:36:48 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:47 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 2/8] io_uring/kbuf: remove legacy kbuf kmem cache Date: Wed, 5 Feb 2025 11:36:43 +0000 Message-ID: <8195c207d8524d94e972c0c82de99282289f7f5c.1738724373.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Remove the kmem cache used by legacy provided buffers. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 -- io_uring/io_uring.h | 1 - io_uring/kbuf.c | 6 ++---- 3 files changed, 2 insertions(+), 7 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e34a92c73a5d8..6fa1e88e40fbe 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3927,8 +3927,6 @@ static int __init io_uring_init(void) req_cachep = kmem_cache_create("io_kiocb", sizeof(struct io_kiocb), &kmem_args, SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT | SLAB_TYPESAFE_BY_RCU); - io_buf_cachep = KMEM_CACHE(io_buffer, - SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); iou_wq = alloc_workqueue("iou_exit", WQ_UNBOUND, 64); BUG_ON(!iou_wq); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index ab619e63ef39c..85bc8f76ca190 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -418,7 +418,6 @@ static inline bool io_req_cache_empty(struct io_ring_ctx *ctx) } extern struct kmem_cache *req_cachep; -extern struct kmem_cache *io_buf_cachep; static inline struct io_kiocb *io_extract_req(struct io_ring_ctx *ctx) { diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 0bed40f6fe3a5..ea9fb3c124e56 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -20,8 +20,6 @@ /* BIDs are addressed by a 16-bit field in a CQE */ #define MAX_BIDS_PER_BGID (1 << 16) -struct kmem_cache *io_buf_cachep; - struct io_provide_buf { struct file *file; __u64 addr; @@ -411,7 +409,7 @@ void io_destroy_buffers(struct io_ring_ctx *ctx) list_for_each_safe(item, tmp, &ctx->io_buffers_cache) { buf = list_entry(item, struct io_buffer, list); - kmem_cache_free(io_buf_cachep, buf); + kfree(buf); } } @@ -514,7 +512,7 @@ static int io_refill_buffer_cache(struct io_ring_ctx *ctx) spin_unlock(&ctx->completion_lock); } - buf = kmem_cache_alloc(io_buf_cachep, GFP_KERNEL); + buf = kmalloc(sizeof(*buf), GFP_KERNEL_ACCOUNT); if (!buf) return -ENOMEM; list_add_tail(&buf->list, &ctx->io_buffers_cache); From patchwork Wed Feb 5 11:36:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960881 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 670E322ACD4 for ; Wed, 5 Feb 2025 11:36:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755414; cv=none; b=ucz/ChFjjEFQTh909IIKupxYHsb5X7UxNEfGU45P8k1cGS+2FuUymmrHbD/G8U+VWDduo5rPW8bCGRbuHJ2T5CQ+Bl24J5mcAEf7T1eVllaVztjUKmTcV8LjLDMqtzXz02eN6SCgtFepDfQmLc5+B085r+gc0zxgHl62ukqMwIQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755414; c=relaxed/simple; bh=iaw6TIBLlPMBaodBJtkdRTF4CRl/fopRKMMhD3NaP7Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZcwOtfeZOIYUzEjNG4eCYc+JTMGfNXz75tSi3MaaFB4M0kfckh4H0l/wTeah79AtfZC+Bdcnng3pQpp31w9DMVTnkMO8BzAJkYZhdIcq3bpSSjULw3nrVb94UQ9B3bM4gm8GxPY51758RSAZdaTq5ltzZqlT1ZhATVSzQ0OpzAE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Khg0/mv0; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Khg0/mv0" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-436249df846so45733935e9.3 for ; Wed, 05 Feb 2025 03:36:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755410; x=1739360210; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eHXTHoM8l2ZmKFgYXPBLo6mbwPh89FeyMs6aOYQdBMc=; b=Khg0/mv0NneRvqi/bD3VFXBGH2UAszysI66FlJT2JtxMvQ+EXyj3m7ye2cY3ZofQDT Tf26YvYzL9OJ9kbt/p+4WvWDGhPhmjHQ/WiXBOkgAOZc31iXnqE4J9I5MLfCNvE336xn fod+Q6f2rcX87g3d5h7OiCmEbfuiCi5xSAn+KepsqESO9zYcA/HIyeeqL1ee5Q5YTGZv /5OeBYXJQVja4MaWASDdNOambbtvIBiHzZKEPf4xWXpIHtrbQX4+FJYeSk4t/kobghhN Q5WxIwLPY5nmjreN3j0qbWBO4IJNlW7Q5lGlCiyApK4EL65LaUmXM3Agi2aZjTgWnDtN KImA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755410; x=1739360210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eHXTHoM8l2ZmKFgYXPBLo6mbwPh89FeyMs6aOYQdBMc=; b=oqoN710BmqqILtDFSQ6i6I192zkTDXggl+o3/s3r+9lWZvJ2R5QHVNwvzYYQkEMFZf mqJPMEV71RBq3cl2uUmlCZWlQTBSaOzLRmvC/afYJvmiUPjTdF0/ns55vZz2UTcv9Dsa 6PQ2iClKVmZ6+it+wkpUX3moN7V5kbhyEMdk0WyEk1VCvJCX3MQ3/DEOHaSgMN7Mtor7 0t7aZAmDyyc0VYJJRINGpLmJcSf+Bvi3/hBR9PNKLJlt1rBZ0uelXfIG+UMrhPhOpKUd tUxa7T35uHbrhBuEZcTjyfiVHjCZ68p/EUezHdtVFrRNqkJPHVhvt15do8nLMrp0nnzk 7uJQ== X-Gm-Message-State: AOJu0YwJZJanJgpDnSRIVxmdE4dn4iQCwM6E30psqfgydkuSa3aT96pl q5KXTk+NM4uyfcobzOVt2ioIkEgl+AAF+nTpFx0Wbey5efUT+XLYqTa1Dg== X-Gm-Gg: ASbGncsDXVGkDZpjoRUlpGZ8gC/RYxasma2vfk38hwhkLJFgtjH4ess/SwIQKugSEC8 PANNsPCLO5nY+spv7JWyA88rnKxv7DXh4d5eQjUklS0thNdoF3JPH67xQGGgWY5y3o65ZwyHmf2 /2cN+iP2DKgt05PKsDnOS5F6QGbjb2fj5/MPI0sf9hQ8ZFe1a4It2bSnhvdbC1mzVPYQshdJJ/i mOMVwbB3ttg8KEOVTJy5rB1+hIsJHTH7KuAAGBUsZsbwgQ04vfk7Y6IErbzqZ9iEncB5XaYJx7R iUX1hoOImYwgvvhOFDxZFDn7Is8= X-Google-Smtp-Source: AGHT+IF5lKj1gvYfwQ6QbPjxTt9korXkWA3k++sXEiuGM6O092Oax29zRJdj0YVFDsxGgF2v/DDLXw== X-Received: by 2002:a05:600c:198b:b0:436:5fc9:30ba with SMTP id 5b1f17b1804b1-4390d5a46f1mr18339015e9.29.1738755409934; Wed, 05 Feb 2025 03:36:49 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:48 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 3/8] io_uring/kbuf: move locking into io_kbuf_drop() Date: Wed, 5 Feb 2025 11:36:44 +0000 Message-ID: <530f0cf1f06963029399f819a9a58b1a34bebef3.1738724373.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the burden of locking out of the caller into io_kbuf_drop(), that will help with furher refactoring. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 5 +---- io_uring/kbuf.h | 4 ++-- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6fa1e88e40fbe..ed7c9081352a4 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -398,11 +398,8 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) static void io_clean_op(struct io_kiocb *req) { - if (req->flags & REQ_F_BUFFER_SELECTED) { - spin_lock(&req->ctx->completion_lock); + if (unlikely(req->flags & REQ_F_BUFFER_SELECTED)) io_kbuf_drop(req); - spin_unlock(&req->ctx->completion_lock); - } if (req->flags & REQ_F_NEED_CLEANUP) { const struct io_cold_def *def = &io_cold_defs[req->opcode]; diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index bd80c44c5af1e..310f94a0727a6 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -174,13 +174,13 @@ static inline void __io_put_kbuf_list(struct io_kiocb *req, int len, static inline void io_kbuf_drop(struct io_kiocb *req) { - lockdep_assert_held(&req->ctx->completion_lock); - if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) return; + spin_lock(&req->ctx->completion_lock); /* len == 0 is fine here, non-ring will always drop all of it */ __io_put_kbuf_list(req, 0, &req->ctx->io_buffers_comp); + spin_unlock(&req->ctx->completion_lock); } static inline unsigned int __io_put_kbufs(struct io_kiocb *req, int len, From patchwork Wed Feb 5 11:36:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960882 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87E0122A4F2 for ; Wed, 5 Feb 2025 11:36:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755415; cv=none; b=CUYQUg12ZiaYVwfIb2f9nymrTG0i/2wlK5Vazh3bjmt8pc4l6NWN3/jdC6H8zlM8j59U8sCmNpEiDPboPp40ANjMfp9ApmZVel/v4O9HxUcqE2DKNVGKN3Q+FGlXPbZT86OE1eP4VXHQu97Eibez1ipjTwRFQ072z3CmcGgva9A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755415; c=relaxed/simple; bh=G3CfUWxMfny6vrqzl01oo0Ss+bMdT1rg/x0pNzSy+4M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cXjz7zBNbEZ1ol6HUUXpT7uORgzfmPpEdO0NMDe55V1a2H6ahvi0u7+C7eXzAEjFhQHjogl259+Y3L2YKLX2SXMILKOhHCQvv4vhHQxVA1+w6sLQ1TzJ8XKKHsBkLi0pnL8OU9rrvEZpDXHsaOkhJkwBflB5oNA/mqS0ofCFcZQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cWeioakO; arc=none smtp.client-ip=209.85.128.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cWeioakO" Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-43634b570c1so47120535e9.0 for ; Wed, 05 Feb 2025 03:36:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755411; x=1739360211; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7fmmYIE47lfgA6A2mQ8rveu6ZE5mZQb5KMBp3G7/DaM=; b=cWeioakOPwru4ZvhLzzc9Rs8Wd+IjYTWJeFFsJ+sZJhbfjFf2s8E8sE7tHAt59kCPv TtVr6Tj8P6TazHtEjY5KJVvGLoxyU2s2UUORdL2pN+pBBj3OD02jKr1ZjUz2peTCFWHc a5TP/GJIrfB589SMWqpkctxIN5NRFSJ+XsrP2VtfZNDwQLdTe7CpekI1QD8QI1TGgNmr 3kgq19kxUu5eK+iP0Jo/bp+shreeDY2i4U3/Get60MZ53yygW9Pvguax4x6UD0+WF9Y/ tUUx1tgjMc7bxoAEOnLHvZUtkPMs5r8htHe2SnFCt8USJCkIMPfg3hckqpCPGPfoCcQp 1ZlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755411; x=1739360211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7fmmYIE47lfgA6A2mQ8rveu6ZE5mZQb5KMBp3G7/DaM=; b=QJz8icjPRiL7HdAnrNc79fgw5Jl5+B6MiP1F4lSJlUFH6MbS4ymFLpK7iOvxrCl6cY XzG/Aftr4HkW6RbonIY1b+t7IAjsSYEtWmKrm7Jyl5+9PjxD/WVEQzzq1lAV/bz8txxp zYIsNFiOHyRE5pdJOszFl9pwPSeSQe5c1EoI20qqXIhHAlqUpetfrAqVlHTMed4nlV0H ssnC0xCX317HNN/4fKp+BxmrGjm6N/EgnTSxC2zxARarCCUXZFlJjkCtLKPu2UAHtEQb fLTaJL0C+DN/6Ec/Um2tAU3HF9AlDD+SVCqbDrly8Ln6X0hOBIYcgSS9zqcXCnpYMZ6I cv5Q== X-Gm-Message-State: AOJu0Yy3fa7Opk3Qe6yPOe0yMb8OsM+jpy19u8R5BhTeI3fhjVRUYoZd NAbax7QDeUWdv2Ia4lQo+KnmNpG8huvTX4btitDcsN9g2LijEy6KlpraTw== X-Gm-Gg: ASbGncvZh4wgU+7A0OTQhMWQy4SjYRSTy3Wx0jqBhJQqwv4XH0k1/r2FyPFIm4ct3WD eaZuu+9VxZAuwyp+5x/+duGBdddtzoGTwdTbHamQOg2aVVt5T9FxCYLpjztMMsJpLLQK9rVntk8 ZvLXBe9I03AyAe/cbhWZ/ehMCPEYJf6+1tlbJUhHODuTS/if/I50gzYEO+wIy8XumgCsYjHhP4P +wjZ8dwEQgGe7dSD8xZaFVj3JYNsD/RnyPoQh9iDUvJRPErpfvffF4htFp0Qz1MmcEpeIZv99+Q sTeoITD1sdsQhXifm7gWqTRonFc= X-Google-Smtp-Source: AGHT+IG2kEeBu8e+cE188/NyEcsK3Rgf2Y5opRrd8AmvCdk5Hev/SPeq9F2UUZTPkUqkHpkBRhEToA== X-Received: by 2002:a05:600c:3508:b0:434:fb65:ebbb with SMTP id 5b1f17b1804b1-4390d43da6fmr18080345e9.17.1738755411205; Wed, 05 Feb 2025 03:36:51 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:50 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 4/8] io_uring/kbuf: simplify __io_put_kbuf Date: Wed, 5 Feb 2025 11:36:45 +0000 Message-ID: <1b7f1394ec4afc7f96b35a61f5992e27c49fd067.1738724373.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 As a preparation step remove an optimisation from __io_put_kbuf() trying to use the locked cache. With that __io_put_kbuf_list() is only used with ->io_buffers_comp, and we remove the explicit list argument. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 26 +++----------------------- io_uring/kbuf.h | 7 +++---- 2 files changed, 6 insertions(+), 27 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index ea9fb3c124e56..eae6cf502b57f 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -70,29 +70,9 @@ bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags) void __io_put_kbuf(struct io_kiocb *req, int len, unsigned issue_flags) { - /* - * We can add this buffer back to two lists: - * - * 1) The io_buffers_cache list. This one is protected by the - * ctx->uring_lock. If we already hold this lock, add back to this - * list as we can grab it from issue as well. - * 2) The io_buffers_comp list. This one is protected by the - * ctx->completion_lock. - * - * We migrate buffers from the comp_list to the issue cache list - * when we need one. - */ - if (issue_flags & IO_URING_F_UNLOCKED) { - struct io_ring_ctx *ctx = req->ctx; - - spin_lock(&ctx->completion_lock); - __io_put_kbuf_list(req, len, &ctx->io_buffers_comp); - spin_unlock(&ctx->completion_lock); - } else { - lockdep_assert_held(&req->ctx->uring_lock); - - __io_put_kbuf_list(req, len, &req->ctx->io_buffers_cache); - } + spin_lock(&req->ctx->completion_lock); + __io_put_kbuf_list(req, len); + spin_unlock(&req->ctx->completion_lock); } static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 310f94a0727a6..1f28770648298 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -160,14 +160,13 @@ static inline bool __io_put_kbuf_ring(struct io_kiocb *req, int len, int nr) return ret; } -static inline void __io_put_kbuf_list(struct io_kiocb *req, int len, - struct list_head *list) +static inline void __io_put_kbuf_list(struct io_kiocb *req, int len) { if (req->flags & REQ_F_BUFFER_RING) { __io_put_kbuf_ring(req, len, 1); } else { req->buf_index = req->kbuf->bgid; - list_add(&req->kbuf->list, list); + list_add(&req->kbuf->list, &req->ctx->io_buffers_comp); req->flags &= ~REQ_F_BUFFER_SELECTED; } } @@ -179,7 +178,7 @@ static inline void io_kbuf_drop(struct io_kiocb *req) spin_lock(&req->ctx->completion_lock); /* len == 0 is fine here, non-ring will always drop all of it */ - __io_put_kbuf_list(req, 0, &req->ctx->io_buffers_comp); + __io_put_kbuf_list(req, 0); spin_unlock(&req->ctx->completion_lock); } From patchwork Wed Feb 5 11:36:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960883 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E75E22ACD4 for ; Wed, 5 Feb 2025 11:36:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755416; cv=none; b=BsfO13dVyKH3zlopWmLiFoBdC4Dh1bTzhKvVGeTe5XFS+Dumhxv8WNvy15T8apDDNI+ZaAA68alMa4rvEEZ7g5jkzcbLo+3LXGu0d+2UAvReSZSoz1WjvMoz3j/VBsM2zKBpakfxcZGl5Y0LF1ToNg6i5dNAb6vt4RR2FhDLa/s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755416; c=relaxed/simple; bh=ieQ5RJtNYpG3cYQtlgg0tvsBWaNrxS5OnuBbL0M+NgE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C47TwyHw1qys3pNKLXTdLfb23GbmuvtD3qkoGF139gqgn70wn57/BKGWNIgKt8ouogS6hUq8ueEx7OiNpooGEXmEXmFh9UN93jkmPWdDI7rd65N3cMb3T3Xs9pFqifqmc1ZO7KsLbX1TjiHMArF2WfVjR7jAASa/RrirArhUebw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hyifcIC4; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hyifcIC4" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-4361f796586so77748585e9.3 for ; Wed, 05 Feb 2025 03:36:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755413; x=1739360213; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nvfjTyzYru3H2UPC+6JsGGKZyTnjPYYlWkSpj0N0sAA=; b=hyifcIC4Szsr9m5ZSAti0p4Z6FCxNdffQsaTD9PxAYEnMPiU+zmSfI/u9Hxa2n4vpM XBO5kNUkRI/HdYOowrdlMar+5jtb5nU4nuGO1HBdD4ql8nXzRkGKeYXVifE+C6HM39r2 jiaKVdELSLRhY5PGM87cAQN4uh4K50FB0vTHU6pYM2G0pEGvw70G16+FqqwEQJkKI4bW goCWNe02x9TfSdxMz/6TRHzZylS68LVQEEBa/8k7uth9ztutkzV1mHDC5ETxY/jfDxzf JdB/XBN8MkGcw3A0AjlmggkVAU08CnpK8gV8XY/ih+pzER+c3Jo9Y2z1O6Xa4FR0clC6 /kdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755413; x=1739360213; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nvfjTyzYru3H2UPC+6JsGGKZyTnjPYYlWkSpj0N0sAA=; b=sVdJkK3LPYhn4TQam5FgvhogGnv0e3HmevV265Uh9ahreMb4wVAs1XCxb5DO8gDPP9 zb/aBjJNKKuclnizq3aIfv1rwiTDwv4WmyzDhZ6vlUf8wrfLbF8t+jIMZ8IL1Pea6X5q 0UatSb+UIMg3YOim9BDuf8skVoxrVoWPVKvwObBKC579ZaKsQMQPVFbFBMCGel6Gbaw8 +B/HQu522b1RorIhV4sBTr/KX0k/J6VleEccIZfK4QnqqDr2MGX8OraOTOCjsxaPA7wT rw4eRpAXTK8UA5XOBuLRTDdBbbcJzKY4XrUV3dGyUqTea0xaYtxz9zswoME56zreTY4T arSg== X-Gm-Message-State: AOJu0YxW1swJ1wG5eZ9sp211IawRGvuFbR7xHVT8hxjVyPvjY2AIZWmd nL2sBvv5MSW+dfzsp5hhpZumMBpaI+YA37bFMA9GDpL6IFWLOXEjikip1g== X-Gm-Gg: ASbGnctks9kKthnR4QmcWVItOejVAO/7nNOja1tXIc6VEczjiqlrJeQEJDrpyTg3+l6 KxGKNQEtcbAaVbDnY1P1OwY5Y48CYEh4SV8JwUc2D3C7vG8xFamOocF6q/k4pqSRIa0ZhmkDsL1 Cc7KNbCTXDQT1qtC29UAXsIOb8g+LcM0bbLlQksVo7dWWlia2CwAUE7NJbnGH/kCTPNCE8LuKEK 8EnyI7MSjPAr618+VarvD07Jn3+vRkHlm+wNDpWZbQm9reIi5W5e9vB0T6Ohb9qA01u939R6Y2R wMjoCdrPwHS2VuQy/3GIQViZgUE= X-Google-Smtp-Source: AGHT+IGMLybgnH7DkczH23rfnoEyuwj0+d76Thfe8Uyds7kuy605k/TkXkSsKHHuExOp5hg8hFxjjQ== X-Received: by 2002:a05:600c:3baa:b0:434:f817:4492 with SMTP id 5b1f17b1804b1-4390d591b4emr21591285e9.31.1738755412422; Wed, 05 Feb 2025 03:36:52 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:51 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 5/8] io_uring/kbuf: remove legacy kbuf caching Date: Wed, 5 Feb 2025 11:36:46 +0000 Message-ID: <18287217466ee2576ea0b1e72daccf7b22c7e856.1738724373.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Remove all struct io_buffer caches. It makes it a fair bit simpler. Apart from from killing a bunch of lines and juggling between lists, __io_put_kbuf_list() doesn't need ->completion_lock locking now. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 -- io_uring/io_uring.c | 2 -- io_uring/kbuf.c | 57 +++++----------------------------- io_uring/kbuf.h | 5 ++- 4 files changed, 9 insertions(+), 58 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 3def525a1da37..e2fef264ff8b8 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -360,7 +360,6 @@ struct io_ring_ctx { spinlock_t completion_lock; - struct list_head io_buffers_comp; struct list_head cq_overflow_list; struct hlist_head waitid_list; @@ -379,8 +378,6 @@ struct io_ring_ctx { unsigned int file_alloc_start; unsigned int file_alloc_end; - struct list_head io_buffers_cache; - /* Keep this last, we don't need it for the fast path */ struct wait_queue_head poll_wq; struct io_restriction restrictions; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ed7c9081352a4..969caaccce9d8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -323,7 +323,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) init_waitqueue_head(&ctx->sqo_sq_wait); INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); - INIT_LIST_HEAD(&ctx->io_buffers_cache); ret = io_alloc_cache_init(&ctx->apoll_cache, IO_POLL_ALLOC_CACHE_MAX, sizeof(struct async_poll), 0); ret |= io_alloc_cache_init(&ctx->netmsg_cache, IO_ALLOC_CACHE_MAX, @@ -348,7 +347,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) spin_lock_init(&ctx->completion_lock); raw_spin_lock_init(&ctx->timeout_lock); INIT_WQ_LIST(&ctx->iopoll_list); - INIT_LIST_HEAD(&ctx->io_buffers_comp); INIT_LIST_HEAD(&ctx->defer_list); INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index eae6cf502b57f..ef0c06d1bc86f 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -70,9 +70,7 @@ bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags) void __io_put_kbuf(struct io_kiocb *req, int len, unsigned issue_flags) { - spin_lock(&req->ctx->completion_lock); __io_put_kbuf_list(req, len); - spin_unlock(&req->ctx->completion_lock); } static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, @@ -345,7 +343,9 @@ static int __io_remove_buffers(struct io_ring_ctx *ctx, struct io_buffer *nxt; nxt = list_first_entry(&bl->buf_list, struct io_buffer, list); - list_move(&nxt->list, &ctx->io_buffers_cache); + list_del(&nxt->list); + kfree(nxt); + if (++i == nbufs) return i; cond_resched(); @@ -363,8 +363,6 @@ static void io_put_bl(struct io_ring_ctx *ctx, struct io_buffer_list *bl) void io_destroy_buffers(struct io_ring_ctx *ctx) { struct io_buffer_list *bl; - struct list_head *item, *tmp; - struct io_buffer *buf; while (1) { unsigned long index = 0; @@ -378,19 +376,6 @@ void io_destroy_buffers(struct io_ring_ctx *ctx) break; io_put_bl(ctx, bl); } - - /* - * Move deferred locked entries to cache before pruning - */ - spin_lock(&ctx->completion_lock); - if (!list_empty(&ctx->io_buffers_comp)) - list_splice_init(&ctx->io_buffers_comp, &ctx->io_buffers_cache); - spin_unlock(&ctx->completion_lock); - - list_for_each_safe(item, tmp, &ctx->io_buffers_cache) { - buf = list_entry(item, struct io_buffer, list); - kfree(buf); - } } int io_remove_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) @@ -472,33 +457,6 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe return 0; } -static int io_refill_buffer_cache(struct io_ring_ctx *ctx) -{ - struct io_buffer *buf; - - /* - * Completions that don't happen inline (eg not under uring_lock) will - * add to ->io_buffers_comp. If we don't have any free buffers, check - * the completion list and splice those entries first. - */ - if (!list_empty_careful(&ctx->io_buffers_comp)) { - spin_lock(&ctx->completion_lock); - if (!list_empty(&ctx->io_buffers_comp)) { - list_splice_init(&ctx->io_buffers_comp, - &ctx->io_buffers_cache); - spin_unlock(&ctx->completion_lock); - return 0; - } - spin_unlock(&ctx->completion_lock); - } - - buf = kmalloc(sizeof(*buf), GFP_KERNEL_ACCOUNT); - if (!buf) - return -ENOMEM; - list_add_tail(&buf->list, &ctx->io_buffers_cache); - return 0; -} - static int io_add_buffers(struct io_ring_ctx *ctx, struct io_provide_buf *pbuf, struct io_buffer_list *bl) { @@ -507,12 +465,11 @@ static int io_add_buffers(struct io_ring_ctx *ctx, struct io_provide_buf *pbuf, int i, bid = pbuf->bid; for (i = 0; i < pbuf->nbufs; i++) { - if (list_empty(&ctx->io_buffers_cache) && - io_refill_buffer_cache(ctx)) + buf = kmalloc(sizeof(*buf), GFP_KERNEL_ACCOUNT); + if (!buf) break; - buf = list_first_entry(&ctx->io_buffers_cache, struct io_buffer, - list); - list_move_tail(&buf->list, &bl->buf_list); + + list_add_tail(&buf->list, &bl->buf_list); buf->addr = addr; buf->len = min_t(__u32, pbuf->len, MAX_RW_COUNT); buf->bid = bid; diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 1f28770648298..c0b9636c5c4ae 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -166,8 +166,9 @@ static inline void __io_put_kbuf_list(struct io_kiocb *req, int len) __io_put_kbuf_ring(req, len, 1); } else { req->buf_index = req->kbuf->bgid; - list_add(&req->kbuf->list, &req->ctx->io_buffers_comp); req->flags &= ~REQ_F_BUFFER_SELECTED; + kfree(req->kbuf); + req->kbuf = NULL; } } @@ -176,10 +177,8 @@ static inline void io_kbuf_drop(struct io_kiocb *req) if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) return; - spin_lock(&req->ctx->completion_lock); /* len == 0 is fine here, non-ring will always drop all of it */ __io_put_kbuf_list(req, 0); - spin_unlock(&req->ctx->completion_lock); } static inline unsigned int __io_put_kbufs(struct io_kiocb *req, int len, From patchwork Wed Feb 5 11:36:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960884 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52C4B22ACD4 for ; Wed, 5 Feb 2025 11:36:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755419; cv=none; b=TBWgneZ53ak5x7+Ovn53LwWx2BUt6/BaUk/p9J3PjitLSojTkLLZfyZucSkATt4Q6pgPR+tL+DESpipMdt7zNWXlLLCZ7Y0t4dHaZ+brKZoxvHE8rfJy44R2TRGn1TBnTej69V3GO87ODFJJFkNkAkdVEdx8gICGbrMcoh1WoLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755419; c=relaxed/simple; bh=TuwebxuLayIpG5s1JE+nmTZZQ7IcGyYFTxgLxNjKK7w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JmJH/7066nRXSBIeD0bFhAa3EtI6PosO5PmUW9mRA2XWklxDlNb7DieroGEb2Nn1/9+LNfDHz4ksyjtc/ds5dUaHwuSuSb/PAunsrRce0Y10QeXBXoXceCBx96YkCdicmW33AOvQBLzXdA7wz0ImE12VdPrAj/abAYKVVLrwX18= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m1Pe77WP; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m1Pe77WP" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-436202dd730so47005435e9.2 for ; Wed, 05 Feb 2025 03:36:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755415; x=1739360215; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3mldhM/2N7G4MxKkTBk2LFxpjmmTPM9Ga/RGmVFsZrY=; b=m1Pe77WPR8EnACmKU1mnVrDM9cg+SesRr04i205g+alX0to1B8Nw3i9VMfptEHMtox TLFdqFkZkHoEjuvYexNOUYX/HPuTinX9T6qjp1/jCwbbDX/m+EtpgrtBc4oSK4dzPf9J e5eXQy2MPxzJ0EMdCeJGKR/Ro65VzScl+smgTfgZHvacqukUXRfKBuDVbKWx9G+Bb4FY DwAWOhhiSnwn7UDNg9+feTzeDNwQ0vCIcjS5guFQ6Yn9ju3wkIVdNSf4QZc5vB1mQLUK FJywoVS4lJlBmIAkFEFRRiltOhZYte+6qJwwzAPHfqoH3cNInmYclpTs53gRsFUrL0MO GwXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755415; x=1739360215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3mldhM/2N7G4MxKkTBk2LFxpjmmTPM9Ga/RGmVFsZrY=; b=w8aVcMrrE+9laLp5euXAEnGhhkMyutFrViDW7uZLz44od0vWCD9v5DZXTCDFQENJTT MNqEo62SOIyYgl47F3wiNAp2Cs0wM2nJrd8bYVfL/rMN2XlIfV3rFrvv/y8IbU86XH8s HOQh1VZ7XhZwj1Q3hRykm7IorBdgA75Of9vRPMXuhDIQAehM0WbrKRvrbCS+gOL2fpiL 5fZIBYhdffFVwsZiTMDsjRrGffU6luhDkRmz+lpJj/RIKjsTeK3AskC1GFyUJH/qV0CS OibYWDQJHyw7DWP7TeUqLy3dOiM5NFpKZAeGMK2rZ3mhBOLXXqP7uSTB9QLOvQbvfLdS Mk0w== X-Gm-Message-State: AOJu0YzHaIAEhmTpLc4Jn1TGb2oumE7EjUrVKymZOv3fIzI9OOaoEgqa fU6l1u75B7sUd18kej8DVOeioKTAshZZtjOpv5aFC+xQiR06EHuKiKW9MA== X-Gm-Gg: ASbGnctEQsT2L7psJqk3BwQUfJ/4tX0ejx8k158ckctCFWrUkUmzzRw1kqJUa3rsCbC DfnzoasWIqbEJgoJJLkjORszMcunO0ZAhLPMfCbjWwIA7Y1q85BtKCrmWaHGZIZ1e1egYz7bIC8 iJ+UyT+/QXXa9wasX8TE+cZ42fx5uWYRXdkeR68XWCoMD3nDC7retcGEG7QthxR5UxgVBR+8F0a t/6+82taRYig2OWCfaHZFb8J+0KacnrwdJ0Eao6llqpHgu5v4k98RSZOBE9GdkowLMjkdRrxBWl ZdySxDEJ7A4M1Ceg6Ci1dsbqRFE= X-Google-Smtp-Source: AGHT+IEfRL7tOvJWCYumhVI2ixkw4mblXtyeNOA38KdTOiI98WuRP+/QG602YWtiN9IOQtgdHfjYug== X-Received: by 2002:a05:600c:5254:b0:438:a913:a99 with SMTP id 5b1f17b1804b1-4390d5b2fa8mr19414765e9.31.1738755414788; Wed, 05 Feb 2025 03:36:54 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:54 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 6/8] io_uring/kbuf: open code __io_put_kbuf() Date: Wed, 5 Feb 2025 11:36:47 +0000 Message-ID: <9dc17380272b48d56c95992c6f9eaacd5546e1d3.1738724373.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 __io_put_kbuf() is a trivial wrapper, open code it into __io_put_kbufs(). Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 5 ----- io_uring/kbuf.h | 4 +--- 2 files changed, 1 insertion(+), 8 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index ef0c06d1bc86f..f41c141ae8eda 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -68,11 +68,6 @@ bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags) return true; } -void __io_put_kbuf(struct io_kiocb *req, int len, unsigned issue_flags) -{ - __io_put_kbuf_list(req, len); -} - static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, struct io_buffer_list *bl) { diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index c0b9636c5c4ae..055b7a672f2e0 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -74,8 +74,6 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg); -void __io_put_kbuf(struct io_kiocb *req, int len, unsigned issue_flags); - bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags); struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, @@ -194,7 +192,7 @@ static inline unsigned int __io_put_kbufs(struct io_kiocb *req, int len, if (!__io_put_kbuf_ring(req, len, nbufs)) ret |= IORING_CQE_F_BUF_MORE; } else { - __io_put_kbuf(req, len, issue_flags); + __io_put_kbuf_list(req, len); } return ret; } From patchwork Wed Feb 5 11:36:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960885 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55E9422ACD7 for ; Wed, 5 Feb 2025 11:36:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755421; cv=none; b=J9uI/U08sVJTZ2/ElT1ub7GlaWFwmNSipLES+PDPhWhIFdKgfk14+w7iFCmViVPM92ar90O/zQXcdfjKRe5plsS4INrp7DUAfmsKlgXQsh8EN8qcA3gVzkUotr8rjux7OtRhHVDLMsR1+B9piHAmhDbAls+gyk/+DfmRfkxSAbo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755421; c=relaxed/simple; bh=fO3WIHRlQugLysD9UNt7Mq+Lb7PdSKoQ2Hw07m6RHq4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rAELt7rByqa70EJXxrVw4uRRXqbuJBx3wgEfL9zSK4isPG84rjomBJKzgOaHtpCIABtl52iY096XRTafZv3Yy0cslhlAnYkMBpJg+Fw3n2223pimyfwn665srLOLpGqnZKzFEz01o4QZkt3IeLGrDOCKAvuibn84VGmJZsxWgu8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NH7bT2T8; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NH7bT2T8" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-4364a37a1d7so66020735e9.3 for ; Wed, 05 Feb 2025 03:36:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755417; x=1739360217; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HhExhce8Aiyglbiuep/ta6W+YhgIu1iQn6uz5vzEfYk=; b=NH7bT2T8oTQDr12DFfZnmrapqxNRFgQe7DHuas+c4h/kZMZmb8gOxMnddDlipQdPY+ dvlpKhgoVcCjHmgzEoPZXWW4tOFuWqrEfYbRezTH8CefrLu7kYCwBVqyZfKHf/Oh4lDe AaMY5U4M7QYW7kEI7sgI4yAS/G0C8XcjklyoKbVp8izGfEFHZhmD63347qvAKzknnFDS BfGvbpBQjzcicRM5vnALA5fB48rIeYB+h2qo7UgQDOlfZDsr2fCkt6xQN7wjXn7S3jou ncFc9EKwQ8un6JCfAh6zWZO1KTME8UtuYpn/2HnZ989PZ0JL8zZkxSPiWwDaU3IBEzuZ Wn/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755417; x=1739360217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HhExhce8Aiyglbiuep/ta6W+YhgIu1iQn6uz5vzEfYk=; b=Gr8a837yqkkhzKerI5zcavcv6s4a28/IzaLgYU8pqk7gxZhbLM5Qy+rgqoI9LCZ8gI VAvRsewSnX7kp3ymOnC3PWOpcswK8hQ/S9W4HVOE4o8/IUPXut7wzYqdl08CsKRHh6Ym LgtJZNNS4peIk3DQ2h1bEgrxaaLl7MbC9qb4aFyYHXwVaSZzIznyh6YUCw4fF564f6fQ 3GcFbxb01fXJJpO60l6LAhEIIfYMzaGhY08C+vxsZv1a1kUSk2tCpyEdN13xg9vOeirq yclhdNhOy17VNuXvJIOxXx7zk642mNAmvqrkD+IUy9oOtIEgeWfvlYgmesR98/HchaqG MOJQ== X-Gm-Message-State: AOJu0YwTwkYE4U3Kg99Ot4OKP1eGEiGe7hx9eW+XkErpIyjjgkf4eqXy 7rZXt5MyeGE6Aq5ua3/RRt7JPWLdVxb3/CRjj7u9v2mU77whHBgGqVX5nQ== X-Gm-Gg: ASbGncsNp1/MjdhndISNIqsUVl9ng6m24GWOn/9GWP8RFJ8i4I2VX8ctooPxIHdFNMm XM7GJQ8ndbFKgfbMghu4BYtzf/LM9D3LjRN7KHhkqOqdJHiDZxUrhdGcJbBZoqWdJCOCu8VwQi0 6kgjNzAfaAkHnBzPHks8C+HQZOWyuC8380n3urw5JxBGBCtruv2nLMK27KbziaCikV28Naheray WDRPNaH4eniXmPVxnWp8aaHnUPQrn7JfR7OmvcG+Y5+gK5Ce+6UK84nh2WqNtSqdr16NgGAcz03 cA8BTyVpfB7JQTiM4SF5Ze22QS8= X-Google-Smtp-Source: AGHT+IFdKhn0SfGvIFa8dvxYguULcGBNa9fF7bA1pOk4J99pvcl6zOMNQAd87PRMoWD3kPwXA64t1A== X-Received: by 2002:a05:600c:1f8f:b0:434:ff30:a159 with SMTP id 5b1f17b1804b1-4390d34b326mr18629285e9.0.1738755416857; Wed, 05 Feb 2025 03:36:56 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:55 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 7/8] io_uring/kbuf: introduce io_kbuf_drop_legacy() Date: Wed, 5 Feb 2025 11:36:48 +0000 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_kbuf_drop() is only used for legacy provided buffers, and so __io_put_kbuf_list() is never called for REQ_F_BUFFER_RING. Remove the dead branch out of __io_put_kbuf_list(), rename it into io_kbuf_drop_legacy() and use it directly instead of io_kbuf_drop(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- io_uring/kbuf.c | 10 ++++++++++ io_uring/kbuf.h | 24 ++---------------------- 3 files changed, 13 insertions(+), 23 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 969caaccce9d8..ec98a0ec6f34e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -397,7 +397,7 @@ static bool req_need_defer(struct io_kiocb *req, u32 seq) static void io_clean_op(struct io_kiocb *req) { if (unlikely(req->flags & REQ_F_BUFFER_SELECTED)) - io_kbuf_drop(req); + io_kbuf_drop_legacy(req); if (req->flags & REQ_F_NEED_CLEANUP) { const struct io_cold_def *def = &io_cold_defs[req->opcode]; diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index f41c141ae8eda..6d76108b67e03 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -50,6 +50,16 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } +void io_kbuf_drop_legacy(struct io_kiocb *req) +{ + if (WARN_ON_ONCE(!(req->flags & REQ_F_BUFFER_SELECTED))) + return; + req->buf_index = req->kbuf->bgid; + req->flags &= ~REQ_F_BUFFER_SELECTED; + kfree(req->kbuf); + req->kbuf = NULL; +} + bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 055b7a672f2e0..3e18c916afc60 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -75,6 +75,7 @@ int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg); bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags); +void io_kbuf_drop_legacy(struct io_kiocb *req); struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, unsigned int bgid); @@ -158,27 +159,6 @@ static inline bool __io_put_kbuf_ring(struct io_kiocb *req, int len, int nr) return ret; } -static inline void __io_put_kbuf_list(struct io_kiocb *req, int len) -{ - if (req->flags & REQ_F_BUFFER_RING) { - __io_put_kbuf_ring(req, len, 1); - } else { - req->buf_index = req->kbuf->bgid; - req->flags &= ~REQ_F_BUFFER_SELECTED; - kfree(req->kbuf); - req->kbuf = NULL; - } -} - -static inline void io_kbuf_drop(struct io_kiocb *req) -{ - if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) - return; - - /* len == 0 is fine here, non-ring will always drop all of it */ - __io_put_kbuf_list(req, 0); -} - static inline unsigned int __io_put_kbufs(struct io_kiocb *req, int len, int nbufs, unsigned issue_flags) { @@ -192,7 +172,7 @@ static inline unsigned int __io_put_kbufs(struct io_kiocb *req, int len, if (!__io_put_kbuf_ring(req, len, nbufs)) ret |= IORING_CQE_F_BUF_MORE; } else { - __io_put_kbuf_list(req, len); + io_kbuf_drop_legacy(req); } return ret; } From patchwork Wed Feb 5 11:36:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13960886 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB24722A4E1 for ; Wed, 5 Feb 2025 11:37:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755422; cv=none; b=nYpcmuUtN2xwlKDNg+3xkUwEaqAuugP1CAGu8OidbOEahU/TGuoqfEref9Dn5+to6XDpCeO078cxqlv4XkqHe5Ybft3RDWhSuJRTPyLDb3r9/AG9d0sxEY3bobUwDgocFVRE8T1bPnBDri6AJOlBRwly1QG8Z3jk4t3/VSy4vQc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738755422; c=relaxed/simple; bh=M3QeKR3rEadmhAMGo5LEhYRK14X+WHtxcXUeoL4bX1I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OajXvnkal/E9KGzXLO9RNLQtw0AKUlh7gAuy4fKYk6XFXtljcEFsvA/UNuAozSHG3ZNJ3Bm04E8NmCbiiXl6MFTQU9JTN0vV6d/8h3RAB6Gg9HwK4G1kOG4LnHbaJE2WOIar1xGRlFvXKeXxQecg6hROA8OAIAgJLKbEUA1nVG8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lwplOp/s; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lwplOp/s" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-436a39e4891so45783665e9.1 for ; Wed, 05 Feb 2025 03:37:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738755419; x=1739360219; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tE+cKByEWWXThcjMWTmzJSol4RltuG6Uf0Z6BAELz0M=; b=lwplOp/s4oAvFvGsOQMw1Xe0v6H6HKsgSlfUuHeBN5rNFudbzqOroKmEc4hG6i3xxt U+epbEUlConA50cA6iIEqM2lWQbfgIe4emDtTcu5pOm9Ou0ea2uy4Utz28enHuwd0Ddq DEpjZ7HPW0u3bKKtzbuTAaBGfV466BgN0sjn7O2Uny50v9uIFcdR2IUBvENDjUJ+Dt66 vAEXZFFOqUSwMS8SBanJJgG/dljdH1qbU3ZFpqbYUtZ1iCaGdyf8yaYg0czZooeqnQd7 2vI8nZOfYq7mZV0crbDJenqd3UEi3b8rItCzJh5KwO1qrjrFDnYzqQrEl0OcAHM13J2a H9fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738755419; x=1739360219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tE+cKByEWWXThcjMWTmzJSol4RltuG6Uf0Z6BAELz0M=; b=SGUOo7AINCYXw2q70JVA1V8WY/cHNyFP/RzWWJoq08E7WuJbIJps9tkrvurxgofTgG 5Y6Qx6U2fCLKJbLZ4ZwUgz0kfgeWNQcD9xSHy9x7gMLwP/+x43TwLQuRqhYTCoUhbi+Z m+y2h/sXw/GlF3Opf8Kx7XK1/FLakL2NZ8MblM8+/0iUq5O366NXhyLPK2oacnI37zP5 ZjJmSxpuQ6xyBi1gTiVVGgra/H/vaZg/cKNO/4nVMLYJtDfS1H0Gi9Xgn7mN0o5gyKW+ 10zmjODMN3tB/4gdUSqjph66aiggXPYcMhwMhZjd9Gog/oF7pQtNk8rUkN1qTMyjXChd TGwA== X-Gm-Message-State: AOJu0YzSYf5Hn1d9nFOQCl4CJJKFVg8z7rtgk/0QxspK6BIWUXCKOA0m 2hhjUoJokDzCGQD//G+C5xIX48abYqd9ebUcLlSK7qdfjEB/JjCo/EVhDA== X-Gm-Gg: ASbGnctlXiWzu2WHDb4lcAZ1jVThjl8oevVVnS/K6pbrsKpti1ZVhz6ITsE6DpwlSJh yCQx+tIYOslWQSYTssG9vOnfMmo3SkabNWxF+5W9OpOQUnpQAtJHURLgWdzHKWmbp22A4jVoV15 j8PDQc+f0b17WROZjmnVzTkgw0dJBhDdQ9FbGJ+KjFn0lI+iF4tl0E/RFDPk54EQPAQ8S3AddWL c7jemi9f7jN8J/YtskgXZ6/Wg3kUXCB7dAPh1eOXmBazrNh9Z9Fx5fKvj4ThICT0iRbZ791sV8u cyfpOKgbDAbvrowdv/in4NOS0fY= X-Google-Smtp-Source: AGHT+IHAdSKYB6RqLjr3zmaDlCpIlrXJVn0QdBTos7/yUD7pf9cDN5CSYae/+u2L5/NwH9nMeNkcSQ== X-Received: by 2002:a05:600c:500d:b0:434:f1d5:1453 with SMTP id 5b1f17b1804b1-4390e88b7fdmr14799485e9.0.1738755418224; Wed, 05 Feb 2025 03:36:58 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.128.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7d4sm18514505e9.10.2025.02.05.03.36.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Feb 2025 03:36:57 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com Subject: [PATCH 8/8] io_uring/kbuf: uninline __io_put_kbufs Date: Wed, 5 Feb 2025 11:36:49 +0000 Message-ID: <3dade7f55ad590e811aff83b1ec55c9c04e17b2b.1738724373.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 __io_put_kbufs() and other helper functions are too large to be inlined, compilers would normally refuse to do so. Uninline it and move together with io_kbuf_commit into kbuf.c. io_kbuf_commitSigned-off-by: Pavel Begunkov Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 60 +++++++++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 74 +++++++------------------------------------------ 2 files changed, 70 insertions(+), 64 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 6d76108b67e03..319c5a25f72db 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -20,6 +20,9 @@ /* BIDs are addressed by a 16-bit field in a CQE */ #define MAX_BIDS_PER_BGID (1 << 16) +/* Mapped buffer ring, return io_uring_buf from head */ +#define io_ring_head_to_buf(br, head, mask) &(br)->bufs[(head) & (mask)] + struct io_provide_buf { struct file *file; __u64 addr; @@ -29,6 +32,34 @@ struct io_provide_buf { __u16 bid; }; +bool io_kbuf_commit(struct io_kiocb *req, + struct io_buffer_list *bl, int len, int nr) +{ + if (unlikely(!(req->flags & REQ_F_BUFFERS_COMMIT))) + return true; + + req->flags &= ~REQ_F_BUFFERS_COMMIT; + + if (unlikely(len < 0)) + return true; + + if (bl->flags & IOBL_INC) { + struct io_uring_buf *buf; + + buf = io_ring_head_to_buf(bl->buf_ring, bl->head, bl->mask); + if (WARN_ON_ONCE(len > buf->len)) + len = buf->len; + buf->len -= len; + if (buf->len) { + buf->addr += len; + return false; + } + } + + bl->head += nr; + return true; +} + static inline struct io_buffer_list *io_buffer_get_list(struct io_ring_ctx *ctx, unsigned int bgid) { @@ -323,6 +354,35 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg) return io_provided_buffers_select(req, &arg->max_len, bl, arg->iovs); } +static inline bool __io_put_kbuf_ring(struct io_kiocb *req, int len, int nr) +{ + struct io_buffer_list *bl = req->buf_list; + bool ret = true; + + if (bl) { + ret = io_kbuf_commit(req, bl, len, nr); + req->buf_index = bl->bgid; + } + req->flags &= ~REQ_F_BUFFER_RING; + return ret; +} + +unsigned int __io_put_kbufs(struct io_kiocb *req, int len, int nbufs) +{ + unsigned int ret; + + ret = IORING_CQE_F_BUFFER | (req->buf_index << IORING_CQE_BUFFER_SHIFT); + + if (unlikely(!(req->flags & REQ_F_BUFFER_RING))) { + io_kbuf_drop_legacy(req); + return ret; + } + + if (!__io_put_kbuf_ring(req, len, nbufs)) + ret |= IORING_CQE_F_BUF_MORE; + return ret; +} + static int __io_remove_buffers(struct io_ring_ctx *ctx, struct io_buffer_list *bl, unsigned nbufs) { diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 3e18c916afc60..2ec0b983ce243 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -77,6 +77,10 @@ int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg); bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags); void io_kbuf_drop_legacy(struct io_kiocb *req); +unsigned int __io_put_kbufs(struct io_kiocb *req, int len, int nbufs); +bool io_kbuf_commit(struct io_kiocb *req, + struct io_buffer_list *bl, int len, int nr); + struct io_mapped_region *io_pbuf_get_region(struct io_ring_ctx *ctx, unsigned int bgid); @@ -115,77 +119,19 @@ static inline bool io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags) return false; } -/* Mapped buffer ring, return io_uring_buf from head */ -#define io_ring_head_to_buf(br, head, mask) &(br)->bufs[(head) & (mask)] - -static inline bool io_kbuf_commit(struct io_kiocb *req, - struct io_buffer_list *bl, int len, int nr) -{ - if (unlikely(!(req->flags & REQ_F_BUFFERS_COMMIT))) - return true; - - req->flags &= ~REQ_F_BUFFERS_COMMIT; - - if (unlikely(len < 0)) - return true; - - if (bl->flags & IOBL_INC) { - struct io_uring_buf *buf; - - buf = io_ring_head_to_buf(bl->buf_ring, bl->head, bl->mask); - if (WARN_ON_ONCE(len > buf->len)) - len = buf->len; - buf->len -= len; - if (buf->len) { - buf->addr += len; - return false; - } - } - - bl->head += nr; - return true; -} - -static inline bool __io_put_kbuf_ring(struct io_kiocb *req, int len, int nr) -{ - struct io_buffer_list *bl = req->buf_list; - bool ret = true; - - if (bl) { - ret = io_kbuf_commit(req, bl, len, nr); - req->buf_index = bl->bgid; - } - req->flags &= ~REQ_F_BUFFER_RING; - return ret; -} - -static inline unsigned int __io_put_kbufs(struct io_kiocb *req, int len, - int nbufs, unsigned issue_flags) -{ - unsigned int ret; - - if (!(req->flags & (REQ_F_BUFFER_RING | REQ_F_BUFFER_SELECTED))) - return 0; - - ret = IORING_CQE_F_BUFFER | (req->buf_index << IORING_CQE_BUFFER_SHIFT); - if (req->flags & REQ_F_BUFFER_RING) { - if (!__io_put_kbuf_ring(req, len, nbufs)) - ret |= IORING_CQE_F_BUF_MORE; - } else { - io_kbuf_drop_legacy(req); - } - return ret; -} - static inline unsigned int io_put_kbuf(struct io_kiocb *req, int len, unsigned issue_flags) { - return __io_put_kbufs(req, len, 1, issue_flags); + if (!(req->flags & (REQ_F_BUFFER_RING | REQ_F_BUFFER_SELECTED))) + return 0; + return __io_put_kbufs(req, len, 1); } static inline unsigned int io_put_kbufs(struct io_kiocb *req, int len, int nbufs, unsigned issue_flags) { - return __io_put_kbufs(req, len, nbufs, issue_flags); + if (!(req->flags & (REQ_F_BUFFER_RING | REQ_F_BUFFER_SELECTED))) + return 0; + return __io_put_kbufs(req, len, nbufs); } #endif