From patchwork Wed Nov 2 15:18:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13028276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36391C43217 for ; Wed, 2 Nov 2022 15:20:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229523AbiKBPUG (ORCPT ); Wed, 2 Nov 2022 11:20:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230343AbiKBPTj (ORCPT ); Wed, 2 Nov 2022 11:19:39 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFF5D1704C for ; Wed, 2 Nov 2022 08:19:37 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id l16-20020a05600c4f1000b003c6c0d2a445so1500249wmq.4 for ; Wed, 02 Nov 2022 08:19:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5vaKIUeUGBd2XcHhMkOucioiZX3YOY/Xwk36yuAh4aE=; b=SMPVoEcnmpjpWpOIfBwY/KnyKs3CRx77848wtFgxK+9C4zYohcvhTNGNGFP5XkChR6 ED/B9ch8j+BAAqYgUtK25sIBqGDVmuHOfEJvQCdIUBAqNwT+GjXTEaUT7RfIrW/zMQwL h7tVJgYoZ3LQKGI0frdwlzmNN8n2RbnU6/NO3R+8ukknLe34s1IWDiFAQS8jHDYE6Ted o/xJvjRBf1VmOrr3j87h5H70rcop8lpfBkfZQNlFKn6F2xuKAeQ9cZEfkrlT8FBj4qBu Mov/Vd1nSG6D6qD7z75nGmdO4KYkU8tyJK2sL3ZDlk1YFrLLo159bSV3il4Mdx4f9pBz KPrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5vaKIUeUGBd2XcHhMkOucioiZX3YOY/Xwk36yuAh4aE=; b=q80T1t4a7UC/FFXF7LnFkcpGCyElIAo1DNc4hxNDqC0nct4plWBW3iuFeCnmuhvpxO 7oT8cioDfvzE+Xd1QHz1BbTMkZaS4Ar9dm/OrJy9SXjLwxzkL+RThuu2BmvAX/49ps03 w5MmYJ1qU7qPDHwgVxUbPs2EE+GNYCj8lHmIWghjQISe4wd+t4J/sYzRed3wty3d+ido sVrJyW0wSsHCx5eeQlI9Ed8mm+G/MF984DHPHLlAubvj/zWZQtaCqJhePhAU7xQP3fPt +JLtyOfrZRz35nvLey+lJemSVq0+JLl7gBHv5I6Mgaz7xgF1LidgKSTmiIx4szTBUA79 1lTQ== X-Gm-Message-State: ACrzQf2bGML5Mo1pbvfhzFFZDIp7b7po4yKwRbg3Gu4wHkEdUSyiyORb TiANlyfiJsah+atRF4Vyudg= X-Google-Smtp-Source: AMsMyM4B3oR7+dgj8lw6zRqTrgMRYIARqQAIzT7kVhBg9U8a1NcckjVFmZxVIrfrvGyXtYQcMDSMsA== X-Received: by 2002:a05:600c:1e2a:b0:3c8:353b:253f with SMTP id ay42-20020a05600c1e2a00b003c8353b253fmr16111973wmb.51.1667402376065; Wed, 02 Nov 2022 08:19:36 -0700 (PDT) Received: from 127.0.0.1localhost ([82.132.186.241]) by smtp.gmail.com with ESMTPSA id a14-20020adff7ce000000b0022e66749437sm13043232wrq.93.2022.11.02.08.19.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 08:19:35 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org Cc: kernel-team@fb.com, Pavel Begunkov Subject: [PATCH for-next v4 1/6] mempool: introduce mempool_is_saturated Date: Wed, 2 Nov 2022 15:18:19 +0000 Message-Id: <636aed30be8c35d78f45e244998bc6209283cccc.1667384020.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Introduce a helper mempool_is_saturated(), which tells if the mempool is under-filled or not. We need it to figure out whether it should be freed right into the mempool or could be cached with top level caches. Signed-off-by: Pavel Begunkov --- include/linux/mempool.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 0c964ac107c2..4aae6c06c5f2 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -30,6 +30,11 @@ static inline bool mempool_initialized(mempool_t *pool) return pool->elements != NULL; } +static inline bool mempool_is_saturated(mempool_t *pool) +{ + return READ_ONCE(pool->curr_nr) >= pool->min_nr; +} + void mempool_exit(mempool_t *pool); int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, From patchwork Wed Nov 2 15:18:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13028278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19087C4332F for ; Wed, 2 Nov 2022 15:20:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230297AbiKBPUQ (ORCPT ); Wed, 2 Nov 2022 11:20:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230232AbiKBPTk (ORCPT ); Wed, 2 Nov 2022 11:19:40 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1793C17E38 for ; Wed, 2 Nov 2022 08:19:39 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id v7so5099529wmn.0 for ; Wed, 02 Nov 2022 08:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z9Y20rpLVvo5TBJHmlhFfyIhAn0kzicFV4WKChu5otc=; b=filVyL0Cr2NsXxSa7V2aV1vUQN0Kw/mMrkgXtu5BhQpaf1I2TO7CHTslMgukI7Qv55 xXpZ4lvwmqSFgeduVh+u0T78+t7uPlPHj+qIPJi/4BejHhynBgNYydk0Ql1xQO2uGJpS wiT/7XkQqO1yffwfuZAmmlWN08gQLul+8EqC5npZnD5sFePzwsipyIpIUZ3Jn2nXfw+/ qi5sCoJQhp97LqwRhkYz/RguBr9SbqjyPQTrZzxBG+2XvKMX0NeQKyjKY9kqt17KuCwP 8EsFVb4E+0NiLxzk+MkfnnKbs2jMH3B+zVNz0xOIL+jlwW4eWdBxrlAyjX/5d/ZMsDpe /4fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z9Y20rpLVvo5TBJHmlhFfyIhAn0kzicFV4WKChu5otc=; b=j3d9pK+0iaPO/liJ5GA6avqqD4Z3uTy1jG5IvxdfExasJnZ6hxB0kbm8vQ1kPVlp5e aj2Z7pf+2k1We7iA8n8cZmeSgP8bXzEPXH7rU4l1H/cerzTmY6VOjva/3t9nSSvfNM/m Q4M2JQwJ9cpqm4PPQGx2sfS0vjBhJe76l9ScDomJEhcnpNVVStzZS3+iVp1c5AOo5Nnr cM+2my6+BjRxjC3H/g/xaHMPaB8mHLnBO/gqkKXqkPdPszN6YMM0wjlj1yHYJ1R8N4qE k7r8K1VX5VAV+5485ZAmJf9ny5hUE4WTcq/gnSMCDUvyA+mTgjgeT5+aOq4Pue1YHct8 YVLA== X-Gm-Message-State: ACrzQf0wB24jylltwrjAlmHd8h2YKPtqFIVe3Rp2jFrmodnJRcuvf7qU NfO0AIoHuDD5qjetI4bFK3g12YHGggY= X-Google-Smtp-Source: AMsMyM7s+P+ZhTLHW4fnCpyqJDlvHmS83I3SBnor0krwMUjC9K5stQA7oDM0G8t5X7GVlEmShN9w4Q== X-Received: by 2002:a05:600c:6023:b0:3cf:7dc1:e08e with SMTP id az35-20020a05600c602300b003cf7dc1e08emr6673920wmb.154.1667402377536; Wed, 02 Nov 2022 08:19:37 -0700 (PDT) Received: from 127.0.0.1localhost ([82.132.186.241]) by smtp.gmail.com with ESMTPSA id a14-20020adff7ce000000b0022e66749437sm13043232wrq.93.2022.11.02.08.19.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 08:19:36 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org Cc: kernel-team@fb.com, Pavel Begunkov Subject: [PATCH for-next v4 2/6] bio: don't rob starving biosets of bios Date: Wed, 2 Nov 2022 15:18:20 +0000 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Biosets keep a mempool, so as long as requests complete we can always can allocate and have forward progress. Percpu bio caches break that assumptions as we may complete into the cache of one CPU and after try and fail to allocate with another CPU. We also can't grab from another CPU's cache without tricky sync. If we're allocating with a bio while the mempool is undersaturated, remove REQ_ALLOC_CACHE flag, so on put it will go straight to mempool. It might try to free into mempool more requests than required, but assuming than there is no memory starvation in the system it'll stabilise and never hit that path. Signed-off-by: Pavel Begunkov --- block/bio.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/block/bio.c b/block/bio.c index 57c2f327225b..8afc3e78beff 100644 --- a/block/bio.c +++ b/block/bio.c @@ -526,6 +526,8 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, } if (unlikely(!p)) return NULL; + if (!mempool_is_saturated(&bs->bio_pool)) + opf &= ~REQ_ALLOC_CACHE; bio = p + bs->front_pad; if (nr_vecs > BIO_INLINE_VECS) { From patchwork Wed Nov 2 15:18:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13028279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D10EC43219 for ; Wed, 2 Nov 2022 15:20:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231138AbiKBPUR (ORCPT ); Wed, 2 Nov 2022 11:20:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229875AbiKBPTm (ORCPT ); Wed, 2 Nov 2022 11:19:42 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 648B121809 for ; Wed, 2 Nov 2022 08:19:40 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id k8so25064251wrh.1 for ; Wed, 02 Nov 2022 08:19:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ONzzojRePq37xqITluKuxgMnGfoyFw7UZB2xd8qRqeg=; b=nJOC2eRTqccm5tiEXuF3S74F0fXQIJkraiPb/7wmM3t/+1LVP3RUviGFsLCq020dc5 Y6RS0TZSUDQ2S2mZFKIMuoNsI+otylZqpddVi9EBlm/gS5N1uqn9v4tgExycziBZei1F mC/Osn/firG2s6Sc5sb954PcQDcrDFcbA+EL2ybt5AMF+BiQmypmG/ALn2Jk5BIMp8/H QZb/aFYFolQ3KGaPzc9S1t4kafWti5s4xDXy7MKGo+jXv4EQGe9642k4O/tT9XPMYHb/ qSMqpyhhdVxPdeDmyE1LIu5J57pzrBFUNAGfJCBY1gP1R/GsstcNWtQUENw1VIK5llJo Pf0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ONzzojRePq37xqITluKuxgMnGfoyFw7UZB2xd8qRqeg=; b=5nxHft9QU6seWYcV924cgSleAuSU2YkAd6+DVP/HU7IWxFH86wvnyUyXMZQkhEnxFY w5Wi4FkpWK0vXJMQxiEVoECD5vIltrPgAmNEyNr5fpHMLYa0Gf9Yy1n5GMDsAKzhfdfg UMqZdbAvDg7BXYq2aQ/H/Nhb8xShK7iy81nsLQirS2qynxITUDAGVE9H09Ar6kVfibNj RXjKII7KVD3zTHZKoWLm5rM7ucu6MirqJ+XNI6bHWx3fk+7jR0bZ19yHxC6JdBsSc/OX Jy7FCHyPCXAsFMm9EW377brT9FK+I/Q2j1F0W+EsIglriRrqyvKfL2hcEqCwlIaiBAvO /QWA== X-Gm-Message-State: ACrzQf0Rj3Su8xtRjxL3lWOld2PdIX41mBWRIAIzF1QIxLAgh7AHDI0T IRWl/z7tBg46nHJ1WS1jiuw= X-Google-Smtp-Source: AMsMyM4Ts6+x9ECdoG/fZRQDkLVO5/sgP9xDj5EHV/t80y3TgI+WHEQUHfwgLk4Jt2HpOVs4h1xGIQ== X-Received: by 2002:a5d:6dcd:0:b0:235:f087:fec2 with SMTP id d13-20020a5d6dcd000000b00235f087fec2mr14938340wrz.444.1667402378925; Wed, 02 Nov 2022 08:19:38 -0700 (PDT) Received: from 127.0.0.1localhost ([82.132.186.241]) by smtp.gmail.com with ESMTPSA id a14-20020adff7ce000000b0022e66749437sm13043232wrq.93.2022.11.02.08.19.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 08:19:38 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org Cc: kernel-team@fb.com, Pavel Begunkov Subject: [PATCH for-next v4 3/6] bio: split pcpu cache part of bio_put into a helper Date: Wed, 2 Nov 2022 15:18:21 +0000 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Extract a helper out of bio_put for recycling into percpu caches. It's a preparation patch without functional changes. Signed-off-by: Pavel Begunkov --- block/bio.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/block/bio.c b/block/bio.c index 8afc3e78beff..f99d27566839 100644 --- a/block/bio.c +++ b/block/bio.c @@ -727,6 +727,28 @@ static void bio_alloc_cache_destroy(struct bio_set *bs) bs->cache = NULL; } +static inline void bio_put_percpu_cache(struct bio *bio) +{ + struct bio_alloc_cache *cache; + + cache = per_cpu_ptr(bio->bi_pool->cache, get_cpu()); + bio_uninit(bio); + + if ((bio->bi_opf & REQ_POLLED) && !WARN_ON_ONCE(in_interrupt())) { + bio->bi_next = cache->free_list; + cache->free_list = bio; + cache->nr++; + } else { + put_cpu(); + bio_free(bio); + return; + } + + if (cache->nr > ALLOC_CACHE_MAX + ALLOC_CACHE_SLACK) + bio_alloc_cache_prune(cache, ALLOC_CACHE_SLACK); + put_cpu(); +} + /** * bio_put - release a reference to a bio * @bio: bio to release reference to @@ -742,20 +764,10 @@ void bio_put(struct bio *bio) if (!atomic_dec_and_test(&bio->__bi_cnt)) return; } - - if ((bio->bi_opf & REQ_ALLOC_CACHE) && !WARN_ON_ONCE(in_interrupt())) { - struct bio_alloc_cache *cache; - - bio_uninit(bio); - cache = per_cpu_ptr(bio->bi_pool->cache, get_cpu()); - bio->bi_next = cache->free_list; - cache->free_list = bio; - if (++cache->nr > ALLOC_CACHE_MAX + ALLOC_CACHE_SLACK) - bio_alloc_cache_prune(cache, ALLOC_CACHE_SLACK); - put_cpu(); - } else { + if (bio->bi_opf & REQ_ALLOC_CACHE) + bio_put_percpu_cache(bio); + else bio_free(bio); - } } EXPORT_SYMBOL(bio_put); From patchwork Wed Nov 2 15:18:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13028280 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D329CC43217 for ; Wed, 2 Nov 2022 15:20:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231150AbiKBPUS (ORCPT ); Wed, 2 Nov 2022 11:20:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230303AbiKBPTm (ORCPT ); Wed, 2 Nov 2022 11:19:42 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB22618E35 for ; Wed, 2 Nov 2022 08:19:41 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id g12so25026363wrs.10 for ; Wed, 02 Nov 2022 08:19:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/sXPnj4a9cJY4OJxSEMI079XGw7qQy2cvmQCwAb8viI=; b=Q85W0CcBLrlcP50D+nHEUMuY4/X0PGYXoPtGUBzWXWl/ArV0P4d9fxndcAy8EconoO +PLrdKPACnQPrD113Nng4gwsbFEoiVuy09J8rCB/zAXtHycRMd/AZtSk+l74omaOSADB JR/AKnlRJhRtCjKMHkOcxc6A+8qGNpZaWxQMYPJCeXEJaIfjz6wz0kxs/54CPKrapqOq Dtnz+yugsZcS9eamk+lvvW/jBfEk8G1LgpTuhWU5KlB2ngxtGXKtRghbKLd49aw7UOrb zHAUQbDekfoOl7aIk3tuxSjv3I8Eh0r/bsZ7tvAHV7EojlXPC4BJ1XcrC+CQBX108Gey 136w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/sXPnj4a9cJY4OJxSEMI079XGw7qQy2cvmQCwAb8viI=; b=z0VR+V4dOWGDSQvLINTy4HwyZ9KVhcZkPad3dq7VZtOMOmTg2kA9Nk+Nb+JwHJzJj8 VNP8TETDxHs3sPYoayrdbOU6V/Gim5t/eIbEf0/+c41ki7ftZYaHpnOgUhBZhcIRJNLI QYrYITf9zbb1+LUUOucl6UM9nZPJVgKzCyn9JgE1Z+nwescUy0YvQZtfH03Nq6B2KzXb /T+42kd19T/5g1IdeAsgBUUnlWhpo7pUskqaXxG0wpW9OP8HV45gP2YHLATQKW9e+MiU CIPdUYqFUfnJqgYDGnfzWiwFHHMjLM13GjiRq2rLl59iaE7GEjUY898+wnHdOnYjiz3f t4Nw== X-Gm-Message-State: ACrzQf0amcJAIMx9x0T4NRWyCjnLQ/356saEdj0nB+Z2sHbGxPBbBXZP XCRF97sgBjABmqYp3WJuWqM= X-Google-Smtp-Source: AMsMyM6C44jUz8zG1iiIyL2p37iZd32LcQL2/FL8pkNEucxJG0+tJfRUia8SDZlRIhp6AJfgSn0saw== X-Received: by 2002:a5d:47c7:0:b0:236:64a4:6d5a with SMTP id o7-20020a5d47c7000000b0023664a46d5amr15328841wrc.666.1667402380281; Wed, 02 Nov 2022 08:19:40 -0700 (PDT) Received: from 127.0.0.1localhost ([82.132.186.241]) by smtp.gmail.com with ESMTPSA id a14-20020adff7ce000000b0022e66749437sm13043232wrq.93.2022.11.02.08.19.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 08:19:39 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org Cc: kernel-team@fb.com, Pavel Begunkov Subject: [PATCH for-next v4 4/6] bio: add pcpu caching for non-polling bio_put Date: Wed, 2 Nov 2022 15:18:22 +0000 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org This patch extends REQ_ALLOC_CACHE to IRQ completions, whenever currently it's only limited to iopoll. Instead of guarding the list with irq toggling on alloc, which is expensive, it keeps an additional irq-safe list from which bios are spliced in batches to ammortise overhead. On the put side it toggles irqs, but in many cases they're already disabled and so cheap. Signed-off-by: Pavel Begunkov --- block/bio.c | 70 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 54 insertions(+), 16 deletions(-) diff --git a/block/bio.c b/block/bio.c index f99d27566839..d989e45583ac 100644 --- a/block/bio.c +++ b/block/bio.c @@ -25,9 +25,15 @@ #include "blk-rq-qos.h" #include "blk-cgroup.h" +#define ALLOC_CACHE_THRESHOLD 16 +#define ALLOC_CACHE_SLACK 64 +#define ALLOC_CACHE_MAX 512 + struct bio_alloc_cache { struct bio *free_list; + struct bio *free_list_irq; unsigned int nr; + unsigned int nr_irq; }; static struct biovec_slab { @@ -408,6 +414,22 @@ static void punt_bios_to_rescuer(struct bio_set *bs) queue_work(bs->rescue_workqueue, &bs->rescue_work); } +static void bio_alloc_irq_cache_splice(struct bio_alloc_cache *cache) +{ + unsigned long flags; + + /* cache->free_list must be empty */ + if (WARN_ON_ONCE(cache->free_list)) + return; + + local_irq_save(flags); + cache->free_list = cache->free_list_irq; + cache->free_list_irq = NULL; + cache->nr += cache->nr_irq; + cache->nr_irq = 0; + local_irq_restore(flags); +} + static struct bio *bio_alloc_percpu_cache(struct block_device *bdev, unsigned short nr_vecs, blk_opf_t opf, gfp_t gfp, struct bio_set *bs) @@ -417,8 +439,12 @@ static struct bio *bio_alloc_percpu_cache(struct block_device *bdev, cache = per_cpu_ptr(bs->cache, get_cpu()); if (!cache->free_list) { - put_cpu(); - return NULL; + if (READ_ONCE(cache->nr_irq) >= ALLOC_CACHE_THRESHOLD) + bio_alloc_irq_cache_splice(cache); + if (!cache->free_list) { + put_cpu(); + return NULL; + } } bio = cache->free_list; cache->free_list = bio->bi_next; @@ -462,9 +488,6 @@ static struct bio *bio_alloc_percpu_cache(struct block_device *bdev, * submit_bio_noacct() should be avoided - instead, use bio_set's front_pad * for per bio allocations. * - * If REQ_ALLOC_CACHE is set, the final put of the bio MUST be done from process - * context, not hard/soft IRQ. - * * Returns: Pointer to new bio on success, NULL on failure. */ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, @@ -678,11 +701,8 @@ void guard_bio_eod(struct bio *bio) bio_truncate(bio, maxsector << 9); } -#define ALLOC_CACHE_MAX 512 -#define ALLOC_CACHE_SLACK 64 - -static void bio_alloc_cache_prune(struct bio_alloc_cache *cache, - unsigned int nr) +static int __bio_alloc_cache_prune(struct bio_alloc_cache *cache, + unsigned int nr) { unsigned int i = 0; struct bio *bio; @@ -694,6 +714,17 @@ static void bio_alloc_cache_prune(struct bio_alloc_cache *cache, if (++i == nr) break; } + return i; +} + +static void bio_alloc_cache_prune(struct bio_alloc_cache *cache, + unsigned int nr) +{ + nr -= __bio_alloc_cache_prune(cache, nr); + if (!READ_ONCE(cache->free_list)) { + bio_alloc_irq_cache_splice(cache); + __bio_alloc_cache_prune(cache, nr); + } } static int bio_cpu_dead(unsigned int cpu, struct hlist_node *node) @@ -732,6 +763,12 @@ static inline void bio_put_percpu_cache(struct bio *bio) struct bio_alloc_cache *cache; cache = per_cpu_ptr(bio->bi_pool->cache, get_cpu()); + if (READ_ONCE(cache->nr_irq) + cache->nr > ALLOC_CACHE_MAX) { + put_cpu(); + bio_free(bio); + return; + } + bio_uninit(bio); if ((bio->bi_opf & REQ_POLLED) && !WARN_ON_ONCE(in_interrupt())) { @@ -739,13 +776,14 @@ static inline void bio_put_percpu_cache(struct bio *bio) cache->free_list = bio; cache->nr++; } else { - put_cpu(); - bio_free(bio); - return; - } + unsigned long flags; - if (cache->nr > ALLOC_CACHE_MAX + ALLOC_CACHE_SLACK) - bio_alloc_cache_prune(cache, ALLOC_CACHE_SLACK); + local_irq_save(flags); + bio->bi_next = cache->free_list_irq; + cache->free_list_irq = bio; + cache->nr_irq++; + local_irq_restore(flags); + } put_cpu(); } From patchwork Wed Nov 2 15:18:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13028281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC790C4332F for ; Wed, 2 Nov 2022 15:20:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230345AbiKBPUT (ORCPT ); Wed, 2 Nov 2022 11:20:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230353AbiKBPTp (ORCPT ); Wed, 2 Nov 2022 11:19:45 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7042F24BFA for ; Wed, 2 Nov 2022 08:19:43 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id bs21so25058102wrb.4 for ; Wed, 02 Nov 2022 08:19:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=12WSPtjWIVh7qXdWOGW7CSPnEqHMACEW2HlugAwiwI4=; b=EGzrKLjizyg+Wb0WMbXX6YC8X94uwHc4vswQAVzFwsSRBUt48R1WwSazu6XrpwjJfG qAf3qrQe+kl2eCpyFnXNvWxKXCXJWPOCcPGK/5BkX13WfsfcROLi4ZXJoBcjrYiOV2Kl U4+/6zUWGND2YiF1m7+pp53bo09G/ykP9Ymn7MwVDX+lr7AuDqlPIh0H2HB1RplsOVkG huTVH5i4UtHYCzxDExiSTbJLxRr1llYeBlhBVnt43eSILAA4g+AvVbuVccgR/WZvMX0g thDiwfPZLQ1vr/sfdyT2z2s8rZQXY2JT9dArHD92VgJB0cV3z4T4JyrfKTMuik7uJ+iu wwaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=12WSPtjWIVh7qXdWOGW7CSPnEqHMACEW2HlugAwiwI4=; b=QVESCel0Ys5nNsb725ie4AV/UvQbirSDG/ELcDBISwoRdtR/UgUk0VinnqWvOUE8EB suPLdb3cwqvmoS7xnTr5oi4J3e9Jd184zun4yr11z8NVdpA+rHkLO1iieCBoMN0bD7l1 Fw9FDk6vBvKIHp4U6vGNEHOmPCVXiQd6AbEzW8421jzv1pMZPTw1f1nHwEzWSgPQJyBu az6Sju/QeekwlI9hp7g/PRWWOLZIrptbP2/Cy/8eYThUGhxkJcDp4z+NHA3PIG3hMfqV KTgKFpXPpCkkJBW2RCggalWsO7Ncvt9G4q2OntSvRdw+MSX6K+CcmVWex4F9WfUpPdIF kKqw== X-Gm-Message-State: ACrzQf16sMcTo6+BEGEeLNRKWB0LYxd2vQmpv70KaJCJBjHh1v/P670x 0jUkf5UPORtL3Ta7vaWLc8A= X-Google-Smtp-Source: AMsMyM7Jt123QhX2Yy3qpqB0kFuVji3Gbf4u/851EAj9PzV/ltsUddkJVBQdGMx1ANuVGoR3R78KJA== X-Received: by 2002:a5d:5233:0:b0:236:b893:9d81 with SMTP id i19-20020a5d5233000000b00236b8939d81mr13712590wra.503.1667402381958; Wed, 02 Nov 2022 08:19:41 -0700 (PDT) Received: from 127.0.0.1localhost ([82.132.186.241]) by smtp.gmail.com with ESMTPSA id a14-20020adff7ce000000b0022e66749437sm13043232wrq.93.2022.11.02.08.19.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 08:19:41 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org Cc: kernel-team@fb.com, Pavel Begunkov Subject: [PATCH for-next v4 5/6] bio: shrink max number of pcpu cached bios Date: Wed, 2 Nov 2022 15:18:23 +0000 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The downside of the bio pcpu cache is that bios of a cpu will be never freed unless there is new I/O issued from that cpu. We currently keep max 512 bios, which feels too much, half it. Signed-off-by: Pavel Begunkov --- block/bio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/bio.c b/block/bio.c index d989e45583ac..6277a2f68ab8 100644 --- a/block/bio.c +++ b/block/bio.c @@ -27,7 +27,7 @@ #define ALLOC_CACHE_THRESHOLD 16 #define ALLOC_CACHE_SLACK 64 -#define ALLOC_CACHE_MAX 512 +#define ALLOC_CACHE_MAX 256 struct bio_alloc_cache { struct bio *free_list; From patchwork Wed Nov 2 15:18:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13028282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EB66C4321E for ; Wed, 2 Nov 2022 15:20:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230366AbiKBPUU (ORCPT ); Wed, 2 Nov 2022 11:20:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230369AbiKBPTr (ORCPT ); Wed, 2 Nov 2022 11:19:47 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F80228719 for ; Wed, 2 Nov 2022 08:19:46 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id y16so25013154wrt.12 for ; Wed, 02 Nov 2022 08:19:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oYX5VxLpbP1y6B9vClLwT36AajgL5l3hqA33mKwDa80=; b=Dyvhz/y6n4d8Ycbw/UyTmAyy/UB2ocHmfyo0fIzIwDnsvcEgdEhOUorn0ggeI0ToxS lMKqE+B4+VvFq6OzO3P91IZ9yW2n1g6f/KFW5Y8eyi+yUklJmmXsqiso1IdxsARE3QAJ QmswKil8IZ6AKAHgh28e4ZAkyhmr+lrLnL7OVfeZhVVfmupi7Oae/Tm5298Xa9k1yaeD UEeMh0HFdvAFWmNGcf5dg50LA8dplZR+YVLIZe242JN/sGOiUsHvfah08bzMSObvFHuc FR6elDM/lYaIiU36+FwWq+5AwH3X/vXDgP9FTZBRxx+lXF3wnS8JE+RarirVnqtmlcz1 ClXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oYX5VxLpbP1y6B9vClLwT36AajgL5l3hqA33mKwDa80=; b=ASI1Tyl4Xlc9PjkD0Kk47xhOQZSDws46jjnYWgcxBW+7UHjZyD0JDRk/OkgsJDMoSo piMqcdbbWwB/oqnlwF1Rhe3IjbOIXyqbFOGGLV2SrY9Sj1WMYbLv0zg/KJo9Ueqg40VP PeaX0PPq2JmqoDAYsUCuvY7tEc92wl7yztoP2k+dioWccKe//5F6DUroXwu6xNd0Jmln a92p8tK+x0wFAFpKRO2zyIxa0MWwp/1PkuClC83ou5+TroYbRuWS7TYKbA5WlbotQjww pomcy1Xj+lRIIx2P5eHtjGtrfgDQeiGsgAckLE4zxFpR2Isit7uDWmZmbG0esC1lc+bO gdDw== X-Gm-Message-State: ACrzQf2DOhpLW8SRgqOulqj+BhVfts+TGDTqq2CZCs9msIlOermqstlk 9fJIHiL27o1nCToLUzScjuNoSVK3BeE= X-Google-Smtp-Source: AMsMyM5BSSsNeoQSsriMoKxz+H2CiX+1AiBmPZbQ1SMx2gUn7/uFGk/0UpAtTq8t9Gtr4s5ohHuc+Q== X-Received: by 2002:a05:6000:10a:b0:236:6a79:f5cf with SMTP id o10-20020a056000010a00b002366a79f5cfmr15066385wrx.470.1667402384629; Wed, 02 Nov 2022 08:19:44 -0700 (PDT) Received: from 127.0.0.1localhost ([82.132.186.241]) by smtp.gmail.com with ESMTPSA id a14-20020adff7ce000000b0022e66749437sm13043232wrq.93.2022.11.02.08.19.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 08:19:44 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , linux-block@vger.kernel.org Cc: kernel-team@fb.com, Pavel Begunkov Subject: [PATCH for-next v4 6/6] io_uring/rw: enable bio caches for IRQ rw Date: Wed, 2 Nov 2022 15:18:24 +0000 Message-Id: X-Mailer: git-send-email 2.38.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Now we can use IOCB_ALLOC_CACHE not only for iopoll'ed reads/write but also for normal IRQ driven I/O. Signed-off-by: Pavel Begunkov --- io_uring/rw.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index bb47cc4da713..5c91cc80b348 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -665,6 +665,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode) ret = kiocb_set_rw_flags(kiocb, rw->flags); if (unlikely(ret)) return ret; + kiocb->ki_flags |= IOCB_ALLOC_CACHE; /* * If the file is marked O_NONBLOCK, still allow retry for it if it @@ -680,7 +681,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode) return -EOPNOTSUPP; kiocb->private = NULL; - kiocb->ki_flags |= IOCB_HIPRI | IOCB_ALLOC_CACHE; + kiocb->ki_flags |= IOCB_HIPRI; kiocb->ki_complete = io_complete_rw_iopoll; req->iopoll_completed = 0; } else {