From patchwork Wed Aug 11 19:35:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12431853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E56EC4320E for ; Wed, 11 Aug 2021 19:35:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 772BC6105A for ; Wed, 11 Aug 2021 19:35:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229655AbhHKTgG (ORCPT ); Wed, 11 Aug 2021 15:36:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229946AbhHKTgD (ORCPT ); Wed, 11 Aug 2021 15:36:03 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20196C0613D5 for ; Wed, 11 Aug 2021 12:35:40 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id w14so5169981pjh.5 for ; Wed, 11 Aug 2021 12:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=L7nEJ4dK4R1jH+/WHvia5muthOy9g65ru5z/3wVHRoE=; b=tv9oqAVejrrGokNmOqO0Ms1rxAVj+vm5DTRXkCXDIf4klCeIBkEvJ4v9RTKTYG01WA iaLLB7o99dK5AjUtzEsxBZkzttLdIAjjtSIW+uyTyiwdB41IxGMfkoU4PCSOIHTNA2EK NhbodCJYjx9KFeGQLxfaaPnsGyLRVMVk+hFukuEpiTdpV87HWiUWWzfP/CxKJBCjdiwf IXQFAPHwzhu0JvR12tb3pNmvAhVSq20IdOALpL8AhfCR2ISSGR1Ay5lxUmnwHR83cyec 2Reh+xVaziPbWMdEyAskXuFvzgHUUKcOloNiYL1YRE64KKOtZ22KNrazLzaLn9EElkm/ Qj4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=L7nEJ4dK4R1jH+/WHvia5muthOy9g65ru5z/3wVHRoE=; b=JHgzqpvAhCNJ+RKhXFgkhpkSm1C5JQYuruVrVM3tl7mM6xzjwms6WonR9mjaHlszp7 W7m/A5emtPdWI+djpI4yYinWUAyhQIBBSNPj0LFV+0xO+yvZxar9JUUL+41KFB/9K9NV 99/EEoy1jlM2vOP/82CQdrtlKzjzXAGbpBBvvSiLKAD7/Rcj63PXY/2q5s8evfqmcPEW qF0U6bWZ17ezXya1wywC9pmZQkIYUfiM/zp84k8UC7JcKzoaJiLQPT4qiogYsNnfR2dF K4Dlq4HN4e502jekTsXgD8sdu6ceZCrAX9INP3wtPkiVNq7XBUaacvUQ214C7J9RLNQ3 j+JQ== X-Gm-Message-State: AOAM533EEh4YsvKHVn+7j3NuS4NlsiFu4TbOcouX4z/f5i1gfjhsVu77 u01rNwnw055i/Z4rzdfI34wP0R+WmGEAEcrV X-Google-Smtp-Source: ABdhPJx7HRsnH9aZkRlFR+my4k725bNl8KLhfQEjijU09X1B1Ld8cYqbDcQuwAYKhjXdKQUPm5m19Q== X-Received: by 2002:a17:90a:2c0c:: with SMTP id m12mr153114pjd.107.1628710539670; Wed, 11 Aug 2021 12:35:39 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id u20sm286487pgm.4.2021.08.11.12.35.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 12:35:39 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 1/6] bio: optimize initialization of a bio Date: Wed, 11 Aug 2021 13:35:28 -0600 Message-Id: <20210811193533.766613-2-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210811193533.766613-1-axboe@kernel.dk> References: <20210811193533.766613-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The memset() used is measurably slower in targeted benchmarks. Get rid of it and fill in the bio manually, in a separate helper. Signed-off-by: Jens Axboe --- block/bio.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/block/bio.c b/block/bio.c index 1fab762e079b..0b1025899131 100644 --- a/block/bio.c +++ b/block/bio.c @@ -238,6 +238,35 @@ static void bio_free(struct bio *bio) } } +static inline void __bio_init(struct bio *bio) +{ + bio->bi_next = NULL; + bio->bi_bdev = NULL; + bio->bi_opf = 0; + bio->bi_flags = bio->bi_ioprio = bio->bi_write_hint = 0; + bio->bi_status = 0; + bio->bi_iter.bi_sector = 0; + bio->bi_iter.bi_size = 0; + bio->bi_iter.bi_idx = 0; + bio->bi_iter.bi_bvec_done = 0; + bio->bi_end_io = NULL; + bio->bi_private = NULL; +#ifdef CONFIG_BLK_CGROUP + bio->bi_blkg = NULL; + bio->bi_issue.value = 0; +#ifdef CONFIG_BLK_CGROUP_IOCOST + bio->bi_iocost_cost = 0; +#endif +#endif +#ifdef CONFIG_BLK_INLINE_ENCRYPTION + bio->bi_crypt_context = NULL; +#endif +#ifdef CONFIG_BLK_DEV_INTEGRITY + bio->bi_integrity = NULL; +#endif + bio->bi_vcnt = 0; +} + /* * Users of this function have their own bio allocation. Subsequently, * they must remember to pair any call to bio_init() with bio_uninit() @@ -246,7 +275,7 @@ static void bio_free(struct bio *bio) void bio_init(struct bio *bio, struct bio_vec *table, unsigned short max_vecs) { - memset(bio, 0, sizeof(*bio)); + __bio_init(bio); atomic_set(&bio->__bi_remaining, 1); atomic_set(&bio->__bi_cnt, 1); From patchwork Wed Aug 11 19:35:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12431855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56C57C41537 for ; Wed, 11 Aug 2021 19:35:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2F2596105A for ; Wed, 11 Aug 2021 19:35:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231655AbhHKTgH (ORCPT ); Wed, 11 Aug 2021 15:36:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231403AbhHKTgF (ORCPT ); Wed, 11 Aug 2021 15:36:05 -0400 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A280C061765 for ; Wed, 11 Aug 2021 12:35:41 -0700 (PDT) Received: by mail-pl1-x62b.google.com with SMTP id e15so4030187plh.8 for ; Wed, 11 Aug 2021 12:35:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3qJ0cLXPwzuYEK5Ou0Wuu6u7LZDeLGYslbPHi5c/pHk=; b=SfbYM3OkBemATUFedu3RcmN2KNt+PrN56St6aEdv1p2MLCaQbOwLIQA7l9X3Fp3/PL VNbtizRGhOpevgv9CFl9hmXh+McvLeafA3DvzphN/LRlEOuLOYpEDVxxfj4fqwICpaWL mFur6X+dISuCkMsVCvHSDO5q7U7nqlJUxDmjs9G068CVZJYIL9c3qvXeRsvt1FYvfm/t OQO43sqweqGWLQD8/M930uAPaGl3xd0fetGWLU9KJcDIxmizS6CWGqaoqkLYUl4Ch6LJ jTGMfGbY2Tn4zdnYOe28+MnGtHSvSDZx/4MseKVbTNkYJ0enGK37QbxyC2ZupUwzx/dZ /tNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3qJ0cLXPwzuYEK5Ou0Wuu6u7LZDeLGYslbPHi5c/pHk=; b=N09TIt8yThpMIVX86BdV7B1LK9kIFWS65t6wR/cqKVY4bh/j4Rzuaajr8GyNxTLV1E EYQVEhUkqu+C/hhGBhn13lPpd8ugU6CPLRfT60Wve5Fu4KOc+31Z/inOPow18oI80ioM 1DXb4RLvh3+6IzIChvcPX0LsgI7HQXrB4B+ZA76gypYDUz7qz52wHn5WmUntcgSF00F2 orWEkntNXVIXb6VWZLAJuOjq9dimQLWbsYuYuZ9y6ZaAk7R2g6B5XHV9tYE29sLXMhpr /8KspA+XwzDMmZBDykgSZrd3Vpna0agUvmhsf+bI3GGNq6K4AGcBvMi2ehNqMS0Bh0OH Omlg== X-Gm-Message-State: AOAM532muSzLrXfVVdbRaopGLmtdGoWvoMS+Gjk+p3c19JrtIpgA69Bl vfe6J0xIN/b7nxo6lY5cBBsmPA== X-Google-Smtp-Source: ABdhPJw+pjLBfJhRyRCDOoT+oXV+bgpnlxCfXS+YcKGx6KPwhG3u9kPWOt3xTeufIVtCKF26mC90rg== X-Received: by 2002:a17:90b:f10:: with SMTP id br16mr3171552pjb.65.1628710540759; Wed, 11 Aug 2021 12:35:40 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id u20sm286487pgm.4.2021.08.11.12.35.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 12:35:40 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 2/6] fs: add kiocb alloc cache flag Date: Wed, 11 Aug 2021 13:35:29 -0600 Message-Id: <20210811193533.766613-3-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210811193533.766613-1-axboe@kernel.dk> References: <20210811193533.766613-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If this kiocb can safely use the polled bio allocation cache, then this flag must be set. Signed-off-by: Jens Axboe --- include/linux/fs.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index 640574294216..0dcc5de779c9 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -319,6 +319,8 @@ enum rw_hint { /* iocb->ki_waitq is valid */ #define IOCB_WAITQ (1 << 19) #define IOCB_NOIO (1 << 20) +/* can use bio alloc cache */ +#define IOCB_ALLOC_CACHE (1 << 21) struct kiocb { struct file *ki_filp; From patchwork Wed Aug 11 19:35:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12431857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91C34C00143 for ; Wed, 11 Aug 2021 19:35:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6919F60F21 for ; Wed, 11 Aug 2021 19:35:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231553AbhHKTgI (ORCPT ); Wed, 11 Aug 2021 15:36:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231585AbhHKTgG (ORCPT ); Wed, 11 Aug 2021 15:36:06 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35598C06179B for ; Wed, 11 Aug 2021 12:35:42 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id t3so4029064plg.9 for ; Wed, 11 Aug 2021 12:35:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tDPWk1yBG9VXyK3+TgNLLGFSPQaPVX6OpfLB0cZ8UbY=; b=N2SQk0jWGzefn7ZryjJ+8B+QR0pYyx4832XiD6ixNsbDDZCpt6DYBxR2dxyldJKzdY VCB9KfHGLTD0aJUvMCZR9U8AnmMFBAHxvDfp3OQoK4r+I/b5T4sZbd5UUu5V2Dd8+DD7 hJD2LKuwpYPXzdp3a3/L1Gc/VFh7W5u4+IsCaqDczxooSruYkJL6Qunr3HipCQhbNSXc 6sVmdNmH989spY+tC4SxRFGmxnlSMMZbnPhwLdPiCuipIUiire6Qomwz7nbwD/kQKbTK EerDk1VUGMtqWIx6p/Q0do7egCZWgsrR1CUGdhvSATQd1nH7mJkFIqK9L1r0+WVHYl11 G9Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tDPWk1yBG9VXyK3+TgNLLGFSPQaPVX6OpfLB0cZ8UbY=; b=ARB4Vy/epsvIL+wgPfYcAYLJGG0SZKg/GbEhCFvg6gjIrD2mOpI2n6+0DbmYW8FBBN iRjaXl04KmB/Q57kHvVuxnLRs3oifla7imJj1eDeJklm6pBZcegxlxptAMZA1CtbTjKo +NUtB3nF2WF2YBS50KAcFGZ4EWXiQ+6RtztojR79GsDamTTb73YJuREMafoNhi0I6apB qauE8N8N2fOlb8DhsT4atfQSM3+fxu3OpdUbSkFuEa5M/B2RhyPHwhiPKTlFalzS0H5m ghSnIrYuNBnXaTN2T2C6cV+aMI54Q21aULo5GcdUzeAr3B8aF0XkibT6KNFXRfNtbeU5 2oUA== X-Gm-Message-State: AOAM531yC6Tn9DGwvY5x7GBtdAeumnZLivtKCJH/8D3EHkLAGX+Kk2el Zla34sWLrfRwVp5F7MluFosLnA== X-Google-Smtp-Source: ABdhPJww6eH4MuL0ZbF6qoK+twZet8gFfzBwTPEE1tkzFOCe9g9QtblnV8z3DFLRqEzevVBtsuizRg== X-Received: by 2002:a63:e116:: with SMTP id z22mr278412pgh.361.1628710541727; Wed, 11 Aug 2021 12:35:41 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id u20sm286487pgm.4.2021.08.11.12.35.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 12:35:41 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 3/6] bio: add allocation cache abstraction Date: Wed, 11 Aug 2021 13:35:30 -0600 Message-Id: <20210811193533.766613-4-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210811193533.766613-1-axboe@kernel.dk> References: <20210811193533.766613-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add a per-cpu bio_set cache for bio allocations, enabling us to quickly recycle them instead of going through the slab allocator. This cache isn't IRQ safe, and hence is only really suitable for polled IO. Very simple - keeps a count of bio's in the cache, and maintains a max of 512 with a slack of 64. If we get above max + slack, we drop slack number of bio's. Signed-off-by: Jens Axboe --- block/bio.c | 139 +++++++++++++++++++++++++++++++++---- include/linux/bio.h | 23 ++++-- include/linux/blk_types.h | 1 + include/linux/cpuhotplug.h | 1 + 4 files changed, 144 insertions(+), 20 deletions(-) diff --git a/block/bio.c b/block/bio.c index 0b1025899131..689335c00937 100644 --- a/block/bio.c +++ b/block/bio.c @@ -25,6 +25,11 @@ #include "blk.h" #include "blk-rq-qos.h" +struct bio_alloc_cache { + struct bio_list free_list; + unsigned int nr; +}; + static struct biovec_slab { int nr_vecs; char *name; @@ -620,6 +625,69 @@ void guard_bio_eod(struct bio *bio) bio_truncate(bio, maxsector << 9); } +#define ALLOC_CACHE_MAX 512 +#define ALLOC_CACHE_SLACK 64 + +static void bio_alloc_cache_prune(struct bio_alloc_cache *cache, + unsigned int nr) +{ + struct bio *bio; + unsigned int i; + + i = 0; + while ((bio = bio_list_pop(&cache->free_list)) != NULL) { + cache->nr--; + bio_free(bio); + if (++i == nr) + break; + } +} + +static int bio_cpu_dead(unsigned int cpu, struct hlist_node *node) +{ + struct bio_set *bs; + + bs = hlist_entry_safe(node, struct bio_set, cpuhp_dead); + if (bs->cache) { + struct bio_alloc_cache *cache = per_cpu_ptr(bs->cache, cpu); + + bio_alloc_cache_prune(cache, -1U); + } + return 0; +} + +static void bio_alloc_cache_destroy(struct bio_set *bs) +{ + int cpu; + + if (!bs->cache) + return; + + preempt_disable(); + cpuhp_state_remove_instance_nocalls(CPUHP_BIO_DEAD, &bs->cpuhp_dead); + for_each_possible_cpu(cpu) { + struct bio_alloc_cache *cache; + + cache = per_cpu_ptr(bs->cache, cpu); + bio_alloc_cache_prune(cache, -1U); + } + preempt_enable(); + free_percpu(bs->cache); +} + +static inline bool __bio_put(struct bio *bio) +{ + if (!bio_flagged(bio, BIO_REFFED)) + return true; + + BIO_BUG_ON(!atomic_read(&bio->__bi_cnt)); + + /* + * last put frees it + */ + return atomic_dec_and_test(&bio->__bi_cnt); +} + /** * bio_put - release a reference to a bio * @bio: bio to release reference to @@ -630,16 +698,21 @@ void guard_bio_eod(struct bio *bio) **/ void bio_put(struct bio *bio) { - if (!bio_flagged(bio, BIO_REFFED)) - bio_free(bio); - else { - BIO_BUG_ON(!atomic_read(&bio->__bi_cnt)); + if (unlikely(!__bio_put(bio))) + return; - /* - * last put frees it - */ - if (atomic_dec_and_test(&bio->__bi_cnt)) - bio_free(bio); + if (bio_flagged(bio, BIO_PERCPU_CACHE)) { + struct bio_alloc_cache *cache; + + bio_uninit(bio); + cache = per_cpu_ptr(bio->bi_pool->cache, get_cpu()); + bio_list_add_head(&cache->free_list, bio); + cache->nr++; + if (cache->nr > ALLOC_CACHE_MAX + ALLOC_CACHE_SLACK) + bio_alloc_cache_prune(cache, ALLOC_CACHE_SLACK); + put_cpu(); + } else { + bio_free(bio); } } EXPORT_SYMBOL(bio_put); @@ -1531,6 +1604,7 @@ int biovec_init_pool(mempool_t *pool, int pool_entries) */ void bioset_exit(struct bio_set *bs) { + bio_alloc_cache_destroy(bs); if (bs->rescue_workqueue) destroy_workqueue(bs->rescue_workqueue); bs->rescue_workqueue = NULL; @@ -1592,12 +1666,18 @@ int bioset_init(struct bio_set *bs, biovec_init_pool(&bs->bvec_pool, pool_size)) goto bad; - if (!(flags & BIOSET_NEED_RESCUER)) - return 0; - - bs->rescue_workqueue = alloc_workqueue("bioset", WQ_MEM_RECLAIM, 0); - if (!bs->rescue_workqueue) - goto bad; + if (flags & BIOSET_NEED_RESCUER) { + bs->rescue_workqueue = alloc_workqueue("bioset", + WQ_MEM_RECLAIM, 0); + if (!bs->rescue_workqueue) + goto bad; + } + if (flags & BIOSET_PERCPU_CACHE) { + bs->cache = alloc_percpu(struct bio_alloc_cache); + if (!bs->cache) + goto bad; + cpuhp_state_add_instance_nocalls(CPUHP_BIO_DEAD, &bs->cpuhp_dead); + } return 0; bad: @@ -1624,6 +1704,32 @@ int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) } EXPORT_SYMBOL(bioset_init_from_src); +struct bio *bio_alloc_kiocb(struct kiocb *kiocb, gfp_t gfp, + unsigned short nr_vecs, struct bio_set *bs) +{ + struct bio_alloc_cache *cache = NULL; + struct bio *bio; + + if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) || nr_vecs > BIO_INLINE_VECS) + goto normal_alloc; + + cache = per_cpu_ptr(bs->cache, get_cpu()); + bio = bio_list_pop(&cache->free_list); + if (bio) { + cache->nr--; + put_cpu(); + bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs); + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; + } + put_cpu(); +normal_alloc: + bio = bio_alloc_bioset(gfp, nr_vecs, bs); + if (cache) + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; +} + static int __init init_bio(void) { int i; @@ -1638,6 +1744,9 @@ static int __init init_bio(void) SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL); } + cpuhp_setup_state_multi(CPUHP_BIO_DEAD, "block/bio:dead", NULL, + bio_cpu_dead); + if (bioset_init(&fs_bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS)) panic("bio: can't allocate bios\n"); diff --git a/include/linux/bio.h b/include/linux/bio.h index 2203b686e1f0..5f4a741ee97d 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -401,6 +401,7 @@ static inline struct bio *bio_next_split(struct bio *bio, int sectors, enum { BIOSET_NEED_BVECS = BIT(0), BIOSET_NEED_RESCUER = BIT(1), + BIOSET_PERCPU_CACHE = BIT(2), }; extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags); extern void bioset_exit(struct bio_set *); @@ -409,6 +410,8 @@ extern int bioset_init_from_src(struct bio_set *bs, struct bio_set *src); struct bio *bio_alloc_bioset(gfp_t gfp, unsigned short nr_iovecs, struct bio_set *bs); +struct bio *bio_alloc_kiocb(struct kiocb *kiocb, gfp_t, unsigned short nr_vecs, + struct bio_set *bs); struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned short nr_iovecs); extern void bio_put(struct bio *); @@ -652,18 +655,22 @@ static inline struct bio *bio_list_peek(struct bio_list *bl) return bl->head; } -static inline struct bio *bio_list_pop(struct bio_list *bl) +static inline void bio_list_del_head(struct bio_list *bl, struct bio *head) { - struct bio *bio = bl->head; - - if (bio) { + if (head) { bl->head = bl->head->bi_next; if (!bl->head) bl->tail = NULL; - bio->bi_next = NULL; + head->bi_next = NULL; } +} + +static inline struct bio *bio_list_pop(struct bio_list *bl) +{ + struct bio *bio = bl->head; + bio_list_del_head(bl, bio); return bio; } @@ -699,6 +706,12 @@ struct bio_set { struct kmem_cache *bio_slab; unsigned int front_pad; + /* + * per-cpu bio alloc cache and notifier + */ + struct bio_alloc_cache __percpu *cache; + struct hlist_node cpuhp_dead; + mempool_t bio_pool; mempool_t bvec_pool; #if defined(CONFIG_BLK_DEV_INTEGRITY) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 290f9061b29a..f68d4e8c775e 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -301,6 +301,7 @@ enum { BIO_TRACKED, /* set if bio goes through the rq_qos path */ BIO_REMAPPED, BIO_ZONE_WRITE_LOCKED, /* Owns a zoned device zone write lock */ + BIO_PERCPU_CACHE, /* can participate in per-cpu alloc cache */ BIO_FLAG_LAST }; diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index f39b34b13871..fe72c8d6c980 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -46,6 +46,7 @@ enum cpuhp_state { CPUHP_ARM_OMAP_WAKE_DEAD, CPUHP_IRQ_POLL_DEAD, CPUHP_BLOCK_SOFTIRQ_DEAD, + CPUHP_BIO_DEAD, CPUHP_ACPI_CPUDRV_DEAD, CPUHP_S390_PFAULT_DEAD, CPUHP_BLK_MQ_DEAD, From patchwork Wed Aug 11 19:35:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12431861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84894C432BE for ; Wed, 11 Aug 2021 19:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 58F11610FC for ; Wed, 11 Aug 2021 19:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231741AbhHKTgJ (ORCPT ); Wed, 11 Aug 2021 15:36:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229802AbhHKTgH (ORCPT ); Wed, 11 Aug 2021 15:36:07 -0400 Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12D10C06179E for ; Wed, 11 Aug 2021 12:35:43 -0700 (PDT) Received: by mail-pj1-x102b.google.com with SMTP id lw7-20020a17090b1807b029017881cc80b7so11299291pjb.3 for ; Wed, 11 Aug 2021 12:35:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gzAcF0zizY0wVeW39ScMbIL+yu4HjhIlQYH8a7v47yo=; b=UP4vySG67B3mWmew1yhsxprWgt1py2v7fnkkpmlxAWoNyfd6VwiwQHfDuPyJBCAT9y CpW0eU4AV0iEKcg7DA7lLDWAlVS3hCZPLjRJx/hdBaiGn2KKYJ3jMGu5fBeSlT73oeDI 4Uro6cMsRsvkQZeamniiMreRLD7wwy1ru1ytSfqrZOnxKmlFsrq+O+0TyHopycKoZOaL JCSihSz+y22bXA/DJ/kRXPF7qkAWYUHfwv9HY4gjILL+NY1Qy7FpKL9Uaa7GShm6n8+R ZzdAY3AMgCLXa3GT/y5g87RcUXs+Onz6aOnsLzBQqwuU/ld5FBkzvCd+V8aGkZGi/XJd 1Feg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gzAcF0zizY0wVeW39ScMbIL+yu4HjhIlQYH8a7v47yo=; b=osiTfpczs0WgLDh9JjhyGS/6uddArYigmJvrIvL68TT3JHRSjIYfT1gOE4Emm/paqL lHwm25dRIXsv4p61H7g/GQtsUV0GYBQAIIIX5W7QtkMf+hCTGtmWzv8i8EfO4ubn5F34 PMxnTe3O1seznC6Wp+Pmbt1cASDkTms7vNoViUapWfC2CI+ilj04lM9m2I1D9Jac1myZ HydXyAUe2qHr+q10UCLes92iUVNlNORh+cS/8Shhf9bvA+6ByVxMZQr5YeFSqpGSBWqw 0QI+rv04ZEFYjvsK+d/wsZn6WP61pmdakhoXqM6L6lMe7XVvc2kzga8IHDKTTZmqjskV XBUg== X-Gm-Message-State: AOAM533e17D9kFR7t6dhwElIv5WIYeip2nMf0HLltTLk5SNN/IkM/+TJ YnPNii/onorb4eArCw8SkrHHDg== X-Google-Smtp-Source: ABdhPJyLozS2daFM9HNJ9WFZc7F7zdNlB81nDYSLw4lvsHkDusDtRKe6nUWUP09eLh+8xl4WDH8wZg== X-Received: by 2002:a05:6a00:18a3:b029:3dd:8ade:9b8c with SMTP id x35-20020a056a0018a3b02903dd8ade9b8cmr309666pfh.63.1628710542623; Wed, 11 Aug 2021 12:35:42 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id u20sm286487pgm.4.2021.08.11.12.35.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 12:35:42 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 4/6] block: clear BIO_PERCPU_CACHE flag if polling isn't supported Date: Wed, 11 Aug 2021 13:35:31 -0600 Message-Id: <20210811193533.766613-5-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210811193533.766613-1-axboe@kernel.dk> References: <20210811193533.766613-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The bio alloc cache relies on the fact that a polled bio will complete in process context, clear the cacheable flag if we disable polling for a given bio. Signed-off-by: Jens Axboe --- block/blk-core.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/block/blk-core.c b/block/blk-core.c index 04477697ee4b..c130206e9961 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -833,8 +833,11 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } - if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) { + /* can't support alloc cache if we turn off polling */ + bio_clear_flag(bio, BIO_PERCPU_CACHE); bio->bi_opf &= ~REQ_HIPRI; + } switch (bio_op(bio)) { case REQ_OP_DISCARD: From patchwork Wed Aug 11 19:35:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12431859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73AB1C00144 for ; Wed, 11 Aug 2021 19:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3BB4C61102 for ; Wed, 11 Aug 2021 19:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231565AbhHKTgJ (ORCPT ); Wed, 11 Aug 2021 15:36:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231636AbhHKTgH (ORCPT ); Wed, 11 Aug 2021 15:36:07 -0400 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08265C061765 for ; Wed, 11 Aug 2021 12:35:44 -0700 (PDT) Received: by mail-pj1-x1034.google.com with SMTP id nt11so5223230pjb.2 for ; Wed, 11 Aug 2021 12:35:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oEanWYqex9JuH3iGdmfjQ00rfPxW60wGQ1NzD9wjGSo=; b=o2n0uXw2A1dkovhZfrRJwk7J04SXPUMV78vc0FOMnSsnuVtqCj2WcPC/Iqmc7x5FwX oann7w01bIooqFyKR1mliZ2wecJ5eTq9TTgSZtiz9Q7oiyIqPG99Hwfl96nNiaONwI4G A+7KOq2RIGqH3RNu30ppI/lDnVw/TQW6olt0IclIfRXmgnVa1xiESGUfpjTBIhjasuLi wS9nLuibcY/fgmtX3wTgsJ0pwhgHwHqU12cySJgucpQkYJzCIsGyCvwbUtr+ipid3t2t DI0GoM/p4Xh9FibPXdpBn0UIYejaoBS+SFQ7OF2Yzdf01y1fGKM/tlkaQXCB/i2bLI5U XxiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oEanWYqex9JuH3iGdmfjQ00rfPxW60wGQ1NzD9wjGSo=; b=HbUgbfOoaGSwn+R7CUKHcVWJ8edMNf05RI8tkoIKA+O4J5IZm2wSgARuGPIDca1aKJ aOeVIIuleNFZIkTFbfSIFbalb7xs+LzG+tO+P/DeYjBeLurvEIJeGe6luYYghRYHsNPW IcBX9oaYdP78kv7nBhnyg11a50hJVoaOmFkgbqv9NQjesW5kvR3VqRR60LQx8hknn3YS evRZ210xPe4sIDVpfEWcJ7oR8A0J8zdW7O68QDVuW13rcugNm79CcTzV1haojiayy2jc Y+8R7xX749clvNOE0bf2E55g/esDJLTrUvqACuiLZ9G6kVJkZXuzubGz8FDIWfTm9x8F ajKA== X-Gm-Message-State: AOAM5306yfYfSlxgundEHXEK3Lgplcyog6t6jBPtC7sjfOK7ZM39n0hK DkS6b8B1eKyCUR9O70c8r4ZGBQ== X-Google-Smtp-Source: ABdhPJwNzpEpb02gfUBt/Jh6uj4oiG5Hz4tPIqqmCtxgyRm82dQwWXlf5CU0LGyPf3PYJFQf4b8e8w== X-Received: by 2002:a05:6a00:1596:b029:3c7:998a:709 with SMTP id u22-20020a056a001596b02903c7998a0709mr399865pfk.68.1628710543569; Wed, 11 Aug 2021 12:35:43 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id u20sm286487pgm.4.2021.08.11.12.35.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 12:35:43 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 5/6] io_uring: enable use of bio alloc cache Date: Wed, 11 Aug 2021 13:35:32 -0600 Message-Id: <20210811193533.766613-6-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210811193533.766613-1-axboe@kernel.dk> References: <20210811193533.766613-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Mark polled IO as being safe for dipping into the bio allocation cache, in case the targeted bio_set has it enabled. This brings an IOPOLL gen2 Optane QD=128 workload from ~3.0M IOPS to ~3.3M IOPS. Signed-off-by: Jens Axboe --- fs/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index bd3f8529fe6f..00da7bdd2b53 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2667,7 +2667,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) !kiocb->ki_filp->f_op->iopoll) return -EOPNOTSUPP; - kiocb->ki_flags |= IOCB_HIPRI; + kiocb->ki_flags |= IOCB_HIPRI | IOCB_ALLOC_CACHE; kiocb->ki_complete = io_complete_rw_iopoll; req->iopoll_completed = 0; } else { From patchwork Wed Aug 11 19:35:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12431863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A35DC19F35 for ; Wed, 11 Aug 2021 19:35:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6BAF0610A4 for ; Wed, 11 Aug 2021 19:35:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231747AbhHKTgK (ORCPT ); Wed, 11 Aug 2021 15:36:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231702AbhHKTgJ (ORCPT ); Wed, 11 Aug 2021 15:36:09 -0400 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 285C3C061765 for ; Wed, 11 Aug 2021 12:35:45 -0700 (PDT) Received: by mail-pl1-x629.google.com with SMTP id e19so4019696pla.10 for ; Wed, 11 Aug 2021 12:35:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CPHR9+/srqubnx1IylV7gQnOmmOqOlKCpGfW9/b0t68=; b=Gn68Gv0X+76XfYrCArMrLcLPqzknELSJKAuyrHdrGzix21DE/eqSRfrWSFN4SWJaCg +JMXaKuFKpMGx0jUV7s64y8UY/0n+9/3xaN2ZQpxjcDgj7wm/KsXdfFfAZO+F0bfMRKy YDZQcSKvqvXYeTmYdAl9KshM1rtEGzvRC4Ad9QY6QlH9tZCW6qUP/Y0gVNCDlrNOX63g 2Q1sUqPz1wd0krTQK5B4gR21aJRh6IFREdrxprE1LuFXzLUVwHpLwymHVOgNn26huGy3 M1iqqJ+JlVpZmsyRCYGmV2xRtaG4AyZiElS1T3Ld4n+Q0OBx2nb9arZGaOyAmOwwDYvj exXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CPHR9+/srqubnx1IylV7gQnOmmOqOlKCpGfW9/b0t68=; b=B1TALRZj4ObZ5d4cHxwHIf3eHrHJqg/S8Igwf6/P9Y/phHlFDMUaYItrl2B/rXM2LP n1jXnMQnxleIpHa/3F2KU0rfcU9m251UpDxUzMrK46/R6VnMjjY92kGgl0jqE52eLtdX ucRREz/F0EbckT193OJ269TylDPu710yGMGOlEmjSuAdSqNvGHjU/Pu0ZdyaVCHpX/8x TgJ4FS2HrpX0icWmk5Ds3AoaBGRKEb8RXbaIx8fduuPhSexZYr8mPKIUw3e8YDJ8GkHz YWA5h0S0kn4meaKJ2ctw0JZqzvYhusbmTL2psqVP8so41VVF7C/IZ81cD5AyAJMUUw32 H/MA== X-Gm-Message-State: AOAM5312JYw4h8yUlR91um1QTMsCnrmW8zDim6V0KqkUUZoC5UznIJax bQHiLqkpGnQg+50FPb/caOXaVw== X-Google-Smtp-Source: ABdhPJwtoG2aWyj8p2ouXV0qKbPEkI2k8GUibvKupEVpZmlq5szola+EWrLsdd7T7R6joLqN3r5/lg== X-Received: by 2002:a05:6a00:888:b029:3c3:ff1:38e with SMTP id q8-20020a056a000888b02903c30ff1038emr317565pfj.17.1628710544550; Wed, 11 Aug 2021 12:35:44 -0700 (PDT) Received: from localhost.localdomain ([66.219.217.159]) by smtp.gmail.com with ESMTPSA id u20sm286487pgm.4.2021.08.11.12.35.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 12:35:44 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 6/6] block: enable use of bio allocation cache Date: Wed, 11 Aug 2021 13:35:33 -0600 Message-Id: <20210811193533.766613-7-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210811193533.766613-1-axboe@kernel.dk> References: <20210811193533.766613-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Initialize the bio_set used for IO with per-cpu bio caching enabled, and use the new bio_alloc_kiocb() helper to dip into that cache. Signed-off-by: Jens Axboe --- fs/block_dev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 9ef4f1fc2cb0..798bb9d8f533 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -385,7 +385,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, (bdev_logical_block_size(bdev) - 1)) return -EINVAL; - bio = bio_alloc_bioset(GFP_KERNEL, nr_pages, &blkdev_dio_pool); + bio = bio_alloc_kiocb(iocb, GFP_KERNEL, nr_pages, &blkdev_dio_pool); dio = container_of(bio, struct blkdev_dio, bio); dio->is_sync = is_sync = is_sync_kiocb(iocb); @@ -513,7 +513,9 @@ blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) static __init int blkdev_init(void) { - return bioset_init(&blkdev_dio_pool, 4, offsetof(struct blkdev_dio, bio), BIOSET_NEED_BVECS); + return bioset_init(&blkdev_dio_pool, 4, + offsetof(struct blkdev_dio, bio), + BIOSET_NEED_BVECS|BIOSET_PERCPU_CACHE); } module_init(blkdev_init);