From patchwork Thu Aug 12 15:41:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12433809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC4D9C4320A for ; Thu, 12 Aug 2021 15:41:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 93C4260D07 for ; Thu, 12 Aug 2021 15:41:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238195AbhHLPmT (ORCPT ); Thu, 12 Aug 2021 11:42:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237375AbhHLPmS (ORCPT ); Thu, 12 Aug 2021 11:42:18 -0400 Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 656CCC061756 for ; Thu, 12 Aug 2021 08:41:53 -0700 (PDT) Received: by mail-ot1-x333.google.com with SMTP id c2-20020a0568303482b029048bcf4c6bd9so8242242otu.8 for ; Thu, 12 Aug 2021 08:41:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mwde66FBJEayMjWLFoOXKXeLTWGiokZGl4UEeh431cU=; b=X+OAoTNalGpdjv+G+zkze8hvEjnVKfAX2xQiw6waJIXyUSRk333/Pz8Sl37J0t+2Hk a6obqFMEOhEhjPzvFgVKWnqd3DeCm4xQdgRJz8b27KbtbEYi9v2jWUqQKR2HMccD/MVP xKMz1EqyaHCnoih7URWAdTrH8cSmaF4uMwh/kGqhybTAa8CURx3NuFbYLcO/FZh2y+4i daIB9Ks2N7cVuuh26dPDjROvoexIyAjQw7DUzQLEDFmvNUsjgJzXNIaH3IDjPfYE8ywb EniSPSCbS5Vyv2tsi6D/ahIPtRgaOjT0AcmDC/0zGuPOSMWRP7k8bKmF2oLQWqWqrSPV lc+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mwde66FBJEayMjWLFoOXKXeLTWGiokZGl4UEeh431cU=; b=elHa5u3uj4m3XFQVgpZ/ki7Hzg6lFFXSx29Wwgsk741oEjORo5HUS9jgVyzId3GKKl WI1E0s5QciMKDFSv9o0pfL+wGmZvuqdd3SuKWeMzgHDmKN/PlQjK4vT9VfXxd1oMHImI F71I2cdeMY+imbU5VjEj3NMYK6+CPUmrLPFqezZxGFbsUFzI5pFMcDXRykbeltmd1GzB eOCTEzgY7p2/OZZ0N6f/+EGGovTgEFqQXLhB72K2J6nGmv1RDECE7GxfcDddXDw1z5Hg Zs3RLbD4Hov8dNBNQ6DTktSsnIckkkmjZZ6mG0dJaEuVZ2MKH017UKsuu1/w8sq4GWiT 6HQQ== X-Gm-Message-State: AOAM53057pcmxbikbozvEJmOyS0jHeW819nOCo5FYxYTIhcubVDJxiRG T/lcSjcBT3kqeY8nr17VKpTDNiRKZ77ck308 X-Google-Smtp-Source: ABdhPJxIWJhHum+jAgMMJAb5jKpUBzhFTmpVFoJzX3tyoiY7z63wOlsHDp9NX8kLw/HQvJ3ouWQVPw== X-Received: by 2002:a9d:2de1:: with SMTP id g88mr4022908otb.84.1628782912735; Thu, 12 Aug 2021 08:41:52 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id w16sm690973oih.19.2021.08.12.08.41.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Aug 2021 08:41:52 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 1/6] bio: optimize initialization of a bio Date: Thu, 12 Aug 2021 09:41:44 -0600 Message-Id: <20210812154149.1061502-2-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210812154149.1061502-1-axboe@kernel.dk> References: <20210812154149.1061502-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The memset() used is measurably slower in targeted benchmarks, wasting about 1% of the total runtime, or 50% of the (later) hot path cached bio alloc. Get rid of it and fill in the bio manually. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/bio.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/block/bio.c b/block/bio.c index 1fab762e079b..9bf98b877aba 100644 --- a/block/bio.c +++ b/block/bio.c @@ -246,7 +246,34 @@ static void bio_free(struct bio *bio) void bio_init(struct bio *bio, struct bio_vec *table, unsigned short max_vecs) { - memset(bio, 0, sizeof(*bio)); + bio->bi_next = NULL; + bio->bi_bdev = NULL; + bio->bi_opf = 0; + bio->bi_flags = 0; + bio->bi_ioprio = 0; + bio->bi_write_hint = 0; + bio->bi_status = 0; + bio->bi_iter.bi_sector = 0; + bio->bi_iter.bi_size = 0; + bio->bi_iter.bi_idx = 0; + bio->bi_iter.bi_bvec_done = 0; + bio->bi_end_io = NULL; + bio->bi_private = NULL; +#ifdef CONFIG_BLK_CGROUP + bio->bi_blkg = NULL; + bio->bi_issue.value = 0; +#ifdef CONFIG_BLK_CGROUP_IOCOST + bio->bi_iocost_cost = 0; +#endif +#endif +#ifdef CONFIG_BLK_INLINE_ENCRYPTION + bio->bi_crypt_context = NULL; +#endif +#ifdef CONFIG_BLK_DEV_INTEGRITY + bio->bi_integrity = NULL; +#endif + bio->bi_vcnt = 0; + atomic_set(&bio->__bi_remaining, 1); atomic_set(&bio->__bi_cnt, 1); From patchwork Thu Aug 12 15:41:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12433807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66522C43214 for ; Thu, 12 Aug 2021 15:41:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4D44060F91 for ; Thu, 12 Aug 2021 15:41:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238168AbhHLPmT (ORCPT ); Thu, 12 Aug 2021 11:42:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238199AbhHLPmT (ORCPT ); Thu, 12 Aug 2021 11:42:19 -0400 Received: from mail-oi1-x230.google.com (mail-oi1-x230.google.com [IPv6:2607:f8b0:4864:20::230]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 221C3C0613D9 for ; Thu, 12 Aug 2021 08:41:54 -0700 (PDT) Received: by mail-oi1-x230.google.com with SMTP id bi32so11031306oib.2 for ; Thu, 12 Aug 2021 08:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LEnopQL4PHusgBDS3f+zKlQOKB9hsKjgx1LSo73cC14=; b=jPHTzAiTHc6FoNV++PtWwTTZ7W0ltfV924DY4oBUi4/BbGIr159yPRx/mDJpIKoA8+ nsZjIHxaD0PBYr11WBs92pL3U8hSE7WeNOnCT97KpAZRVQD3eG8Fd6+DZOebn5Rv74jj f0BfuuGsugOprSHQxTWmfoS8tJ4a3apiLlIPiBOlnpZ8nOWIJ9JPqBrQDFuEfTuJCi+f rpbWbHxrJQBnCuqkUqQYWhIkh5lmLyYEqvRDX1MLAjASwM6yP3YKUzXAPW7ztEPK/UpL PfUaaudJspIaDMWOK3bKtN5pnTa0YBer89Z9UZFhVCxek5baCpCY69/eGKq1qcmxnMfL 0Zzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LEnopQL4PHusgBDS3f+zKlQOKB9hsKjgx1LSo73cC14=; b=PKsKDLEQ//zJ9RlXwToYALx72kdgG4eJXoD96+BZXwikIQ4KSdYa/jj8V+zVV+OUNl 5QzkDSZ4QHozwzGGB6YMcVEnUYTSTrrwe8Qb/s62Xq1KppI99qiPx+GDHviPL5QDMORj OYeGe1JkQnFNr1xPPUaqxI1BzJvLML3qckpZl9jjT2sUYZYiqpOb064i2a47hUV3Rk3e F1URx51pGEs4OlO0Tc4EqA4KRJ5DVntKs7MhemAXIpCfKrjS6NEznMAQF/m5Cwsz7PLc tj/Nrc7O2pAa1LxbChkTt/C6mgvEQG4lAiv3AH3UjIZJGGEID/2T+OWQYdrmp/EZiv7u JT3A== X-Gm-Message-State: AOAM532CGmhY8FKomElOhd7sS7Qn8gYUYRJYlm0qvXLKonDSMQN+Wp1y vPAypwVQWbiWJIpAFs1jBlXL1w== X-Google-Smtp-Source: ABdhPJy6tTDQsmMyTtSwAxGdAXZ4oMRMLnqItaww2JbeqGfzhj0BjF5Z3TZJVYZhpmbVBeBlhvnXww== X-Received: by 2002:a05:6808:2084:: with SMTP id s4mr3827632oiw.167.1628782913507; Thu, 12 Aug 2021 08:41:53 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id w16sm690973oih.19.2021.08.12.08.41.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Aug 2021 08:41:53 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 2/6] fs: add kiocb alloc cache flag Date: Thu, 12 Aug 2021 09:41:45 -0600 Message-Id: <20210812154149.1061502-3-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210812154149.1061502-1-axboe@kernel.dk> References: <20210812154149.1061502-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org If this kiocb can safely use the polled bio allocation cache, then this flag must be set. Generally this can be set for polled IO, where we will not see IRQ completions of the request. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- include/linux/fs.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index 640574294216..0dcc5de779c9 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -319,6 +319,8 @@ enum rw_hint { /* iocb->ki_waitq is valid */ #define IOCB_WAITQ (1 << 19) #define IOCB_NOIO (1 << 20) +/* can use bio alloc cache */ +#define IOCB_ALLOC_CACHE (1 << 21) struct kiocb { struct file *ki_filp; From patchwork Thu Aug 12 15:41:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12433813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D317FC04FE3 for ; Thu, 12 Aug 2021 15:41:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A84BF60FBF for ; Thu, 12 Aug 2021 15:41:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238241AbhHLPmV (ORCPT ); Thu, 12 Aug 2021 11:42:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238261AbhHLPmV (ORCPT ); Thu, 12 Aug 2021 11:42:21 -0400 Received: from mail-ot1-x32c.google.com (mail-ot1-x32c.google.com [IPv6:2607:f8b0:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BE38C0613D9 for ; Thu, 12 Aug 2021 08:41:56 -0700 (PDT) Received: by mail-ot1-x32c.google.com with SMTP id l36-20020a0568302b24b0290517526ce5e3so793473otv.11 for ; Thu, 12 Aug 2021 08:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TLRHeGT8+RAZgeaFPRMZhCfLrpW1DAxjHOSfqTjELLw=; b=ljWjTyij5TeS7JhqwXA0iOGT+ucvLdoOGLydmfS9GD2K8uEDdDgqLX9P+V1g3oxQY9 Gq96HSqQD+op6EH3OpxWG4qY5pVkLiSAkRjhMhr4I9D7V6Q6lzgkVyQs2qa+/z/OzH9G XmQ5oRVvVi9pKyU8Qz5zXpquf6z/H95EIbEZw6TORKrQq3QYQmvIpNp4VuzkknICwEsw 2ZDWQyPWp2RkM6DULlcPsA8Fxdv09M8+gLd/UmliKJ/rhCIj7kjI2T4Wk8kANdezlmCk 5W1WY4357cn5fVlTtBdnJGA35GL+z+QuoROGM1ngxNnlDXn5U1Ox4WRgLYCQmENQKVgN 0gXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TLRHeGT8+RAZgeaFPRMZhCfLrpW1DAxjHOSfqTjELLw=; b=NrGQt2YSQmVdFfp5uRR8VgAwq71FBW5S2/hLsA6LPmoNMXE3YgE8ZNaw+fQD3fUfkN NRBzIviZvaJTCzn1BVTB/2lXz0V3N3xRXmQOqvmK25YL/bSj5jgxevIyn4OhFV9a6+DE WxLMAIbsymwzau8hYowaFHm5oFs/tkt48hbhqMySeCx0jv3kEXod7TUuxDfZrcvNMFfR 52k0mkg4Q1IzbF/r+eNzV2WoJo+ObeKS/Ql2/pMb2gjzu8t7sqE4eU65CO2/eP0ML2l3 m/n+rOztlwAkBjDdXNMDfKksCU+/bfQ0b+uCB9EqkglKXP3Q7fYPw1Qzn27V/wcqxEog dVqQ== X-Gm-Message-State: AOAM532Ildp0thVt3mLvsdku+Gu+tLLLrAGr5VqIDvZtKzue/GVXxtHq K8IJBRSFOaTDdMSKZ3m+m91YqhzQL+G5U/gu X-Google-Smtp-Source: ABdhPJxqjFwV8cVjOOHG1jf7xvGdpcGqEBX+rEXLpzT4YVw+QOHWV2mKqYtMtGYC/LiaXBRszAmT9Q== X-Received: by 2002:a9d:65d9:: with SMTP id z25mr4068996oth.200.1628782914299; Thu, 12 Aug 2021 08:41:54 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id w16sm690973oih.19.2021.08.12.08.41.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Aug 2021 08:41:53 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 3/6] bio: add allocation cache abstraction Date: Thu, 12 Aug 2021 09:41:46 -0600 Message-Id: <20210812154149.1061502-4-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210812154149.1061502-1-axboe@kernel.dk> References: <20210812154149.1061502-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add a per-cpu bio_set cache for bio allocations, enabling us to quickly recycle them instead of going through the slab allocator. This cache isn't IRQ safe, and hence is only really suitable for polled IO. Very simple - keeps a count of bio's in the cache, and maintains a max of 512 with a slack of 64. If we get above max + slack, we drop slack number of bio's. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/bio.c | 135 +++++++++++++++++++++++++++++++++---- include/linux/bio.h | 13 ++++ include/linux/blk_types.h | 1 + include/linux/cpuhotplug.h | 1 + 4 files changed, 136 insertions(+), 14 deletions(-) diff --git a/block/bio.c b/block/bio.c index 9bf98b877aba..0fa2990018de 100644 --- a/block/bio.c +++ b/block/bio.c @@ -25,6 +25,11 @@ #include "blk.h" #include "blk-rq-qos.h" +struct bio_alloc_cache { + struct bio_list free_list; + unsigned int nr; +}; + static struct biovec_slab { int nr_vecs; char *name; @@ -618,6 +623,55 @@ void guard_bio_eod(struct bio *bio) bio_truncate(bio, maxsector << 9); } +#define ALLOC_CACHE_MAX 512 +#define ALLOC_CACHE_SLACK 64 + +static void bio_alloc_cache_prune(struct bio_alloc_cache *cache, + unsigned int nr) +{ + unsigned int i = 0; + struct bio *bio; + + while ((bio = bio_list_pop(&cache->free_list)) != NULL) { + cache->nr--; + bio_free(bio); + if (++i == nr) + break; + } +} + +static int bio_cpu_dead(unsigned int cpu, struct hlist_node *node) +{ + struct bio_set *bs; + + bs = hlist_entry_safe(node, struct bio_set, cpuhp_dead); + if (bs->cache) { + struct bio_alloc_cache *cache = per_cpu_ptr(bs->cache, cpu); + + bio_alloc_cache_prune(cache, -1U); + } + return 0; +} + +static void bio_alloc_cache_destroy(struct bio_set *bs) +{ + int cpu; + + if (!bs->cache) + return; + + preempt_disable(); + cpuhp_state_remove_instance_nocalls(CPUHP_BIO_DEAD, &bs->cpuhp_dead); + for_each_possible_cpu(cpu) { + struct bio_alloc_cache *cache; + + cache = per_cpu_ptr(bs->cache, cpu); + bio_alloc_cache_prune(cache, -1U); + } + preempt_enable(); + free_percpu(bs->cache); +} + /** * bio_put - release a reference to a bio * @bio: bio to release reference to @@ -628,16 +682,23 @@ void guard_bio_eod(struct bio *bio) **/ void bio_put(struct bio *bio) { - if (!bio_flagged(bio, BIO_REFFED)) - bio_free(bio); - else { + if (unlikely(bio_flagged(bio, BIO_REFFED))) { BIO_BUG_ON(!atomic_read(&bio->__bi_cnt)); - - /* - * last put frees it - */ if (atomic_dec_and_test(&bio->__bi_cnt)) - bio_free(bio); + return; + } + + if (bio_flagged(bio, BIO_PERCPU_CACHE)) { + struct bio_alloc_cache *cache; + + bio_uninit(bio); + cache = per_cpu_ptr(bio->bi_pool->cache, get_cpu()); + bio_list_add_head(&cache->free_list, bio); + if (++cache->nr > ALLOC_CACHE_MAX + ALLOC_CACHE_SLACK) + bio_alloc_cache_prune(cache, ALLOC_CACHE_SLACK); + put_cpu(); + } else { + bio_free(bio); } } EXPORT_SYMBOL(bio_put); @@ -1529,6 +1590,7 @@ int biovec_init_pool(mempool_t *pool, int pool_entries) */ void bioset_exit(struct bio_set *bs) { + bio_alloc_cache_destroy(bs); if (bs->rescue_workqueue) destroy_workqueue(bs->rescue_workqueue); bs->rescue_workqueue = NULL; @@ -1590,12 +1652,18 @@ int bioset_init(struct bio_set *bs, biovec_init_pool(&bs->bvec_pool, pool_size)) goto bad; - if (!(flags & BIOSET_NEED_RESCUER)) - return 0; - - bs->rescue_workqueue = alloc_workqueue("bioset", WQ_MEM_RECLAIM, 0); - if (!bs->rescue_workqueue) - goto bad; + if (flags & BIOSET_NEED_RESCUER) { + bs->rescue_workqueue = alloc_workqueue("bioset", + WQ_MEM_RECLAIM, 0); + if (!bs->rescue_workqueue) + goto bad; + } + if (flags & BIOSET_PERCPU_CACHE) { + bs->cache = alloc_percpu(struct bio_alloc_cache); + if (!bs->cache) + goto bad; + cpuhp_state_add_instance_nocalls(CPUHP_BIO_DEAD, &bs->cpuhp_dead); + } return 0; bad: @@ -1622,6 +1690,42 @@ int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) } EXPORT_SYMBOL(bioset_init_from_src); +/** + * bio_alloc_kiocb - Allocate a bio from bio_set based on kiocb + * @kiocb: kiocb describing the IO + * @bs: bio_set to allocate from + * + * Description: + * Like @bio_alloc_bioset, but pass in the kiocb. The kiocb is only + * used to check if we should dip into the per-cpu bio_set allocation + * cache. The allocation uses GFP_KERNEL internally. + * + */ +struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, + struct bio_set *bs) +{ + struct bio_alloc_cache *cache; + struct bio *bio; + + if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) || nr_vecs > BIO_INLINE_VECS) + return bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs); + + cache = per_cpu_ptr(bs->cache, get_cpu()); + bio = bio_list_pop(&cache->free_list); + if (bio) { + cache->nr--; + put_cpu(); + bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs); + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; + } + put_cpu(); + bio = bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs); + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; +} +EXPORT_SYMBOL_GPL(bio_alloc_kiocb); + static int __init init_bio(void) { int i; @@ -1636,6 +1740,9 @@ static int __init init_bio(void) SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL); } + cpuhp_setup_state_multi(CPUHP_BIO_DEAD, "block/bio:dead", NULL, + bio_cpu_dead); + if (bioset_init(&fs_bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS)) panic("bio: can't allocate bios\n"); diff --git a/include/linux/bio.h b/include/linux/bio.h index 2203b686e1f0..89ad28213b1d 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -401,6 +401,7 @@ static inline struct bio *bio_next_split(struct bio *bio, int sectors, enum { BIOSET_NEED_BVECS = BIT(0), BIOSET_NEED_RESCUER = BIT(1), + BIOSET_PERCPU_CACHE = BIT(2), }; extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags); extern void bioset_exit(struct bio_set *); @@ -409,6 +410,8 @@ extern int bioset_init_from_src(struct bio_set *bs, struct bio_set *src); struct bio *bio_alloc_bioset(gfp_t gfp, unsigned short nr_iovecs, struct bio_set *bs); +struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, + struct bio_set *bs); struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned short nr_iovecs); extern void bio_put(struct bio *); @@ -699,6 +702,11 @@ struct bio_set { struct kmem_cache *bio_slab; unsigned int front_pad; + /* + * per-cpu bio alloc cache + */ + struct bio_alloc_cache __percpu *cache; + mempool_t bio_pool; mempool_t bvec_pool; #if defined(CONFIG_BLK_DEV_INTEGRITY) @@ -715,6 +723,11 @@ struct bio_set { struct bio_list rescue_list; struct work_struct rescue_work; struct workqueue_struct *rescue_workqueue; + + /* + * Hot un-plug notifier for the per-cpu cache, if used + */ + struct hlist_node cpuhp_dead; }; static inline bool bioset_initialized(struct bio_set *bs) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 290f9061b29a..f68d4e8c775e 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -301,6 +301,7 @@ enum { BIO_TRACKED, /* set if bio goes through the rq_qos path */ BIO_REMAPPED, BIO_ZONE_WRITE_LOCKED, /* Owns a zoned device zone write lock */ + BIO_PERCPU_CACHE, /* can participate in per-cpu alloc cache */ BIO_FLAG_LAST }; diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index f39b34b13871..fe72c8d6c980 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -46,6 +46,7 @@ enum cpuhp_state { CPUHP_ARM_OMAP_WAKE_DEAD, CPUHP_IRQ_POLL_DEAD, CPUHP_BLOCK_SOFTIRQ_DEAD, + CPUHP_BIO_DEAD, CPUHP_ACPI_CPUDRV_DEAD, CPUHP_S390_PFAULT_DEAD, CPUHP_BLK_MQ_DEAD, From patchwork Thu Aug 12 15:41:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12433811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 019C2C4338F for ; Thu, 12 Aug 2021 15:41:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D4DD360FBF for ; Thu, 12 Aug 2021 15:41:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238219AbhHLPmV (ORCPT ); Thu, 12 Aug 2021 11:42:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238232AbhHLPmU (ORCPT ); Thu, 12 Aug 2021 11:42:20 -0400 Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A58EBC0613D9 for ; Thu, 12 Aug 2021 08:41:55 -0700 (PDT) Received: by mail-ot1-x334.google.com with SMTP id h63-20020a9d14450000b02904ce97efee36so8257827oth.7 for ; Thu, 12 Aug 2021 08:41:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gzAcF0zizY0wVeW39ScMbIL+yu4HjhIlQYH8a7v47yo=; b=TihX5YMJv+Ds4lQ0p4PiRHGk8oWXtLun1JbOkDn9HgtJB+KVVPNtgcEJqWHN7MtssK zPaHp9eZvCzPqmb8Xn5Ht5kPNW67wAp/VeMnxecLfD70hI1QdMfxXKhI8NjgsXNppzR9 IKM7yLkipYJauHpX6/tUBptEh+8XR+eNFblYE2oLiwofyHXRBZzdxNiUvcVxHPHrdLaw kPFZr2tLL4Eiid5AL4qWeha7+227TtjYSFfdvN4Me5ktN16iHpMsdhsI3zuT6ijwBnYy Ech6E+Xxu2DQ755Q3I78ma4wJ5dcrmDnmHPPQ4Zu07gGsrv0pHGJyi92gJlr/ZiDbViT gkcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gzAcF0zizY0wVeW39ScMbIL+yu4HjhIlQYH8a7v47yo=; b=eqiLr/qi+afG2kKu5I2FbXreo6Yth5GDJluujvDrf6sRQL6VXhxUIipdq50WtCk6l4 /p3hmS9sB9UUKD1AqE57XttQRuHMwQ07glSIqBOPD+N9uS2CCgGanY+TNc5b7gTCIcz9 g3ayQu6ssLMsFI1pf7tRsf3sEW5knAFnDO53rjmhj9IkgcBIL8jmoXya8V4moZyZqwfF dleSD8y4v5FgJShmyaeUzFBPjlLbXbCX1VAEIZljhb6tqpUMbwuJUbwDSwnmaI9CgbNB mv0KrUY6YCYQDh5gxrkhRIpTHKN5NmFXQHb97bs6w11Egl74jxNL/0QsUBGQpJoOIQ03 Do+Q== X-Gm-Message-State: AOAM531w80fS30sBl+71zwuuVjTHheAXs6QCDdmF19xATbiYH0o71YZz WQLlMDyG0RtpqvoWQC+mp0XN3w== X-Google-Smtp-Source: ABdhPJw0Sl3w155IOnkFB5DmHVe92pvQEqfNjekV4IPmKjJ6R84Q/0+3dhjbz7vUG5w9NDS9EG6vBg== X-Received: by 2002:a05:6830:4429:: with SMTP id q41mr4022053otv.284.1628782915057; Thu, 12 Aug 2021 08:41:55 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id w16sm690973oih.19.2021.08.12.08.41.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Aug 2021 08:41:54 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 4/6] block: clear BIO_PERCPU_CACHE flag if polling isn't supported Date: Thu, 12 Aug 2021 09:41:47 -0600 Message-Id: <20210812154149.1061502-5-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210812154149.1061502-1-axboe@kernel.dk> References: <20210812154149.1061502-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The bio alloc cache relies on the fact that a polled bio will complete in process context, clear the cacheable flag if we disable polling for a given bio. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- block/blk-core.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/block/blk-core.c b/block/blk-core.c index 04477697ee4b..c130206e9961 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -833,8 +833,11 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } - if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) { + /* can't support alloc cache if we turn off polling */ + bio_clear_flag(bio, BIO_PERCPU_CACHE); bio->bi_opf &= ~REQ_HIPRI; + } switch (bio_op(bio)) { case REQ_OP_DISCARD: From patchwork Thu Aug 12 15:41:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12433815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 777E9C19F31 for ; Thu, 12 Aug 2021 15:41:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5528B60F91 for ; Thu, 12 Aug 2021 15:41:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238262AbhHLPmW (ORCPT ); Thu, 12 Aug 2021 11:42:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238249AbhHLPmW (ORCPT ); Thu, 12 Aug 2021 11:42:22 -0400 Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8524C061756 for ; Thu, 12 Aug 2021 08:41:56 -0700 (PDT) Received: by mail-oi1-x233.google.com with SMTP id u25so10988633oiv.5 for ; Thu, 12 Aug 2021 08:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=46//QmBZH3jrthv766p60WFq7Yim71orBie7qxwdLOk=; b=AHYlNGB1VuUhJsFptE3Rzc+pkikDIsJJHkgmNwB82F+KMUbVB6wmJgwYELxwIseO8i 7MxRO28Sb2nZIjJ40mar67nomEgBQL5s9MJdCf+3Z0baT5XSgk3cxcNJUPDFlncVYULB 7nbiG6CJXH+/5HUQFkcEj8UU3yvgI6erFmzEetW81NJcI3rImhahLR2+sNXzQHrJEL7b KHeKKQArgUvmPJVmt8B8O+ECpJnGYo4qMo4fILzs7SB1K06B5HcJcDJqAnJCBxmmF/5T IZSNCna1OPjpqt00UG0o3AFIZtn8GGlj5+sM0KXS8TeLZFLWsbjbJTnQJkR17xcQ/48t Bxlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=46//QmBZH3jrthv766p60WFq7Yim71orBie7qxwdLOk=; b=PRw11SjD3EGTYG6PFQhqNK3xLIw0jicTTbpWijzALyepIFta65ybl3HvoVoremMkkG 35DKKj12alQKN2RKyglnVI1qsn71ieI/ShGkRUITIH+vjvLj2sX46ikcaUg6Zntnst6U hQyAzvJ4x0jbUzH6T0FBUSUwoV87VZPQmEtyz7+Z5VvO9CiqY7w9hmdPlgxSFW8uwTdf Cn+6FXHjVfBGBlIdJEd+XFpeDGoUR7AyUduRySJiE0zXrqTnY0CUtQ3ZJXOyF9EIWZax 18378lXOUNLWkfwT5sD6U8Z08aq0IJLX1w1MX3sQeQWEK4+dA0KxsM098Z4aXqyiprfK PVsA== X-Gm-Message-State: AOAM533EoHShcCzIYogZMsVgfG0r0IC/RmY2rpaIyt3jnPfS1eYE+jEh IRyZMfcHTv3YFAD4prmV5vGVag== X-Google-Smtp-Source: ABdhPJzMOdA4TfkbjVKZv7BxDailBEKJLEzaXZMOL+3uXynnlUm55GrMLTalK4pu6naC53aDg4sT9g== X-Received: by 2002:a05:6808:bc8:: with SMTP id o8mr3785607oik.2.1628782916274; Thu, 12 Aug 2021 08:41:56 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id w16sm690973oih.19.2021.08.12.08.41.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Aug 2021 08:41:55 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Jens Axboe Subject: [PATCH 5/6] io_uring: enable use of bio alloc cache Date: Thu, 12 Aug 2021 09:41:48 -0600 Message-Id: <20210812154149.1061502-6-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210812154149.1061502-1-axboe@kernel.dk> References: <20210812154149.1061502-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Mark polled IO as being safe for dipping into the bio allocation cache, in case the targeted bio_set has it enabled. This brings an IOPOLL gen2 Optane QD=128 workload from ~3.0M IOPS to ~3.3M IOPS. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- fs/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 6c65c90131cb..ea387b0741b8 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2736,7 +2736,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) !kiocb->ki_filp->f_op->iopoll) return -EOPNOTSUPP; - kiocb->ki_flags |= IOCB_HIPRI; + kiocb->ki_flags |= IOCB_HIPRI | IOCB_ALLOC_CACHE; kiocb->ki_complete = io_complete_rw_iopoll; req->iopoll_completed = 0; } else { From patchwork Thu Aug 12 15:41:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12433817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B9CEC4320E for ; Thu, 12 Aug 2021 15:41:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 403D7610CC for ; Thu, 12 Aug 2021 15:41:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238261AbhHLPmX (ORCPT ); Thu, 12 Aug 2021 11:42:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238249AbhHLPmX (ORCPT ); Thu, 12 Aug 2021 11:42:23 -0400 Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCFD9C061756 for ; Thu, 12 Aug 2021 08:41:57 -0700 (PDT) Received: by mail-oi1-x236.google.com with SMTP id u25so10988685oiv.5 for ; Thu, 12 Aug 2021 08:41:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KGh3DHNk1Pecrp3DYsOdrKTAPnauoq0TCqYdAequptQ=; b=DoiXWpYd2RHBJtHUgUh4Sg78Sfmx9ShUnUChmtkfLaqByRc92ip3oxNL7yYQexaD/C JD4pI0eg2qtQ7FKB9ZwvqN7KtRjvhEB9cOONFqlamaahBTI8tpXX7x+lVm9zJk8A2jqh 349Qb3zWWDWc7QG3Li16aSaaNFnNU7T1EQQwyFsJDPqsloZOG4x/3pLzEVNGHf8jC5NN B3P2VIfL0MaB1StYr/72pS3HUHoTg78KkAh1blNi0uyYWp1qHJPLVTB9daT/ejJvD9nI ETNjPm/CtGeBH/CN8To+Mxo6zIF3cFO9XWv4huH0PzMN0wgg1hv8lZc+wJIWTgCifjgj aSoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KGh3DHNk1Pecrp3DYsOdrKTAPnauoq0TCqYdAequptQ=; b=ojdIt4jCTomJea7HurkHlFNssAF24OnA0Bzwgr6EpnVMaxAhSMmJSVdcQkEP2WjIZD z/JoRe8tQdxLGBq453If0J3KU2SF15gENUuYoW2MiaXPQBQiEK6ChLiK4Fo0Uix8vqCO uTHE2Ac0Rolassa0Zbm/wFI7945ibB8p0QSpzm2lG6nkUBR9gisl3GPy5uHJMtunSIJN S5uBZPPp2GPsEPU0gf5kDlLDk/TyPT+5WhzG592R00x4VHf4LHVvSYrq/q6DcM0310OK CaLW/6AGnWbx1mcN3lt49N23ggnUPeTTI7iCkFCtMsdFAby5QtzMh/j7nTTegs2j4/Do huEQ== X-Gm-Message-State: AOAM532Xrrj/ddWjF/90D2iXSwUmdRsQiph3tMWyslUrVyPsInT1pAbS xUcyNAnzypuHn7WaIVLPke945g== X-Google-Smtp-Source: ABdhPJx9b5fJYV9wguO/TCBrHRqtklbv/5xdkxNojagYy4U/38VaU5evI1UFOdIYrEs7Mxo2EDhWnw== X-Received: by 2002:aca:230e:: with SMTP id e14mr3673238oie.93.1628782917172; Thu, 12 Aug 2021 08:41:57 -0700 (PDT) Received: from p1.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id w16sm690973oih.19.2021.08.12.08.41.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Aug 2021 08:41:56 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, hch@infradead.org, Christoph Hellwig , Jens Axboe Subject: [PATCH 6/6] block: use the percpu bio cache in __blkdev_direct_IO Date: Thu, 12 Aug 2021 09:41:49 -0600 Message-Id: <20210812154149.1061502-7-axboe@kernel.dk> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210812154149.1061502-1-axboe@kernel.dk> References: <20210812154149.1061502-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Christoph Hellwig Use bio_alloc_kiocb to dip into the percpu cache of bios when the caller asks for it. Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/block_dev.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 9ef4f1fc2cb0..3c7fb7106713 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -385,7 +385,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, (bdev_logical_block_size(bdev) - 1)) return -EINVAL; - bio = bio_alloc_bioset(GFP_KERNEL, nr_pages, &blkdev_dio_pool); + bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool); dio = container_of(bio, struct blkdev_dio, bio); dio->is_sync = is_sync = is_sync_kiocb(iocb); @@ -513,7 +513,9 @@ blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) static __init int blkdev_init(void) { - return bioset_init(&blkdev_dio_pool, 4, offsetof(struct blkdev_dio, bio), BIOSET_NEED_BVECS); + return bioset_init(&blkdev_dio_pool, 4, + offsetof(struct blkdev_dio, bio), + BIOSET_NEED_BVECS|BIOSET_PERCPU_CACHE); } module_init(blkdev_init);