From patchwork Wed Mar 23 19:45:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 12790071 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4A6D3C433FE for ; Wed, 23 Mar 2022 19:45:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648064740; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=6Ni4pXHQ9ZAzduCpIg0bAVA8u384qCaa23qmj/rkF88=; b=JpYmAR+FUR6dG6MuGFvYHSceFoHyQpDGCm3FPm2UNK7FRh4l3Dlq0teZUrlESlZ63b0ufK hwuDN9KC3dqFEw64pKcS7Ur4j6fJklfqZDHVYoPCk0R7pbng3Tp4JP5gZ9Mg4sDfdwfglJ zusakAIpLTng5ozAMHtK3oz4Fjci2xs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-654-Kwl6oUs0PBGi-8LLJjW19w-1; Wed, 23 Mar 2022 15:45:36 -0400 X-MC-Unique: Kwl6oUs0PBGi-8LLJjW19w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 57F5D1044563; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 45C11400E420; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id D20331940347; Wed, 23 Mar 2022 19:45:33 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 1AA951940352 for ; Wed, 23 Mar 2022 19:45:32 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id EED7C46A38A; Wed, 23 Mar 2022 19:45:31 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EB1F958BAC8 for ; Wed, 23 Mar 2022 19:45:31 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 574DA83395F for ; Wed, 23 Mar 2022 19:45:31 +0000 (UTC) Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-251-zHx5jnEzMnClkeTQv55wEg-1; Wed, 23 Mar 2022 15:45:28 -0400 X-MC-Unique: zHx5jnEzMnClkeTQv55wEg-1 Received: by mail-qt1-f197.google.com with SMTP id z18-20020ac84552000000b002e201c79cd4so2019716qtn.2 for ; Wed, 23 Mar 2022 12:45:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=AKeDBg4ZdHttOQXh6zOoYZL7eiQ4d/9ADfn1bk7vgsI=; b=P/gmCuZV7m6R5cCu+2Udx4uSoBJ+sXPwrsN5nHkIZbcPR5+Sav+IHhZKkUn5a2lDrS fKAYRgSvCigZLPkhkSk72XWg+5h341pn5rRrpm8xkwWMm5zXWmLw9ws88ByJbOyVpwOK h6R/wv1kOMro5O+/Fvxal4Jk80rMU1ioo5VJ3tj/MDDzssICOc/VyLJfzXByYi/BuCF+ 7NOa1M1VtZZKfu6+IQUiHLUicvDh0Tq6mFjEZktUOXFtTnKblOd1xvgxPaeCzkJ4yy8t d8TJQy4VBFsOEif90KmWG8oUpE3x6UoCwF4+zwJA9PhenQNdeYx2E7LB4S1QcY/5rXdK LMRQ== X-Gm-Message-State: AOAM532tqkeUog9XCBj/BEXyh6WwuOZzs82wpiSXZCk9+6gsZto5BR5u 4yngR7B/oK2ncJW172JeZNOEO9QvFLz99uXo+9pzmAVwp6oVLneUXCPoMELhjuIM+5IWS0rltTd nnamX8PbRRe3olQ== X-Received: by 2002:ad4:5aa9:0:b0:441:3a0a:1aba with SMTP id u9-20020ad45aa9000000b004413a0a1abamr1463208qvg.20.1648064727288; Wed, 23 Mar 2022 12:45:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx14W12RoUy94PnscHvKvwxWAan+U2UFinhkJZ0fo2szicnnDf5lrhasgjpruR6xvmPU2ZYlw== X-Received: by 2002:ad4:5aa9:0:b0:441:3a0a:1aba with SMTP id u9-20020ad45aa9000000b004413a0a1abamr1463195qvg.20.1648064727094; Wed, 23 Mar 2022 12:45:27 -0700 (PDT) Received: from localhost (pool-68-160-176-52.bstnma.fios.verizon.net. [68.160.176.52]) by smtp.gmail.com with ESMTPSA id u129-20020a376087000000b0067e401d7177sm512410qkb.3.2022.03.23.12.45.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 12:45:26 -0700 (PDT) From: Mike Snitzer X-Google-Original-From: Mike Snitzer To: axboe@kernel.dk Date: Wed, 23 Mar 2022 15:45:21 -0400 Message-Id: <20220323194524.5900-2-snitzer@kernel.org> In-Reply-To: <20220323194524.5900-1-snitzer@kernel.org> References: <20220323194524.5900-1-snitzer@kernel.org> X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Subject: [dm-devel] [PATCH v2 1/4] block: allow BIOSET_PERCPU_CACHE use from bio_alloc_clone X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, hch@lst.de, ming.lei@redhat.com MIME-Version: 1.0 Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com These changes allow DM core to make full use of BIOSET_PERCPU_CACHE for REQ_POLLED bios: Factor out bio_alloc_percpu_cache() from bio_alloc_kiocb() to allow use by bio_alloc_clone() too. Update bioset_init_from_src() to set BIOSET_PERCPU_CACHE if bio_src->cache is not NULL. Move bio_clear_polled() to include/linux/bio.h to allow users outside of block core. Signed-off-by: Mike Snitzer --- block/bio.c | 56 +++++++++++++++++++++++++++++++++-------------------- block/blk.h | 7 ------- include/linux/bio.h | 7 +++++++ 3 files changed, 42 insertions(+), 28 deletions(-) diff --git a/block/bio.c b/block/bio.c index b15f5466ce08..a7633aa82d7d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -420,6 +420,33 @@ static void punt_bios_to_rescuer(struct bio_set *bs) queue_work(bs->rescue_workqueue, &bs->rescue_work); } +static struct bio *bio_alloc_percpu_cache(struct block_device *bdev, + unsigned short nr_vecs, unsigned int opf, gfp_t gfp, + struct bio_set *bs) +{ + struct bio_alloc_cache *cache; + struct bio *bio; + + cache = per_cpu_ptr(bs->cache, get_cpu()); + if (cache->free_list) { + bio = cache->free_list; + cache->free_list = bio->bi_next; + cache->nr--; + put_cpu(); + bio_init(bio, bdev, nr_vecs ? bio->bi_inline_vecs : NULL, + nr_vecs, opf); + bio->bi_pool = bs; + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; + } + put_cpu(); + bio = bio_alloc_bioset(bdev, nr_vecs, opf, gfp, bs); + if (!bio) + return NULL; + bio_set_flag(bio, BIO_PERCPU_CACHE); + return bio; +} + /** * bio_alloc_bioset - allocate a bio for I/O * @bdev: block device to allocate the bio for (can be %NULL) @@ -768,7 +795,10 @@ struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src, { struct bio *bio; - bio = bio_alloc_bioset(bdev, 0, bio_src->bi_opf, gfp, bs); + if (bs->cache && bio_src->bi_opf & REQ_POLLED) + bio = bio_alloc_percpu_cache(bdev, 0, bio_src->bi_opf, gfp, bs); + else + bio = bio_alloc_bioset(bdev, 0, bio_src->bi_opf, gfp, bs); if (!bio) return NULL; @@ -1736,6 +1766,8 @@ int bioset_init_from_src(struct bio_set *bs, struct bio_set *src) flags |= BIOSET_NEED_BVECS; if (src->rescue_workqueue) flags |= BIOSET_NEED_RESCUER; + if (src->cache) + flags |= BIOSET_PERCPU_CACHE; return bioset_init(bs, src->bio_pool.min_nr, src->front_pad, flags); } @@ -1753,35 +1785,17 @@ EXPORT_SYMBOL(bioset_init_from_src); * Like @bio_alloc_bioset, but pass in the kiocb. The kiocb is only * used to check if we should dip into the per-cpu bio_set allocation * cache. The allocation uses GFP_KERNEL internally. On return, the - * bio is marked BIO_PERCPU_CACHEABLE, and the final put of the bio + * bio is marked BIO_PERCPU_CACHE, and the final put of the bio * MUST be done from process context, not hard/soft IRQ. * */ struct bio *bio_alloc_kiocb(struct kiocb *kiocb, struct block_device *bdev, unsigned short nr_vecs, unsigned int opf, struct bio_set *bs) { - struct bio_alloc_cache *cache; - struct bio *bio; - if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) || nr_vecs > BIO_INLINE_VECS) return bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs); - cache = per_cpu_ptr(bs->cache, get_cpu()); - if (cache->free_list) { - bio = cache->free_list; - cache->free_list = bio->bi_next; - cache->nr--; - put_cpu(); - bio_init(bio, bdev, nr_vecs ? bio->bi_inline_vecs : NULL, - nr_vecs, opf); - bio->bi_pool = bs; - bio_set_flag(bio, BIO_PERCPU_CACHE); - return bio; - } - put_cpu(); - bio = bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs); - bio_set_flag(bio, BIO_PERCPU_CACHE); - return bio; + return bio_alloc_percpu_cache(bdev, nr_vecs, opf, GFP_KERNEL, bs); } EXPORT_SYMBOL_GPL(bio_alloc_kiocb); diff --git a/block/blk.h b/block/blk.h index ebaa59ca46ca..8e338e76d303 100644 --- a/block/blk.h +++ b/block/blk.h @@ -451,13 +451,6 @@ extern struct device_attribute dev_attr_events; extern struct device_attribute dev_attr_events_async; extern struct device_attribute dev_attr_events_poll_msecs; -static inline void bio_clear_polled(struct bio *bio) -{ - /* can't support alloc cache if we turn off polling */ - bio_clear_flag(bio, BIO_PERCPU_CACHE); - bio->bi_opf &= ~REQ_POLLED; -} - long blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); diff --git a/include/linux/bio.h b/include/linux/bio.h index 7523aba4ddf7..709663ae757a 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -787,6 +787,13 @@ static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb) bio->bi_opf |= REQ_NOWAIT; } +static inline void bio_clear_polled(struct bio *bio) +{ + /* can't support alloc cache if we turn off polling */ + bio_clear_flag(bio, BIO_PERCPU_CACHE); + bio->bi_opf &= ~REQ_POLLED; +} + struct bio *blk_next_bio(struct bio *bio, struct block_device *bdev, unsigned int nr_pages, unsigned int opf, gfp_t gfp); From patchwork Wed Mar 23 19:45:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 12790070 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CC61AC433F5 for ; Wed, 23 Mar 2022 19:45:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648064737; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=dBDXL6EoW//Q4JJ5QU19xDRe+K4qLmsxIV2pEtzSGnw=; b=HfRRVluRZ3A9nwREQpZNVAHkmG+zEaSlIf3PZhl9wehclzvqDDUEr09a7fpn0XNrpnTWee ela5nbB72TIA7GlHJsm9oCphTIldSzUkmKaYpYu0WFvyEaMFa7effi3fUwhpUnuEmuVfVN Mp+E/zkQN5zCchNGgmV+t4cWOJU+5QM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-214-P5YlSLNIMT6AdjIgrsOiJg-1; Wed, 23 Mar 2022 15:45:34 -0400 X-MC-Unique: P5YlSLNIMT6AdjIgrsOiJg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4C9FA299E74B; Wed, 23 Mar 2022 19:45:32 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3AFB85721A1; Wed, 23 Mar 2022 19:45:32 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 056C61940347; Wed, 23 Mar 2022 19:45:32 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 185541949763 for ; Wed, 23 Mar 2022 19:45:31 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 059634010A01; Wed, 23 Mar 2022 19:45:31 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast03.extmail.prod.ext.rdu2.redhat.com [10.11.55.19]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0079D400E420 for ; Wed, 23 Mar 2022 19:45:30 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DA94C811E76 for ; Wed, 23 Mar 2022 19:45:30 +0000 (UTC) Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-668-xwOwnngeMkiVazTmWlGcgg-1; Wed, 23 Mar 2022 15:45:29 -0400 X-MC-Unique: xwOwnngeMkiVazTmWlGcgg-1 Received: by mail-qk1-f199.google.com with SMTP id w200-20020a3762d1000000b0067d2149318dso1702068qkb.1 for ; Wed, 23 Mar 2022 12:45:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=ewXCoGrLUptePB/yghi1giztCZDIQfZAZPwz3EjuzAI=; b=wzIfwnLQBgLdSLDOadXSBTcc2bb2EjN+PWH21MbWsFpTmviHAEVdmRoBGrfEHTdtyb TXI/nQn+ndx4oICESsyEOZVmjbwJDkY77ExntDcsM4BvU+S4AGpFu7pnKhb8dckvy/XF aL0xkAlt+KULsDvwlrS/d58CWzEzuV3ovLfdjT2667NgLQUQe2Glf6+Ry5oILZtxaeXh p9PxqneEYAjbDNKB6tA0R8zHyt5MHE+mzEyy+MbcAOTX9qXfx1k2Owhikrh6lYT3Xa1R uktdAHS/E5PqFyJxWiBQrsEzekUysURILQOe2QXv/vE/sAzWcOXeZyxdwkSa8PaDbN/i 7s4Q== X-Gm-Message-State: AOAM531Fy26TXd+/+0+QzcmOsUmsMZll7Ckd0/WCMql4XP9PfqSLNiJ2 1csTuwtGT0p5ysymAjRj65tR4wRfWzZ1gNtJFHiQDSvIOGdMMVmmgwOJ/BT/pNsStXYEYgzWyB+ 6TRuO2wgjQzNBhg== X-Received: by 2002:ac8:7fcc:0:b0:2e0:684e:42cc with SMTP id b12-20020ac87fcc000000b002e0684e42ccmr1353712qtk.35.1648064729117; Wed, 23 Mar 2022 12:45:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzRo/E6Via9iAEdkuG26l/B0IpzCO611anBQnTlTq6VFsvjqWDfBgpNUyUNJkJuIgGuIsZXHA== X-Received: by 2002:ac8:7fcc:0:b0:2e0:684e:42cc with SMTP id b12-20020ac87fcc000000b002e0684e42ccmr1353698qtk.35.1648064728862; Wed, 23 Mar 2022 12:45:28 -0700 (PDT) Received: from localhost (pool-68-160-176-52.bstnma.fios.verizon.net. [68.160.176.52]) by smtp.gmail.com with ESMTPSA id h62-20020a37b741000000b0067da4164f8fsm425972qkf.126.2022.03.23.12.45.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 12:45:28 -0700 (PDT) From: Mike Snitzer X-Google-Original-From: Mike Snitzer To: axboe@kernel.dk Date: Wed, 23 Mar 2022 15:45:22 -0400 Message-Id: <20220323194524.5900-3-snitzer@kernel.org> In-Reply-To: <20220323194524.5900-1-snitzer@kernel.org> References: <20220323194524.5900-1-snitzer@kernel.org> X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Subject: [dm-devel] [PATCH v2 2/4] block: allow BIOSET_PERCPU_CACHE use from bio_alloc_bioset X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, hch@lst.de, ming.lei@redhat.com MIME-Version: 1.0 Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Add REQ_ALLOC_CACHE and set it in %opf passed to bio_alloc_bioset to inform bio_alloc_bioset (and any stacked block drivers) that bio should be allocated from respective bioset's per-cpu alloc cache if possible. This decouples access control to the alloc cache (via REQ_ALLOC_CACHE) from actual participation in a specific alloc cache (BIO_PERCPU_CACHE). Otherwise an upper layer's bioset may not have an alloc cache, in which case the bio issued to underlying device(s) wouldn't reflect that allocating from an alloc cache warranted (if possible). Signed-off-by: Mike Snitzer --- block/bio.c | 33 ++++++++++++++++++++------------- include/linux/bio.h | 4 +++- include/linux/blk_types.h | 4 +++- 3 files changed, 26 insertions(+), 15 deletions(-) diff --git a/block/bio.c b/block/bio.c index a7633aa82d7d..0b65ea241f54 100644 --- a/block/bio.c +++ b/block/bio.c @@ -440,11 +440,7 @@ static struct bio *bio_alloc_percpu_cache(struct block_device *bdev, return bio; } put_cpu(); - bio = bio_alloc_bioset(bdev, nr_vecs, opf, gfp, bs); - if (!bio) - return NULL; - bio_set_flag(bio, BIO_PERCPU_CACHE); - return bio; + return NULL; } /** @@ -488,11 +484,24 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, gfp_t saved_gfp = gfp_mask; struct bio *bio; void *p; + bool use_alloc_cache; /* should not use nobvec bioset for nr_vecs > 0 */ if (WARN_ON_ONCE(!mempool_initialized(&bs->bvec_pool) && nr_vecs > 0)) return NULL; + use_alloc_cache = (bs->cache && (opf & REQ_ALLOC_CACHE) && + nr_vecs <= BIO_INLINE_VECS); + if (use_alloc_cache) { + bio = bio_alloc_percpu_cache(bdev, nr_vecs, opf, gfp_mask, bs); + if (bio) + return bio; + /* + * No cached bio available, mark bio returned below to + * particpate in per-cpu alloc cache. + */ + } + /* * submit_bio_noacct() converts recursion to iteration; this means if * we're running beneath it, any bios we allocate and submit will not be @@ -546,6 +555,8 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, bio_init(bio, bdev, NULL, 0, opf); } + if (use_alloc_cache) + bio_set_flag(bio, BIO_PERCPU_CACHE); bio->bi_pool = bs; return bio; @@ -795,10 +806,7 @@ struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src, { struct bio *bio; - if (bs->cache && bio_src->bi_opf & REQ_POLLED) - bio = bio_alloc_percpu_cache(bdev, 0, bio_src->bi_opf, gfp, bs); - else - bio = bio_alloc_bioset(bdev, 0, bio_src->bi_opf, gfp, bs); + bio = bio_alloc_bioset(bdev, 0, bio_src->bi_opf, gfp, bs); if (!bio) return NULL; @@ -1792,10 +1800,9 @@ EXPORT_SYMBOL(bioset_init_from_src); struct bio *bio_alloc_kiocb(struct kiocb *kiocb, struct block_device *bdev, unsigned short nr_vecs, unsigned int opf, struct bio_set *bs) { - if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) || nr_vecs > BIO_INLINE_VECS) - return bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs); - - return bio_alloc_percpu_cache(bdev, nr_vecs, opf, GFP_KERNEL, bs); + if (kiocb->ki_flags & IOCB_ALLOC_CACHE) + opf |= REQ_ALLOC_CACHE; + return bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs); } EXPORT_SYMBOL_GPL(bio_alloc_kiocb); diff --git a/include/linux/bio.h b/include/linux/bio.h index 709663ae757a..1be27e87a1f4 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -783,6 +783,8 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page, static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb) { bio->bi_opf |= REQ_POLLED; + if (kiocb->ki_flags & IOCB_ALLOC_CACHE) + bio->bi_opf |= REQ_ALLOC_CACHE; if (!is_sync_kiocb(kiocb)) bio->bi_opf |= REQ_NOWAIT; } @@ -791,7 +793,7 @@ static inline void bio_clear_polled(struct bio *bio) { /* can't support alloc cache if we turn off polling */ bio_clear_flag(bio, BIO_PERCPU_CACHE); - bio->bi_opf &= ~REQ_POLLED; + bio->bi_opf &= ~(REQ_POLLED | REQ_ALLOC_CACHE); } struct bio *blk_next_bio(struct bio *bio, struct block_device *bdev, diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 5561e58d158a..5f9a0c39d4c5 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -327,7 +327,7 @@ enum { BIO_TRACKED, /* set if bio goes through the rq_qos path */ BIO_REMAPPED, BIO_ZONE_WRITE_LOCKED, /* Owns a zoned device zone write lock */ - BIO_PERCPU_CACHE, /* can participate in per-cpu alloc cache */ + BIO_PERCPU_CACHE, /* participates in per-cpu alloc cache */ BIO_FLAG_LAST }; @@ -414,6 +414,7 @@ enum req_flag_bits { __REQ_NOUNMAP, /* do not free blocks when zeroing */ __REQ_POLLED, /* caller polls for completion using bio_poll */ + __REQ_ALLOC_CACHE, /* allocate IO from cache if available */ /* for driver use */ __REQ_DRV, @@ -439,6 +440,7 @@ enum req_flag_bits { #define REQ_NOUNMAP (1ULL << __REQ_NOUNMAP) #define REQ_POLLED (1ULL << __REQ_POLLED) +#define REQ_ALLOC_CACHE (1ULL << __REQ_ALLOC_CACHE) #define REQ_DRV (1ULL << __REQ_DRV) #define REQ_SWAP (1ULL << __REQ_SWAP) From patchwork Wed Mar 23 19:45:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 12790069 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7ED4DC43217 for ; Wed, 23 Mar 2022 19:45:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648064738; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=sMZvs+OFnQm0I5u4UQDD9LkSF5cS3O3CCJCl0QVIgCk=; b=V37MXshR2wfMLz5820CZ2Z2my1yWMkjsjTpPYZ64lbmEEWIY9L5Bt7M1oOEzFyluwJdqnK f/56dddESAVYqwbIz/XX10DosU7vwV3GtC/wHzBgXaXXximNu0JCwkV0INw7ijxtv2GWsX 6HxkrzdFiEWhb9u6mt7UieHMmFSIXz4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-275-eVhtYLIbN8aU8VHjpE0LSw-1; Wed, 23 Mar 2022 15:45:37 -0400 X-MC-Unique: eVhtYLIbN8aU8VHjpE0LSw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 95EE938025F4; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8404440CFD06; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 515DF194035C; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 4F9311949763 for ; Wed, 23 Mar 2022 19:45:33 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 14FF7401E7B; Wed, 23 Mar 2022 19:45:33 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast08.extmail.prod.ext.rdu2.redhat.com [10.11.55.24]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 10429401E4E for ; Wed, 23 Mar 2022 19:45:33 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D2E4B38025FD for ; Wed, 23 Mar 2022 19:45:32 +0000 (UTC) Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-68-fPYdTaxWNq6JQJp6ccpfTw-1; Wed, 23 Mar 2022 15:45:31 -0400 X-MC-Unique: fPYdTaxWNq6JQJp6ccpfTw-1 Received: by mail-qv1-f69.google.com with SMTP id 33-20020a0c8024000000b0043d17ffb0bdso1998269qva.18 for ; Wed, 23 Mar 2022 12:45:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=tfNH+5/HwjgUfw+cvmAnjvE/JE7439RWvRdUdw80No0=; b=aKoy57/XWp4+dCgyirU1WYuNCZ0AVrEM5NL65uQO7YdMsIS3toyfwB1ywcafdVFRXL 93FanOvrr+aBvnK6TPCvzgYg66QMCQxYF+l5IvFyiCrsggAiAuk/J24f7jRUYHXhwmOI BEW6NSAXj0BKC4lN4uvo9GlUjrdQ2maQHfwiCpKDVMWE3h4d5girofSt37MdRQw9AmX2 q6L0vE8gXE+DYmaNl9T3aTpiRKpH3R6O7+W4i1UHqvjMhFywj+IHX1yoY4C2peeoNCw0 eQB9kOMFONlLoJsHqA4UhBdYrAOPJdUj8IstK2pEWQda5quOs42xyUh5Y20dxYd1jvV8 u4ow== X-Gm-Message-State: AOAM532l2KLJljKjzB4JABX5Brkgs9gC0bP4WKGaNRsRZYfkqiRxRRRL pVA98F5B8RNt4pHr3VgbqZ6bOn7WseDPQ4+RPEJRkcAGbl20G1ODJoyzF5sPYSvRctVgBhjGh1h Zqz4QsCYziLxhCg== X-Received: by 2002:a37:64a:0:b0:67d:430e:2a20 with SMTP id 71-20020a37064a000000b0067d430e2a20mr1067536qkg.265.1648064730572; Wed, 23 Mar 2022 12:45:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzkwWOh6Well00wZWN3Pa0vtdo4r0dGuz4vGTky++LCnVcOeEJg91pOXt2yuaRFGau8p357xw== X-Received: by 2002:a37:64a:0:b0:67d:430e:2a20 with SMTP id 71-20020a37064a000000b0067d430e2a20mr1067522qkg.265.1648064730301; Wed, 23 Mar 2022 12:45:30 -0700 (PDT) Received: from localhost (pool-68-160-176-52.bstnma.fios.verizon.net. [68.160.176.52]) by smtp.gmail.com with ESMTPSA id o4-20020a05620a22c400b0067e02a697e0sm481848qki.33.2022.03.23.12.45.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 12:45:29 -0700 (PDT) From: Mike Snitzer X-Google-Original-From: Mike Snitzer To: axboe@kernel.dk Date: Wed, 23 Mar 2022 15:45:23 -0400 Message-Id: <20220323194524.5900-4-snitzer@kernel.org> In-Reply-To: <20220323194524.5900-1-snitzer@kernel.org> References: <20220323194524.5900-1-snitzer@kernel.org> X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Subject: [dm-devel] [PATCH v2 3/4] dm: enable BIOSET_PERCPU_CACHE for dm_io bioset X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, hch@lst.de, ming.lei@redhat.com MIME-Version: 1.0 Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Also change dm_io_complete() to use bio_clear_polled() so that it properly clears all associated bio state (REQ_POLLED, BIO_PERCPU_CACHE, etc). This commit improves DM's hipri bio polling (REQ_POLLED) perf by ~7%. Signed-off-by: Mike Snitzer --- drivers/md/dm.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 1c4d1e12d74b..b3cb2c1aea2a 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -899,9 +899,9 @@ static void dm_io_complete(struct dm_io *io) /* * Upper layer won't help us poll split bio, io->orig_bio * may only reflect a subset of the pre-split original, - * so clear REQ_POLLED in case of requeue + * so clear REQ_POLLED and BIO_PERCPU_CACHE on requeue. */ - bio->bi_opf &= ~REQ_POLLED; + bio_clear_polled(bio); return; } @@ -3016,7 +3016,7 @@ struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_qu pool_size = max(dm_get_reserved_bio_based_ios(), min_pool_size); front_pad = roundup(per_io_data_size, __alignof__(struct dm_target_io)) + DM_TARGET_IO_BIO_OFFSET; io_front_pad = roundup(per_io_data_size, __alignof__(struct dm_io)) + DM_IO_BIO_OFFSET; - ret = bioset_init(&pools->io_bs, pool_size, io_front_pad, 0); + ret = bioset_init(&pools->io_bs, pool_size, io_front_pad, BIOSET_PERCPU_CACHE); if (ret) goto out; if (integrity && bioset_integrity_create(&pools->io_bs, pool_size)) From patchwork Wed Mar 23 19:45:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 12790072 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 81B01C433F5 for ; Wed, 23 Mar 2022 19:45:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648064741; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=JfkWgAKW9mw++aaj4XaL5ywvlKII7sL3jqNobELI58g=; b=AR36Hm0CZ1rXsW1lVbCt+cvo7cAn8lJ+vowMXTbUfrap0jH73yk3KkYK1fEkGDIOM3UCNj v+ZjwR4PltUBVLBxV7/D9yl86FpIfBCwQSFu1VymwR+6TUywFb9nUc4a6Eb/72F4OufmsV V4P50u1Syf8fKf5G11Ox3APDRr1dH3A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-617-Ox1dn2C7P-aUyCh1NphRog-1; Wed, 23 Mar 2022 15:45:38 -0400 X-MC-Unique: Ox1dn2C7P-aUyCh1NphRog-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CDB0618A6585; Wed, 23 Mar 2022 19:45:35 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id BC03540CFD01; Wed, 23 Mar 2022 19:45:35 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 932CD194034E; Wed, 23 Mar 2022 19:45:35 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 400A4194035E for ; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 2CC9248FB14; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast03.extmail.prod.ext.rdu2.redhat.com [10.11.55.19]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 28C5846A38A for ; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0EE82811E90 for ; Wed, 23 Mar 2022 19:45:34 +0000 (UTC) Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-543-4OxaELMuPputS4SKSzB1Wg-1; Wed, 23 Mar 2022 15:45:32 -0400 X-MC-Unique: 4OxaELMuPputS4SKSzB1Wg-1 Received: by mail-qv1-f69.google.com with SMTP id o1-20020a0c9001000000b00440e415a3a2so2011844qvo.13 for ; Wed, 23 Mar 2022 12:45:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=AF9xOLertVL1zPER2DBlMssTlLLbZ7i6J9GUZUZZbT8=; b=QW9FTZO7tUKbhhSIJ83OlL0q5irfivVfWL7dTBXtaynNSbxQoaXew4Km8etDw+bstR UGibKzC1F07LDrbizZ1/y6T1zvIWQo74whJTwm+g1ITFss+DYrGvAQ93lRSp5OT8fD1o BttVQUy5DRTQaBfQ+bI3opUvAlmdOVzxrXowikG7dLwGlCcyWePU3JRZaq1Z38i/0E76 3LUkEX845ohH/wEdWJtH2OWbp4yrcYdnjY9iEUEIc8isXC6eyu2HviIyLoF7LPomghge ttKyfTQdd9JWfVYy3R5FyfVYfuOUDMak33I+VY42HfycByu+8iKjaWT/dYF8qO6/LX5z DHaQ== X-Gm-Message-State: AOAM532dcIRubSX8kKbRqImLL8kX+V725XjItQvbsIsm7dknaRZhqPXH pToiJquZA15xviIVCZpJKdxzaPxHbkq4C/ekt2eSmM39i0fpfbIigDOIWrzjMy0Z25osfLA6dnV 9aPbwUB/IW9uivA== X-Received: by 2002:ad4:5c6f:0:b0:440:cd9b:db9a with SMTP id i15-20020ad45c6f000000b00440cd9bdb9amr1340808qvh.86.1648064732106; Wed, 23 Mar 2022 12:45:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyGZdV6BdMXLbVGDokSS/YFyydCYRxSjfvid2Egnjd81a3tVo6BqGGKuI+kT/R4KaO8Ehs/6Q== X-Received: by 2002:ad4:5c6f:0:b0:440:cd9b:db9a with SMTP id i15-20020ad45c6f000000b00440cd9bdb9amr1340794qvh.86.1648064731874; Wed, 23 Mar 2022 12:45:31 -0700 (PDT) Received: from localhost (pool-68-160-176-52.bstnma.fios.verizon.net. [68.160.176.52]) by smtp.gmail.com with ESMTPSA id y24-20020a37e318000000b0067d43d76184sm445886qki.97.2022.03.23.12.45.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 12:45:31 -0700 (PDT) From: Mike Snitzer X-Google-Original-From: Mike Snitzer To: axboe@kernel.dk Date: Wed, 23 Mar 2022 15:45:24 -0400 Message-Id: <20220323194524.5900-5-snitzer@kernel.org> In-Reply-To: <20220323194524.5900-1-snitzer@kernel.org> References: <20220323194524.5900-1-snitzer@kernel.org> X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Subject: [dm-devel] [PATCH v2 4/4] dm: conditionally enable BIOSET_PERCPU_CACHE for bio-based dm_io bioset X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, hch@lst.de, ming.lei@redhat.com MIME-Version: 1.0 Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com A bioset's percpu cache may have broader utility in the future but for now constrain it to being tightly coupled to QUEUE_FLAG_POLL. Signed-off-by: Mike Snitzer --- drivers/md/dm-table.c | 11 ++++++++--- drivers/md/dm.c | 6 +++--- drivers/md/dm.h | 4 ++-- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index c0be4f60b427..7ebc70e3eb2f 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1002,6 +1002,8 @@ bool dm_table_request_based(struct dm_table *t) return __table_type_request_based(dm_table_get_type(t)); } +static int dm_table_supports_poll(struct dm_table *t); + static int dm_table_alloc_md_mempools(struct dm_table *t, struct mapped_device *md) { enum dm_queue_mode type = dm_table_get_type(t); @@ -1009,21 +1011,24 @@ static int dm_table_alloc_md_mempools(struct dm_table *t, struct mapped_device * unsigned min_pool_size = 0; struct dm_target *ti; unsigned i; + bool poll_supported = false; if (unlikely(type == DM_TYPE_NONE)) { DMWARN("no table type is set, can't allocate mempools"); return -EINVAL; } - if (__table_type_bio_based(type)) + if (__table_type_bio_based(type)) { for (i = 0; i < t->num_targets; i++) { ti = t->targets + i; per_io_data_size = max(per_io_data_size, ti->per_io_data_size); min_pool_size = max(min_pool_size, ti->num_flush_bios); } + poll_supported = !!dm_table_supports_poll(t); + } - t->mempools = dm_alloc_md_mempools(md, type, t->integrity_supported, - per_io_data_size, min_pool_size); + t->mempools = dm_alloc_md_mempools(md, type, per_io_data_size, min_pool_size, + t->integrity_supported, poll_supported); if (!t->mempools) return -ENOMEM; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index b3cb2c1aea2a..ebd7919e555f 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -2999,8 +2999,8 @@ int dm_noflush_suspending(struct dm_target *ti) EXPORT_SYMBOL_GPL(dm_noflush_suspending); struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_queue_mode type, - unsigned integrity, unsigned per_io_data_size, - unsigned min_pool_size) + unsigned per_io_data_size, unsigned min_pool_size, + bool integrity, bool poll) { struct dm_md_mempools *pools = kzalloc_node(sizeof(*pools), GFP_KERNEL, md->numa_node_id); unsigned int pool_size = 0; @@ -3016,7 +3016,7 @@ struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_qu pool_size = max(dm_get_reserved_bio_based_ios(), min_pool_size); front_pad = roundup(per_io_data_size, __alignof__(struct dm_target_io)) + DM_TARGET_IO_BIO_OFFSET; io_front_pad = roundup(per_io_data_size, __alignof__(struct dm_io)) + DM_IO_BIO_OFFSET; - ret = bioset_init(&pools->io_bs, pool_size, io_front_pad, BIOSET_PERCPU_CACHE); + ret = bioset_init(&pools->io_bs, pool_size, io_front_pad, poll ? BIOSET_PERCPU_CACHE : 0); if (ret) goto out; if (integrity && bioset_integrity_create(&pools->io_bs, pool_size)) diff --git a/drivers/md/dm.h b/drivers/md/dm.h index 9013dc1a7b00..3f89664fea01 100644 --- a/drivers/md/dm.h +++ b/drivers/md/dm.h @@ -221,8 +221,8 @@ void dm_kcopyd_exit(void); * Mempool operations */ struct dm_md_mempools *dm_alloc_md_mempools(struct mapped_device *md, enum dm_queue_mode type, - unsigned integrity, unsigned per_bio_data_size, - unsigned min_pool_size); + unsigned per_io_data_size, unsigned min_pool_size, + bool integrity, bool poll); void dm_free_md_mempools(struct dm_md_mempools *pools); /*