From patchwork Wed May 9 01:33:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 10387943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6F2FB60236 for ; Wed, 9 May 2018 01:34:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C2E528FF5 for ; Wed, 9 May 2018 01:34:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 30B6928FF7; Wed, 9 May 2018 01:34:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F2FF28FF5 for ; Wed, 9 May 2018 01:34:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 267E66B030D; Tue, 8 May 2018 21:34:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1F0246B030E; Tue, 8 May 2018 21:34:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0225F6B030F; Tue, 8 May 2018 21:34:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt0-f200.google.com (mail-qt0-f200.google.com [209.85.216.200]) by kanga.kvack.org (Postfix) with ESMTP id C5DCB6B030D for ; Tue, 8 May 2018 21:34:18 -0400 (EDT) Received: by mail-qt0-f200.google.com with SMTP id b10-v6so25728110qto.5 for ; Tue, 08 May 2018 18:34:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=/emOH7XvSPPri8tVw+ICyBNrOlNcFdAXg+itT4xycEc=; b=sc6NLHiZgOpQxePEkb3RXdboWFqJBe8dXVGyEhnnv36R/GLv2jIQuUoi0t+z+oXJ9/ 6GifXWQ+vqTOg+t7/KnFtyLmHW7lZVxOnZkSoJDf/bdhT3XD/CcbqwZ2mUHNZjpGexiH qvdTIjxlkhVczXcAXKPMt4dveYf/ac+3Nu9juKxPx95skSAr3/vkf0Z3mFZ/nd7DlVQL I9VrlXGTX7Sme/VVyfduqEVxFg49KdQiUQ1UVn3gLDm9sO1/5z6K10BUrgzcFEHB5sha HZU1NaERBdXvBKoBO33fBSw31PlladjNrGVkZlcnCopYkTd6bYl7YQyqAN1A6oiZMD8X 9pfA== X-Gm-Message-State: ALQs6tAjA8/AQVk6S3QpIYH+tBUygKMJp5Hhx2XHszqcmBCdJ0JheGBG K3aDJrCSB9QeqopMgOkDiPMSViPIvMDU1Xnft9WztrSmJNjelvZDW/teIlDJLtKZQ8RKx7s3bC7 hXmgosVzG95Q+HJbdIlEgpVAX3yfYe2lWwPKtKA4nAC4VZBQkrM1NiRhIfW1p7Ps8YSjYhX4EOk uWHD/jUDNbskXvbp6N16W8YFXLX4ITUyu7L9TpbKOkJ8ZgfZhAkY+YbdXQjQirJD9VvXWJS+kvv +EL8fN3lZx+tWzGAtYjiNv6ZgcxHosC80u4/KW0qxF1a8XRoJke84so9X4uOWX4f1sBLlMCb9in Hgx8KFkCNJJf+R7EeVibHCNX4jCDuCXMVp/7xmFR+hlMbRbAdasYkm2XN9cz6o0kp/pdoVpkPPj F X-Received: by 10.55.182.198 with SMTP id g189mr30624064qkf.334.1525829658497; Tue, 08 May 2018 18:34:18 -0700 (PDT) X-Received: by 10.55.182.198 with SMTP id g189mr30624029qkf.334.1525829657631; Tue, 08 May 2018 18:34:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525829657; cv=none; d=google.com; s=arc-20160816; b=Fp7Ak5DQxhaHQQzUVjnbZVpW33l7TIK4IJN/oTSMLK16Ts8Md7E06Zgb3Ko2Zz83kD FNH7Qs36WT8t9L5knlcmi4jK9IvjNBxAAe4GevfRzDTaDl33LmYvEVfLR7+6ac9sTmLB Nl4P0nk8yXuCZMmraQ2TXKzO0RK5FtaqDCxu8HRA1DUFb8N1swh7BvnOw6h6ahVaDrsP 4WuztoIzaCe9g+Eg+FG6E04r6TBF04H75QjuIGIFY49PcMLdmgaxVNQ/UI4Wx0R0AIx+ x/TjxOVRCWO6dzJSpLAXwmQrXG2zM3TRquSu/PK4ACctobFdD6OEAJ2FW8NpI38NxXU5 RItQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=/emOH7XvSPPri8tVw+ICyBNrOlNcFdAXg+itT4xycEc=; b=VD+jlg2OS3Oemxa2R+/WNNF/no4ELwOiuW+M9wrjx0Zmyla6kDg4fN/0MsXkAyxqem 3KMgXnauHKWBik4XAx7uzg2Yn5Y1+E+Ump9dcd0hjVTj0eo8Kfxix1mOfGPcgScqYGxj Dxm0L6LUUCpN5mxtxCa9RowrYpyW3qnzGG796t2t5hStRfMnnzcgdoVy2/fElROksjlu n6lylBlPZxAVoYHjPsP+XxpKEfjz1qrUyocHN8vFQVcEBlpKYieUrFRFwgmsDEYJ9z2l fy7iSu9IiN2dbJ4QZQvNHdJ6lGQALdp7ktv/14OpaLZj88Uc1UHNl39bsHCYywEYg5vP 53jA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=C/00x1Eh; spf=pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kent.overstreet@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id h31-v6sor14974356qte.44.2018.05.08.18.34.17 for (Google Transport Security); Tue, 08 May 2018 18:34:17 -0700 (PDT) Received-SPF: pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=C/00x1Eh; spf=pass (google.com: domain of kent.overstreet@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=kent.overstreet@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/emOH7XvSPPri8tVw+ICyBNrOlNcFdAXg+itT4xycEc=; b=C/00x1EhVpru7GtswN8r3piGKOum4Qk4lQ3is/AITaI4YFD8TRW9G6yxc9gK+zAu5n sQFwTxywxjNvHkdSIMRZ+Fb5KRQPqrqws6sksbpAahbZ0XPqZhKiU6BBxMa4GmEpv5ck /xTQWI1o87BttrHD3M5GjTpHB3xmMwv/7ZsoefKD2zB1i67BMN4MX/6KLMwMhWocijV9 KPIWK7GLxZg9Eb4Br3Y0BaihjYAYZdauWwg+JsSKVEV7tNlFRiiuHHoB5Epps1u4/DAJ TERHLIUsD//yi3W1IgOfdqg5s/iP8nZApX9b8ekEUWiYrGYvcrNCadIeeSsq8PHJjtCz Wf/w== X-Google-Smtp-Source: AB8JxZq1+P10dIKDbUzNocS57e6KenIRw/QbEk+Q+S8GRGbM+U6x+m4nz1F6jskGK3NBIO+KGd8cdQ== X-Received: by 2002:aed:3a46:: with SMTP id n64-v6mr30336394qte.118.1525829657203; Tue, 08 May 2018 18:34:17 -0700 (PDT) Received: from localhost.localdomain (c-71-234-172-214.hsd1.vt.comcast.net. [71.234.172.214]) by smtp.gmail.com with ESMTPSA id x28-v6sm23719003qtx.95.2018.05.08.18.34.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 May 2018 18:34:16 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, Jens Axboe , Ingo Molnar Cc: Kent Overstreet Subject: [PATCH 02/10] block: Convert bio_set to mempool_init() Date: Tue, 8 May 2018 21:33:50 -0400 Message-Id: <20180509013358.16399-3-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180509013358.16399-1-kent.overstreet@gmail.com> References: <20180509013358.16399-1-kent.overstreet@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Minor performance improvement by getting rid of pointer indirections from allocation/freeing fastpaths. Signed-off-by: Kent Overstreet --- block/bio-integrity.c | 29 ++++++++++++++--------------- block/bio.c | 36 +++++++++++++++++------------------- include/linux/bio.h | 10 +++++----- 3 files changed, 36 insertions(+), 39 deletions(-) diff --git a/block/bio-integrity.c b/block/bio-integrity.c index 9cfdd6c83b..add7c7c853 100644 --- a/block/bio-integrity.c +++ b/block/bio-integrity.c @@ -56,12 +56,12 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, struct bio_set *bs = bio->bi_pool; unsigned inline_vecs; - if (!bs || !bs->bio_integrity_pool) { + if (!bs || !mempool_initialized(&bs->bio_integrity_pool)) { bip = kmalloc(sizeof(struct bio_integrity_payload) + sizeof(struct bio_vec) * nr_vecs, gfp_mask); inline_vecs = nr_vecs; } else { - bip = mempool_alloc(bs->bio_integrity_pool, gfp_mask); + bip = mempool_alloc(&bs->bio_integrity_pool, gfp_mask); inline_vecs = BIP_INLINE_VECS; } @@ -74,7 +74,7 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, unsigned long idx = 0; bip->bip_vec = bvec_alloc(gfp_mask, nr_vecs, &idx, - bs->bvec_integrity_pool); + &bs->bvec_integrity_pool); if (!bip->bip_vec) goto err; bip->bip_max_vcnt = bvec_nr_vecs(idx); @@ -90,7 +90,7 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, return bip; err: - mempool_free(bip, bs->bio_integrity_pool); + mempool_free(bip, &bs->bio_integrity_pool); return ERR_PTR(-ENOMEM); } EXPORT_SYMBOL(bio_integrity_alloc); @@ -111,10 +111,10 @@ static void bio_integrity_free(struct bio *bio) kfree(page_address(bip->bip_vec->bv_page) + bip->bip_vec->bv_offset); - if (bs && bs->bio_integrity_pool) { - bvec_free(bs->bvec_integrity_pool, bip->bip_vec, bip->bip_slab); + if (bs && mempool_initialized(&bs->bio_integrity_pool)) { + bvec_free(&bs->bvec_integrity_pool, bip->bip_vec, bip->bip_slab); - mempool_free(bip, bs->bio_integrity_pool); + mempool_free(bip, &bs->bio_integrity_pool); } else { kfree(bip); } @@ -465,16 +465,15 @@ EXPORT_SYMBOL(bio_integrity_clone); int bioset_integrity_create(struct bio_set *bs, int pool_size) { - if (bs->bio_integrity_pool) + if (mempool_initialized(&bs->bio_integrity_pool)) return 0; - bs->bio_integrity_pool = mempool_create_slab_pool(pool_size, bip_slab); - if (!bs->bio_integrity_pool) + if (mempool_init_slab_pool(&bs->bio_integrity_pool, + pool_size, bip_slab)) return -1; - bs->bvec_integrity_pool = biovec_create_pool(pool_size); - if (!bs->bvec_integrity_pool) { - mempool_destroy(bs->bio_integrity_pool); + if (biovec_init_pool(&bs->bvec_integrity_pool, pool_size)) { + mempool_exit(&bs->bio_integrity_pool); return -1; } @@ -484,8 +483,8 @@ EXPORT_SYMBOL(bioset_integrity_create); void bioset_integrity_free(struct bio_set *bs) { - mempool_destroy(bs->bio_integrity_pool); - mempool_destroy(bs->bvec_integrity_pool); + mempool_exit(&bs->bio_integrity_pool); + mempool_exit(&bs->bvec_integrity_pool); } EXPORT_SYMBOL(bioset_integrity_free); diff --git a/block/bio.c b/block/bio.c index e1708db482..360e9bcea5 100644 --- a/block/bio.c +++ b/block/bio.c @@ -254,7 +254,7 @@ static void bio_free(struct bio *bio) bio_uninit(bio); if (bs) { - bvec_free(bs->bvec_pool, bio->bi_io_vec, BVEC_POOL_IDX(bio)); + bvec_free(&bs->bvec_pool, bio->bi_io_vec, BVEC_POOL_IDX(bio)); /* * If we have front padding, adjust the bio pointer before freeing @@ -262,7 +262,7 @@ static void bio_free(struct bio *bio) p = bio; p -= bs->front_pad; - mempool_free(p, bs->bio_pool); + mempool_free(p, &bs->bio_pool); } else { /* Bio was allocated by bio_kmalloc() */ kfree(bio); @@ -454,7 +454,8 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, inline_vecs = nr_iovecs; } else { /* should not use nobvec bioset for nr_iovecs > 0 */ - if (WARN_ON_ONCE(!bs->bvec_pool && nr_iovecs > 0)) + if (WARN_ON_ONCE(!mempool_initialized(&bs->bvec_pool) && + nr_iovecs > 0)) return NULL; /* * generic_make_request() converts recursion to iteration; this @@ -483,11 +484,11 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, bs->rescue_workqueue) gfp_mask &= ~__GFP_DIRECT_RECLAIM; - p = mempool_alloc(bs->bio_pool, gfp_mask); + p = mempool_alloc(&bs->bio_pool, gfp_mask); if (!p && gfp_mask != saved_gfp) { punt_bios_to_rescuer(bs); gfp_mask = saved_gfp; - p = mempool_alloc(bs->bio_pool, gfp_mask); + p = mempool_alloc(&bs->bio_pool, gfp_mask); } front_pad = bs->front_pad; @@ -503,11 +504,11 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, if (nr_iovecs > inline_vecs) { unsigned long idx = 0; - bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, bs->bvec_pool); + bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, &bs->bvec_pool); if (!bvl && gfp_mask != saved_gfp) { punt_bios_to_rescuer(bs); gfp_mask = saved_gfp; - bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, bs->bvec_pool); + bvl = bvec_alloc(gfp_mask, nr_iovecs, &idx, &bs->bvec_pool); } if (unlikely(!bvl)) @@ -524,7 +525,7 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, return bio; err_free: - mempool_free(p, bs->bio_pool); + mempool_free(p, &bs->bio_pool); return NULL; } EXPORT_SYMBOL(bio_alloc_bioset); @@ -1848,11 +1849,11 @@ EXPORT_SYMBOL_GPL(bio_trim); * create memory pools for biovec's in a bio_set. * use the global biovec slabs created for general use. */ -mempool_t *biovec_create_pool(int pool_entries) +int biovec_init_pool(mempool_t *pool, int pool_entries) { struct biovec_slab *bp = bvec_slabs + BVEC_POOL_MAX; - return mempool_create_slab_pool(pool_entries, bp->slab); + return mempool_init_slab_pool(pool, pool_entries, bp->slab); } void bioset_free(struct bio_set *bs) @@ -1860,8 +1861,8 @@ void bioset_free(struct bio_set *bs) if (bs->rescue_workqueue) destroy_workqueue(bs->rescue_workqueue); - mempool_destroy(bs->bio_pool); - mempool_destroy(bs->bvec_pool); + mempool_exit(&bs->bio_pool); + mempool_exit(&bs->bvec_pool); bioset_integrity_free(bs); bio_put_slab(bs); @@ -1913,15 +1914,12 @@ struct bio_set *bioset_create(unsigned int pool_size, return NULL; } - bs->bio_pool = mempool_create_slab_pool(pool_size, bs->bio_slab); - if (!bs->bio_pool) + if (mempool_init_slab_pool(&bs->bio_pool, pool_size, bs->bio_slab)) goto bad; - if (flags & BIOSET_NEED_BVECS) { - bs->bvec_pool = biovec_create_pool(pool_size); - if (!bs->bvec_pool) - goto bad; - } + if ((flags & BIOSET_NEED_BVECS) && + biovec_init_pool(&bs->bvec_pool, pool_size)) + goto bad; if (!(flags & BIOSET_NEED_RESCUER)) return bs; diff --git a/include/linux/bio.h b/include/linux/bio.h index ce547a25e8..720f7261d0 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -412,7 +412,7 @@ enum { BIOSET_NEED_RESCUER = BIT(1), }; extern void bioset_free(struct bio_set *); -extern mempool_t *biovec_create_pool(int pool_entries); +extern int biovec_init_pool(mempool_t *pool, int pool_entries); extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set *); extern void bio_put(struct bio *); @@ -722,11 +722,11 @@ struct bio_set { struct kmem_cache *bio_slab; unsigned int front_pad; - mempool_t *bio_pool; - mempool_t *bvec_pool; + mempool_t bio_pool; + mempool_t bvec_pool; #if defined(CONFIG_BLK_DEV_INTEGRITY) - mempool_t *bio_integrity_pool; - mempool_t *bvec_integrity_pool; + mempool_t bio_integrity_pool; + mempool_t bvec_integrity_pool; #endif /*