From patchwork Fri Mar 22 13:13:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 10865639 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 94FBA922 for ; Fri, 22 Mar 2019 13:14:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7738928F2C for ; Fri, 22 Mar 2019 13:14:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6B80A2A79C; Fri, 22 Mar 2019 13:14:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2F3C328F2C for ; Fri, 22 Mar 2019 13:14:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729867AbfCVNOC (ORCPT ); Fri, 22 Mar 2019 09:14:02 -0400 Received: from mx2.suse.de ([195.135.220.15]:55166 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729586AbfCVNOB (ORCPT ); Fri, 22 Mar 2019 09:14:01 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id A27E9ABA1; Fri, 22 Mar 2019 13:13:58 +0000 (UTC) From: Johannes Thumshirn To: Jens Axboe Cc: Hannes Reinecke , Bart Van Assche , Christoph Hellwig , Jan Kara , Linux Block Layer Mailinglist , Linux FSDEVEL Mailinglist , Jens Axboe , Johannes Thumshirn Subject: [PATCH v2 1/3] block: bio: kill BIO_SEG_VALID flag Date: Fri, 22 Mar 2019 14:13:44 +0100 Message-Id: <20190322131346.20169-2-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190322131346.20169-1-jthumshirn@suse.de> References: <20190322131346.20169-1-jthumshirn@suse.de> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jens Axboe Kill the BIO_SEG_VALID flag. We should just use ->bi_phys_segments to tell if it's valid or not. This patch uses -1 to signify it's not. Signed-off-by: Jens Axboe [ jth initialize pc bios with 1 phys segment and WARN if we're submitting bios with -1 segments ] Signed-off-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig Reviewed-by: Hannes Reinecke --- block/bio.c | 15 +++++++++------ block/blk-core.c | 1 + block/blk-merge.c | 13 ++++++------- drivers/md/raid5.c | 2 +- include/linux/blk_types.h | 1 - 5 files changed, 17 insertions(+), 15 deletions(-) diff --git a/block/bio.c b/block/bio.c index 71a78d9fb8b7..a79b2bbcffd0 100644 --- a/block/bio.c +++ b/block/bio.c @@ -280,6 +280,7 @@ void bio_init(struct bio *bio, struct bio_vec *table, unsigned short max_vecs) { memset(bio, 0, sizeof(*bio)); + bio->bi_phys_segments = -1; atomic_set(&bio->__bi_remaining, 1); atomic_set(&bio->__bi_cnt, 1); @@ -305,6 +306,7 @@ void bio_reset(struct bio *bio) bio_uninit(bio); memset(bio, 0, BIO_RESET_BYTES); + bio->bi_phys_segments = -1; bio->bi_flags = flags; atomic_set(&bio->__bi_remaining, 1); } @@ -573,7 +575,7 @@ EXPORT_SYMBOL(bio_put); int bio_phys_segments(struct request_queue *q, struct bio *bio) { - if (unlikely(!bio_flagged(bio, BIO_SEG_VALID))) + if (unlikely(bio->bi_phys_segments == -1)) blk_recount_segments(q, bio); return bio->bi_phys_segments; @@ -712,7 +714,10 @@ int bio_add_pc_page(struct request_queue *q, struct bio *bio, struct page bvec->bv_len = len; bvec->bv_offset = offset; bio->bi_vcnt++; - bio->bi_phys_segments++; + if (bio->bi_phys_segments == -1) + bio->bi_phys_segments = 1; + else + bio->bi_phys_segments++; bio->bi_iter.bi_size += len; /* @@ -731,7 +736,7 @@ int bio_add_pc_page(struct request_queue *q, struct bio *bio, struct page /* If we may be able to merge these biovecs, force a recount */ if (bio->bi_vcnt > 1 && biovec_phys_mergeable(q, bvec - 1, bvec)) - bio_clear_flag(bio, BIO_SEG_VALID); + bio->bi_phys_segments = -1; done: return len; @@ -1913,10 +1918,8 @@ void bio_trim(struct bio *bio, int offset, int size) if (offset == 0 && size == bio->bi_iter.bi_size) return; - bio_clear_flag(bio, BIO_SEG_VALID); - + bio->bi_phys_segments = -1; bio_advance(bio, offset << 9); - bio->bi_iter.bi_size = size; if (bio_integrity(bio)) diff --git a/block/blk-core.c b/block/blk-core.c index 4673ebe42255..53372a16dd7c 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1514,6 +1514,7 @@ void blk_rq_bio_prep(struct request_queue *q, struct request *rq, else if (bio_op(bio) == REQ_OP_DISCARD) rq->nr_phys_segments = 1; + WARN_ON(rq->nr_phys_segments == -1); rq->__data_len = bio->bi_iter.bi_size; rq->bio = rq->biotail = bio; diff --git a/block/blk-merge.c b/block/blk-merge.c index 1c9d4f0f96ea..16c2ae69c46b 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -343,7 +343,6 @@ void blk_queue_split(struct request_queue *q, struct bio **bio) /* physical segments can be figured out during splitting */ res = split ? split : *bio; res->bi_phys_segments = nsegs; - bio_set_flag(res, BIO_SEG_VALID); if (split) { /* there isn't chance to merge the splitted bio */ @@ -440,8 +439,6 @@ void blk_recount_segments(struct request_queue *q, struct bio *bio) bio->bi_next = NULL; bio->bi_phys_segments = __blk_recalc_rq_segments(q, bio); bio->bi_next = nxt; - - bio_set_flag(bio, BIO_SEG_VALID); } static int blk_phys_contig_segment(struct request_queue *q, struct bio *bio, @@ -620,6 +617,8 @@ static inline int ll_new_hw_segment(struct request_queue *q, { int nr_phys_segs = bio_phys_segments(q, bio); + if (WARN_ON(nr_phys_segs == -1)) + nr_phys_segs = 0; if (req->nr_phys_segments + nr_phys_segs > queue_max_segments(q)) goto no_merge; @@ -651,9 +650,9 @@ int ll_back_merge_fn(struct request_queue *q, struct request *req, req_set_nomerge(q, req); return 0; } - if (!bio_flagged(req->biotail, BIO_SEG_VALID)) + if (req->biotail->bi_phys_segments == -1) blk_recount_segments(q, req->biotail); - if (!bio_flagged(bio, BIO_SEG_VALID)) + if (bio->bi_phys_segments == -1) blk_recount_segments(q, bio); return ll_new_hw_segment(q, req, bio); @@ -673,9 +672,9 @@ int ll_front_merge_fn(struct request_queue *q, struct request *req, req_set_nomerge(q, req); return 0; } - if (!bio_flagged(bio, BIO_SEG_VALID)) + if (bio->bi_phys_segments == -1) blk_recount_segments(q, bio); - if (!bio_flagged(req->bio, BIO_SEG_VALID)) + if (req->bio->bi_phys_segments == -1) blk_recount_segments(q, req->bio); return ll_new_hw_segment(q, req, bio); diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index c033bfcb209e..79eb54dcf0f9 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -5247,7 +5247,7 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio) rcu_read_unlock(); raid_bio->bi_next = (void*)rdev; bio_set_dev(align_bi, rdev->bdev); - bio_clear_flag(align_bi, BIO_SEG_VALID); + align_bi->bi_phys_segments = -1; if (is_badblock(rdev, align_bi->bi_iter.bi_sector, bio_sectors(align_bi), diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index d66bf5f32610..472059e92071 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -215,7 +215,6 @@ struct bio { /* * bio flags */ -#define BIO_SEG_VALID 1 /* bi_phys_segments valid */ #define BIO_CLONED 2 /* doesn't own data */ #define BIO_BOUNCED 3 /* bio is a bounce bio */ #define BIO_USER_MAPPED 4 /* contains user pages */ From patchwork Fri Mar 22 13:13:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 10865641 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CAB2C17E0 for ; Fri, 22 Mar 2019 13:14:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BBB7F2A795 for ; Fri, 22 Mar 2019 13:14:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B035128F2C; Fri, 22 Mar 2019 13:14:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EDB402A785 for ; Fri, 22 Mar 2019 13:14:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729737AbfCVNOA (ORCPT ); Fri, 22 Mar 2019 09:14:00 -0400 Received: from mx2.suse.de ([195.135.220.15]:55152 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728773AbfCVNOA (ORCPT ); Fri, 22 Mar 2019 09:14:00 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id A2FB1AC8C; Fri, 22 Mar 2019 13:13:58 +0000 (UTC) From: Johannes Thumshirn To: Jens Axboe Cc: Hannes Reinecke , Bart Van Assche , Christoph Hellwig , Jan Kara , Linux Block Layer Mailinglist , Linux FSDEVEL Mailinglist , Johannes Thumshirn Subject: [PATCH v2 2/3] block: bio: ensure newly added bio flags don't override BVEC_POOL_IDX Date: Fri, 22 Mar 2019 14:13:45 +0100 Message-Id: <20190322131346.20169-3-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190322131346.20169-1-jthumshirn@suse.de> References: <20190322131346.20169-1-jthumshirn@suse.de> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP With the introduction of BIO_ALLOCED we've used up all available bits in bio::bi_flags. Make sure no-one adds a new one and thus overrides the BVEC_POOL_IDX Signed-off-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig Reviewed-by: Hannes Reinecke --- block/bio.c | 3 +++ include/linux/blk_types.h | 25 ++++++++++++++----------- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/block/bio.c b/block/bio.c index a79b2bbcffd0..87a515e93bee 100644 --- a/block/bio.c +++ b/block/bio.c @@ -2198,6 +2198,9 @@ static int __init init_bio(void) bio_slab_nr = 0; bio_slabs = kcalloc(bio_slab_max, sizeof(struct bio_slab), GFP_KERNEL); + + BUILD_BUG_ON(BIO_FLAG_LAST > BVEC_POOL_OFFSET); + if (!bio_slabs) panic("bio: can't allocate bios\n"); diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 472059e92071..1d28992a20f0 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -215,19 +215,22 @@ struct bio { /* * bio flags */ -#define BIO_CLONED 2 /* doesn't own data */ -#define BIO_BOUNCED 3 /* bio is a bounce bio */ -#define BIO_USER_MAPPED 4 /* contains user pages */ -#define BIO_NULL_MAPPED 5 /* contains invalid user pages */ -#define BIO_QUIET 6 /* Make BIO Quiet */ -#define BIO_CHAIN 7 /* chained bio, ->bi_remaining in effect */ -#define BIO_REFFED 8 /* bio has elevated ->bi_cnt */ -#define BIO_THROTTLED 9 /* This bio has already been subjected to +enum { + BIO_CLONED, /* doesn't own data */ + BIO_BOUNCED, /* bio is a bounce bio */ + BIO_USER_MAPPED, /* contains user pages */ + BIO_NULL_MAPPED, /* contains invalid user pages */ + BIO_QUIET, /* Make BIO Quiet */ + BIO_CHAIN, /* chained bio, ->bi_remaining in effect */ + BIO_REFFED, /* bio has elevated ->bi_cnt */ + BIO_THROTTLED, /* This bio has already been subjected to * throttling rules. Don't do it again. */ -#define BIO_TRACE_COMPLETION 10 /* bio_endio() should trace the final completion + BIO_TRACE_COMPLETION, /* bio_endio() should trace the final completion * of this bio. */ -#define BIO_QUEUE_ENTERED 11 /* can use blk_queue_enter_live() */ -#define BIO_TRACKED 12 /* set if bio goes through the rq_qos path */ + BIO_QUEUE_ENTERED, /* can use blk_queue_enter_live() */ + BIO_TRACKED, /* set if bio goes through the rq_qos path */ + BIO_FLAG_LAST +}; /* See BVEC_POOL_OFFSET below before adding new flags */ From patchwork Fri Mar 22 13:13:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 10865635 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1BBD922 for ; Fri, 22 Mar 2019 13:14:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98F172A798 for ; Fri, 22 Mar 2019 13:14:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9732C2A79F; Fri, 22 Mar 2019 13:14:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 412292A798 for ; Fri, 22 Mar 2019 13:14:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729696AbfCVNOA (ORCPT ); Fri, 22 Mar 2019 09:14:00 -0400 Received: from mx2.suse.de ([195.135.220.15]:55160 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729682AbfCVNOA (ORCPT ); Fri, 22 Mar 2019 09:14:00 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id B3893AC94; Fri, 22 Mar 2019 13:13:58 +0000 (UTC) From: Johannes Thumshirn To: Jens Axboe Cc: Hannes Reinecke , Bart Van Assche , Christoph Hellwig , Jan Kara , Linux Block Layer Mailinglist , Linux FSDEVEL Mailinglist , Johannes Thumshirn Subject: [PATCH v2 3/3] block: bio: introduce BIO_ALLOCED flag and check it in bio_free Date: Fri, 22 Mar 2019 14:13:46 +0100 Message-Id: <20190322131346.20169-4-jthumshirn@suse.de> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190322131346.20169-1-jthumshirn@suse.de> References: <20190322131346.20169-1-jthumshirn@suse.de> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When we're submitting a bio from stack and this ends up being split, we call bio_put(). bio_put() will eventually call bio_free() if the reference count drops to 0. But freeing the bio is wrong, as it was never allocated out of the bio's mempool. Flag each normally allocated bio as 'BIO_ALLOCATED' and skip freeing if the flag isn't set. Fixes: 189ce2b9dcc3 ("block: fast-path for small and simple direct I/O requests") Signed-off-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig Reviewed-by: Hannes Reinecke --- block/bio.c | 4 ++++ include/linux/blk_types.h | 1 + 2 files changed, 5 insertions(+) diff --git a/block/bio.c b/block/bio.c index 87a515e93bee..ba6949f111b7 100644 --- a/block/bio.c +++ b/block/bio.c @@ -255,6 +255,9 @@ static void bio_free(struct bio *bio) bio_uninit(bio); + if (!bio_flagged(bio, BIO_ALLOCED)) + return; + if (bs) { bvec_free(&bs->bvec_pool, bio->bi_io_vec, BVEC_POOL_IDX(bio)); @@ -523,6 +526,7 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs, bvl = bio->bi_inline_vecs; } + bio_set_flag(bio, BIO_ALLOCED); bio->bi_pool = bs; bio->bi_max_vecs = nr_iovecs; bio->bi_io_vec = bvl; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 1d28992a20f0..19d7402a9af3 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -229,6 +229,7 @@ enum { * of this bio. */ BIO_QUEUE_ENTERED, /* can use blk_queue_enter_live() */ BIO_TRACKED, /* set if bio goes through the rq_qos path */ + BIO_ALLOCED, /* bio allocated by bio_alloc_bioset */ BIO_FLAG_LAST };