diff mbox series

[01/14] block: inline fast path of driver tag allocation

Message ID 20211017013748.76461-2-axboe@kernel.dk (mailing list archive)
State New, archived
Headers show
Series Various block layer optimizations | expand

Commit Message

Jens Axboe Oct. 17, 2021, 1:37 a.m. UTC 
If we don't use an IO scheduler or have shared tags, then we don't need
to call into this external function at all. This saves ~2% for such
a setup.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c |  8 +++-----
 block/blk-mq.h | 15 ++++++++++++++-
 2 files changed, 17 insertions(+), 6 deletions(-)

Comments

Christoph Hellwig Oct. 18, 2021, 8:42 a.m. UTC | #1 
On Sat, Oct 16, 2021 at 07:37:35PM -0600, Jens Axboe wrote:
> If we don't use an IO scheduler or have shared tags, then we don't need
> to call into this external function at all. This saves ~2% for such
> a setup.

This looks correct, although the call chain gets a little confusing
now.  How much difference is this over basically inlining the
whole old blk_mq_get_driver_tag?
Jens Axboe Oct. 18, 2021, 2:38 p.m. UTC | #2 
On 10/18/21 2:42 AM, Christoph Hellwig wrote:
> On Sat, Oct 16, 2021 at 07:37:35PM -0600, Jens Axboe wrote:
>> If we don't use an IO scheduler or have shared tags, then we don't need
>> to call into this external function at all. This saves ~2% for such
>> a setup.
> 
> This looks correct, although the call chain gets a little confusing
> now.  How much difference is this over basically inlining the
> whole old blk_mq_get_driver_tag?

It just condenses the fast path into the checks upfront. I can run the
numbers, but it'll be extra work.
diff mbox series

Patch

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 1bbe5de66c40..90bc93fe373e 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1145,7 +1145,7 @@  static inline unsigned int queued_to_index(unsigned int queued)
 	return min(BLK_MQ_MAX_DISPATCH_ORDER - 1, ilog2(queued) + 1);
 }
 
-static bool __blk_mq_get_driver_tag(struct request *rq)
+static bool __blk_mq_alloc_driver_tag(struct request *rq)
 {
 	struct sbitmap_queue *bt = &rq->mq_hctx->tags->bitmap_tags;
 	unsigned int tag_offset = rq->mq_hctx->tags->nr_reserved_tags;
@@ -1169,11 +1169,9 @@  static bool __blk_mq_get_driver_tag(struct request *rq)
 	return true;
 }
 
-bool blk_mq_get_driver_tag(struct request *rq)
+bool __blk_mq_get_driver_tag(struct blk_mq_hw_ctx *hctx, struct request *rq)
 {
-	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
-
-	if (rq->tag == BLK_MQ_NO_TAG && !__blk_mq_get_driver_tag(rq))
+	if (rq->tag == BLK_MQ_NO_TAG && !__blk_mq_alloc_driver_tag(rq))
 		return false;
 
 	if ((hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED) &&
diff --git a/block/blk-mq.h b/block/blk-mq.h
index 8be447995106..ceed0a001c76 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -264,7 +264,20 @@  static inline void blk_mq_put_driver_tag(struct request *rq)
 	__blk_mq_put_driver_tag(rq->mq_hctx, rq);
 }
 
-bool blk_mq_get_driver_tag(struct request *rq);
+bool __blk_mq_get_driver_tag(struct blk_mq_hw_ctx *hctx, struct request *rq);
+
+static inline bool blk_mq_get_driver_tag(struct request *rq)
+{
+	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
+
+	if (rq->tag != BLK_MQ_NO_TAG &&
+	    !(hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED)) {
+		hctx->tags->rqs[rq->tag] = rq;
+		return true;
+	}
+
+	return __blk_mq_get_driver_tag(hctx, rq);
+}
 
 static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap)
 {