From patchwork Thu Oct 25 21:16:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10656607 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 24E7214DE for ; Thu, 25 Oct 2018 21:16:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19CDB2C463 for ; Thu, 25 Oct 2018 21:16:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E38C2C64C; Thu, 25 Oct 2018 21:16:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8CA742C463 for ; Thu, 25 Oct 2018 21:16:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726448AbeJZFu7 (ORCPT ); Fri, 26 Oct 2018 01:50:59 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:34474 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726065AbeJZFu7 (ORCPT ); Fri, 26 Oct 2018 01:50:59 -0400 Received: by mail-it1-f193.google.com with SMTP id e81-v6so3054159itc.1 for ; Thu, 25 Oct 2018 14:16:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9JynGiYF6jE1B6dobU9g3isZq38bzidmWC1EEymc56M=; b=pF3fnpXtNTJddMktxezMPbGE8vIJPJSS9k05L4J/mN2JCAVPTP0YEslMIOfLn+sS3Z RofOnLz94L1N2tpHA5LxCcu67fKgjWIbYGxjY3naJY4EUUO/TTTNofCbd0+VWX8Ycw+B GG4WbjyWaDbubcQprNdA+KTjxcjYOXELQwgFIiH8WeFyluJaarU662eNrzyl58bs4T8N KHbqW2XDwSJOgW7fvkhjgKZFPjK77YBfWBD+wF5PHOJr0eNr3pzK8CvIxQbhdcF/WqH/ 3wHkVTzlrClWVRPoyS9VTfXmNARX9eN1APLcEPK40YmfF4HYGg0ggEP3pz4IX2YCw3mu EsOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9JynGiYF6jE1B6dobU9g3isZq38bzidmWC1EEymc56M=; b=YBl2CLkMTNBAmcKm6VU/u/a6EPOYiM/WVSBZMf2MjP+YVw3uqPTOg7GFJuG4YeV5EL jNBrY2oQHRDTufK2bPFB8p7ka4/a/kTgo5nabAhvjp32ncDuxGwOz+ul4CN0XQbpPO6x OOaaoDp8+uvc41BmWgfY7izT2EIYvmndbwax8XxMFowlNwainHj+HbV/Lf/YhgJ91AUt oRnxTF4oOfY3rCcxVoAHTMFT+s43AFhLb4GDz3GDK+1H6UHntsyX/Qg8u5pBBpt0xZgQ DkLzgpKn0prRZLVWbL4ayg8qYQtnGCxvEyBtgaPa82kGgH5qa04lUlZR7N11mnrfYLqQ S32Q== X-Gm-Message-State: AGRZ1gLf+yF7xSXYEJbf4ULgStWpg9hV12U2eW+DNjvj0ZuL1DmuH/QO fQS1cwMStTTJXo5TVRT0J1ApSfNz/BBpmA== X-Google-Smtp-Source: AJdET5fSYSZeliIAbjevWpeJcn5mdtyKQqHoKt+ki6PAukjDrZLeXHxcC0jsAbx7OsjLzJNZH0jTtA== X-Received: by 2002:a24:6fc6:: with SMTP id x189-v6mr2068494itb.149.1540502199395; Thu, 25 Oct 2018 14:16:39 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id l23-v6sm2831890ioj.40.2018.10.25.14.16.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 Oct 2018 14:16:38 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Jens Axboe Subject: [PATCH 05/14] blk-mq: allow software queue to map to multiple hardware queues Date: Thu, 25 Oct 2018 15:16:17 -0600 Message-Id: <20181025211626.12692-6-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181025211626.12692-1-axboe@kernel.dk> References: <20181025211626.12692-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The mapping used to be dependent on just the CPU location, but now it's a tuple of { type, cpu} instead. This is a prep patch for allowing a single software queue to map to multiple hardware queues. No functional changes in this patch. Signed-off-by: Jens Axboe Reviewed-by: Hannes Reinecke --- block/blk-mq-sched.c | 2 +- block/blk-mq.c | 18 ++++++++++++------ block/blk-mq.h | 2 +- block/kyber-iosched.c | 6 +++--- include/linux/blk-mq.h | 3 ++- 5 files changed, 19 insertions(+), 12 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 8125e9393ec2..d232ecf3290c 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -110,7 +110,7 @@ static void blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) static struct blk_mq_ctx *blk_mq_next_ctx(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx) { - unsigned idx = ctx->index_hw; + unsigned short idx = ctx->index_hw[hctx->type]; if (++idx == hctx->nr_ctx) idx = 0; diff --git a/block/blk-mq.c b/block/blk-mq.c index e6ea7da99125..fab84c6bda18 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -75,14 +75,18 @@ static bool blk_mq_hctx_has_pending(struct blk_mq_hw_ctx *hctx) static void blk_mq_hctx_mark_pending(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx) { - if (!sbitmap_test_bit(&hctx->ctx_map, ctx->index_hw)) - sbitmap_set_bit(&hctx->ctx_map, ctx->index_hw); + const int bit = ctx->index_hw[hctx->type]; + + if (!sbitmap_test_bit(&hctx->ctx_map, bit)) + sbitmap_set_bit(&hctx->ctx_map, bit); } static void blk_mq_hctx_clear_pending(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx) { - sbitmap_clear_bit(&hctx->ctx_map, ctx->index_hw); + const int bit = ctx->index_hw[hctx->type]; + + sbitmap_clear_bit(&hctx->ctx_map, bit); } struct mq_inflight { @@ -954,7 +958,7 @@ static bool dispatch_rq_from_ctx(struct sbitmap *sb, unsigned int bitnr, struct request *blk_mq_dequeue_from_ctx(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *start) { - unsigned off = start ? start->index_hw : 0; + unsigned off = start ? start->index_hw[hctx->type] : 0; struct dispatch_rq_data data = { .hctx = hctx, .rq = NULL, @@ -2342,10 +2346,12 @@ static void blk_mq_map_swqueue(struct request_queue *q) ctx = per_cpu_ptr(q->queue_ctx, i); hctx = blk_mq_map_queue_type(q, 0, i); - + hctx->type = 0; cpumask_set_cpu(i, hctx->cpumask); - ctx->index_hw = hctx->nr_ctx; + ctx->index_hw[hctx->type] = hctx->nr_ctx; hctx->ctxs[hctx->nr_ctx++] = ctx; + /* wrap */ + BUG_ON(!hctx->nr_ctx); } mutex_unlock(&q->sysfs_lock); diff --git a/block/blk-mq.h b/block/blk-mq.h index 55428b92c019..7b5a790acdbf 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -17,7 +17,7 @@ struct blk_mq_ctx { } ____cacheline_aligned_in_smp; unsigned int cpu; - unsigned int index_hw; + unsigned short index_hw[HCTX_MAX_TYPES]; /* incremented at dispatch time */ unsigned long rq_dispatched[2]; diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index 728757a34fa0..b824a639d5d4 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -576,7 +576,7 @@ static bool kyber_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio) { struct kyber_hctx_data *khd = hctx->sched_data; struct blk_mq_ctx *ctx = blk_mq_get_ctx(hctx->queue); - struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw]; + struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw[hctx->type]]; unsigned int sched_domain = kyber_sched_domain(bio->bi_opf); struct list_head *rq_list = &kcq->rq_list[sched_domain]; bool merged; @@ -602,7 +602,7 @@ static void kyber_insert_requests(struct blk_mq_hw_ctx *hctx, list_for_each_entry_safe(rq, next, rq_list, queuelist) { unsigned int sched_domain = kyber_sched_domain(rq->cmd_flags); - struct kyber_ctx_queue *kcq = &khd->kcqs[rq->mq_ctx->index_hw]; + struct kyber_ctx_queue *kcq = &khd->kcqs[rq->mq_ctx->index_hw[hctx->type]]; struct list_head *head = &kcq->rq_list[sched_domain]; spin_lock(&kcq->lock); @@ -611,7 +611,7 @@ static void kyber_insert_requests(struct blk_mq_hw_ctx *hctx, else list_move_tail(&rq->queuelist, head); sbitmap_set_bit(&khd->kcq_map[sched_domain], - rq->mq_ctx->index_hw); + rq->mq_ctx->index_hw[hctx->type]); blk_mq_sched_request_inserted(rq); spin_unlock(&kcq->lock); } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index c992069bb3ee..72b36faf182d 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -37,7 +37,8 @@ struct blk_mq_hw_ctx { struct blk_mq_ctx *dispatch_from; unsigned int dispatch_busy; - unsigned int nr_ctx; + unsigned short type; + unsigned short nr_ctx; struct blk_mq_ctx **ctxs; spinlock_t dispatch_wait_lock;