From patchwork Mon Oct 29 16:37:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659747 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4339113B5 for ; Mon, 29 Oct 2018 16:39:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2967729BC9 for ; Mon, 29 Oct 2018 16:39:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1E12929BCE; Mon, 29 Oct 2018 16:39:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B466429BC9 for ; Mon, 29 Oct 2018 16:39:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728025AbeJ3B1I (ORCPT ); Mon, 29 Oct 2018 21:27:08 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:52159 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728024AbeJ3B1G (ORCPT ); Mon, 29 Oct 2018 21:27:06 -0400 Received: by mail-it1-f193.google.com with SMTP id y123-v6so4107528itc.1 for ; Mon, 29 Oct 2018 09:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fb0X3O/E/9T8J1C6SoUrkl44F/lVWoM3/BiHGQqatJw=; b=Dr/WvqsDe4prXI3xbC+Nav/VrVMySw+xGxDEHttOprLECOE+mQ/z7G1iBAQttO0YIx 2BYuEGGl5ONSQCjof3IueTI0cJUhUJUde/2ttFgXlEnuTEB0zRv/3wxSGayLRLx1MgMZ Oo1RM/cjoKB7I2trz27JGGvEblPoa6yIEWcXckIrYNQOECEMz9ObwWjJcppQdtbi4dSJ lZTa+km8Z8f10kEmH7Dizt2J/bSdKjeTQrQjbQA/QOxCsX6TQLM/ZanofrN07dfZUrha AVQ3acVQOnnK++h318jwVzEF6MBfjBWrDTkDt+ykjJl7TkDRg/9KVAi8Xv1z543N/t8J Bp0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=fb0X3O/E/9T8J1C6SoUrkl44F/lVWoM3/BiHGQqatJw=; b=obLlU0OS+0raj3O1lQOCIp3zabreYz7eBKBUZUqMk1TfZRFzL1RmrNCT4fD4CtkEaX QesNsEKu2HGeGTmlhdI1XDn+CmpP0yof0A4mD4RbOmM1USp/9Ol49tXLRTr00sRgnbv8 6vtnD5KI2DXyzX0f+cFqTTAWfRY6XEGyAPNRfydWJ0Kh0X2xx0zQuwKO/VoJ0F15jV4r /RmxU05CVzO6GmwLasiZYjrHephJYnr2Zs53KUKOPuq5R3ns3yxpKUVbZhknlg2jZZ/K lk8zn3YBiyvWvXOWtaTftMhG/InSWD5PqXC9+yjSID8ueS9zcJGrTpWn07k9MHV8qxxb 7QHw== X-Gm-Message-State: AGRZ1gLYzNKcBDV+VWZiTIVzCZye8LttxmgEbnXbCbGf6lb7HgYFd3PP d0WIdEd+0RnQe5WhTQSR9U7clpeuM18= X-Google-Smtp-Source: AJdET5dol1JlA+W/SgeBOkRXCGtqnXIcJID0gqbW+4yNWwVSdYPlsRPE4zQRTRE1VrxPSlfkgWb9cQ== X-Received: by 2002:a24:21d5:: with SMTP id e204-v6mr10430754ita.127.1540831064266; Mon, 29 Oct 2018 09:37:44 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:43 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 01/14] blk-mq: kill q->mq_map Date: Mon, 29 Oct 2018 10:37:25 -0600 Message-Id: <20181029163738.10172-2-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It's just a pointer to set->mq_map, use that instead. Reviewed-by: Christoph Hellwig Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe Reviewed-by: Bart Van Assche --- block/blk-mq.c | 13 ++++--------- block/blk-mq.h | 4 +++- include/linux/blkdev.h | 2 -- 3 files changed, 7 insertions(+), 12 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 21e4147c4810..22d5beaab5a0 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2321,7 +2321,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) * If the cpu isn't present, the cpu is mapped to first hctx. */ for_each_possible_cpu(i) { - hctx_idx = q->mq_map[i]; + hctx_idx = set->mq_map[i]; /* unmapped hw queue can be remapped after CPU topo changed */ if (!set->tags[hctx_idx] && !__blk_mq_alloc_rq_map(set, hctx_idx)) { @@ -2331,7 +2331,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) * case, remap the current ctx to hctx[0] which * is guaranteed to always have tags allocated */ - q->mq_map[i] = 0; + set->mq_map[i] = 0; } ctx = per_cpu_ptr(q->queue_ctx, i); @@ -2429,8 +2429,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q) static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set, struct request_queue *q) { - q->tag_set = set; - mutex_lock(&set->tag_list_lock); /* @@ -2467,8 +2465,6 @@ void blk_mq_release(struct request_queue *q) kobject_put(&hctx->kobj); } - q->mq_map = NULL; - kfree(q->queue_hw_ctx); /* @@ -2588,7 +2584,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, int node; struct blk_mq_hw_ctx *hctx; - node = blk_mq_hw_queue_to_node(q->mq_map, i); + node = blk_mq_hw_queue_to_node(set->mq_map, i); /* * If the hw queue has been mapped to another numa node, * we need to realloc the hctx. If allocation fails, fallback @@ -2665,8 +2661,6 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, if (!q->queue_hw_ctx) goto err_percpu; - q->mq_map = set->mq_map; - blk_mq_realloc_hw_ctxs(set, q); if (!q->nr_hw_queues) goto err_hctxs; @@ -2675,6 +2669,7 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, blk_queue_rq_timeout(q, set->timeout ? set->timeout : 30 * HZ); q->nr_queues = nr_cpu_ids; + q->tag_set = set; q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT; diff --git a/block/blk-mq.h b/block/blk-mq.h index 9497b47e2526..9536be06d022 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -75,7 +75,9 @@ extern int blk_mq_hw_queue_to_node(unsigned int *map, unsigned int); static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, int cpu) { - return q->queue_hw_ctx[q->mq_map[cpu]]; + struct blk_mq_tag_set *set = q->tag_set; + + return q->queue_hw_ctx[set->mq_map[cpu]]; } /* diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index c675e2b5af62..4223ae2d2198 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -412,8 +412,6 @@ struct request_queue { const struct blk_mq_ops *mq_ops; - unsigned int *mq_map; - /* sw queues */ struct blk_mq_ctx __percpu *queue_ctx; unsigned int nr_queues; From patchwork Mon Oct 29 16:37:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 253531751 for ; Mon, 29 Oct 2018 16:39:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A7D729BC1 for ; Mon, 29 Oct 2018 16:39:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F303C29BCA; Mon, 29 Oct 2018 16:39:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E36D829BC1 for ; Mon, 29 Oct 2018 16:39:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728027AbeJ3B2f (ORCPT ); Mon, 29 Oct 2018 21:28:35 -0400 Received: from mail-it1-f196.google.com ([209.85.166.196]:39226 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728074AbeJ3B1J (ORCPT ); Mon, 29 Oct 2018 21:27:09 -0400 Received: by mail-it1-f196.google.com with SMTP id m15so10170305itl.4 for ; Mon, 29 Oct 2018 09:37:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=70QpW4qhrpAmGnXsQsDjnn6bIgpmJRF0zjYtC71FW9o=; b=k+P7FirqGigtseHfvGKxPU/J5QVhPUuJv7u93h1DHf+aRE+DrdaY0W9Eb61woyEIMn 5YCT3jLJ5mldik5dzSrbnw9/TzDRmBZGQOZdhleVKaaIj4/uX7+jMZCilDkcU2Rk1u2t JOOkbcjCmS/6AOK/dUAUr1shEaYy4lXn40p1CaaLSDKhvnhItFJTFWUlBI4elEE9Ea/O eSdbJW5FO3LhTPCa2iOfSGtUQA5lYZRTLqfcwJpOhhwIy+3ezWuvWi1eAm2RE4mTaLl5 6ZPFK7GX58GUf8HujoBXnT2SZsdJnoVTXVNBnMJuk/zrwSzVTufM5wjIHqY3dGl77weC bKIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=70QpW4qhrpAmGnXsQsDjnn6bIgpmJRF0zjYtC71FW9o=; b=BBnRaRYK/YyEByWunKNFn0IUVrI8uvBNLbnc+AeUzsEui+9BkMIQWkaaq1Y97j9SGD RrvnyM1eISVEaNhOAsfv00qKja5BtMvBh6g2ghu7MdsH26s1oYsDgeTAsZLLMW+LKLiL jaxP0LHYapM382ZwniQHbQL7YUOoh/srNvkdyiOACrXLafFOn68Lri+TfILrESlHDoHr 4DaD6JMNHjwFnxynaeceukHbatGHlv66vpHcpL61YJmN8hWNpUml3eFpPCNDJO43xXLk ZkNFsvShyNVnpaOdWGCIWCEIbiJD3AO8bbONaLjj/ByrrNXpCCTe1mbSE9R0VLcu3Xst pEfw== X-Gm-Message-State: AGRZ1gJpqc+6tUo9OAOWQVpT3VY5LAqM7gv/3K3cQY0ti6xwLb2X3Sc+ WOotyn2lIEoIsKYQaWbuTbYEKpyIc3M= X-Google-Smtp-Source: AJdET5d6pUA1WrENd3A2FkaQ6GS+WlQXw82BanmZio7MyXTYlaQcdwZ0ks1budZBQNSdljXowngPww== X-Received: by 2002:a24:3c0a:: with SMTP id m10-v6mr10622219ita.15.1540831066027; Mon, 29 Oct 2018 09:37:46 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:44 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 02/14] blk-mq: abstract out queue map Date: Mon, 29 Oct 2018 10:37:26 -0600 Message-Id: <20181029163738.10172-3-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is in preparation for allowing multiple sets of maps per queue, if so desired. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe Reviewed-by: Bart Van Assche --- block/blk-mq-cpumap.c | 10 ++++---- block/blk-mq-pci.c | 10 ++++---- block/blk-mq-rdma.c | 4 ++-- block/blk-mq-virtio.c | 8 +++---- block/blk-mq.c | 34 ++++++++++++++------------- block/blk-mq.h | 8 +++---- drivers/block/virtio_blk.c | 2 +- drivers/nvme/host/pci.c | 2 +- drivers/scsi/qla2xxx/qla_os.c | 5 ++-- drivers/scsi/scsi_lib.c | 2 +- drivers/scsi/smartpqi/smartpqi_init.c | 3 ++- drivers/scsi/virtio_scsi.c | 3 ++- include/linux/blk-mq-pci.h | 4 ++-- include/linux/blk-mq-virtio.h | 4 ++-- include/linux/blk-mq.h | 13 ++++++++-- 15 files changed, 63 insertions(+), 49 deletions(-) diff --git a/block/blk-mq-cpumap.c b/block/blk-mq-cpumap.c index 3eb169f15842..6e6686c55984 100644 --- a/block/blk-mq-cpumap.c +++ b/block/blk-mq-cpumap.c @@ -30,10 +30,10 @@ static int get_first_sibling(unsigned int cpu) return cpu; } -int blk_mq_map_queues(struct blk_mq_tag_set *set) +int blk_mq_map_queues(struct blk_mq_queue_map *qmap) { - unsigned int *map = set->mq_map; - unsigned int nr_queues = set->nr_hw_queues; + unsigned int *map = qmap->mq_map; + unsigned int nr_queues = qmap->nr_queues; unsigned int cpu, first_sibling; for_each_possible_cpu(cpu) { @@ -62,12 +62,12 @@ EXPORT_SYMBOL_GPL(blk_mq_map_queues); * We have no quick way of doing reverse lookups. This is only used at * queue init time, so runtime isn't important. */ -int blk_mq_hw_queue_to_node(unsigned int *mq_map, unsigned int index) +int blk_mq_hw_queue_to_node(struct blk_mq_queue_map *qmap, unsigned int index) { int i; for_each_possible_cpu(i) { - if (index == mq_map[i]) + if (index == qmap->mq_map[i]) return local_memory_node(cpu_to_node(i)); } diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c index db644ec624f5..40333d60a850 100644 --- a/block/blk-mq-pci.c +++ b/block/blk-mq-pci.c @@ -31,26 +31,26 @@ * that maps a queue to the CPUs that have irq affinity for the corresponding * vector. */ -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, +int blk_mq_pci_map_queues(struct blk_mq_queue_map *qmap, struct pci_dev *pdev, int offset) { const struct cpumask *mask; unsigned int queue, cpu; - for (queue = 0; queue < set->nr_hw_queues; queue++) { + for (queue = 0; queue < qmap->nr_queues; queue++) { mask = pci_irq_get_affinity(pdev, queue + offset); if (!mask) goto fallback; for_each_cpu(cpu, mask) - set->mq_map[cpu] = queue; + qmap->mq_map[cpu] = queue; } return 0; fallback: - WARN_ON_ONCE(set->nr_hw_queues > 1); - blk_mq_clear_mq_map(set); + WARN_ON_ONCE(qmap->nr_queues > 1); + blk_mq_clear_mq_map(qmap); return 0; } EXPORT_SYMBOL_GPL(blk_mq_pci_map_queues); diff --git a/block/blk-mq-rdma.c b/block/blk-mq-rdma.c index 996167f1de18..a71576aff3a5 100644 --- a/block/blk-mq-rdma.c +++ b/block/blk-mq-rdma.c @@ -41,12 +41,12 @@ int blk_mq_rdma_map_queues(struct blk_mq_tag_set *set, goto fallback; for_each_cpu(cpu, mask) - set->mq_map[cpu] = queue; + set->map[0].mq_map[cpu] = queue; } return 0; fallback: - return blk_mq_map_queues(set); + return blk_mq_map_queues(&set->map[0]); } EXPORT_SYMBOL_GPL(blk_mq_rdma_map_queues); diff --git a/block/blk-mq-virtio.c b/block/blk-mq-virtio.c index c3afbca11299..661fbfef480f 100644 --- a/block/blk-mq-virtio.c +++ b/block/blk-mq-virtio.c @@ -29,7 +29,7 @@ * that maps a queue to the CPUs that have irq affinity for the corresponding * vector. */ -int blk_mq_virtio_map_queues(struct blk_mq_tag_set *set, +int blk_mq_virtio_map_queues(struct blk_mq_queue_map *qmap, struct virtio_device *vdev, int first_vec) { const struct cpumask *mask; @@ -38,17 +38,17 @@ int blk_mq_virtio_map_queues(struct blk_mq_tag_set *set, if (!vdev->config->get_vq_affinity) goto fallback; - for (queue = 0; queue < set->nr_hw_queues; queue++) { + for (queue = 0; queue < qmap->nr_queues; queue++) { mask = vdev->config->get_vq_affinity(vdev, first_vec + queue); if (!mask) goto fallback; for_each_cpu(cpu, mask) - set->mq_map[cpu] = queue; + qmap->mq_map[cpu] = queue; } return 0; fallback: - return blk_mq_map_queues(set); + return blk_mq_map_queues(qmap); } EXPORT_SYMBOL_GPL(blk_mq_virtio_map_queues); diff --git a/block/blk-mq.c b/block/blk-mq.c index 22d5beaab5a0..fa2e5176966e 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1974,7 +1974,7 @@ struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, struct blk_mq_tags *tags; int node; - node = blk_mq_hw_queue_to_node(set->mq_map, hctx_idx); + node = blk_mq_hw_queue_to_node(&set->map[0], hctx_idx); if (node == NUMA_NO_NODE) node = set->numa_node; @@ -2030,7 +2030,7 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, size_t rq_size, left; int node; - node = blk_mq_hw_queue_to_node(set->mq_map, hctx_idx); + node = blk_mq_hw_queue_to_node(&set->map[0], hctx_idx); if (node == NUMA_NO_NODE) node = set->numa_node; @@ -2321,7 +2321,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) * If the cpu isn't present, the cpu is mapped to first hctx. */ for_each_possible_cpu(i) { - hctx_idx = set->mq_map[i]; + hctx_idx = set->map[0].mq_map[i]; /* unmapped hw queue can be remapped after CPU topo changed */ if (!set->tags[hctx_idx] && !__blk_mq_alloc_rq_map(set, hctx_idx)) { @@ -2331,7 +2331,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) * case, remap the current ctx to hctx[0] which * is guaranteed to always have tags allocated */ - set->mq_map[i] = 0; + set->map[0].mq_map[i] = 0; } ctx = per_cpu_ptr(q->queue_ctx, i); @@ -2584,7 +2584,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, int node; struct blk_mq_hw_ctx *hctx; - node = blk_mq_hw_queue_to_node(set->mq_map, i); + node = blk_mq_hw_queue_to_node(&set->map[0], i); /* * If the hw queue has been mapped to another numa node, * we need to realloc the hctx. If allocation fails, fallback @@ -2793,18 +2793,18 @@ static int blk_mq_update_queue_map(struct blk_mq_tag_set *set) * for (queue = 0; queue < set->nr_hw_queues; queue++) { * mask = get_cpu_mask(queue) * for_each_cpu(cpu, mask) - * set->mq_map[cpu] = queue; + * set->map.mq_map[cpu] = queue; * } * * When we need to remap, the table has to be cleared for * killing stale mapping since one CPU may not be mapped * to any hw queue. */ - blk_mq_clear_mq_map(set); + blk_mq_clear_mq_map(&set->map[0]); return set->ops->map_queues(set); } else - return blk_mq_map_queues(set); + return blk_mq_map_queues(&set->map[0]); } /* @@ -2859,10 +2859,12 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) return -ENOMEM; ret = -ENOMEM; - set->mq_map = kcalloc_node(nr_cpu_ids, sizeof(*set->mq_map), - GFP_KERNEL, set->numa_node); - if (!set->mq_map) + set->map[0].mq_map = kcalloc_node(nr_cpu_ids, + sizeof(*set->map[0].mq_map), + GFP_KERNEL, set->numa_node); + if (!set->map[0].mq_map) goto out_free_tags; + set->map[0].nr_queues = set->nr_hw_queues; ret = blk_mq_update_queue_map(set); if (ret) @@ -2878,8 +2880,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) return 0; out_free_mq_map: - kfree(set->mq_map); - set->mq_map = NULL; + kfree(set->map[0].mq_map); + set->map[0].mq_map = NULL; out_free_tags: kfree(set->tags); set->tags = NULL; @@ -2894,8 +2896,8 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set) for (i = 0; i < nr_cpu_ids; i++) blk_mq_free_map_and_requests(set, i); - kfree(set->mq_map); - set->mq_map = NULL; + kfree(set->map[0].mq_map); + set->map[0].mq_map = NULL; kfree(set->tags); set->tags = NULL; @@ -3056,7 +3058,7 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, pr_warn("Increasing nr_hw_queues to %d fails, fallback to %d\n", nr_hw_queues, prev_nr_hw_queues); set->nr_hw_queues = prev_nr_hw_queues; - blk_mq_map_queues(set); + blk_mq_map_queues(&set->map[0]); goto fallback; } blk_mq_map_swqueue(q); diff --git a/block/blk-mq.h b/block/blk-mq.h index 9536be06d022..889f0069dd80 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -70,14 +70,14 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, /* * CPU -> queue mappings */ -extern int blk_mq_hw_queue_to_node(unsigned int *map, unsigned int); +extern int blk_mq_hw_queue_to_node(struct blk_mq_queue_map *qmap, unsigned int); static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, int cpu) { struct blk_mq_tag_set *set = q->tag_set; - return q->queue_hw_ctx[set->mq_map[cpu]]; + return q->queue_hw_ctx[set->map[0].mq_map[cpu]]; } /* @@ -206,12 +206,12 @@ static inline void blk_mq_put_driver_tag(struct request *rq) __blk_mq_put_driver_tag(hctx, rq); } -static inline void blk_mq_clear_mq_map(struct blk_mq_tag_set *set) +static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap) { int cpu; for_each_possible_cpu(cpu) - set->mq_map[cpu] = 0; + qmap->mq_map[cpu] = 0; } #endif diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 086c6bb12baa..6e869d05f91e 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -624,7 +624,7 @@ static int virtblk_map_queues(struct blk_mq_tag_set *set) { struct virtio_blk *vblk = set->driver_data; - return blk_mq_virtio_map_queues(set, vblk->vdev, 0); + return blk_mq_virtio_map_queues(&set->map[0], vblk->vdev, 0); } #ifdef CONFIG_VIRTIO_BLK_SCSI diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index f30031945ee4..e5d783cb6937 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -435,7 +435,7 @@ static int nvme_pci_map_queues(struct blk_mq_tag_set *set) { struct nvme_dev *dev = set->driver_data; - return blk_mq_pci_map_queues(set, to_pci_dev(dev->dev), + return blk_mq_pci_map_queues(&set->map[0], to_pci_dev(dev->dev), dev->num_vecs > 1 ? 1 /* admin queue */ : 0); } diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c index 3e2665c66bc4..ca9ac124f218 100644 --- a/drivers/scsi/qla2xxx/qla_os.c +++ b/drivers/scsi/qla2xxx/qla_os.c @@ -6934,11 +6934,12 @@ static int qla2xxx_map_queues(struct Scsi_Host *shost) { int rc; scsi_qla_host_t *vha = (scsi_qla_host_t *)shost->hostdata; + struct blk_mq_queue_map *qmap = &shost->tag_set.map[0]; if (USER_CTRL_IRQ(vha->hw)) - rc = blk_mq_map_queues(&shost->tag_set); + rc = blk_mq_map_queues(qmap); else - rc = blk_mq_pci_map_queues(&shost->tag_set, vha->hw->pdev, 0); + rc = blk_mq_pci_map_queues(qmap, vha->hw->pdev, 0); return rc; } diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 651be30ba96a..ed81b8e74cfe 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -1812,7 +1812,7 @@ static int scsi_map_queues(struct blk_mq_tag_set *set) if (shost->hostt->map_queues) return shost->hostt->map_queues(shost); - return blk_mq_map_queues(set); + return blk_mq_map_queues(&set->map[0]); } void __scsi_init_queue(struct Scsi_Host *shost, struct request_queue *q) diff --git a/drivers/scsi/smartpqi/smartpqi_init.c b/drivers/scsi/smartpqi/smartpqi_init.c index a25a07a0b7f0..bac084260d80 100644 --- a/drivers/scsi/smartpqi/smartpqi_init.c +++ b/drivers/scsi/smartpqi/smartpqi_init.c @@ -5319,7 +5319,8 @@ static int pqi_map_queues(struct Scsi_Host *shost) { struct pqi_ctrl_info *ctrl_info = shost_to_hba(shost); - return blk_mq_pci_map_queues(&shost->tag_set, ctrl_info->pci_dev, 0); + return blk_mq_pci_map_queues(&shost->tag_set.map[0], + ctrl_info->pci_dev, 0); } static int pqi_getpciinfo_ioctl(struct pqi_ctrl_info *ctrl_info, diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c index 1c72db94270e..c3c95b314286 100644 --- a/drivers/scsi/virtio_scsi.c +++ b/drivers/scsi/virtio_scsi.c @@ -719,8 +719,9 @@ static void virtscsi_target_destroy(struct scsi_target *starget) static int virtscsi_map_queues(struct Scsi_Host *shost) { struct virtio_scsi *vscsi = shost_priv(shost); + struct blk_mq_queue_map *qmap = &shost->tag_set.map[0]; - return blk_mq_virtio_map_queues(&shost->tag_set, vscsi->vdev, 2); + return blk_mq_virtio_map_queues(qmap, vscsi->vdev, 2); } /* diff --git a/include/linux/blk-mq-pci.h b/include/linux/blk-mq-pci.h index 9f4c17f0d2d8..0b1f45c62623 100644 --- a/include/linux/blk-mq-pci.h +++ b/include/linux/blk-mq-pci.h @@ -2,10 +2,10 @@ #ifndef _LINUX_BLK_MQ_PCI_H #define _LINUX_BLK_MQ_PCI_H -struct blk_mq_tag_set; +struct blk_mq_queue_map; struct pci_dev; -int blk_mq_pci_map_queues(struct blk_mq_tag_set *set, struct pci_dev *pdev, +int blk_mq_pci_map_queues(struct blk_mq_queue_map *qmap, struct pci_dev *pdev, int offset); #endif /* _LINUX_BLK_MQ_PCI_H */ diff --git a/include/linux/blk-mq-virtio.h b/include/linux/blk-mq-virtio.h index 69b4da262c45..687ae287e1dc 100644 --- a/include/linux/blk-mq-virtio.h +++ b/include/linux/blk-mq-virtio.h @@ -2,10 +2,10 @@ #ifndef _LINUX_BLK_MQ_VIRTIO_H #define _LINUX_BLK_MQ_VIRTIO_H -struct blk_mq_tag_set; +struct blk_mq_queue_map; struct virtio_device; -int blk_mq_virtio_map_queues(struct blk_mq_tag_set *set, +int blk_mq_virtio_map_queues(struct blk_mq_queue_map *qmap, struct virtio_device *vdev, int first_vec); #endif /* _LINUX_BLK_MQ_VIRTIO_H */ diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 5c8418ebbfd6..71fd205b4213 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -74,8 +74,17 @@ struct blk_mq_hw_ctx { struct srcu_struct srcu[0]; }; +struct blk_mq_queue_map { + unsigned int *mq_map; + unsigned int nr_queues; +}; + +enum { + HCTX_MAX_TYPES = 1, +}; + struct blk_mq_tag_set { - unsigned int *mq_map; + struct blk_mq_queue_map map[HCTX_MAX_TYPES]; const struct blk_mq_ops *ops; unsigned int nr_hw_queues; unsigned int queue_depth; /* max hw supported */ @@ -294,7 +303,7 @@ void blk_mq_freeze_queue_wait(struct request_queue *q); int blk_mq_freeze_queue_wait_timeout(struct request_queue *q, unsigned long timeout); -int blk_mq_map_queues(struct blk_mq_tag_set *set); +int blk_mq_map_queues(struct blk_mq_queue_map *qmap); void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues); void blk_mq_quiesce_queue_nowait(struct request_queue *q); From patchwork Mon Oct 29 16:37:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659693 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 324FE13B5 for ; Mon, 29 Oct 2018 16:37:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 173A92967A for ; Mon, 29 Oct 2018 16:37:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 09C5329C45; Mon, 29 Oct 2018 16:37:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B38232967A for ; Mon, 29 Oct 2018 16:37:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728152AbeJ3B1K (ORCPT ); Mon, 29 Oct 2018 21:27:10 -0400 Received: from mail-it1-f194.google.com ([209.85.166.194]:54857 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728064AbeJ3B1K (ORCPT ); Mon, 29 Oct 2018 21:27:10 -0400 Received: by mail-it1-f194.google.com with SMTP id d6so4717286itl.4 for ; Mon, 29 Oct 2018 09:37:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7tpaRVSjhaK88jexqA/z4LLCkTo0WudeZhco39P0CL8=; b=aAiOocQM/qr+dkvhRoy7DTtw2TGJZ8QxlsS1xfpmfB+Y1LESvD+yGnFlQCdmTlmgpG zTaEJGBAvpy0+INyAHzCkDpFdw0Kr+ECbtU3feolq5l/akofreFA9e54SMCBZ5q0FYO8 Eh1RrH/qV7RAJucf6gy4E+L0LF/Jo+AK2adjHQxM99CttRVTH+MCx31WgfQFeMpOqJrR /6lV6K5mno6EWFoK8SWDgtHx8vZQKeeUwdneUzcE9mWzqGu7qmTRsimK/fW15PBEpqM/ HigkDuP1q5z7fj9kVM9XOduKJGnTZDu4hyU74ayOryozKpx3Gg9zDffpFyVvo5EIML5K +/wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7tpaRVSjhaK88jexqA/z4LLCkTo0WudeZhco39P0CL8=; b=mtM+MIs+eyZ+xFWUJqs70hdVk+UuipNp6M8oBTDL0oZNRNoeclc1nL06XGG3WLaxGp nqyGitbzRRWoDHbMJcU+TlHEsczATS8eeWNtJhacA2aKVqsHgWIw2ADersSq07Di/CBk 5QWemTNoUP6se+htf6y5++8LATey/OWhixOUGhPa4AxHtVdTTfMYTKD9Dc0bOUhODyBa 7AnS8wTbHdXlvMRZI0dqvGC2kKnKXoN2lnN20QF6Nxq3YHvtNOYyrGzfLls9un55UCYz kay4xVIlR7YTlkd5r+Xmyag9wp+m0al9gXGnXv9T/jden+PPk3KjthXMd8s3vxTTxZjp 4a9Q== X-Gm-Message-State: AGRZ1gIpPfvgWiWUSxDzJOU185BrXpNLPLnAMF+Dmk6i3DEvCKKewrFr XAdng3s0qlpiE6KyqjviJA5KcIc6Eyc= X-Google-Smtp-Source: AJdET5dRoGp+h4o2frTysI1QUlAJzxsn4/JpObChfmceUa8YbO8e4XAnZTHQrwNDzUBsDQ/H4+woxA== X-Received: by 2002:a24:ce47:: with SMTP id v68-v6mr3818711itg.163.1540831067893; Mon, 29 Oct 2018 09:37:47 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:46 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 03/14] blk-mq: provide dummy blk_mq_map_queue_type() helper Date: Mon, 29 Oct 2018 10:37:27 -0600 Message-Id: <20181029163738.10172-4-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Doesn't do anything right now, but it's needed as a prep patch to get the interfaces right. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- block/blk-mq.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/block/blk-mq.h b/block/blk-mq.h index 889f0069dd80..79c300faa7ce 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -80,6 +80,12 @@ static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, return q->queue_hw_ctx[set->map[0].mq_map[cpu]]; } +static inline struct blk_mq_hw_ctx *blk_mq_map_queue_type(struct request_queue *q, + int type, int cpu) +{ + return blk_mq_map_queue(q, cpu); +} + /* * sysfs helpers */ From patchwork Mon Oct 29 16:37:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659737 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0C631751 for ; Mon, 29 Oct 2018 16:39:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 971F329BC1 for ; Mon, 29 Oct 2018 16:39:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B5BF29BCA; Mon, 29 Oct 2018 16:39:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A449D29BC1 for ; Mon, 29 Oct 2018 16:39:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727775AbeJ3B21 (ORCPT ); Mon, 29 Oct 2018 21:28:27 -0400 Received: from mail-it1-f194.google.com ([209.85.166.194]:50947 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728177AbeJ3B1M (ORCPT ); Mon, 29 Oct 2018 21:27:12 -0400 Received: by mail-it1-f194.google.com with SMTP id k206-v6so10318429ite.0 for ; Mon, 29 Oct 2018 09:37:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WJ3jzDMA+8g6uXbFr710wBI4sLV3/EeQMxzeZ89+Y4o=; b=xzLPJazW/CJXYasmJZQniPtEagsfplGfyfMikASZQMs5RKEp+wMvzjwGcJxCFh1cpB Gn0at+AL2hEvBs0qMXAwaZzM6b5NhyYNOcQpsLNep8l5EEGo+UsfREI2mYPFXVEomelg VZrTfzF77FyaGjws7HALmVnmMBCuwU6Eql2Rf5/FqHmE3eHexhf3EqC22i8A3+mWm8hR k6CphFTzR+NPzGPhyK+nuye0aNVZ18fMCjbQZDpoWMyuV5gyXx4gON9wC97y4L0XYM+p DyZ7Lcd1JRE1Pp69KG3bsVJ7gLDAj8VgrJ0IZWW2SIvbsVIMhyHiosk6x+DFtQCp/Riw VvYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WJ3jzDMA+8g6uXbFr710wBI4sLV3/EeQMxzeZ89+Y4o=; b=cAueLCenqqcNTsqByB+VMa35XiEGF0bkjwmYjYP1o5e45FpQAfDMZ5ZK4dxC7evp2l 6uuWYY0clKWpzpwEKZOw7r6tUCjCnqzKaUdDRcs+fJodUY+ZAnJvJd2GTfCRwvLziZBn +FlNa4k6Am0T6m8w2R799JliWSoIFv0Lk53sx95nE2Sa+XiYiysWcuOOsYWvkh+8V/Sl kBbp5SHMLZKDVN1RLFjfttuFcyNfSiUHJ72ITVTP+Bg/DG9ufdtYX42KZKRj+JjNMXo+ GhfVlcZeLXA78JZZGuMFsb4VYMcMJ5Lk2xt/4haRYYZi6d1MZIUH2mz/BVX3tM7mWQAb 3ZTw== X-Gm-Message-State: AGRZ1gIo6d7JhxiL1Q4+tIvQCNK1a0ICZNnvxGT2G7xh8lMaVHkJ/F2i qf+3hE1MEU7TjdtL1P41iNLFz4KAyt4= X-Google-Smtp-Source: AJdET5eFFHS0J9CfAhSFB31/oCr/ndTizeJjAL40KmElrVimQPHz8ISaP4MIsJ/KWQ3toMTGYkdKTg== X-Received: by 2002:a24:bcc1:: with SMTP id n184-v6mr10406957ite.174.1540831070189; Mon, 29 Oct 2018 09:37:50 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:48 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 04/14] blk-mq: pass in request/bio flags to queue mapping Date: Mon, 29 Oct 2018 10:37:28 -0600 Message-Id: <20181029163738.10172-5-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Prep patch for being able to place request based not just on CPU location, but also on the type of request. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- block/blk-flush.c | 7 +++--- block/blk-mq-debugfs.c | 4 +++- block/blk-mq-sched.c | 16 ++++++++++---- block/blk-mq-tag.c | 5 +++-- block/blk-mq.c | 50 +++++++++++++++++++++++------------------- block/blk-mq.h | 8 ++++--- block/blk.h | 6 ++--- 7 files changed, 58 insertions(+), 38 deletions(-) diff --git a/block/blk-flush.c b/block/blk-flush.c index 9baa9a119447..7922dba81497 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -219,7 +219,7 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error) /* release the tag's ownership to the req cloned from */ spin_lock_irqsave(&fq->mq_flush_lock, flags); - hctx = blk_mq_map_queue(q, flush_rq->mq_ctx->cpu); + hctx = blk_mq_map_queue(q, flush_rq->cmd_flags, flush_rq->mq_ctx->cpu); if (!q->elevator) { blk_mq_tag_set_rq(hctx, flush_rq->tag, fq->orig_rq); flush_rq->tag = -1; @@ -307,7 +307,8 @@ static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, if (!q->elevator) { fq->orig_rq = first_rq; flush_rq->tag = first_rq->tag; - hctx = blk_mq_map_queue(q, first_rq->mq_ctx->cpu); + hctx = blk_mq_map_queue(q, first_rq->cmd_flags, + first_rq->mq_ctx->cpu); blk_mq_tag_set_rq(hctx, first_rq->tag, flush_rq); } else { flush_rq->internal_tag = first_rq->internal_tag; @@ -330,7 +331,7 @@ static void mq_flush_data_end_io(struct request *rq, blk_status_t error) unsigned long flags; struct blk_flush_queue *fq = blk_get_flush_queue(q, ctx); - hctx = blk_mq_map_queue(q, ctx->cpu); + hctx = blk_mq_map_queue(q, rq->cmd_flags, ctx->cpu); if (q->elevator) { WARN_ON(rq->tag < 0); diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 9ed43a7c70b5..fac70c81b7de 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -427,8 +427,10 @@ struct show_busy_params { static void hctx_show_busy_rq(struct request *rq, void *data, bool reserved) { const struct show_busy_params *params = data; + struct blk_mq_hw_ctx *hctx; - if (blk_mq_map_queue(rq->q, rq->mq_ctx->cpu) == params->hctx) + hctx = blk_mq_map_queue(rq->q, rq->cmd_flags, rq->mq_ctx->cpu); + if (hctx == params->hctx) __blk_mq_debugfs_rq_show(params->m, list_entry_rq(&rq->queuelist)); } diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 29bfe8017a2d..8125e9393ec2 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -311,7 +311,7 @@ bool __blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio) { struct elevator_queue *e = q->elevator; struct blk_mq_ctx *ctx = blk_mq_get_ctx(q); - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu); + struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, bio->bi_opf, ctx->cpu); bool ret = false; if (e && e->type->ops.mq.bio_merge) { @@ -367,7 +367,9 @@ void blk_mq_sched_insert_request(struct request *rq, bool at_head, struct request_queue *q = rq->q; struct elevator_queue *e = q->elevator; struct blk_mq_ctx *ctx = rq->mq_ctx; - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu); + struct blk_mq_hw_ctx *hctx; + + hctx = blk_mq_map_queue(q, rq->cmd_flags, ctx->cpu); /* flush rq in flush machinery need to be dispatched directly */ if (!(rq->rq_flags & RQF_FLUSH_SEQ) && op_is_flush(rq->cmd_flags)) { @@ -400,9 +402,15 @@ void blk_mq_sched_insert_requests(struct request_queue *q, struct blk_mq_ctx *ctx, struct list_head *list, bool run_queue_async) { - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu); - struct elevator_queue *e = hctx->queue->elevator; + struct blk_mq_hw_ctx *hctx; + struct elevator_queue *e; + struct request *rq; + + /* For list inserts, requests better be on the same hw queue */ + rq = list_first_entry(list, struct request, queuelist); + hctx = blk_mq_map_queue(q, rq->cmd_flags, ctx->cpu); + e = hctx->queue->elevator; if (e && e->type->ops.mq.insert_requests) e->type->ops.mq.insert_requests(hctx, list, false); else { diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 4254e74c1446..478a959357f5 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -168,7 +168,8 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) io_schedule(); data->ctx = blk_mq_get_ctx(data->q); - data->hctx = blk_mq_map_queue(data->q, data->ctx->cpu); + data->hctx = blk_mq_map_queue(data->q, data->cmd_flags, + data->ctx->cpu); tags = blk_mq_tags_from_data(data); if (data->flags & BLK_MQ_REQ_RESERVED) bt = &tags->breserved_tags; @@ -530,7 +531,7 @@ u32 blk_mq_unique_tag(struct request *rq) struct blk_mq_hw_ctx *hctx; int hwq = 0; - hctx = blk_mq_map_queue(q, rq->mq_ctx->cpu); + hctx = blk_mq_map_queue(q, rq->cmd_flags, rq->mq_ctx->cpu); hwq = hctx->queue_num; return (hwq << BLK_MQ_UNIQUE_TAG_BITS) | diff --git a/block/blk-mq.c b/block/blk-mq.c index fa2e5176966e..e6ea7da99125 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -332,8 +332,8 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data, } static struct request *blk_mq_get_request(struct request_queue *q, - struct bio *bio, unsigned int op, - struct blk_mq_alloc_data *data) + struct bio *bio, + struct blk_mq_alloc_data *data) { struct elevator_queue *e = q->elevator; struct request *rq; @@ -347,8 +347,9 @@ static struct request *blk_mq_get_request(struct request_queue *q, put_ctx_on_error = true; } if (likely(!data->hctx)) - data->hctx = blk_mq_map_queue(q, data->ctx->cpu); - if (op & REQ_NOWAIT) + data->hctx = blk_mq_map_queue(q, data->cmd_flags, + data->ctx->cpu); + if (data->cmd_flags & REQ_NOWAIT) data->flags |= BLK_MQ_REQ_NOWAIT; if (e) { @@ -359,9 +360,10 @@ static struct request *blk_mq_get_request(struct request_queue *q, * dispatch list. Don't include reserved tags in the * limiting, as it isn't useful. */ - if (!op_is_flush(op) && e->type->ops.mq.limit_depth && + if (!op_is_flush(data->cmd_flags) && + e->type->ops.mq.limit_depth && !(data->flags & BLK_MQ_REQ_RESERVED)) - e->type->ops.mq.limit_depth(op, data); + e->type->ops.mq.limit_depth(data->cmd_flags, data); } else { blk_mq_tag_busy(data->hctx); } @@ -376,8 +378,8 @@ static struct request *blk_mq_get_request(struct request_queue *q, return NULL; } - rq = blk_mq_rq_ctx_init(data, tag, op); - if (!op_is_flush(op)) { + rq = blk_mq_rq_ctx_init(data, tag, data->cmd_flags); + if (!op_is_flush(data->cmd_flags)) { rq->elv.icq = NULL; if (e && e->type->ops.mq.prepare_request) { if (e->type->icq_cache && rq_ioc(bio)) @@ -394,7 +396,7 @@ static struct request *blk_mq_get_request(struct request_queue *q, struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op, blk_mq_req_flags_t flags) { - struct blk_mq_alloc_data alloc_data = { .flags = flags }; + struct blk_mq_alloc_data alloc_data = { .flags = flags, .cmd_flags = op }; struct request *rq; int ret; @@ -402,7 +404,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, unsigned int op, if (ret) return ERR_PTR(ret); - rq = blk_mq_get_request(q, NULL, op, &alloc_data); + rq = blk_mq_get_request(q, NULL, &alloc_data); blk_queue_exit(q); if (!rq) @@ -420,7 +422,7 @@ EXPORT_SYMBOL(blk_mq_alloc_request); struct request *blk_mq_alloc_request_hctx(struct request_queue *q, unsigned int op, blk_mq_req_flags_t flags, unsigned int hctx_idx) { - struct blk_mq_alloc_data alloc_data = { .flags = flags }; + struct blk_mq_alloc_data alloc_data = { .flags = flags, .cmd_flags = op }; struct request *rq; unsigned int cpu; int ret; @@ -453,7 +455,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, cpu = cpumask_first_and(alloc_data.hctx->cpumask, cpu_online_mask); alloc_data.ctx = __blk_mq_get_ctx(q, cpu); - rq = blk_mq_get_request(q, NULL, op, &alloc_data); + rq = blk_mq_get_request(q, NULL, &alloc_data); blk_queue_exit(q); if (!rq) @@ -467,7 +469,7 @@ static void __blk_mq_free_request(struct request *rq) { struct request_queue *q = rq->q; struct blk_mq_ctx *ctx = rq->mq_ctx; - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu); + struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, rq->cmd_flags, ctx->cpu); const int sched_tag = rq->internal_tag; blk_pm_mark_last_busy(rq); @@ -484,7 +486,7 @@ void blk_mq_free_request(struct request *rq) struct request_queue *q = rq->q; struct elevator_queue *e = q->elevator; struct blk_mq_ctx *ctx = rq->mq_ctx; - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, ctx->cpu); + struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(q, rq->cmd_flags, ctx->cpu); if (rq->rq_flags & RQF_ELVPRIV) { if (e && e->type->ops.mq.finish_request) @@ -976,8 +978,9 @@ bool blk_mq_get_driver_tag(struct request *rq) { struct blk_mq_alloc_data data = { .q = rq->q, - .hctx = blk_mq_map_queue(rq->q, rq->mq_ctx->cpu), + .hctx = blk_mq_map_queue(rq->q, rq->cmd_flags, rq->mq_ctx->cpu), .flags = BLK_MQ_REQ_NOWAIT, + .cmd_flags = rq->cmd_flags, }; bool shared; @@ -1141,7 +1144,7 @@ bool blk_mq_dispatch_rq_list(struct request_queue *q, struct list_head *list, rq = list_first_entry(list, struct request, queuelist); - hctx = blk_mq_map_queue(rq->q, rq->mq_ctx->cpu); + hctx = blk_mq_map_queue(rq->q, rq->cmd_flags, rq->mq_ctx->cpu); if (!got_budget && !blk_mq_get_dispatch_budget(hctx)) break; @@ -1572,7 +1575,8 @@ void __blk_mq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, void blk_mq_request_bypass_insert(struct request *rq, bool run_queue) { struct blk_mq_ctx *ctx = rq->mq_ctx; - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(rq->q, ctx->cpu); + struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(rq->q, rq->cmd_flags, + ctx->cpu); spin_lock(&hctx->lock); list_add_tail(&rq->queuelist, &hctx->dispatch); @@ -1782,7 +1786,8 @@ blk_status_t blk_mq_request_issue_directly(struct request *rq) int srcu_idx; blk_qc_t unused_cookie; struct blk_mq_ctx *ctx = rq->mq_ctx; - struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(rq->q, ctx->cpu); + struct blk_mq_hw_ctx *hctx = blk_mq_map_queue(rq->q, rq->cmd_flags, + ctx->cpu); hctx_lock(hctx, &srcu_idx); ret = __blk_mq_try_issue_directly(hctx, rq, &unused_cookie, true); @@ -1816,7 +1821,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) { const int is_sync = op_is_sync(bio->bi_opf); const int is_flush_fua = op_is_flush(bio->bi_opf); - struct blk_mq_alloc_data data = { .flags = 0 }; + struct blk_mq_alloc_data data = { .flags = 0, .cmd_flags = bio->bi_opf }; struct request *rq; unsigned int request_count = 0; struct blk_plug *plug; @@ -1839,7 +1844,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) rq_qos_throttle(q, bio, NULL); - rq = blk_mq_get_request(q, bio, bio->bi_opf, &data); + rq = blk_mq_get_request(q, bio, &data); if (unlikely(!rq)) { rq_qos_cleanup(q, bio); if (bio->bi_opf & REQ_NOWAIT) @@ -1908,6 +1913,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) if (same_queue_rq) { data.hctx = blk_mq_map_queue(q, + same_queue_rq->cmd_flags, same_queue_rq->mq_ctx->cpu); blk_mq_try_issue_directly(data.hctx, same_queue_rq, &cookie); @@ -2262,7 +2268,7 @@ static void blk_mq_init_cpu_queues(struct request_queue *q, * Set local node, IFF we have more than one hw queue. If * not, we remain on the home node of the device */ - hctx = blk_mq_map_queue(q, i); + hctx = blk_mq_map_queue_type(q, 0, i); if (nr_hw_queues > 1 && hctx->numa_node == NUMA_NO_NODE) hctx->numa_node = local_memory_node(cpu_to_node(i)); } @@ -2335,7 +2341,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) } ctx = per_cpu_ptr(q->queue_ctx, i); - hctx = blk_mq_map_queue(q, i); + hctx = blk_mq_map_queue_type(q, 0, i); cpumask_set_cpu(i, hctx->cpumask); ctx->index_hw = hctx->nr_ctx; diff --git a/block/blk-mq.h b/block/blk-mq.h index 79c300faa7ce..55428b92c019 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -73,7 +73,8 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, extern int blk_mq_hw_queue_to_node(struct blk_mq_queue_map *qmap, unsigned int); static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, - int cpu) + unsigned int flags, + int cpu) { struct blk_mq_tag_set *set = q->tag_set; @@ -83,7 +84,7 @@ static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, static inline struct blk_mq_hw_ctx *blk_mq_map_queue_type(struct request_queue *q, int type, int cpu) { - return blk_mq_map_queue(q, cpu); + return blk_mq_map_queue(q, type, cpu); } /* @@ -134,6 +135,7 @@ struct blk_mq_alloc_data { struct request_queue *q; blk_mq_req_flags_t flags; unsigned int shallow_depth; + unsigned int cmd_flags; /* input & output parameter */ struct blk_mq_ctx *ctx; @@ -208,7 +210,7 @@ static inline void blk_mq_put_driver_tag(struct request *rq) if (rq->tag == -1 || rq->internal_tag == -1) return; - hctx = blk_mq_map_queue(rq->q, rq->mq_ctx->cpu); + hctx = blk_mq_map_queue(rq->q, rq->cmd_flags, rq->mq_ctx->cpu); __blk_mq_put_driver_tag(hctx, rq); } diff --git a/block/blk.h b/block/blk.h index 2bf1cfeeb9c0..78ae94886acf 100644 --- a/block/blk.h +++ b/block/blk.h @@ -104,10 +104,10 @@ static inline void queue_flag_clear(unsigned int flag, struct request_queue *q) __clear_bit(flag, &q->queue_flags); } -static inline struct blk_flush_queue *blk_get_flush_queue( - struct request_queue *q, struct blk_mq_ctx *ctx) +static inline struct blk_flush_queue * +blk_get_flush_queue(struct request_queue *q, struct blk_mq_ctx *ctx) { - return blk_mq_map_queue(q, ctx->cpu)->fq; + return blk_mq_map_queue(q, REQ_OP_FLUSH, ctx->cpu)->fq; } static inline void __blk_get_queue(struct request_queue *q) From patchwork Mon Oct 29 16:37:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659733 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0A85B1751 for ; Mon, 29 Oct 2018 16:39:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E540A29BC1 for ; Mon, 29 Oct 2018 16:39:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9E3C29BCA; Mon, 29 Oct 2018 16:39:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6309229BC1 for ; Mon, 29 Oct 2018 16:39:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728037AbeJ3B1P (ORCPT ); Mon, 29 Oct 2018 21:27:15 -0400 Received: from mail-it1-f195.google.com ([209.85.166.195]:37203 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728081AbeJ3B1O (ORCPT ); Mon, 29 Oct 2018 21:27:14 -0400 Received: by mail-it1-f195.google.com with SMTP id e74-v6so10182492ita.2 for ; Mon, 29 Oct 2018 09:37:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YdW9jHQFmkDw/Bx6HtFcVTIU0iWJR3rZpnd+EOU2uNE=; b=Se1GMmG2yUzRhbD6TQn47Wm6Mee88P1kJovqFVWM7pKN2lcYE4ysLQlWzeR/jEwJQ0 BRcOgm+2E5PxTpm+rsRhwKfhtYu7Bqhl/+FMIrYs+aRF90qGsN1Sl30uEvOZYLswyPN8 vno2j1NzUIwYCx6r1z/jQw8sCJmbJ0IpbbPLbIaLSB9ZkagDLQYJ8LOAdzBlF+l7tLdQ 0uaC6kVqYQVk1MIcMI4kbPJgAsDewrGbNol9L1negqXeNU4t2XIfcNIMktWxGw9v0zOV d5tgOURAnl2PNR+AM47T/PwDqTksT12A8MrMN2I1/Aux8JBxja+TstFqNB+2PfZmXm5+ EFlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YdW9jHQFmkDw/Bx6HtFcVTIU0iWJR3rZpnd+EOU2uNE=; b=QkT9PNhdcDmgL3NYYVFztiiuJ6QqcCNXRQ7KgS4V8LqZDQl+qomHnggAuGVEPEjHkV JHRhqqHRMLl4ljuvyFr5CgTTFL2GNNn1OBZJWrkxEtJinlvBMZIdgEHgE5Sys1KULKth jR3aUiynqfqxTRcp5SxzrS8FyYLK3ivvcPRnIAq39KLkFGo9ba4de6NtPgxBSr3KdQOi XVibmG7JGRMGo0St2r32p5H9hzPMyHeRRDDP6f/uGvgwSpaxfPP+vQzxFikI2035jgGp nn3G+CPjkRCjLBVx9ZuwftSClaXV5CC5u8AES5wSZNXas1ZM6GqSvC6orUhLy9Sj7FhF 02uQ== X-Gm-Message-State: AGRZ1gLzgLKMQ8iVbB7hBwQuqmiA1EI9DPDBspkoGk5nHGWB9n94c9g6 7R8ebQ21Cb7Oy7BKAuf5G/KyZ85FJRU= X-Google-Smtp-Source: AJdET5dtYFrGLX0W5FVQua2DTuWr3i7tr8VKZFl8/0RjGUECTIzyNRmXNZ+nKoAqcc1bD0B6cDpkoA== X-Received: by 2002:a05:660c:441:: with SMTP id d1mr10654181itl.22.1540831072079; Mon, 29 Oct 2018 09:37:52 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:50 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 05/14] blk-mq: allow software queue to map to multiple hardware queues Date: Mon, 29 Oct 2018 10:37:29 -0600 Message-Id: <20181029163738.10172-6-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The mapping used to be dependent on just the CPU location, but now it's a tuple of { type, cpu} instead. This is a prep patch for allowing a single software queue to map to multiple hardware queues. No functional changes in this patch. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- block/blk-mq-sched.c | 2 +- block/blk-mq.c | 18 ++++++++++++------ block/blk-mq.h | 2 +- block/kyber-iosched.c | 6 +++--- include/linux/blk-mq.h | 3 ++- 5 files changed, 19 insertions(+), 12 deletions(-) diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index 8125e9393ec2..d232ecf3290c 100644 --- a/block/blk-mq-sched.c +++ b/block/blk-mq-sched.c @@ -110,7 +110,7 @@ static void blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) static struct blk_mq_ctx *blk_mq_next_ctx(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx) { - unsigned idx = ctx->index_hw; + unsigned short idx = ctx->index_hw[hctx->type]; if (++idx == hctx->nr_ctx) idx = 0; diff --git a/block/blk-mq.c b/block/blk-mq.c index e6ea7da99125..fab84c6bda18 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -75,14 +75,18 @@ static bool blk_mq_hctx_has_pending(struct blk_mq_hw_ctx *hctx) static void blk_mq_hctx_mark_pending(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx) { - if (!sbitmap_test_bit(&hctx->ctx_map, ctx->index_hw)) - sbitmap_set_bit(&hctx->ctx_map, ctx->index_hw); + const int bit = ctx->index_hw[hctx->type]; + + if (!sbitmap_test_bit(&hctx->ctx_map, bit)) + sbitmap_set_bit(&hctx->ctx_map, bit); } static void blk_mq_hctx_clear_pending(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *ctx) { - sbitmap_clear_bit(&hctx->ctx_map, ctx->index_hw); + const int bit = ctx->index_hw[hctx->type]; + + sbitmap_clear_bit(&hctx->ctx_map, bit); } struct mq_inflight { @@ -954,7 +958,7 @@ static bool dispatch_rq_from_ctx(struct sbitmap *sb, unsigned int bitnr, struct request *blk_mq_dequeue_from_ctx(struct blk_mq_hw_ctx *hctx, struct blk_mq_ctx *start) { - unsigned off = start ? start->index_hw : 0; + unsigned off = start ? start->index_hw[hctx->type] : 0; struct dispatch_rq_data data = { .hctx = hctx, .rq = NULL, @@ -2342,10 +2346,12 @@ static void blk_mq_map_swqueue(struct request_queue *q) ctx = per_cpu_ptr(q->queue_ctx, i); hctx = blk_mq_map_queue_type(q, 0, i); - + hctx->type = 0; cpumask_set_cpu(i, hctx->cpumask); - ctx->index_hw = hctx->nr_ctx; + ctx->index_hw[hctx->type] = hctx->nr_ctx; hctx->ctxs[hctx->nr_ctx++] = ctx; + /* wrap */ + BUG_ON(!hctx->nr_ctx); } mutex_unlock(&q->sysfs_lock); diff --git a/block/blk-mq.h b/block/blk-mq.h index 55428b92c019..7b5a790acdbf 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -17,7 +17,7 @@ struct blk_mq_ctx { } ____cacheline_aligned_in_smp; unsigned int cpu; - unsigned int index_hw; + unsigned short index_hw[HCTX_MAX_TYPES]; /* incremented at dispatch time */ unsigned long rq_dispatched[2]; diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index 728757a34fa0..b824a639d5d4 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -576,7 +576,7 @@ static bool kyber_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio) { struct kyber_hctx_data *khd = hctx->sched_data; struct blk_mq_ctx *ctx = blk_mq_get_ctx(hctx->queue); - struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw]; + struct kyber_ctx_queue *kcq = &khd->kcqs[ctx->index_hw[hctx->type]]; unsigned int sched_domain = kyber_sched_domain(bio->bi_opf); struct list_head *rq_list = &kcq->rq_list[sched_domain]; bool merged; @@ -602,7 +602,7 @@ static void kyber_insert_requests(struct blk_mq_hw_ctx *hctx, list_for_each_entry_safe(rq, next, rq_list, queuelist) { unsigned int sched_domain = kyber_sched_domain(rq->cmd_flags); - struct kyber_ctx_queue *kcq = &khd->kcqs[rq->mq_ctx->index_hw]; + struct kyber_ctx_queue *kcq = &khd->kcqs[rq->mq_ctx->index_hw[hctx->type]]; struct list_head *head = &kcq->rq_list[sched_domain]; spin_lock(&kcq->lock); @@ -611,7 +611,7 @@ static void kyber_insert_requests(struct blk_mq_hw_ctx *hctx, else list_move_tail(&rq->queuelist, head); sbitmap_set_bit(&khd->kcq_map[sched_domain], - rq->mq_ctx->index_hw); + rq->mq_ctx->index_hw[hctx->type]); blk_mq_sched_request_inserted(rq); spin_unlock(&kcq->lock); } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 71fd205b4213..f9e19962a22f 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -37,7 +37,8 @@ struct blk_mq_hw_ctx { struct blk_mq_ctx *dispatch_from; unsigned int dispatch_busy; - unsigned int nr_ctx; + unsigned short type; + unsigned short nr_ctx; struct blk_mq_ctx **ctxs; spinlock_t dispatch_wait_lock; From patchwork Mon Oct 29 16:37:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659697 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D0DD13B5 for ; Mon, 29 Oct 2018 16:37:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31D6C2967A for ; Mon, 29 Oct 2018 16:37:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 261D529C45; Mon, 29 Oct 2018 16:37:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C83662967A for ; Mon, 29 Oct 2018 16:37:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727806AbeJ3B1R (ORCPT ); Mon, 29 Oct 2018 21:27:17 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:34822 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728062AbeJ3B1Q (ORCPT ); Mon, 29 Oct 2018 21:27:16 -0400 Received: by mail-it1-f193.google.com with SMTP id p64-v6so10180819itp.0 for ; Mon, 29 Oct 2018 09:37:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ibssz+RTYOgnABH8iblTDxiQJXKqVGK071Agvxpp/w8=; b=Euwd+xGSRZSC+Jh6OV9lsCHuG6H4fzJeU0B8PY06fsrW/ClKm3j1ZubHWn/hbqk+MX i9O7SucqpPAUU6rGYbR3uKT3JjhuS5fzzmycCZmcZNo59RLfAcvLQDChoU48oCaEm6g3 sNDmMCiAJPp4xKZzvupi6Zd5W11NpbFBnzQeNP+gIg46fSti73Qm2nENXuzAwE/HCJWm HIPaIiZAnm/JBHPN9+K88/zdPiR/BwsYmV3mmkrO3SVjY1soX1ooT4mTEvpkxs2+bnQv ChVrII9kfhvhl+IKVfmmeb3vt4M17tjvqH6/xVPDodP169qoLWGmSLPTvIB6jvp6hcjP cqYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ibssz+RTYOgnABH8iblTDxiQJXKqVGK071Agvxpp/w8=; b=VkUSljgr+k0l4L/a0RYhiHnaSj+V9VkJ8RsarMBOv0c9kvZPr9qNPebWawgvsmltHi T0L3Q6Q0xQYxIvfuEpFp+M/2/WSS2awlvSofQOPuqLXiUViUeWoSNQuawsWtE0wJMVGi ACDvKYWzCsfUcyico4/9+OLtXZrYWtm5Hebe3EzM5+ILRgAZiwd4XzayBFBg6xkSY21d FJ4snQaZpplk8lw5yZQv17+OmcqWiPI9v8o7dAaaawD/uxGVuslJr9v90mOJ79IYx9W7 dMKDthMtubevqUJ1JCqjwgaQGiIklP/eQdm8ZEZD++0yb0TmPqAFzbjJDplcDF5UZbUA ND+w== X-Gm-Message-State: AGRZ1gK3pAs0vyt8FaghTQvnEGo1v/onkuf2owAu1u8pVNLpkNrtTbLF cJJI0qb+x6UYLe3lsoodn4NPa5UXAis= X-Google-Smtp-Source: AJdET5faGUrC0AwyueMFKr4H1ego6W+WoFGqiU8vFsgz6fuMjIPXRnSB/jMgScRdMdmN+lzkCuIJlA== X-Received: by 2002:a24:b341:: with SMTP id z1-v6mr11211615iti.62.1540831074546; Mon, 29 Oct 2018 09:37:54 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:52 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 06/14] blk-mq: add 'type' attribute to the sysfs hctx directory Date: Mon, 29 Oct 2018 10:37:30 -0600 Message-Id: <20181029163738.10172-7-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It can be useful for a user to verify what type a given hardware queue is, expose this information in sysfs. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe Reviewed-by: Bart Van Assche --- block/blk-mq-sysfs.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c index aafb44224c89..2d737f9e7ba7 100644 --- a/block/blk-mq-sysfs.c +++ b/block/blk-mq-sysfs.c @@ -161,6 +161,11 @@ static ssize_t blk_mq_hw_sysfs_cpus_show(struct blk_mq_hw_ctx *hctx, char *page) return ret; } +static ssize_t blk_mq_hw_sysfs_type_show(struct blk_mq_hw_ctx *hctx, char *page) +{ + return sprintf(page, "%u\n", hctx->type); +} + static struct attribute *default_ctx_attrs[] = { NULL, }; @@ -177,11 +182,16 @@ static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_cpus = { .attr = {.name = "cpu_list", .mode = 0444 }, .show = blk_mq_hw_sysfs_cpus_show, }; +static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_type = { + .attr = {.name = "type", .mode = 0444 }, + .show = blk_mq_hw_sysfs_type_show, +}; static struct attribute *default_hw_ctx_attrs[] = { &blk_mq_hw_sysfs_nr_tags.attr, &blk_mq_hw_sysfs_nr_reserved_tags.attr, &blk_mq_hw_sysfs_cpus.attr, + &blk_mq_hw_sysfs_type.attr, NULL, }; From patchwork Mon Oct 29 16:37:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659701 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9EF083CF1 for ; Mon, 29 Oct 2018 16:38:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 84D662967A for ; Mon, 29 Oct 2018 16:38:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7971529C45; Mon, 29 Oct 2018 16:38:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D732F29C43 for ; Mon, 29 Oct 2018 16:38:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728269AbeJ3B1T (ORCPT ); Mon, 29 Oct 2018 21:27:19 -0400 Received: from mail-it1-f194.google.com ([209.85.166.194]:50969 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728205AbeJ3B1T (ORCPT ); Mon, 29 Oct 2018 21:27:19 -0400 Received: by mail-it1-f194.google.com with SMTP id k206-v6so10318984ite.0 for ; Mon, 29 Oct 2018 09:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NWJxWMqyk1M1/IETmc0oKYYYDdFixaxyAysUlEs0bIs=; b=WNemvEDjEaWbcNiCXfVnZNGP1djRdhzgzbIl7F0+MZfjKPccs81t9q73XC1Z8pLN8w MdBgc0KZc+hNsXds6fuyiv0/gYHTIBBP/+3IRTl0vEgR9RRft17EzHUpw8clbW1Tq7Ya 43Xp2IlPMco8pJw68dAy7PiA4E8K+hvt40g6Qt7FidBPAM+nQiAdZiKCH7up0HxOLz4p ZM1kz+0kA+/jzyitCm0hBxXzvRapcB3P24wNDof6RbZ283ZZqzhECFBrLHCIjDOgTPTH iR5ggOdqMavrlF4BUOjX/7HGNai1RdBLc1q5Z77kEmLCDZFzk2RAJw0Jy3mEgDL1gpBh BgAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NWJxWMqyk1M1/IETmc0oKYYYDdFixaxyAysUlEs0bIs=; b=DjvWQU1bACe3r00nepbPSYy4N53EOJZnfjUuVk3lSFaJcYW9DKh6rH56izfwaKvAN0 Ix4zh0jGlSct8456CU3bv/IDcZ2BTOSPDtLum0udUL1askYy+j4GK2t1ACHz1QopST7l IypL2X37M/ytT8WdKOmdiHMVjUjpFeQ1XbWNVbXjaPkeHyAx690iS0Aa2FKjQNUMim4k yZv/Yr86TtvVq1hGMwOM1hVoNek98PhVSsN95iD8teeNqqp11CtJBx2l3BFNSVsYF8a2 j6jL8yNJ4W+GwbVArwutQPTMNVvge6UVQB3ij72QINOp5+lky/z4Nn9WJjDTLADCGNX2 Z0DA== X-Gm-Message-State: AGRZ1gLiQ0CGJ29/sIdYUenxL4rTWF+s4PbOgyEt1H1k0zmhCmNVY94l X4qIElVY1h5AHJvRbtenwwBfqnn06dk= X-Google-Smtp-Source: AJdET5dbOiUH3M5o2aDYFVySQikm4urYV+9V3lC42fx8Q33/fy8xiqzxEDvuO8V0pADjcn+mDDcmhw== X-Received: by 2002:a24:21d5:: with SMTP id e204-v6mr10431270ita.127.1540831076533; Mon, 29 Oct 2018 09:37:56 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:55 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 07/14] blk-mq: support multiple hctx maps Date: Mon, 29 Oct 2018 10:37:31 -0600 Message-Id: <20181029163738.10172-8-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add support for the tag set carrying multiple queue maps, and for the driver to inform blk-mq how many it wishes to support through setting set->nr_maps. This adds an mq_ops helper for drivers that support more than 1 map, mq_ops->flags_to_type(). The function takes request/bio flags and CPU, and returns a queue map index for that. We then use the type information in blk_mq_map_queue() to index the map set. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- block/blk-mq.c | 85 ++++++++++++++++++++++++++++-------------- block/blk-mq.h | 19 ++++++---- include/linux/blk-mq.h | 7 ++++ 3 files changed, 76 insertions(+), 35 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index fab84c6bda18..0fab36372ace 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2257,7 +2257,8 @@ static int blk_mq_init_hctx(struct request_queue *q, static void blk_mq_init_cpu_queues(struct request_queue *q, unsigned int nr_hw_queues) { - unsigned int i; + struct blk_mq_tag_set *set = q->tag_set; + unsigned int i, j; for_each_possible_cpu(i) { struct blk_mq_ctx *__ctx = per_cpu_ptr(q->queue_ctx, i); @@ -2272,9 +2273,11 @@ static void blk_mq_init_cpu_queues(struct request_queue *q, * Set local node, IFF we have more than one hw queue. If * not, we remain on the home node of the device */ - hctx = blk_mq_map_queue_type(q, 0, i); - if (nr_hw_queues > 1 && hctx->numa_node == NUMA_NO_NODE) - hctx->numa_node = local_memory_node(cpu_to_node(i)); + for (j = 0; j < set->nr_maps; j++) { + hctx = blk_mq_map_queue_type(q, j, i); + if (nr_hw_queues > 1 && hctx->numa_node == NUMA_NO_NODE) + hctx->numa_node = local_memory_node(cpu_to_node(i)); + } } } @@ -2309,7 +2312,7 @@ static void blk_mq_free_map_and_requests(struct blk_mq_tag_set *set, static void blk_mq_map_swqueue(struct request_queue *q) { - unsigned int i, hctx_idx; + unsigned int i, j, hctx_idx; struct blk_mq_hw_ctx *hctx; struct blk_mq_ctx *ctx; struct blk_mq_tag_set *set = q->tag_set; @@ -2345,13 +2348,23 @@ static void blk_mq_map_swqueue(struct request_queue *q) } ctx = per_cpu_ptr(q->queue_ctx, i); - hctx = blk_mq_map_queue_type(q, 0, i); - hctx->type = 0; - cpumask_set_cpu(i, hctx->cpumask); - ctx->index_hw[hctx->type] = hctx->nr_ctx; - hctx->ctxs[hctx->nr_ctx++] = ctx; - /* wrap */ - BUG_ON(!hctx->nr_ctx); + for (j = 0; j < set->nr_maps; j++) { + hctx = blk_mq_map_queue_type(q, j, i); + hctx->type = j; + + /* + * If the CPU is already set in the mask, then we've + * mapped this one already. This can happen if + * devices share queues across queue maps. + */ + if (cpumask_test_cpu(i, hctx->cpumask)) + continue; + cpumask_set_cpu(i, hctx->cpumask); + ctx->index_hw[hctx->type] = hctx->nr_ctx; + hctx->ctxs[hctx->nr_ctx++] = ctx; + /* wrap */ + BUG_ON(!hctx->nr_ctx); + } } mutex_unlock(&q->sysfs_lock); @@ -2519,6 +2532,7 @@ struct request_queue *blk_mq_init_sq_queue(struct blk_mq_tag_set *set, memset(set, 0, sizeof(*set)); set->ops = ops; set->nr_hw_queues = 1; + set->nr_maps = 1; set->queue_depth = queue_depth; set->numa_node = NUMA_NO_NODE; set->flags = set_flags; @@ -2798,6 +2812,8 @@ static int blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set) static int blk_mq_update_queue_map(struct blk_mq_tag_set *set) { if (set->ops->map_queues) { + int i; + /* * transport .map_queues is usually done in the following * way: @@ -2805,18 +2821,21 @@ static int blk_mq_update_queue_map(struct blk_mq_tag_set *set) * for (queue = 0; queue < set->nr_hw_queues; queue++) { * mask = get_cpu_mask(queue) * for_each_cpu(cpu, mask) - * set->map.mq_map[cpu] = queue; + * set->map[x].mq_map[cpu] = queue; * } * * When we need to remap, the table has to be cleared for * killing stale mapping since one CPU may not be mapped * to any hw queue. */ - blk_mq_clear_mq_map(&set->map[0]); + for (i = 0; i < set->nr_maps; i++) + blk_mq_clear_mq_map(&set->map[i]); return set->ops->map_queues(set); - } else + } else { + BUG_ON(set->nr_maps > 1); return blk_mq_map_queues(&set->map[0]); + } } /* @@ -2827,7 +2846,7 @@ static int blk_mq_update_queue_map(struct blk_mq_tag_set *set) */ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) { - int ret; + int i, ret; BUILD_BUG_ON(BLK_MQ_MAX_DEPTH > 1 << BLK_MQ_UNIQUE_TAG_BITS); @@ -2850,6 +2869,11 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) set->queue_depth = BLK_MQ_MAX_DEPTH; } + if (!set->nr_maps) + set->nr_maps = 1; + else if (set->nr_maps > HCTX_MAX_TYPES) + return -EINVAL; + /* * If a crashdump is active, then we are potentially in a very * memory constrained environment. Limit us to 1 queue and @@ -2871,12 +2895,14 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) return -ENOMEM; ret = -ENOMEM; - set->map[0].mq_map = kcalloc_node(nr_cpu_ids, - sizeof(*set->map[0].mq_map), - GFP_KERNEL, set->numa_node); - if (!set->map[0].mq_map) - goto out_free_tags; - set->map[0].nr_queues = set->nr_hw_queues; + for (i = 0; i < set->nr_maps; i++) { + set->map[i].mq_map = kcalloc_node(nr_cpu_ids, + sizeof(struct blk_mq_queue_map), + GFP_KERNEL, set->numa_node); + if (!set->map[i].mq_map) + goto out_free_mq_map; + set->map[i].nr_queues = set->nr_hw_queues; + } ret = blk_mq_update_queue_map(set); if (ret) @@ -2892,9 +2918,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) return 0; out_free_mq_map: - kfree(set->map[0].mq_map); - set->map[0].mq_map = NULL; -out_free_tags: + for (i = 0; i < set->nr_maps; i++) { + kfree(set->map[i].mq_map); + set->map[i].mq_map = NULL; + } kfree(set->tags); set->tags = NULL; return ret; @@ -2903,13 +2930,15 @@ EXPORT_SYMBOL(blk_mq_alloc_tag_set); void blk_mq_free_tag_set(struct blk_mq_tag_set *set) { - int i; + int i, j; for (i = 0; i < nr_cpu_ids; i++) blk_mq_free_map_and_requests(set, i); - kfree(set->map[0].mq_map); - set->map[0].mq_map = NULL; + for (j = 0; j < set->nr_maps; j++) { + kfree(set->map[j].mq_map); + set->map[j].mq_map = NULL; + } kfree(set->tags); set->tags = NULL; diff --git a/block/blk-mq.h b/block/blk-mq.h index 7b5a790acdbf..e27c6f8dc86c 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -72,19 +72,24 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, */ extern int blk_mq_hw_queue_to_node(struct blk_mq_queue_map *qmap, unsigned int); -static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, - unsigned int flags, - int cpu) +static inline struct blk_mq_hw_ctx *blk_mq_map_queue_type(struct request_queue *q, + int type, int cpu) { struct blk_mq_tag_set *set = q->tag_set; - return q->queue_hw_ctx[set->map[0].mq_map[cpu]]; + return q->queue_hw_ctx[set->map[type].mq_map[cpu]]; } -static inline struct blk_mq_hw_ctx *blk_mq_map_queue_type(struct request_queue *q, - int type, int cpu) +static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q, + unsigned int flags, + int cpu) { - return blk_mq_map_queue(q, type, cpu); + int type = 0; + + if (q->mq_ops->flags_to_type) + type = q->mq_ops->flags_to_type(q, flags); + + return blk_mq_map_queue_type(q, type, cpu); } /* diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index f9e19962a22f..837087cf07cc 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -86,6 +86,7 @@ enum { struct blk_mq_tag_set { struct blk_mq_queue_map map[HCTX_MAX_TYPES]; + unsigned int nr_maps; const struct blk_mq_ops *ops; unsigned int nr_hw_queues; unsigned int queue_depth; /* max hw supported */ @@ -109,6 +110,7 @@ struct blk_mq_queue_data { typedef blk_status_t (queue_rq_fn)(struct blk_mq_hw_ctx *, const struct blk_mq_queue_data *); +typedef int (flags_to_type_fn)(struct request_queue *, unsigned int); typedef bool (get_budget_fn)(struct blk_mq_hw_ctx *); typedef void (put_budget_fn)(struct blk_mq_hw_ctx *); typedef enum blk_eh_timer_return (timeout_fn)(struct request *, bool); @@ -133,6 +135,11 @@ struct blk_mq_ops { */ queue_rq_fn *queue_rq; + /* + * Return a queue map type for the given request/bio flags + */ + flags_to_type_fn *flags_to_type; + /* * Reserve budget before queue request, once .queue_rq is * run, it is driver's responsibility to release the From patchwork Mon Oct 29 16:37:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659705 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 980D91751 for ; Mon, 29 Oct 2018 16:38:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E3B22967A for ; Mon, 29 Oct 2018 16:38:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 727C529C45; Mon, 29 Oct 2018 16:38:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01ECF2967A for ; Mon, 29 Oct 2018 16:38:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728266AbeJ3B1W (ORCPT ); Mon, 29 Oct 2018 21:27:22 -0400 Received: from mail-it1-f195.google.com ([209.85.166.195]:32999 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728273AbeJ3B1V (ORCPT ); Mon, 29 Oct 2018 21:27:21 -0400 Received: by mail-it1-f195.google.com with SMTP id p11-v6so876870itf.0 for ; Mon, 29 Oct 2018 09:38:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CCxQ1w7oEfeYNp97sRQwoZGVpyu0hoStQAxbQ0x4cMs=; b=1x9QymbKSfQAQsEaeunWt42kHkrjMbrs37kzbMSUPQ6kQQVhCOLo4tC9Q72IzkoGen /y4yFpjcwH3Kvhkcu1BrpwfjByT85VqZX3zMwQ+fcxS2SI/d4XcOOmwdLVCtH8DEWvro Y8BygKmSvW4drwMIfl9l7x/tIDz+fXl1nx0WaNXZH5B/pYlwvoix1PAQGftREplo4TLe 6uGNwji5R2mRnR7Dt9jkEv9i9DEtPuhAHGeeTEaL5VfAouZF3i01okwpF9lU8VI5+z6T Vvto4i83rswwrASfhDL3trpevQ4lkcC8OUX9i59RmY6wvrwb0j+Hs4ZEus4phy++AiAk ycvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CCxQ1w7oEfeYNp97sRQwoZGVpyu0hoStQAxbQ0x4cMs=; b=Cfk5V14hRms1FM9nqJeISS/XsMm9gmXJQ8W93Wq8sHFDCRJ3vYIVwpYJxsUWpc2bFI uOpD4Jl5WUlZ8vO83Hu2MymGwnvkAf2xQSOCgXvPWn0WsX1yT0JlWUszo07VeWXqXY3s wgARDaTHGvnhSq9+dK4hdHGxv94mU5VUIl8cZVHgfXK/hNJ8v9wSf2v2FIp1RiVfPTpn OfC8D3pq0uJ1xuTkf2Y5WWwcQolzW8x2UVhUb6t6Ch1m6wX4aYHoXMrqW2n5oFyKkACc pf9QAttr1NZySDi0EtVcY+eVOJBnlcYvnKhpdb+F4JAgX4PkG46BP2VMAj2Gd+LScbZk OdLA== X-Gm-Message-State: AGRZ1gKXWbQxVhs6bkdjiLZmXocPF98qvU1GUT1KyRPQUzsXRZWEV9bt n/BU5SPuLLq35pZOrVHS3jjw+jYsBV8= X-Google-Smtp-Source: AJdET5fmxR4SW5z/LgB6ktrnLBgw6BrY449jiuxnWGCX46++qkiTSZHw10Ttf2eU3HjE+B9sXt3d8w== X-Received: by 2002:a24:1a56:: with SMTP id 83-v6mr6138029iti.135.1540831079149; Mon, 29 Oct 2018 09:37:59 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:37:57 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 08/14] blk-mq: separate number of hardware queues from nr_cpu_ids Date: Mon, 29 Oct 2018 10:37:32 -0600 Message-Id: <20181029163738.10172-9-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP With multiple maps, nr_cpu_ids is no longer the maximum number of hardware queues we support on a given devices. The initializer of the tag_set can have set ->nr_hw_queues larger than the available number of CPUs, since we can exceed that with multiple queue maps. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe Reviewed-by: Bart Van Assche --- block/blk-mq.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 0fab36372ace..60a951c4934c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2663,6 +2663,19 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, mutex_unlock(&q->sysfs_lock); } +/* + * Maximum number of queues we support. For single sets, we'll never have + * more than the CPUs (software queues). For multiple sets, the tag_set + * user may have set ->nr_hw_queues larger. + */ +static unsigned int nr_hw_queues(struct blk_mq_tag_set *set) +{ + if (set->nr_maps == 1) + return nr_cpu_ids; + + return max(set->nr_hw_queues, nr_cpu_ids); +} + struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, struct request_queue *q) { @@ -2682,7 +2695,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, /* init q->mq_kobj and sw queues' kobjects */ blk_mq_sysfs_init(q); - q->queue_hw_ctx = kcalloc_node(nr_cpu_ids, sizeof(*(q->queue_hw_ctx)), + q->nr_queues = nr_hw_queues(set); + q->queue_hw_ctx = kcalloc_node(q->nr_queues, sizeof(*(q->queue_hw_ctx)), GFP_KERNEL, set->numa_node); if (!q->queue_hw_ctx) goto err_percpu; @@ -2694,7 +2708,6 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, INIT_WORK(&q->timeout_work, blk_mq_timeout_work); blk_queue_rq_timeout(q, set->timeout ? set->timeout : 30 * HZ); - q->nr_queues = nr_cpu_ids; q->tag_set = set; q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT; @@ -2884,12 +2897,13 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) set->queue_depth = min(64U, set->queue_depth); } /* - * There is no use for more h/w queues than cpus. + * There is no use for more h/w queues than cpus if we just have + * a single map */ - if (set->nr_hw_queues > nr_cpu_ids) + if (set->nr_maps == 1 && set->nr_hw_queues > nr_cpu_ids) set->nr_hw_queues = nr_cpu_ids; - set->tags = kcalloc_node(nr_cpu_ids, sizeof(struct blk_mq_tags *), + set->tags = kcalloc_node(nr_hw_queues(set), sizeof(struct blk_mq_tags *), GFP_KERNEL, set->numa_node); if (!set->tags) return -ENOMEM; @@ -2932,7 +2946,7 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set) { int i, j; - for (i = 0; i < nr_cpu_ids; i++) + for (i = 0; i < nr_hw_queues(set); i++) blk_mq_free_map_and_requests(set, i); for (j = 0; j < set->nr_maps; j++) { @@ -3064,7 +3078,7 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, lockdep_assert_held(&set->tag_list_lock); - if (nr_hw_queues > nr_cpu_ids) + if (set->nr_maps == 1 && nr_hw_queues > nr_cpu_ids) nr_hw_queues = nr_cpu_ids; if (nr_hw_queues < 1 || nr_hw_queues == set->nr_hw_queues) return; From patchwork Mon Oct 29 16:37:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659707 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C26D01751 for ; Mon, 29 Oct 2018 16:38:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A4CC82967A for ; Mon, 29 Oct 2018 16:38:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 98EE429C45; Mon, 29 Oct 2018 16:38:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 391912967A for ; Mon, 29 Oct 2018 16:38:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728292AbeJ3B1Y (ORCPT ); Mon, 29 Oct 2018 21:27:24 -0400 Received: from mail-it1-f194.google.com ([209.85.166.194]:32999 "EHLO mail-it1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728293AbeJ3B1Y (ORCPT ); Mon, 29 Oct 2018 21:27:24 -0400 Received: by mail-it1-f194.google.com with SMTP id p11-v6so876925itf.0 for ; Mon, 29 Oct 2018 09:38:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+Ubma7dzFX73Hbh4PnPWob3RQ4KFeUeAMfxQdKC8go8=; b=Yd47KbApQEd/1cn4fhtp/9aMGY4JyemCI0tK+Q3VU6uBVkHWnexp8wnOthKXO+1d1j 9pMn/MKd2iXdtc8P4Vw6EWw+pO1ml0OCkCkNGhCYZCldTvBu+6Sy16wC6SvFIjD2Kso9 VKnBp1DJyZXRbgOiqfxitizZnxYZWZPMfNkb6eTRg0KT+FcIABMzMyt3FjRCOuLEXT/p fXzRL/Wu3kLTZLY/7Is9jLBlMWbhNCb/DgcqLstVvPOZkVwlTGB4ZTj45WRUBd88/4y1 1VxqnBj49AAzg7DO0BZfYlFfC0YtmUyApqDGNOllLHUbjQm0gwGXVNJ9SKYM3DCbD3Vd fjNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+Ubma7dzFX73Hbh4PnPWob3RQ4KFeUeAMfxQdKC8go8=; b=tdDJskU1uc6On1ynHAy4A433LWKfJ9YetkfA3vUDpSWkKDhrg3hWnypAr9F/ckwqdL 7mXRCBr60UhBQlRn7mSD/TesJyq9SRcQwfY+qpKUt/ry6gzh32Qxj/xz0XsT4BHsViBR NVtp/KfIgYyI9T8mSe5hp5Q8nqspbHCI4qZdlYPLIf3v8XWYMRWdbqKuLMmowgpFnJFx CkDxHKQsaCVa5O3V+2UnCmnlkaOSEgi73vxs1toaTLV51sYl3U6nio2/kt/1vtjYJbRa ks3o0RCdCSlhIo8dpzikgQt6argvUp2yyxFBibHGzLsUbRJGvOSviaKnwSOn3feCdlOQ UYYg== X-Gm-Message-State: AGRZ1gJLj7rwRugANezvuv6QVSsZeCcXL1RMRiX8UlVM33GJsgwJzhdH HOA2slmFutwLWF7Nx4AJmM1Hmz/I3Wg= X-Google-Smtp-Source: AJdET5dAOXg1i6IzXCnAtCwU9nkIkH03E0vPTE6obve10Xk/y4S1tX8YKXzwn1TwKyIhxmv8FvMk7A== X-Received: by 2002:a02:9f85:: with SMTP id a5-v6mr3533728jam.2.1540831081595; Mon, 29 Oct 2018 09:38:01 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.37.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:38:00 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 09/14] blk-mq: ensure that plug lists don't straddle hardware queues Date: Mon, 29 Oct 2018 10:37:33 -0600 Message-Id: <20181029163738.10172-10-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since we insert per hardware queue, we have to ensure that every request on the plug list being inserted belongs to the same hardware queue. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- block/blk-mq.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 60a951c4934c..52b07188b39a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1621,6 +1621,27 @@ static int plug_ctx_cmp(void *priv, struct list_head *a, struct list_head *b) blk_rq_pos(rqa) < blk_rq_pos(rqb))); } +/* + * Need to ensure that the hardware queue matches, so we don't submit + * a list of requests that end up on different hardware queues. + */ +static bool ctx_match(struct request *req, struct blk_mq_ctx *ctx, + unsigned int flags) +{ + if (req->mq_ctx != ctx) + return false; + + /* + * If we just have one map, then we know the hctx will match + * if the ctx matches + */ + if (req->q->tag_set->nr_maps == 1) + return true; + + return blk_mq_map_queue(req->q, req->cmd_flags, ctx->cpu) == + blk_mq_map_queue(req->q, flags, ctx->cpu); +} + void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) { struct blk_mq_ctx *this_ctx; @@ -1628,7 +1649,7 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) struct request *rq; LIST_HEAD(list); LIST_HEAD(ctx_list); - unsigned int depth; + unsigned int depth, this_flags; list_splice_init(&plug->mq_list, &list); @@ -1636,13 +1657,14 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) this_q = NULL; this_ctx = NULL; + this_flags = 0; depth = 0; while (!list_empty(&list)) { rq = list_entry_rq(list.next); list_del_init(&rq->queuelist); BUG_ON(!rq->q); - if (rq->mq_ctx != this_ctx) { + if (!ctx_match(rq, this_ctx, this_flags)) { if (this_ctx) { trace_block_unplug(this_q, depth, !from_schedule); blk_mq_sched_insert_requests(this_q, this_ctx, @@ -1650,6 +1672,7 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) from_schedule); } + this_flags = rq->cmd_flags; this_ctx = rq->mq_ctx; this_q = rq->q; depth = 0; From patchwork Mon Oct 29 16:37:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659727 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5FE931751 for ; Mon, 29 Oct 2018 16:38:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4590229485 for ; Mon, 29 Oct 2018 16:38:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3A70229494; Mon, 29 Oct 2018 16:38:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D682F29485 for ; Mon, 29 Oct 2018 16:38:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728322AbeJ3B10 (ORCPT ); Mon, 29 Oct 2018 21:27:26 -0400 Received: from mail-it1-f196.google.com ([209.85.166.196]:38268 "EHLO mail-it1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728315AbeJ3B10 (ORCPT ); Mon, 29 Oct 2018 21:27:26 -0400 Received: by mail-it1-f196.google.com with SMTP id i76-v6so10174456ita.3 for ; Mon, 29 Oct 2018 09:38:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Hu4SRygQduE3Ah39+YjJ5DfLsSb40qLLfETVJ69MQec=; b=Qne8Q/R59peKayL//DkDF0zKocJo81nHGGU6Fx1JzYRYHOhZbze0azcwjFJabpP8X+ ABtnKdE1fmHihK1tir7me+SNQzUN69X46KPPXfWirjWcjAl/snCbDzghbEWBXZ0wGu6a K+KyoiV6bTtnvR5u2FBiJBc0rwRf36pbB/b3kxLW/S3egAKFjwnfqFfoQZr5ZBLnsCJj XBxrogqmOy9kcIiSb6xYhnhD8DROhBh1AchvYOctmYtdbiqxkJmn7yKfiXBRXssTD1Js m3B2mgsr7WD4VttVAHgF1XEhU/YAmAcB3H2syrBFi+7t5JKqztU7PHAgFY4nUmH4bSQ0 8wCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Hu4SRygQduE3Ah39+YjJ5DfLsSb40qLLfETVJ69MQec=; b=cyyNMFWCKaMv9cbRwVKpHdOKzawB2yxBVq0X3JeeawMpMG9IWyIe6up3UmYhsRSvcr KteSYVty6GZ+EgqCspwZb/Soo0tckV7jnT2nQ5R96NGQCZvhHa6Bo73sg6SNRhO/21Yo SLGHYd/TZRftkZEeL7PUIza6WmUP1dpVbCgOAZ9dDu1cbqFxkpxduKmGnigRBPgyJuCW W8g2o0Vrkm4EnZzVjJF0/U86V5BZ0lGDSuUI8nDPSIeLI3uqQ6iqQpda2n2J6ChW/Pc8 4v//sFINcE1ATLQ4N8j1wNVGyN0NKHGhEwy2XP8GijW7JsDh9aj7Ay447L7UAmDY5kxx GnZg== X-Gm-Message-State: AGRZ1gJl/QDuJexCums5mrr/UUKIgCiXCNyBKBaV2x6JNeKFJv9sNFQb KX088jc03M7lmQHniVD4x++3im+GjSE= X-Google-Smtp-Source: AJdET5dyPe1b17iC0D8wHHWNBSOj0Y/26jftcPO2w99nVKDSRdrIi60lcBEFoh16qhUPi0PYmD7ciQ== X-Received: by 2002:a05:660c:484:: with SMTP id a4mr3990245itk.116.1540831083626; Mon, 29 Oct 2018 09:38:03 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.38.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:38:02 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 10/14] blk-mq: initial support for multiple queue maps Date: Mon, 29 Oct 2018 10:37:34 -0600 Message-Id: <20181029163738.10172-11-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add a queue offset to the tag map. This enables users to map iteratively, for each queue map type they support. Bump maximum number of supported maps to 2, we're now fully able to support more than 1 map. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- block/blk-mq-cpumap.c | 9 +++++---- block/blk-mq-pci.c | 2 +- block/blk-mq-virtio.c | 2 +- include/linux/blk-mq.h | 3 ++- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/block/blk-mq-cpumap.c b/block/blk-mq-cpumap.c index 6e6686c55984..03a534820271 100644 --- a/block/blk-mq-cpumap.c +++ b/block/blk-mq-cpumap.c @@ -14,9 +14,10 @@ #include "blk.h" #include "blk-mq.h" -static int cpu_to_queue_index(unsigned int nr_queues, const int cpu) +static int cpu_to_queue_index(struct blk_mq_queue_map *qmap, + unsigned int nr_queues, const int cpu) { - return cpu % nr_queues; + return qmap->queue_offset + (cpu % nr_queues); } static int get_first_sibling(unsigned int cpu) @@ -44,11 +45,11 @@ int blk_mq_map_queues(struct blk_mq_queue_map *qmap) * performace optimizations. */ if (cpu < nr_queues) { - map[cpu] = cpu_to_queue_index(nr_queues, cpu); + map[cpu] = cpu_to_queue_index(qmap, nr_queues, cpu); } else { first_sibling = get_first_sibling(cpu); if (first_sibling == cpu) - map[cpu] = cpu_to_queue_index(nr_queues, cpu); + map[cpu] = cpu_to_queue_index(qmap, nr_queues, cpu); else map[cpu] = map[first_sibling]; } diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c index 40333d60a850..1dce18553984 100644 --- a/block/blk-mq-pci.c +++ b/block/blk-mq-pci.c @@ -43,7 +43,7 @@ int blk_mq_pci_map_queues(struct blk_mq_queue_map *qmap, struct pci_dev *pdev, goto fallback; for_each_cpu(cpu, mask) - qmap->mq_map[cpu] = queue; + qmap->mq_map[cpu] = qmap->queue_offset + queue; } return 0; diff --git a/block/blk-mq-virtio.c b/block/blk-mq-virtio.c index 661fbfef480f..370827163835 100644 --- a/block/blk-mq-virtio.c +++ b/block/blk-mq-virtio.c @@ -44,7 +44,7 @@ int blk_mq_virtio_map_queues(struct blk_mq_queue_map *qmap, goto fallback; for_each_cpu(cpu, mask) - qmap->mq_map[cpu] = queue; + qmap->mq_map[cpu] = qmap->queue_offset + queue; } return 0; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 837087cf07cc..b5ae2b5677c1 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -78,10 +78,11 @@ struct blk_mq_hw_ctx { struct blk_mq_queue_map { unsigned int *mq_map; unsigned int nr_queues; + unsigned int queue_offset; }; enum { - HCTX_MAX_TYPES = 1, + HCTX_MAX_TYPES = 2, }; struct blk_mq_tag_set { From patchwork Mon Oct 29 16:37:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659725 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 980603CF1 for ; Mon, 29 Oct 2018 16:38:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EF2629493 for ; Mon, 29 Oct 2018 16:38:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 72E8329494; Mon, 29 Oct 2018 16:38:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1319429495 for ; Mon, 29 Oct 2018 16:38:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728007AbeJ3B1y (ORCPT ); Mon, 29 Oct 2018 21:27:54 -0400 Received: from mail-it1-f193.google.com ([209.85.166.193]:53590 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728337AbeJ3B13 (ORCPT ); Mon, 29 Oct 2018 21:27:29 -0400 Received: by mail-it1-f193.google.com with SMTP id f16-v6so2767428ita.3 for ; Mon, 29 Oct 2018 09:38:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=eukhLpnpSqFgkXd6fwxIBdknpGfhnAOi4PUxG7MjAPc=; b=AIPNHXKuRPpEDkKj0gOHLJv8h1wUyO1OQ649DAib/5VXwcfXfApINpexfzi/OEmyWM ByVl2GjNjeRztMNXg8Nyfy9/yUcgo6B7WCKskznq3ja+rMAiKsLtT8+g+TMxf2Wdzmre DY0XBnYmjCeq3goRB6agwt8eSaoAMvG3tbN9sCh8tKkk4woixyCzuoZjlcvT/QtQEQAy gZl2tI48OD3FxQU1SElBNX8arISa5ZFZWUQNjHJV9BtYIRZhF3hK7Ug1nmHFsSk20/Oa JOpmGFtEhvRgHRRU6Kd5uIpJandGXpTdZ6u4nVS9tcyLrM+E00tOx/lSfWerxze27lof QMPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=eukhLpnpSqFgkXd6fwxIBdknpGfhnAOi4PUxG7MjAPc=; b=eadaEsCsBBL7EpFedes1gWC97V8h7j4lN9P4ywyiZk50EIRNMIEi+jV13T9N5guAdG +Fesn2WKxwxZI6MY22+PBg7KfZd+CjxtQVfzS97+lBBvXOQoQuR5ZOPKyMeqdVWurj3g x6cc6vSBmFQJgllt09VtXDu9tapdMp10ecXDEPIb4k8d7l4VFJb17B7Kn7hiM5r7JKwz KQ0ybiei6Y8YsKIZ9IDAB6jPwtSDFJmlBTeo4sGSdRQ1maDLJLsssrINTd66pWuXFCCV /ko/R4xPzJ2C04md8F6UZrvr6BS3/sSJX6eX8s+zfud877LohSB/L/HldFOwwXG4M1c5 PtbA== X-Gm-Message-State: AGRZ1gKH9KkQYxIJ+HkXqnvAJ0MeIwotpLsOvWVVHg7nsTglXP6Rb3AH i1xjZitXFsrO4YobYexdUqKp2zvi66U= X-Google-Smtp-Source: AJdET5fd+fbMZ5osHM5Yz/a2ZBO184Xx/g/2uQxElWT1m4lx66oTjIL/cDOBeMax1x6UOI1kliXEJg== X-Received: by 2002:a02:3b2c:: with SMTP id c44-v6mr10944100jaa.31.1540831087063; Mon, 29 Oct 2018 09:38:07 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.38.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:38:04 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe , Thomas Gleixner Subject: [PATCH 11/14] irq: add support for allocating (and affinitizing) sets of IRQs Date: Mon, 29 Oct 2018 10:37:35 -0600 Message-Id: <20181029163738.10172-12-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A driver may have a need to allocate multiple sets of MSI/MSI-X interrupts, and have them appropriately affinitized. Add support for defining a number of sets in the irq_affinity structure, of varying sizes, and get each set affinitized correctly across the machine. Cc: Thomas Gleixner Cc: linux-kernel@vger.kernel.org Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe Reviewed-by: Ming Lei Reviewed-by: Keith Busch --- include/linux/interrupt.h | 4 ++++ kernel/irq/affinity.c | 40 ++++++++++++++++++++++++++++++--------- 2 files changed, 35 insertions(+), 9 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index 1d6711c28271..ca397ff40836 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -247,10 +247,14 @@ struct irq_affinity_notify { * the MSI(-X) vector space * @post_vectors: Don't apply affinity to @post_vectors at end of * the MSI(-X) vector space + * @nr_sets: Length of passed in *sets array + * @sets: Number of affinitized sets */ struct irq_affinity { int pre_vectors; int post_vectors; + int nr_sets; + int *sets; }; #if defined(CONFIG_SMP) diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c index f4f29b9d90ee..2046a0f0f0f1 100644 --- a/kernel/irq/affinity.c +++ b/kernel/irq/affinity.c @@ -180,6 +180,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd) int curvec, usedvecs; cpumask_var_t nmsk, npresmsk, *node_to_cpumask; struct cpumask *masks = NULL; + int i, nr_sets; /* * If there aren't any vectors left after applying the pre/post @@ -210,10 +211,23 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd) get_online_cpus(); build_node_to_cpumask(node_to_cpumask); - /* Spread on present CPUs starting from affd->pre_vectors */ - usedvecs = irq_build_affinity_masks(affd, curvec, affvecs, - node_to_cpumask, cpu_present_mask, - nmsk, masks); + /* + * Spread on present CPUs starting from affd->pre_vectors. If we + * have multiple sets, build each sets affinity mask separately. + */ + nr_sets = affd->nr_sets; + if (!nr_sets) + nr_sets = 1; + + for (i = 0, usedvecs = 0; i < nr_sets; i++) { + int this_vecs = affd->sets ? affd->sets[i] : affvecs; + int nr; + + nr = irq_build_affinity_masks(affd, curvec, this_vecs, + node_to_cpumask, cpu_present_mask, + nmsk, masks + usedvecs); + usedvecs += nr; + } /* * Spread on non present CPUs starting from the next vector to be @@ -258,13 +272,21 @@ int irq_calc_affinity_vectors(int minvec, int maxvec, const struct irq_affinity { int resv = affd->pre_vectors + affd->post_vectors; int vecs = maxvec - resv; - int ret; + int set_vecs; if (resv > minvec) return 0; - get_online_cpus(); - ret = min_t(int, cpumask_weight(cpu_possible_mask), vecs) + resv; - put_online_cpus(); - return ret; + if (affd->nr_sets) { + int i; + + for (i = 0, set_vecs = 0; i < affd->nr_sets; i++) + set_vecs += affd->sets[i]; + } else { + get_online_cpus(); + set_vecs = cpumask_weight(cpu_possible_mask); + put_online_cpus(); + } + + return resv + min(set_vecs, vecs); } From patchwork Mon Oct 29 16:37:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659721 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B42313B5 for ; Mon, 29 Oct 2018 16:38:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3117429485 for ; Mon, 29 Oct 2018 16:38:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 244F129494; Mon, 29 Oct 2018 16:38:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8172829485 for ; Mon, 29 Oct 2018 16:38:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727841AbeJ3B1v (ORCPT ); Mon, 29 Oct 2018 21:27:51 -0400 Received: from mail-io1-f67.google.com ([209.85.166.67]:33582 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728338AbeJ3B1b (ORCPT ); Mon, 29 Oct 2018 21:27:31 -0400 Received: by mail-io1-f67.google.com with SMTP id f12-v6so2111643iog.0 for ; Mon, 29 Oct 2018 09:38:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BdwO0LohnhfWerZuHJKSr7FFsXgq9TwdYLAuiZgg/AU=; b=g8udK3zQPJXvHAuqmR0bLnaG09AbP1RMdB43PoUGlGEDRFBC+4NG0+datDoeJH7dUC s9T9up2g22UgUZSIUE+ax/93Qa1HuM7HS9iZvMVjvMoWVX8FQLKg4PVsC9Hp6VvuKbRH dn7/+MsAdXRIKvt5K8MSAsk3+JjQ506wkpE0hVa9jgOWXOS/YfN/UwErMFp0G1nPQqrj 2rGoyxuMpAD0QmOjZw6hwC5f49m8M8JxHn1HasI8j8D6k6Aqp/gLRr6bDNLGZLAEL9Cz 1L6KmEyrJyiAdt6wP5OrSFAFGhzlfYBbWRF7iYha/fPjJdbU5R7RwhYzWMKJXn6qjD/K Dk9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BdwO0LohnhfWerZuHJKSr7FFsXgq9TwdYLAuiZgg/AU=; b=rLCYWlUBXX5mAmailsGg0XUn/kgyTS1SpS6AMo2S5VqZf7rOL6631kISDHUzFZfm8d 5Z4U+T/irmdyLUXhunKg3GasvExRwUtSp78aJ5SqJZK++QOEjQ6emmzwfhDkjkgwCD/N wOR09sXP+1igw758JNDoRY9Rf72eppsMrZ+KnKVx4sCiZNJNzEZqve+c5Xkj9dLpvqWn cAFhFBzmc/qhMIFCsQBt5Ti325YLCnlRykvwRGGB4AqQ1NZh57TK3dHoJ8ACxy5iKCRA O9iY5C2lHAWWBFmdpjcO8enDYrU1NHBL09SrThtRT602Wy1EYL4nf/nL7yOKg/qdxMnR a8kw== X-Gm-Message-State: AGRZ1gJq5bft7DM6yTMqTymBePcWwA9JaHDKlF3mhzdtg6C3eqxX8Rgl zp7E8d7rLNyyZaUV6OVEVbfDuXaIGq4= X-Google-Smtp-Source: AJdET5drGBSDuDIaaFxhXsP+Ty4iMHQ0CMn/dMlbYERKCnS819QI4BiLzuU1VcvMAKZpmM5evweLnA== X-Received: by 2002:a6b:9286:: with SMTP id u128-v6mr8236839iod.264.1540831089328; Mon, 29 Oct 2018 09:38:09 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.38.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:38:08 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 12/14] nvme: utilize two queue maps, one for reads and one for writes Date: Mon, 29 Oct 2018 10:37:36 -0600 Message-Id: <20181029163738.10172-13-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP NVMe does round-robin between queues by default, which means that sharing a queue map for both reads and writes can be problematic in terms of read servicing. It's much easier to flood the queue with writes and reduce the read servicing. Implement two queue maps, one for reads and one for writes. The write queue count is configurable through the 'write_queues' parameter. By default, we retain the previous behavior of having a single queue set, shared between reads and writes. Setting 'write_queues' to a non-zero value will create two queue sets, one for reads and one for writes, the latter using the configurable number of queues (hardware queue counts permitting). Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- drivers/nvme/host/pci.c | 139 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 131 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index e5d783cb6937..658c9a2f4114 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -74,11 +74,29 @@ static int io_queue_depth = 1024; module_param_cb(io_queue_depth, &io_queue_depth_ops, &io_queue_depth, 0644); MODULE_PARM_DESC(io_queue_depth, "set io queue depth, should >= 2"); +static int queue_count_set(const char *val, const struct kernel_param *kp); +static const struct kernel_param_ops queue_count_ops = { + .set = queue_count_set, + .get = param_get_int, +}; + +static int write_queues; +module_param_cb(write_queues, &queue_count_ops, &write_queues, 0644); +MODULE_PARM_DESC(write_queues, + "Number of queues to use for writes. If not set, reads and writes " + "will share a queue set."); + struct nvme_dev; struct nvme_queue; static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown); +enum { + NVMEQ_TYPE_READ, + NVMEQ_TYPE_WRITE, + NVMEQ_TYPE_NR, +}; + /* * Represents an NVM Express device. Each nvme_dev is a PCI function. */ @@ -92,6 +110,7 @@ struct nvme_dev { struct dma_pool *prp_small_pool; unsigned online_queues; unsigned max_qid; + unsigned io_queues[NVMEQ_TYPE_NR]; unsigned int num_vecs; int q_depth; u32 db_stride; @@ -134,6 +153,17 @@ static int io_queue_depth_set(const char *val, const struct kernel_param *kp) return param_set_int(val, kp); } +static int queue_count_set(const char *val, const struct kernel_param *kp) +{ + int n = 0, ret; + + ret = kstrtoint(val, 10, &n); + if (n > num_possible_cpus()) + n = num_possible_cpus(); + + return param_set_int(val, kp); +} + static inline unsigned int sq_idx(unsigned int qid, u32 stride) { return qid * 2 * stride; @@ -218,9 +248,20 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64); } +static unsigned int max_io_queues(void) +{ + return num_possible_cpus() + write_queues; +} + +static unsigned int max_queue_count(void) +{ + /* IO queues + admin queue */ + return 1 + max_io_queues(); +} + static inline unsigned int nvme_dbbuf_size(u32 stride) { - return ((num_possible_cpus() + 1) * 8 * stride); + return (max_queue_count() * 8 * stride); } static int nvme_dbbuf_dma_alloc(struct nvme_dev *dev) @@ -431,12 +472,41 @@ static int nvme_init_request(struct blk_mq_tag_set *set, struct request *req, return 0; } +static int queue_irq_offset(struct nvme_dev *dev) +{ + /* if we have more than 1 vec, admin queue offsets us 1 */ + if (dev->num_vecs > 1) + return 1; + + return 0; +} + static int nvme_pci_map_queues(struct blk_mq_tag_set *set) { struct nvme_dev *dev = set->driver_data; + int i, qoff, offset; + + offset = queue_irq_offset(dev); + for (i = 0, qoff = 0; i < set->nr_maps; i++) { + struct blk_mq_queue_map *map = &set->map[i]; + + map->nr_queues = dev->io_queues[i]; + if (!map->nr_queues) { + BUG_ON(i == NVMEQ_TYPE_READ); - return blk_mq_pci_map_queues(&set->map[0], to_pci_dev(dev->dev), - dev->num_vecs > 1 ? 1 /* admin queue */ : 0); + /* shared set, resuse read set parameters */ + map->nr_queues = dev->io_queues[NVMEQ_TYPE_READ]; + qoff = 0; + offset = queue_irq_offset(dev); + } + + map->queue_offset = qoff; + blk_mq_pci_map_queues(map, to_pci_dev(dev->dev), offset); + qoff += map->nr_queues; + offset += map->nr_queues; + } + + return 0; } /** @@ -849,6 +919,14 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; } +static int nvme_flags_to_type(struct request_queue *q, unsigned int flags) +{ + if ((flags & REQ_OP_MASK) == REQ_OP_READ) + return NVMEQ_TYPE_READ; + + return NVMEQ_TYPE_WRITE; +} + static void nvme_pci_complete_rq(struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); @@ -1476,6 +1554,7 @@ static const struct blk_mq_ops nvme_mq_admin_ops = { static const struct blk_mq_ops nvme_mq_ops = { .queue_rq = nvme_queue_rq, + .flags_to_type = nvme_flags_to_type, .complete = nvme_pci_complete_rq, .init_hctx = nvme_init_hctx, .init_request = nvme_init_request, @@ -1888,18 +1967,53 @@ static int nvme_setup_host_mem(struct nvme_dev *dev) return ret; } +static void nvme_calc_io_queues(struct nvme_dev *dev, unsigned int nr_io_queues) +{ + unsigned int this_w_queues = write_queues; + + /* + * Setup read/write queue split + */ + if (nr_io_queues == 1) { + dev->io_queues[NVMEQ_TYPE_READ] = 1; + dev->io_queues[NVMEQ_TYPE_WRITE] = 0; + return; + } + + /* + * If 'write_queues' is set, ensure it leaves room for at least + * one read queue + */ + if (this_w_queues >= nr_io_queues) + this_w_queues = nr_io_queues - 1; + + /* + * If 'write_queues' is set to zero, reads and writes will share + * a queue set. + */ + if (!this_w_queues) { + dev->io_queues[NVMEQ_TYPE_WRITE] = 0; + dev->io_queues[NVMEQ_TYPE_READ] = nr_io_queues; + } else { + dev->io_queues[NVMEQ_TYPE_WRITE] = this_w_queues; + dev->io_queues[NVMEQ_TYPE_READ] = nr_io_queues - this_w_queues; + } +} + static int nvme_setup_io_queues(struct nvme_dev *dev) { struct nvme_queue *adminq = &dev->queues[0]; struct pci_dev *pdev = to_pci_dev(dev->dev); int result, nr_io_queues; unsigned long size; - + int irq_sets[2]; struct irq_affinity affd = { - .pre_vectors = 1 + .pre_vectors = 1, + .nr_sets = ARRAY_SIZE(irq_sets), + .sets = irq_sets, }; - nr_io_queues = num_possible_cpus(); + nr_io_queues = max_io_queues(); result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues); if (result < 0) return result; @@ -1929,6 +2043,12 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) /* Deregister the admin queue's interrupt */ pci_free_irq(pdev, 0, adminq); + nvme_calc_io_queues(dev, nr_io_queues); + irq_sets[0] = dev->io_queues[NVMEQ_TYPE_READ]; + irq_sets[1] = dev->io_queues[NVMEQ_TYPE_WRITE]; + if (!irq_sets[1]) + affd.nr_sets = 1; + /* * If we enable msix early due to not intx, disable it again before * setting up the full range we need. @@ -1941,6 +2061,8 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) dev->num_vecs = result; dev->max_qid = max(result - 1, 1); + nvme_calc_io_queues(dev, dev->max_qid); + /* * Should investigate if there's a performance win from allocating * more queues than interrupt vectors; it might allow the submission @@ -2042,6 +2164,7 @@ static int nvme_dev_add(struct nvme_dev *dev) if (!dev->ctrl.tagset) { dev->tagset.ops = &nvme_mq_ops; dev->tagset.nr_hw_queues = dev->online_queues - 1; + dev->tagset.nr_maps = NVMEQ_TYPE_NR; dev->tagset.timeout = NVME_IO_TIMEOUT; dev->tagset.numa_node = dev_to_node(dev->dev); dev->tagset.queue_depth = @@ -2489,8 +2612,8 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (!dev) return -ENOMEM; - dev->queues = kcalloc_node(num_possible_cpus() + 1, - sizeof(struct nvme_queue), GFP_KERNEL, node); + dev->queues = kcalloc_node(max_queue_count(), sizeof(struct nvme_queue), + GFP_KERNEL, node); if (!dev->queues) goto free; From patchwork Mon Oct 29 16:37:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659711 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 571F413B5 for ; Mon, 29 Oct 2018 16:38:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3CC612967A for ; Mon, 29 Oct 2018 16:38:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3163B29C4A; Mon, 29 Oct 2018 16:38:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC5D82967A for ; Mon, 29 Oct 2018 16:38:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727769AbeJ3B1f (ORCPT ); Mon, 29 Oct 2018 21:27:35 -0400 Received: from mail-io1-f67.google.com ([209.85.166.67]:36063 "EHLO mail-io1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728046AbeJ3B1d (ORCPT ); Mon, 29 Oct 2018 21:27:33 -0400 Received: by mail-io1-f67.google.com with SMTP id o19-v6so5379729iod.3 for ; Mon, 29 Oct 2018 09:38:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Q05dSOPfwm/UowrXt9AMeUMp3mZmjRCMcDVjG5ZA724=; b=COF20vKrj63JShQizBSeV2pci2HC1D41bwvx4tlSvmxUL68aRt2QDcWMN19XF7+rjs HF621Qbl3ilDuNe9zDt/4g8b2bSUZuw30VNTOaHxZZIkXCZ1G6sC0CSJk+cTUiHZVeXM 72rj/ZKwgcvhSlKkYz6s96R0KGRJ1MKNgdTxb0T/cO3Q0XML/YmwI8ct4Htok4Df2Ded 0ST/kzTllASH4rCbj5cOzABgYqLOhvpgGR2My/tVRYxMoEJUgFjceEIKIlYQqIbF96Jw 175VKDPkz9zW8oK8+oxKnodMmUqFi1dq/2o2B/c4Hqd85mjI45A4W9e1SH5EmBdZbwap 926g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Q05dSOPfwm/UowrXt9AMeUMp3mZmjRCMcDVjG5ZA724=; b=tFfN4YSmsdO6FQdNyUCmvFBLgRy7f11uwC6u/TxMWvkpF9cRGce06O82z7qGggLYB/ 8OQgaHuzhsMGqlEkfq9HBq7K5aql83SjtLmGyGRG5Sj532eRmcqTcVX9nIFjb/rQ4VtP 2w9VKu1QP4752qQnE7NO7LAwTT0xXLS8zs4uh+EgPi3/pgoF9vhuWFPWpltgaTDzlvho PkFvoSdeKZ8UW6QEy832QMzkaTvJmfFTYvpsUFXsv5NVEiXIbvi3PEaF1Htxu+DLOvHU wIskRa3ToPKeROFqugxu6K84Ff4d+9WXGYXObG1X53PG9p0irsO8rGsEqfcEvclivQBz qfZw== X-Gm-Message-State: AGRZ1gIxP+zrt64PFPbh7YueHwP3A5AXAokdeCjoIHWeE+swD1J2rvkE 7LRmGEv1fbPsrUP4DFZGPvv3ZVyl2To= X-Google-Smtp-Source: AJdET5dD0eA5AJlugFKS6rUc7yreRdJuV2LCZCeHZocHlVpxudSJPTCWefQyvXPJSc8a/aKoDik/Qg== X-Received: by 2002:a6b:8ec9:: with SMTP id q192-v6mr8719024iod.248.1540831091352; Mon, 29 Oct 2018 09:38:11 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.38.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:38:09 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 13/14] block: add REQ_HIPRI and inherit it from IOCB_HIPRI Date: Mon, 29 Oct 2018 10:37:37 -0600 Message-Id: <20181029163738.10172-14-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We use IOCB_HIPRI to poll for IO in the caller instead of scheduling. This information is not available for (or after) IO submission. The driver may make different queue choices based on the type of IO, so make the fact that we will poll for this IO known to the lower layers as well. Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- fs/block_dev.c | 2 ++ fs/direct-io.c | 2 ++ fs/iomap.c | 9 ++++++++- include/linux/blk_types.h | 4 +++- 4 files changed, 15 insertions(+), 2 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 38b8ce05cbc7..8bb8090c57a7 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -232,6 +232,8 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter, bio.bi_opf = dio_bio_write_op(iocb); task_io_account_write(ret); } + if (iocb->ki_flags & IOCB_HIPRI) + bio.bi_opf |= REQ_HIPRI; qc = submit_bio(&bio); for (;;) { diff --git a/fs/direct-io.c b/fs/direct-io.c index 093fb54cd316..ffb46b7aa5f7 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -1265,6 +1265,8 @@ do_blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, } else { dio->op = REQ_OP_READ; } + if (iocb->ki_flags & IOCB_HIPRI) + dio->op_flags |= REQ_HIPRI; /* * For AIO O_(D)SYNC writes we need to defer completions to a workqueue diff --git a/fs/iomap.c b/fs/iomap.c index ec15cf2ec696..50ad8c8d1dcb 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1554,6 +1554,7 @@ iomap_dio_zero(struct iomap_dio *dio, struct iomap *iomap, loff_t pos, unsigned len) { struct page *page = ZERO_PAGE(0); + int flags = REQ_SYNC | REQ_IDLE; struct bio *bio; bio = bio_alloc(GFP_KERNEL, 1); @@ -1562,9 +1563,12 @@ iomap_dio_zero(struct iomap_dio *dio, struct iomap *iomap, loff_t pos, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; + if (dio->iocb->ki_flags & IOCB_HIPRI) + flags |= REQ_HIPRI; + get_page(page); __bio_add_page(bio, page, len, 0); - bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_SYNC | REQ_IDLE); + bio_set_op_attrs(bio, REQ_OP_WRITE, flags); atomic_inc(&dio->ref); return submit_bio(bio); @@ -1663,6 +1667,9 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, bio_set_pages_dirty(bio); } + if (dio->iocb->ki_flags & IOCB_HIPRI) + bio->bi_opf |= REQ_HIPRI; + iov_iter_advance(dio->submit.iter, n); dio->size += n; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 093a818c5b68..d6c2558d6b73 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -322,6 +322,8 @@ enum req_flag_bits { /* command specific flags for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ + __REQ_HIPRI, + /* for driver use */ __REQ_DRV, __REQ_SWAP, /* swapping request. */ @@ -342,8 +344,8 @@ enum req_flag_bits { #define REQ_RAHEAD (1ULL << __REQ_RAHEAD) #define REQ_BACKGROUND (1ULL << __REQ_BACKGROUND) #define REQ_NOWAIT (1ULL << __REQ_NOWAIT) - #define REQ_NOUNMAP (1ULL << __REQ_NOUNMAP) +#define REQ_HIPRI (1ULL << __REQ_HIPRI) #define REQ_DRV (1ULL << __REQ_DRV) #define REQ_SWAP (1ULL << __REQ_SWAP) From patchwork Mon Oct 29 16:37:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 10659719 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A46FF13B5 for ; Mon, 29 Oct 2018 16:38:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8940D293D5 for ; Mon, 29 Oct 2018 16:38:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7CF53293E3; Mon, 29 Oct 2018 16:38:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C6111293D5 for ; Mon, 29 Oct 2018 16:38:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728395AbeJ3B1g (ORCPT ); Mon, 29 Oct 2018 21:27:36 -0400 Received: from mail-io1-f65.google.com ([209.85.166.65]:45355 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728394AbeJ3B1g (ORCPT ); Mon, 29 Oct 2018 21:27:36 -0400 Received: by mail-io1-f65.google.com with SMTP id p83-v6so5351938iod.12 for ; Mon, 29 Oct 2018 09:38:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AUS9BOQg07ils7vChzxqsumgK0NWQMbJPJfbvlLn1gk=; b=0JHuNA87KYzh6lRQ2LOHX3wG3VqdlV9EzpTDtSBUhoste0/Mxr9KvlF292x2gnkIZx hv7xvMzJRtrnYnquPb3ov1fC4hbzJ3DqUjnx53d/AnYJM8XDtqtzIJt+4xWHV4JP3zN9 w3rTAZONd5uFYdRyLLZqCoB7dSVHgTqimL9NN1UFoPzf0d+n61Qz7Y+vEBIkZfqGHWWB vFFMPQXX++UndcgeKgafwSoEV1uMdd3KHJxECo/gG1G1G/YC5y55uQlJK8qhbz0OMzON XWkexd85WNPczSbsi3+76Aq5Y/dWq7jxbWnnjY6kfPAQjdSyVdTKVlDwNPhe0YrnvXlP lrLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AUS9BOQg07ils7vChzxqsumgK0NWQMbJPJfbvlLn1gk=; b=AUW1tOsVllppeDfj6W67whGl0WQVIGYzxhHSIQcVd6toGwpGMGTC7iXc+uDuz5dP3b WRrQv7DeQDoNbbGNQ/HCtrnHqHwIOzVLh5u/uCiFtjBMF6MlkYAnAhegZPNt5bZLmBsn vqRPJXE5c64Af32bEgwHZ7jU9o5MfM7HVMN3p1aSeu/Me+aJ2vFa0HLBceEJu7eZUjc8 SG6ecBUQuER+bxsq3eo1abfj7AfE2DaNvpDRNQzHzVhW+l5mZ3MKMsd+tYWlSXBBp+d2 sLVDnK4OwdZEXBk35gxog9/IgrFMYbGrQksSZS47MbyEyBF1Sln8/EuMAG/+y4Z02ndZ udMg== X-Gm-Message-State: AGRZ1gLg5eP7Zh2nvbAGgKMsManJAqIL0K5X01KZV3BiZPiMZqQRVkV9 j79Uq9Rt1ODzsoMGOa7ZgLCcJz+Ap0w= X-Google-Smtp-Source: AJdET5eea/KHqFKX5AfZFCdCqgzjkmuE+ffPIM+OCd5TjrvCaSG4eJL4pjrM1Ud/EoB8XhJC7eb6RQ== X-Received: by 2002:a6b:3c08:: with SMTP id k8-v6mr8304024iob.219.1540831094017; Mon, 29 Oct 2018 09:38:14 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id q15-v6sm3367019itc.38.2018.10.29.09.38.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 09:38:12 -0700 (PDT) From: Jens Axboe To: linux-block@vger.kernel.org, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 14/14] nvme: add separate poll queue map Date: Mon, 29 Oct 2018 10:37:38 -0600 Message-Id: <20181029163738.10172-15-axboe@kernel.dk> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181029163738.10172-1-axboe@kernel.dk> References: <20181029163738.10172-1-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Adds support for defining a variable number of poll queues, currently configurable with the 'poll_queues' module parameter. Defaults to a single poll queue. And now we finally have poll support without triggering interrupts! Reviewed-by: Hannes Reinecke Signed-off-by: Jens Axboe --- drivers/nvme/host/pci.c | 103 +++++++++++++++++++++++++++++++++------- include/linux/blk-mq.h | 2 +- 2 files changed, 88 insertions(+), 17 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 658c9a2f4114..cce5d06f11c5 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -86,6 +86,10 @@ MODULE_PARM_DESC(write_queues, "Number of queues to use for writes. If not set, reads and writes " "will share a queue set."); +static int poll_queues = 1; +module_param_cb(poll_queues, &queue_count_ops, &poll_queues, 0644); +MODULE_PARM_DESC(poll_queues, "Number of queues to use for polled IO."); + struct nvme_dev; struct nvme_queue; @@ -94,6 +98,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown); enum { NVMEQ_TYPE_READ, NVMEQ_TYPE_WRITE, + NVMEQ_TYPE_POLL, NVMEQ_TYPE_NR, }; @@ -202,6 +207,7 @@ struct nvme_queue { u16 last_cq_head; u16 qid; u8 cq_phase; + u8 polled; u32 *dbbuf_sq_db; u32 *dbbuf_cq_db; u32 *dbbuf_sq_ei; @@ -250,7 +256,7 @@ static inline void _nvme_check_size(void) static unsigned int max_io_queues(void) { - return num_possible_cpus() + write_queues; + return num_possible_cpus() + write_queues + poll_queues; } static unsigned int max_queue_count(void) @@ -500,8 +506,15 @@ static int nvme_pci_map_queues(struct blk_mq_tag_set *set) offset = queue_irq_offset(dev); } + /* + * The poll queue(s) doesn't have an IRQ (and hence IRQ + * affinity), so use the regular blk-mq cpu mapping + */ map->queue_offset = qoff; - blk_mq_pci_map_queues(map, to_pci_dev(dev->dev), offset); + if (i != NVMEQ_TYPE_POLL) + blk_mq_pci_map_queues(map, to_pci_dev(dev->dev), offset); + else + blk_mq_map_queues(map); qoff += map->nr_queues; offset += map->nr_queues; } @@ -892,7 +905,7 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, * We should not need to do this, but we're still using this to * ensure we can drain requests on a dying queue. */ - if (unlikely(nvmeq->cq_vector < 0)) + if (unlikely(nvmeq->cq_vector < 0 && !nvmeq->polled)) return BLK_STS_IOERR; ret = nvme_setup_cmd(ns, req, &cmnd); @@ -921,6 +934,8 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, static int nvme_flags_to_type(struct request_queue *q, unsigned int flags) { + if (flags & REQ_HIPRI) + return NVMEQ_TYPE_POLL; if ((flags & REQ_OP_MASK) == REQ_OP_READ) return NVMEQ_TYPE_READ; @@ -1094,7 +1109,10 @@ static int adapter_alloc_cq(struct nvme_dev *dev, u16 qid, struct nvme_queue *nvmeq, s16 vector) { struct nvme_command c; - int flags = NVME_QUEUE_PHYS_CONTIG | NVME_CQ_IRQ_ENABLED; + int flags = NVME_QUEUE_PHYS_CONTIG; + + if (vector != -1) + flags |= NVME_CQ_IRQ_ENABLED; /* * Note: we (ab)use the fact that the prp fields survive if no data @@ -1106,7 +1124,10 @@ static int adapter_alloc_cq(struct nvme_dev *dev, u16 qid, c.create_cq.cqid = cpu_to_le16(qid); c.create_cq.qsize = cpu_to_le16(nvmeq->q_depth - 1); c.create_cq.cq_flags = cpu_to_le16(flags); - c.create_cq.irq_vector = cpu_to_le16(vector); + if (vector != -1) + c.create_cq.irq_vector = cpu_to_le16(vector); + else + c.create_cq.irq_vector = 0; return nvme_submit_sync_cmd(dev->ctrl.admin_q, &c, NULL, 0); } @@ -1348,13 +1369,14 @@ static int nvme_suspend_queue(struct nvme_queue *nvmeq) int vector; spin_lock_irq(&nvmeq->cq_lock); - if (nvmeq->cq_vector == -1) { + if (nvmeq->cq_vector == -1 && !nvmeq->polled) { spin_unlock_irq(&nvmeq->cq_lock); return 1; } vector = nvmeq->cq_vector; nvmeq->dev->online_queues--; nvmeq->cq_vector = -1; + nvmeq->polled = false; spin_unlock_irq(&nvmeq->cq_lock); /* @@ -1366,7 +1388,8 @@ static int nvme_suspend_queue(struct nvme_queue *nvmeq) if (!nvmeq->qid && nvmeq->dev->ctrl.admin_q) blk_mq_quiesce_queue(nvmeq->dev->ctrl.admin_q); - pci_free_irq(to_pci_dev(nvmeq->dev->dev), vector, nvmeq); + if (vector != -1) + pci_free_irq(to_pci_dev(nvmeq->dev->dev), vector, nvmeq); return 0; } @@ -1500,7 +1523,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid) spin_unlock_irq(&nvmeq->cq_lock); } -static int nvme_create_queue(struct nvme_queue *nvmeq, int qid) +static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled) { struct nvme_dev *dev = nvmeq->dev; int result; @@ -1510,7 +1533,11 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid) * A queue's vector matches the queue identifier unless the controller * has only one vector available. */ - vector = dev->num_vecs == 1 ? 0 : qid; + if (!polled) + vector = dev->num_vecs == 1 ? 0 : qid; + else + vector = -1; + result = adapter_alloc_cq(dev, qid, nvmeq, vector); if (result) return result; @@ -1527,15 +1554,20 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid) * xxx' warning if the create CQ/SQ command times out. */ nvmeq->cq_vector = vector; + nvmeq->polled = polled; nvme_init_queue(nvmeq, qid); - result = queue_request_irq(nvmeq); - if (result < 0) - goto release_sq; + + if (vector != -1) { + result = queue_request_irq(nvmeq); + if (result < 0) + goto release_sq; + } return result; release_sq: nvmeq->cq_vector = -1; + nvmeq->polled = false; dev->online_queues--; adapter_delete_sq(dev, qid); release_cq: @@ -1686,7 +1718,7 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev) static int nvme_create_io_queues(struct nvme_dev *dev) { - unsigned i, max; + unsigned i, max, rw_queues; int ret = 0; for (i = dev->ctrl.queue_count; i <= dev->max_qid; i++) { @@ -1697,8 +1729,17 @@ static int nvme_create_io_queues(struct nvme_dev *dev) } max = min(dev->max_qid, dev->ctrl.queue_count - 1); + if (max != 1 && dev->io_queues[NVMEQ_TYPE_POLL]) { + rw_queues = dev->io_queues[NVMEQ_TYPE_READ] + + dev->io_queues[NVMEQ_TYPE_WRITE]; + } else { + rw_queues = max; + } + for (i = dev->online_queues; i <= max; i++) { - ret = nvme_create_queue(&dev->queues[i], i); + bool polled = i > rw_queues; + + ret = nvme_create_queue(&dev->queues[i], i, polled); if (ret) break; } @@ -1970,6 +2011,7 @@ static int nvme_setup_host_mem(struct nvme_dev *dev) static void nvme_calc_io_queues(struct nvme_dev *dev, unsigned int nr_io_queues) { unsigned int this_w_queues = write_queues; + unsigned int this_p_queues = poll_queues; /* * Setup read/write queue split @@ -1977,9 +2019,28 @@ static void nvme_calc_io_queues(struct nvme_dev *dev, unsigned int nr_io_queues) if (nr_io_queues == 1) { dev->io_queues[NVMEQ_TYPE_READ] = 1; dev->io_queues[NVMEQ_TYPE_WRITE] = 0; + dev->io_queues[NVMEQ_TYPE_POLL] = 0; return; } + /* + * Configure number of poll queues, if set + */ + if (this_p_queues) { + /* + * We need at least one queue left. With just one queue, we'll + * have a single shared read/write set. + */ + if (this_p_queues >= nr_io_queues) { + this_w_queues = 0; + this_p_queues = nr_io_queues - 1; + } + + dev->io_queues[NVMEQ_TYPE_POLL] = this_p_queues; + nr_io_queues -= this_p_queues; + } else + dev->io_queues[NVMEQ_TYPE_POLL] = 0; + /* * If 'write_queues' is set, ensure it leaves room for at least * one read queue @@ -2049,19 +2110,29 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) if (!irq_sets[1]) affd.nr_sets = 1; + /* + * Need IRQs for read+write queues, and one for the admin queue + */ + nr_io_queues = irq_sets[0] + irq_sets[1] + 1; + /* * If we enable msix early due to not intx, disable it again before * setting up the full range we need. */ pci_free_irq_vectors(pdev); - result = pci_alloc_irq_vectors_affinity(pdev, 1, nr_io_queues + 1, + result = pci_alloc_irq_vectors_affinity(pdev, 1, nr_io_queues, PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY, &affd); if (result <= 0) return -EIO; dev->num_vecs = result; - dev->max_qid = max(result - 1, 1); + result = max(result - 1, 1); + dev->max_qid = result + dev->io_queues[NVMEQ_TYPE_POLL]; nvme_calc_io_queues(dev, dev->max_qid); + dev_info(dev->ctrl.device, "%d/%d/%d r/w/p queues\n", + dev->io_queues[NVMEQ_TYPE_READ], + dev->io_queues[NVMEQ_TYPE_WRITE], + dev->io_queues[NVMEQ_TYPE_POLL]); /* * Should investigate if there's a performance win from allocating diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index b5ae2b5677c1..6ee1d19c6dec 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -82,7 +82,7 @@ struct blk_mq_queue_map { }; enum { - HCTX_MAX_TYPES = 2, + HCTX_MAX_TYPES = 3, }; struct blk_mq_tag_set {