From patchwork Wed Mar 22 00:23:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13183410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0048C761A6 for ; Wed, 22 Mar 2023 00:24:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229459AbjCVAYK (ORCPT ); Tue, 21 Mar 2023 20:24:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229657AbjCVAYJ (ORCPT ); Tue, 21 Mar 2023 20:24:09 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E6E52FCE0 for ; Tue, 21 Mar 2023 17:24:08 -0700 (PDT) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32M01vwm016441 for ; Tue, 21 Mar 2023 17:24:08 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=J4Zpi1yqSm85ldGR5wpBzPETmvR1yC8F/Ig/J1L9XYY=; b=euAeIyuJWR9HFG1S4M7WA83K5xFtDPCi5iuLluB7qZyvRNdh9izTPOvcW+5du18JN7+v bWsjs8LVb153Q4VYJ8XabN4RSC6QIWpGhjxj2bq0T7fhW1G+HKqsqsu6upGQBkb0lx+p KIhh+eNWzQCVOK1UWpj6nsE1be5FQ6PIxOliuj9ETB0ud/hX1l3rmuAGBui2BwGk+FJE D3ub/STaPAJMQySYDCgXVSPEDG6FzfGWow/8JMBbyikm32hSU0a6uutPni/4z/NOweZQ /+LcfKb22tEyNdCR3ZyprCyMyZIYZe0Di3BT/ysVLTQjGRqvPPNmV4Dt3R4HDQ+x3Rsy hQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3pfgr8aqdb-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 21 Mar 2023 17:24:07 -0700 Received: from twshared4298.02.ash9.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Tue, 21 Mar 2023 17:24:06 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id DF30D1419697F; Tue, 21 Mar 2023 17:23:51 -0700 (PDT) From: Keith Busch To: , , , , CC: Keith Busch Subject: [PATCH 1/3] nvme-fabrics: add queue setup helpers Date: Tue, 21 Mar 2023 17:23:48 -0700 Message-ID: <20230322002350.4038048-2-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322002350.4038048-1-kbusch@meta.com> References: <20230322002350.4038048-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: sR82SuWav2FqGdKDNWK-RecePNwa4q_0 X-Proofpoint-GUID: sR82SuWav2FqGdKDNWK-RecePNwa4q_0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-21_11,2023-03-21_01,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch tcp and rdma transports have lots of duplicate code setting up the different queue mappings. Add common helpers. Signed-off-by: Keith Busch Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/host/fabrics.c | 95 +++++++++++++++++++++++++++++++++++++ drivers/nvme/host/fabrics.h | 5 ++ drivers/nvme/host/rdma.c | 81 ++----------------------------- drivers/nvme/host/tcp.c | 92 ++--------------------------------- 4 files changed, 109 insertions(+), 164 deletions(-) diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c index bbaa04a0c502b..9dcb56ea1cc00 100644 --- a/drivers/nvme/host/fabrics.c +++ b/drivers/nvme/host/fabrics.c @@ -4,6 +4,7 @@ * Copyright (c) 2015-2016 HGST, a Western Digital Company. */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include #include #include #include @@ -957,6 +958,100 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, return ret; } +unsigned int nvme_nr_io_queues(struct nvmf_ctrl_options *opts) +{ + unsigned int nr_io_queues; + + nr_io_queues = min(opts->nr_io_queues, num_online_cpus()); + nr_io_queues += min(opts->nr_write_queues, num_online_cpus()); + nr_io_queues += min(opts->nr_poll_queues, num_online_cpus()); + + return nr_io_queues; +} +EXPORT_SYMBOL_GPL(nvme_nr_io_queues); + +void nvme_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues, + u32 io_queues[HCTX_MAX_TYPES]) +{ + if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) { + /* + * separate read/write queues + * hand out dedicated default queues only after we have + * sufficient read queues. + */ + io_queues[HCTX_TYPE_READ] = opts->nr_io_queues; + nr_io_queues -= io_queues[HCTX_TYPE_READ]; + io_queues[HCTX_TYPE_DEFAULT] = + min(opts->nr_write_queues, nr_io_queues); + nr_io_queues -= io_queues[HCTX_TYPE_DEFAULT]; + } else { + /* + * shared read/write queues + * either no write queues were requested, or we don't have + * sufficient queue count to have dedicated default queues. + */ + io_queues[HCTX_TYPE_DEFAULT] = + min(opts->nr_io_queues, nr_io_queues); + nr_io_queues -= io_queues[HCTX_TYPE_DEFAULT]; + } + + if (opts->nr_poll_queues && nr_io_queues) { + /* map dedicated poll queues only if we have queues left */ + io_queues[HCTX_TYPE_POLL] = + min(opts->nr_poll_queues, nr_io_queues); + } +} +EXPORT_SYMBOL_GPL(nvme_set_io_queues); + +void nvme_map_queues(struct blk_mq_tag_set *set, struct nvme_ctrl *ctrl, + struct ib_device *dev, u32 io_queues[HCTX_MAX_TYPES]) +{ + struct nvmf_ctrl_options *opts = ctrl->opts; + + if (opts->nr_write_queues && io_queues[HCTX_TYPE_READ]) { + /* separate read/write queues */ + set->map[HCTX_TYPE_DEFAULT].nr_queues = + io_queues[HCTX_TYPE_DEFAULT]; + set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; + set->map[HCTX_TYPE_READ].nr_queues = + io_queues[HCTX_TYPE_READ]; + set->map[HCTX_TYPE_READ].queue_offset = + io_queues[HCTX_TYPE_DEFAULT]; + } else { + /* shared read/write queues */ + set->map[HCTX_TYPE_DEFAULT].nr_queues = + io_queues[HCTX_TYPE_DEFAULT]; + set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; + set->map[HCTX_TYPE_READ].nr_queues = + io_queues[HCTX_TYPE_DEFAULT]; + set->map[HCTX_TYPE_READ].queue_offset = 0; + } + + if (dev) { + blk_mq_rdma_map_queues(&set->map[HCTX_TYPE_DEFAULT], dev, 0); + blk_mq_rdma_map_queues(&set->map[HCTX_TYPE_READ], dev, 0); + } else { + blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]); + blk_mq_map_queues(&set->map[HCTX_TYPE_READ]); + } + + if (opts->nr_poll_queues && io_queues[HCTX_TYPE_POLL]) { + /* map dedicated poll queues only if we have queues left */ + set->map[HCTX_TYPE_POLL].nr_queues = io_queues[HCTX_TYPE_POLL]; + set->map[HCTX_TYPE_POLL].queue_offset = + io_queues[HCTX_TYPE_DEFAULT] + + io_queues[HCTX_TYPE_READ]; + blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]); + } + + dev_info(ctrl->device, + "mapped %d/%d/%d default/read/poll queues.\n", + io_queues[HCTX_TYPE_DEFAULT], + io_queues[HCTX_TYPE_READ], + io_queues[HCTX_TYPE_POLL]); +} +EXPORT_SYMBOL_GPL(nvme_map_queues); + static int nvmf_check_required_opts(struct nvmf_ctrl_options *opts, unsigned int required_opts) { diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h index dcac3df8a5f76..bad3e21ffb8ba 100644 --- a/drivers/nvme/host/fabrics.h +++ b/drivers/nvme/host/fabrics.h @@ -215,5 +215,10 @@ int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size); bool nvmf_should_reconnect(struct nvme_ctrl *ctrl); bool nvmf_ip_options_match(struct nvme_ctrl *ctrl, struct nvmf_ctrl_options *opts); +void nvme_set_io_queues(struct nvmf_ctrl_options *opts, u32 nr_io_queues, + u32 io_queues[HCTX_MAX_TYPES]); +unsigned int nvme_nr_io_queues(struct nvmf_ctrl_options *opts); +void nvme_map_queues(struct blk_mq_tag_set *set, struct nvme_ctrl *ctrl, + struct ib_device *dev, u32 io_queues[HCTX_MAX_TYPES]); #endif /* _NVME_FABRICS_H */ diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index bbad26b82b56d..cbec566db2761 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -718,18 +718,10 @@ static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl, static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) { struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; - struct ib_device *ibdev = ctrl->device->dev; - unsigned int nr_io_queues, nr_default_queues; - unsigned int nr_read_queues, nr_poll_queues; + unsigned int nr_io_queues; int i, ret; - nr_read_queues = min_t(unsigned int, ibdev->num_comp_vectors, - min(opts->nr_io_queues, num_online_cpus())); - nr_default_queues = min_t(unsigned int, ibdev->num_comp_vectors, - min(opts->nr_write_queues, num_online_cpus())); - nr_poll_queues = min(opts->nr_poll_queues, num_online_cpus()); - nr_io_queues = nr_read_queues + nr_default_queues + nr_poll_queues; - + nr_io_queues = nvme_nr_io_queues(opts); ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues); if (ret) return ret; @@ -744,34 +736,7 @@ static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) dev_info(ctrl->ctrl.device, "creating %d I/O queues.\n", nr_io_queues); - if (opts->nr_write_queues && nr_read_queues < nr_io_queues) { - /* - * separate read/write queues - * hand out dedicated default queues only after we have - * sufficient read queues. - */ - ctrl->io_queues[HCTX_TYPE_READ] = nr_read_queues; - nr_io_queues -= ctrl->io_queues[HCTX_TYPE_READ]; - ctrl->io_queues[HCTX_TYPE_DEFAULT] = - min(nr_default_queues, nr_io_queues); - nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT]; - } else { - /* - * shared read/write queues - * either no write queues were requested, or we don't have - * sufficient queue count to have dedicated default queues. - */ - ctrl->io_queues[HCTX_TYPE_DEFAULT] = - min(nr_read_queues, nr_io_queues); - nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT]; - } - - if (opts->nr_poll_queues && nr_io_queues) { - /* map dedicated poll queues only if we have queues left */ - ctrl->io_queues[HCTX_TYPE_POLL] = - min(nr_poll_queues, nr_io_queues); - } - + nvme_set_io_queues(opts, nr_io_queues, ctrl->io_queues); for (i = 1; i < ctrl->ctrl.queue_count; i++) { ret = nvme_rdma_alloc_queue(ctrl, i, ctrl->ctrl.sqsize + 1); @@ -2143,46 +2108,8 @@ static void nvme_rdma_complete_rq(struct request *rq) static void nvme_rdma_map_queues(struct blk_mq_tag_set *set) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(set->driver_data); - struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; - if (opts->nr_write_queues && ctrl->io_queues[HCTX_TYPE_READ]) { - /* separate read/write queues */ - set->map[HCTX_TYPE_DEFAULT].nr_queues = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; - set->map[HCTX_TYPE_READ].nr_queues = - ctrl->io_queues[HCTX_TYPE_READ]; - set->map[HCTX_TYPE_READ].queue_offset = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - } else { - /* shared read/write queues */ - set->map[HCTX_TYPE_DEFAULT].nr_queues = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; - set->map[HCTX_TYPE_READ].nr_queues = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - set->map[HCTX_TYPE_READ].queue_offset = 0; - } - blk_mq_rdma_map_queues(&set->map[HCTX_TYPE_DEFAULT], - ctrl->device->dev, 0); - blk_mq_rdma_map_queues(&set->map[HCTX_TYPE_READ], - ctrl->device->dev, 0); - - if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL]) { - /* map dedicated poll queues only if we have queues left */ - set->map[HCTX_TYPE_POLL].nr_queues = - ctrl->io_queues[HCTX_TYPE_POLL]; - set->map[HCTX_TYPE_POLL].queue_offset = - ctrl->io_queues[HCTX_TYPE_DEFAULT] + - ctrl->io_queues[HCTX_TYPE_READ]; - blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]); - } - - dev_info(ctrl->ctrl.device, - "mapped %d/%d/%d default/read/poll queues.\n", - ctrl->io_queues[HCTX_TYPE_DEFAULT], - ctrl->io_queues[HCTX_TYPE_READ], - ctrl->io_queues[HCTX_TYPE_POLL]); + nvme_map_queues(set, &ctrl->ctrl, ctrl->device->dev, ctrl->io_queues); } static const struct blk_mq_ops nvme_rdma_mq_ops = { diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 7723a49895244..96298f8794e1a 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -1781,58 +1781,12 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) return ret; } -static unsigned int nvme_tcp_nr_io_queues(struct nvme_ctrl *ctrl) -{ - unsigned int nr_io_queues; - - nr_io_queues = min(ctrl->opts->nr_io_queues, num_online_cpus()); - nr_io_queues += min(ctrl->opts->nr_write_queues, num_online_cpus()); - nr_io_queues += min(ctrl->opts->nr_poll_queues, num_online_cpus()); - - return nr_io_queues; -} - -static void nvme_tcp_set_io_queues(struct nvme_ctrl *nctrl, - unsigned int nr_io_queues) -{ - struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); - struct nvmf_ctrl_options *opts = nctrl->opts; - - if (opts->nr_write_queues && opts->nr_io_queues < nr_io_queues) { - /* - * separate read/write queues - * hand out dedicated default queues only after we have - * sufficient read queues. - */ - ctrl->io_queues[HCTX_TYPE_READ] = opts->nr_io_queues; - nr_io_queues -= ctrl->io_queues[HCTX_TYPE_READ]; - ctrl->io_queues[HCTX_TYPE_DEFAULT] = - min(opts->nr_write_queues, nr_io_queues); - nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT]; - } else { - /* - * shared read/write queues - * either no write queues were requested, or we don't have - * sufficient queue count to have dedicated default queues. - */ - ctrl->io_queues[HCTX_TYPE_DEFAULT] = - min(opts->nr_io_queues, nr_io_queues); - nr_io_queues -= ctrl->io_queues[HCTX_TYPE_DEFAULT]; - } - - if (opts->nr_poll_queues && nr_io_queues) { - /* map dedicated poll queues only if we have queues left */ - ctrl->io_queues[HCTX_TYPE_POLL] = - min(opts->nr_poll_queues, nr_io_queues); - } -} - static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) { unsigned int nr_io_queues; int ret; - nr_io_queues = nvme_tcp_nr_io_queues(ctrl); + nr_io_queues = nvme_nr_io_queues(ctrl->opts); ret = nvme_set_queue_count(ctrl, &nr_io_queues); if (ret) return ret; @@ -1847,8 +1801,8 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) dev_info(ctrl->device, "creating %d I/O queues.\n", nr_io_queues); - nvme_tcp_set_io_queues(ctrl, nr_io_queues); - + nvme_set_io_queues(ctrl->opts, nr_io_queues, + to_tcp_ctrl(ctrl)->io_queues); return __nvme_tcp_alloc_io_queues(ctrl); } @@ -2428,44 +2382,8 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, static void nvme_tcp_map_queues(struct blk_mq_tag_set *set) { struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(set->driver_data); - struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; - - if (opts->nr_write_queues && ctrl->io_queues[HCTX_TYPE_READ]) { - /* separate read/write queues */ - set->map[HCTX_TYPE_DEFAULT].nr_queues = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; - set->map[HCTX_TYPE_READ].nr_queues = - ctrl->io_queues[HCTX_TYPE_READ]; - set->map[HCTX_TYPE_READ].queue_offset = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - } else { - /* shared read/write queues */ - set->map[HCTX_TYPE_DEFAULT].nr_queues = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - set->map[HCTX_TYPE_DEFAULT].queue_offset = 0; - set->map[HCTX_TYPE_READ].nr_queues = - ctrl->io_queues[HCTX_TYPE_DEFAULT]; - set->map[HCTX_TYPE_READ].queue_offset = 0; - } - blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]); - blk_mq_map_queues(&set->map[HCTX_TYPE_READ]); - - if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL]) { - /* map dedicated poll queues only if we have queues left */ - set->map[HCTX_TYPE_POLL].nr_queues = - ctrl->io_queues[HCTX_TYPE_POLL]; - set->map[HCTX_TYPE_POLL].queue_offset = - ctrl->io_queues[HCTX_TYPE_DEFAULT] + - ctrl->io_queues[HCTX_TYPE_READ]; - blk_mq_map_queues(&set->map[HCTX_TYPE_POLL]); - } - - dev_info(ctrl->ctrl.device, - "mapped %d/%d/%d default/read/poll queues.\n", - ctrl->io_queues[HCTX_TYPE_DEFAULT], - ctrl->io_queues[HCTX_TYPE_READ], - ctrl->io_queues[HCTX_TYPE_POLL]); + + nvme_map_queues(set, &ctrl->ctrl, NULL, ctrl->io_queues); } static int nvme_tcp_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob) From patchwork Wed Mar 22 00:23:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13183409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E254C6FD1D for ; Wed, 22 Mar 2023 00:24:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229657AbjCVAYK (ORCPT ); Tue, 21 Mar 2023 20:24:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229694AbjCVAYJ (ORCPT ); Tue, 21 Mar 2023 20:24:09 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE5FD30E9E for ; Tue, 21 Mar 2023 17:24:08 -0700 (PDT) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32M01f2n007274 for ; Tue, 21 Mar 2023 17:24:08 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=4Ik0Oblobgnkp3JuFO1WlPrz7GWLxMCFomlHihR68iw=; b=F3j7ewUhyYGNJg0quBeURSO8RpR2K7nRN0evdAubAek7JrTV8eK4LLnYQAA5nLNyG4MY eggx3XcOeEApibBUUBhGePmF2j2FwUYup06Wp5KKfsNsnlsHdfmoP/DINdguBALLM5ff eV2Q++EYNFiyP4uEFjdz+XQiwdU88VQuz0y5pNTfkX/3P0hELdWfJSHR1cnc1jziA8ZO j93DN9mL0keLB74MWpSPvSqC5goCNdmuLYmDShkXofKhmHNgMcyb/Z4GH9JaUmfokrQl q7FJGZ6SS3du9VSNEeYDq+ToW0NjJRuqR7yOW1cii74tQb/C+hJnvcCKEJKg3SYe+me4 lQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3pfdx9v42d-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 21 Mar 2023 17:24:08 -0700 Received: from twshared4298.02.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Tue, 21 Mar 2023 17:24:06 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 3B06814196982; Tue, 21 Mar 2023 17:23:52 -0700 (PDT) From: Keith Busch To: , , , , CC: Keith Busch Subject: [PATCH 2/3] nvme: add polling options for loop target Date: Tue, 21 Mar 2023 17:23:49 -0700 Message-ID: <20230322002350.4038048-3-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322002350.4038048-1-kbusch@meta.com> References: <20230322002350.4038048-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: vqb5JE5JhrAmBFFNrIhvVsDvVZ6Rhtv0 X-Proofpoint-ORIG-GUID: vqb5JE5JhrAmBFFNrIhvVsDvVZ6Rhtv0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-21_11,2023-03-21_01,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch This is for mostly for testing purposes. Signed-off-by: Keith Busch Reviewed-by: Chaitanya Kulkarni --- drivers/nvme/target/loop.c | 63 +++++++++++++++++++++++++++++++++++--- 1 file changed, 58 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f8..0587ead60b09e 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -22,6 +22,7 @@ struct nvme_loop_iod { struct nvmet_req req; struct nvme_loop_queue *queue; struct work_struct work; + struct work_struct poll; struct sg_table sg_table; struct scatterlist first_sgl[]; }; @@ -37,6 +38,7 @@ struct nvme_loop_ctrl { struct nvme_ctrl ctrl; struct nvmet_port *port; + u32 io_queues[HCTX_MAX_TYPES]; }; static inline struct nvme_loop_ctrl *to_loop_ctrl(struct nvme_ctrl *ctrl) @@ -76,7 +78,11 @@ static void nvme_loop_complete_rq(struct request *req) struct nvme_loop_iod *iod = blk_mq_rq_to_pdu(req); sg_free_table_chained(&iod->sg_table, NVME_INLINE_SG_CNT); - nvme_complete_rq(req); + + if (req->mq_hctx->type != HCTX_TYPE_POLL || !in_interrupt()) + nvme_complete_rq(req); + else + queue_work(nvmet_wq, &iod->poll); } static struct blk_mq_tags *nvme_loop_tagset(struct nvme_loop_queue *queue) @@ -120,6 +126,15 @@ static void nvme_loop_queue_response(struct nvmet_req *req) } } +static void nvme_loop_poll_work(struct work_struct *work) +{ + struct nvme_loop_iod *iod = + container_of(work, struct nvme_loop_iod, poll); + struct request *req = blk_mq_rq_from_pdu(iod); + + nvme_complete_rq(req); +} + static void nvme_loop_execute_work(struct work_struct *work) { struct nvme_loop_iod *iod = @@ -170,6 +185,30 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx, return BLK_STS_OK; } +static bool nvme_loop_poll_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data) +{ + struct blk_mq_hw_ctx *hctx = data; + struct nvme_loop_iod *iod; + struct request *rq; + + rq = blk_mq_tag_to_rq(hctx->tags, bitnr); + if (!rq) + return true; + + iod = blk_mq_rq_to_pdu(rq); + flush_work(&iod->poll); + return true; +} + +static int nvme_loop_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob) +{ + struct blk_mq_tags *tags = hctx->tags; + struct sbitmap_queue *btags = &tags->bitmap_tags; + + sbitmap_for_each_set(&btags->sb, nvme_loop_poll_iter, hctx); + return 1; +} + static void nvme_loop_submit_async_event(struct nvme_ctrl *arg) { struct nvme_loop_ctrl *ctrl = to_loop_ctrl(arg); @@ -197,6 +236,7 @@ static int nvme_loop_init_iod(struct nvme_loop_ctrl *ctrl, iod->req.cqe = &iod->cqe; iod->queue = &ctrl->queues[queue_idx]; INIT_WORK(&iod->work, nvme_loop_execute_work); + INIT_WORK(&iod->poll, nvme_loop_poll_work); return 0; } @@ -247,11 +287,20 @@ static int nvme_loop_init_admin_hctx(struct blk_mq_hw_ctx *hctx, void *data, return 0; } +static void nvme_loop_map_queues(struct blk_mq_tag_set *set) +{ + struct nvme_loop_ctrl *ctrl = to_loop_ctrl(set->driver_data); + + nvme_map_queues(set, &ctrl->ctrl, NULL, ctrl->io_queues); +} + static const struct blk_mq_ops nvme_loop_mq_ops = { .queue_rq = nvme_loop_queue_rq, .complete = nvme_loop_complete_rq, .init_request = nvme_loop_init_request, .init_hctx = nvme_loop_init_hctx, + .map_queues = nvme_loop_map_queues, + .poll = nvme_loop_poll, }; static const struct blk_mq_ops nvme_loop_admin_mq_ops = { @@ -305,7 +354,7 @@ static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl) unsigned int nr_io_queues; int ret, i; - nr_io_queues = min(opts->nr_io_queues, num_online_cpus()); + nr_io_queues = nvme_nr_io_queues(opts); ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues); if (ret || !nr_io_queues) return ret; @@ -321,6 +370,7 @@ static int nvme_loop_init_io_queues(struct nvme_loop_ctrl *ctrl) ctrl->ctrl.queue_count++; } + nvme_set_io_queues(opts, nr_io_queues, ctrl->io_queues); return 0; out_destroy_queues: @@ -494,7 +544,7 @@ static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl) return ret; ret = nvme_alloc_io_tag_set(&ctrl->ctrl, &ctrl->tag_set, - &nvme_loop_mq_ops, 1, + &nvme_loop_mq_ops, ctrl->ctrl.opts->nr_poll_queues ? 3 : 2, sizeof(struct nvme_loop_iod) + NVME_INLINE_SG_CNT * sizeof(struct scatterlist)); if (ret) @@ -534,6 +584,7 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev, struct nvmf_ctrl_options *opts) { struct nvme_loop_ctrl *ctrl; + unsigned int nr_io_queues; int ret; ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL); @@ -559,7 +610,8 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev, ctrl->ctrl.kato = opts->kato; ctrl->port = nvme_loop_find_port(&ctrl->ctrl); - ctrl->queues = kcalloc(opts->nr_io_queues + 1, sizeof(*ctrl->queues), + nr_io_queues = nvme_nr_io_queues(ctrl->ctrl.opts);; + ctrl->queues = kcalloc(nr_io_queues + 1, sizeof(*ctrl->queues), GFP_KERNEL); if (!ctrl->queues) goto out_uninit_ctrl; @@ -648,7 +700,8 @@ static struct nvmf_transport_ops nvme_loop_transport = { .name = "loop", .module = THIS_MODULE, .create_ctrl = nvme_loop_create_ctrl, - .allowed_opts = NVMF_OPT_TRADDR, + .allowed_opts = NVMF_OPT_TRADDR | NVMF_OPT_NR_WRITE_QUEUES | + NVMF_OPT_NR_POLL_QUEUES, }; static int __init nvme_loop_init_module(void) From patchwork Wed Mar 22 00:23:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13183408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3046C76195 for ; Wed, 22 Mar 2023 00:24:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229713AbjCVAYJ (ORCPT ); Tue, 21 Mar 2023 20:24:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229459AbjCVAYJ (ORCPT ); Tue, 21 Mar 2023 20:24:09 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B9A0311FC for ; Tue, 21 Mar 2023 17:24:08 -0700 (PDT) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32M01qAX007380 for ; Tue, 21 Mar 2023 17:24:08 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=s2048-2021-q4; bh=FYTiwteue9O3GyNlt9lJQtg5O7g1VL7645gn3rkoBdg=; b=Femxjp8ZBf+AP1MKe5ApM5+TqruQRtaXRjyg/fFfn3Qw3C8Agg4c9AiYRtEqMIerDVvk 6pK8v8VGM8zbWOrOv1uqdCYMyCIl3KkmZAfnavkbb1bFaGefWTDnsC9G7+OwXgS2BuXZ f42mmI9Ha7ZzCXn81k/eMT0ONFBkJEl5cTtB0MKrM2OXRA0AM1p/F6meXKDy/3heRRmO bWQpfZzmFa27dNnQhm9dv51Be6HUsh7eG6nKaNZ+2yf1jEEZQLNR+8FZLeZWu/zmfdhZ NmcRCNzg2ealrRqfq/JJgDtED2BhGOIK/Nwu9qH7B408W3mtrOq9Og1bNNblEMmoaCG6 jQ== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3pfdx9v42c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 21 Mar 2023 17:24:08 -0700 Received: from twshared4298.02.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Tue, 21 Mar 2023 17:24:06 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 80A5214196985; Tue, 21 Mar 2023 17:23:52 -0700 (PDT) From: Keith Busch To: , , , , CC: Keith Busch , Martin Belanger , Daniel Wagner Subject: [PATCH 3/3] blk-mq: directly poll requests Date: Tue, 21 Mar 2023 17:23:50 -0700 Message-ID: <20230322002350.4038048-4-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322002350.4038048-1-kbusch@meta.com> References: <20230322002350.4038048-1-kbusch@meta.com> X-FB-Internal: Safe X-Proofpoint-GUID: Kgexc-yDTK7n-WIVtlJWoj1U1ijCtI_n X-Proofpoint-ORIG-GUID: Kgexc-yDTK7n-WIVtlJWoj1U1ijCtI_n X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-21_11,2023-03-21_01,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Polling needs a bio with a valid bi_bdev, but neither of those are guaranteed for polled driver requests. Make request based polling use directly use blk-mq's polling function. When executing a request from a polled hctx, we know the request's cookie, and that it's from a live multi-queue that supports polling, so we can safely skip everything that bio_poll provides. Link: http://lists.infradead.org/pipermail/linux-nvme/2023-March/038340.html Reported-by: Martin Belanger Reported-by: Daniel Wagner Signed-off-by: Keith Busch Reviewed-by: Sagi Grimberg Reviewed-by: Christoph Hellwig Tested-by: Daniel Wagner Reviewed-by: Chaitanya Kulkarni Tested-by: Shin'ichiro Kawasaki --- block/blk-mq.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 4e30459df8151..932d2e95392e6 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1334,8 +1334,6 @@ bool blk_rq_is_poll(struct request *rq) return false; if (rq->mq_hctx->type != HCTX_TYPE_POLL) return false; - if (WARN_ON_ONCE(!rq->bio)) - return false; return true; } EXPORT_SYMBOL_GPL(blk_rq_is_poll); @@ -1343,7 +1341,7 @@ EXPORT_SYMBOL_GPL(blk_rq_is_poll); static void blk_rq_poll_completion(struct request *rq, struct completion *wait) { do { - bio_poll(rq->bio, NULL, 0); + blk_mq_poll(rq->q, blk_rq_to_qc(rq), NULL, 0); cond_resched(); } while (!completion_done(wait)); }