From patchwork Tue Dec 11 10:49:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10723525 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7DF1C91E for ; Tue, 11 Dec 2018 10:50:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E9752A04C for ; Tue, 11 Dec 2018 10:50:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 62F6A2A0A8; Tue, 11 Dec 2018 10:50:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 54D702A04C for ; Tue, 11 Dec 2018 10:50:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726406AbeLKKuE (ORCPT ); Tue, 11 Dec 2018 05:50:04 -0500 Received: from bombadil.infradead.org ([198.137.202.133]:58342 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726146AbeLKKuE (ORCPT ); Tue, 11 Dec 2018 05:50:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=x2fHuMXueCupvBf/7U16ncCO4y9PsHCbCPYk+KWCe5w=; b=NouLAFfhXzcpy12T6HuixZ1i0 vtFFRtM2ffDZPcksiUk6PE6mkLbSpE5VHm9NzAsZU5PHxSeSkBNhbTDm8LXhoxyGqf+8u3r/QmbRW D259OSVgh7ROWhkyzZjbQMffkYy0eEk263pSvRNBPOe11mKJqOsNjByNd3wmD+7k4sa4JtsWI3zQ3 mRN0Ax+KgcoUnsvpAsToPCZJ0cePPg93kmz3oSB7pjZWybvFpwuGXFKicHTsK5XHR21/Lpe9q+s0P HDl5eX4VjXv098I6JtAt50jM/T4pC8G9868C7TyWwLU8F/yz/rp1UTAgjSMndOpiFM51Rq/D5EJmu bFPdUDxjA==; Received: from [2601:647:4800:973f:7888:b13c:bff:87b0] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1gWfbd-0004p6-MZ; Tue, 11 Dec 2018 10:49:37 +0000 From: Sagi Grimberg To: linux-nvme@lists.infradead.org Cc: Christoph Hellwig , Keith Busch , linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 1/5] blk-mq-rdma: pass in queue map to blk_mq_rdma_map_queues Date: Tue, 11 Dec 2018 02:49:31 -0800 Message-Id: <20181211104936.25333-2-sagi@grimberg.me> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181211104936.25333-1-sagi@grimberg.me> References: <20181211104936.25333-1-sagi@grimberg.me> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Sagi Grimberg --- block/blk-mq-rdma.c | 8 ++++---- drivers/nvme/host/rdma.c | 2 +- include/linux/blk-mq-rdma.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/block/blk-mq-rdma.c b/block/blk-mq-rdma.c index a71576aff3a5..45030a81a1ed 100644 --- a/block/blk-mq-rdma.c +++ b/block/blk-mq-rdma.c @@ -29,24 +29,24 @@ * @set->nr_hw_queues, or @dev does not provide an affinity mask for a * vector, we fallback to the naive mapping. */ -int blk_mq_rdma_map_queues(struct blk_mq_tag_set *set, +int blk_mq_rdma_map_queues(struct blk_mq_queue_map *map, struct ib_device *dev, int first_vec) { const struct cpumask *mask; unsigned int queue, cpu; - for (queue = 0; queue < set->nr_hw_queues; queue++) { + for (queue = 0; queue < map->nr_queues; queue++) { mask = ib_get_vector_affinity(dev, first_vec + queue); if (!mask) goto fallback; for_each_cpu(cpu, mask) - set->map[0].mq_map[cpu] = queue; + map->mq_map[cpu] = map->queue_offset + queue; } return 0; fallback: - return blk_mq_map_queues(&set->map[0]); + return blk_mq_map_queues(map); } EXPORT_SYMBOL_GPL(blk_mq_rdma_map_queues); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index f2db848f6985..5057d5ab5aaa 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1751,7 +1751,7 @@ static int nvme_rdma_map_queues(struct blk_mq_tag_set *set) { struct nvme_rdma_ctrl *ctrl = set->driver_data; - return blk_mq_rdma_map_queues(set, ctrl->device->dev, 0); + return blk_mq_rdma_map_queues(&set->map[0], ctrl->device->dev, 0); } static const struct blk_mq_ops nvme_rdma_mq_ops = { diff --git a/include/linux/blk-mq-rdma.h b/include/linux/blk-mq-rdma.h index b4ade198007d..7b6ecf9ac4c3 100644 --- a/include/linux/blk-mq-rdma.h +++ b/include/linux/blk-mq-rdma.h @@ -4,7 +4,7 @@ struct blk_mq_tag_set; struct ib_device; -int blk_mq_rdma_map_queues(struct blk_mq_tag_set *set, +int blk_mq_rdma_map_queues(struct blk_mq_queue_map *map, struct ib_device *dev, int first_vec); #endif /* _LINUX_BLK_MQ_RDMA_H */ From patchwork Tue Dec 11 10:49:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10723529 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5FE4C91E for ; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 505242A04C for ; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 44F852A076; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0666E2A051 for ; Tue, 11 Dec 2018 10:50:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726446AbeLKKuF (ORCPT ); Tue, 11 Dec 2018 05:50:05 -0500 Received: from bombadil.infradead.org ([198.137.202.133]:58348 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726437AbeLKKuF (ORCPT ); Tue, 11 Dec 2018 05:50:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=DZnGGEQEW+ELH9ziWAnOxRUmjXA/uLxYrXCAK1/zhFc=; b=TOXz0S1a1DFM+/Q/AFvaZ7yI5 +830dvZqvw7jZBOFM9PAOf9gjvvbvz4iwddc82uPejrfPUT2MBwN5DF5MyS2Wz8FkDOSjYuRDXlyf nuZZlfamdhDBoNuYPHOjWYWBW3/Sp071l1+Fgz4FjFhUxyhedVmbsvc1Gp0FQ2f9C46uY2aNc045f kn5FrCbXyA2/b+d0OWo98DxT0eOtyJ9qEBoYcbtSYx3t/+19EJtj8zEflJ8yWEYeFPoi3X+r/cuRG UyMMRSejOvRWJ/4i9kvom52plWYNviL9ifY/LQf/vr6WpadUrhuYME/2WMc8s8hibs594pgEAIQ3G KT6Aa7m3Q==; Received: from [2601:647:4800:973f:7888:b13c:bff:87b0] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1gWfbd-0004p6-RT; Tue, 11 Dec 2018 10:49:37 +0000 From: Sagi Grimberg To: linux-nvme@lists.infradead.org Cc: Christoph Hellwig , Keith Busch , linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 2/5] nvme-fabrics: add missing nvmf_ctrl_options documentation Date: Tue, 11 Dec 2018 02:49:32 -0800 Message-Id: <20181211104936.25333-3-sagi@grimberg.me> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181211104936.25333-1-sagi@grimberg.me> References: <20181211104936.25333-1-sagi@grimberg.me> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Sagi Grimberg Reviewed-by: Christoph Hellwig --- drivers/nvme/host/fabrics.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h index 524a02a67817..28dc916ef26b 100644 --- a/drivers/nvme/host/fabrics.h +++ b/drivers/nvme/host/fabrics.h @@ -88,6 +88,9 @@ enum { * @max_reconnects: maximum number of allowed reconnect attempts before removing * the controller, (-1) means reconnect forever, zero means remove * immediately; + * @disable_sqflow: disable controller sq flow control + * @hdr_digest: generate/verify header digest (TCP) + * @data_digest: generate/verify data digest (TCP) */ struct nvmf_ctrl_options { unsigned mask; From patchwork Tue Dec 11 10:49:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10723533 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0200A13AF for ; Tue, 11 Dec 2018 10:50:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E677A2A1DA for ; Tue, 11 Dec 2018 10:50:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DAD162A275; Tue, 11 Dec 2018 10:50:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9CB32A215 for ; Tue, 11 Dec 2018 10:50:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726349AbeLKKu2 (ORCPT ); Tue, 11 Dec 2018 05:50:28 -0500 Received: from bombadil.infradead.org ([198.137.202.133]:58418 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726242AbeLKKu2 (ORCPT ); Tue, 11 Dec 2018 05:50:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=NfkvWaje1JF4/C816yy7hq2c3C9hUwkFcxe82o7/7sE=; b=dOgcWRdG8vbmEPy3PyMbQgyVL mY8mvMFIwDPP0qrRcg8GV2qtKpbbFOCSrLRgZg29koS4pIW2kzUr8TVB2m78ABqPrwmABEhSSlORA VvJa9u0pdyNKcyDC8wra+ybcNuUfO7Hz0d7hqbnbN5ABRwnJp17j+iIWvdySjoU97QtyFZACTd8zo 6GsMqS8WnmS2SpsZPRUe0WfEqm/M9JuIjdLJKIQnIz68ZlspofeTYGN5Znws8G+khlap9wcL4/XEV kEnKmXwpkMbrhhes062E9DK/5M+OgBGZzAhKkZemmEFvDiPa56e4K41p9b8lmFUBJviSNW8XvKQOI 79zcEqJig==; Received: from [2601:647:4800:973f:7888:b13c:bff:87b0] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1gWfbe-0004p6-07; Tue, 11 Dec 2018 10:49:38 +0000 From: Sagi Grimberg To: linux-nvme@lists.infradead.org Cc: Christoph Hellwig , Keith Busch , linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 3/5] nvme-fabrics: allow user to set nr_write_queues for separate queue maps Date: Tue, 11 Dec 2018 02:49:33 -0800 Message-Id: <20181211104936.25333-4-sagi@grimberg.me> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181211104936.25333-1-sagi@grimberg.me> References: <20181211104936.25333-1-sagi@grimberg.me> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This argument will specify how many I/O queues will be connected in create_ctrl in addition to nr_io_queues. With this configuration, I/O that carries payload from the host to the target, will use the default hctx queue map, and I/O that involves target to host transfers will use the read hctx queue map. Signed-off-by: Sagi Grimberg Reviewed-by: Christoph Hellwig --- drivers/nvme/host/fabrics.c | 15 ++++++++++++++- drivers/nvme/host/fabrics.h | 3 +++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c index 9c62c6838b76..066c3a02e08b 100644 --- a/drivers/nvme/host/fabrics.c +++ b/drivers/nvme/host/fabrics.c @@ -616,6 +616,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_DISABLE_SQFLOW, "disable_sqflow" }, { NVMF_OPT_HDR_DIGEST, "hdr_digest" }, { NVMF_OPT_DATA_DIGEST, "data_digest" }, + { NVMF_OPT_NR_WRITE_QUEUES, "nr_write_queues=%d" }, { NVMF_OPT_ERR, NULL } }; @@ -837,6 +838,18 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, case NVMF_OPT_DATA_DIGEST: opts->data_digest = true; break; + case NVMF_OPT_NR_WRITE_QUEUES: + if (match_int(args, &token)) { + ret = -EINVAL; + goto out; + } + if (token <= 0) { + pr_err("Invalid nr_write_queues %d\n", token); + ret = -EINVAL; + goto out; + } + opts->nr_write_queues = token; + break; default: pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n", p); @@ -954,7 +967,7 @@ EXPORT_SYMBOL_GPL(nvmf_free_options); #define NVMF_ALLOWED_OPTS (NVMF_OPT_QUEUE_SIZE | NVMF_OPT_NR_IO_QUEUES | \ NVMF_OPT_KATO | NVMF_OPT_HOSTNQN | \ NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\ - NVMF_OPT_DISABLE_SQFLOW) + NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_NR_WRITE_QUEUES) static struct nvme_ctrl * nvmf_create_ctrl(struct device *dev, const char *buf, size_t count) diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h index 28dc916ef26b..81b8fd1c0c5d 100644 --- a/drivers/nvme/host/fabrics.h +++ b/drivers/nvme/host/fabrics.h @@ -61,6 +61,7 @@ enum { NVMF_OPT_DISABLE_SQFLOW = 1 << 14, NVMF_OPT_HDR_DIGEST = 1 << 15, NVMF_OPT_DATA_DIGEST = 1 << 16, + NVMF_OPT_NR_WRITE_QUEUES = 1 << 17, }; /** @@ -91,6 +92,7 @@ enum { * @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP) * @data_digest: generate/verify data digest (TCP) + * @nr_write_queues: number of queues for write I/O */ struct nvmf_ctrl_options { unsigned mask; @@ -110,6 +112,7 @@ struct nvmf_ctrl_options { bool disable_sqflow; bool hdr_digest; bool data_digest; + unsigned int nr_write_queues; }; /* From patchwork Tue Dec 11 10:49:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10723527 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 232151869 for ; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 111AC2A04C for ; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 05B262A0A8; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 881A12A04C for ; Tue, 11 Dec 2018 10:50:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726437AbeLKKuF (ORCPT ); Tue, 11 Dec 2018 05:50:05 -0500 Received: from bombadil.infradead.org ([198.137.202.133]:58356 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726439AbeLKKuF (ORCPT ); Tue, 11 Dec 2018 05:50:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BGXGAtLJ071y3udVrUEE3he2KIN4yUu10pMvRzrD4Ts=; b=CJ3tcibvD6f+YT4220UHw0Zdd 1Q1RUlYWAsIfB2RMAYhSLGSeWX+TRWkTDO2DEkcREj+krSVxZ2rK/LR8vNcqYlCHM7oImca5H4piL gafSIErmbGkdi6Vx/GaD54I2XmW+9kMX3P/GJtFSiG40eGbDur61v4QvE+iUsPXcPUYAoOe5UCHDt TXV01AYrwKVU6zUgizcec/htw4VIzIMtqfPZgDh57ggWh93t15tyim0Ac6Lz8b0DRvBtDE5mVaypF THB0S0c2B4gT2oJhyNmP6PoDbd3E+TOCeRdjPJn/qId57+tkMYHs/n05zwD1KQ5X0hbwwxtf322IU v8NPYpn8A==; Received: from [2601:647:4800:973f:7888:b13c:bff:87b0] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1gWfbe-0004p6-4k; Tue, 11 Dec 2018 10:49:38 +0000 From: Sagi Grimberg To: linux-nvme@lists.infradead.org Cc: Christoph Hellwig , Keith Busch , linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 4/5] nvme-tcp: support separate queue maps for read and write Date: Tue, 11 Dec 2018 02:49:34 -0800 Message-Id: <20181211104936.25333-5-sagi@grimberg.me> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181211104936.25333-1-sagi@grimberg.me> References: <20181211104936.25333-1-sagi@grimberg.me> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Sagi Grimberg --- drivers/nvme/host/tcp.c | 53 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 47 insertions(+), 6 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 15543358e245..5c0ba99fb105 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -1215,7 +1215,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; struct linger sol = { .l_onoff = 1, .l_linger = 0 }; - int ret, opt, rcv_pdu_size; + int ret, opt, rcv_pdu_size, n; queue->ctrl = ctrl; INIT_LIST_HEAD(&queue->send_list); @@ -1271,7 +1271,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, } queue->sock->sk->sk_allocation = GFP_ATOMIC; - queue->io_cpu = (qid == 0) ? 0 : qid - 1; + n = (qid ? qid - 1 : 0) % num_online_cpus(); + queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false); queue->request = NULL; queue->data_remaining = 0; queue->ddgst_remaining = 0; @@ -1433,6 +1434,7 @@ static struct blk_mq_tag_set *nvme_tcp_alloc_tagset(struct nvme_ctrl *nctrl, set->driver_data = ctrl; set->nr_hw_queues = nctrl->queue_count - 1; set->timeout = NVME_IO_TIMEOUT; + set->nr_maps = 2 /* default + read */; } ret = blk_mq_alloc_tag_set(set); @@ -1527,7 +1529,12 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) static unsigned int nvme_tcp_nr_io_queues(struct nvme_ctrl *ctrl) { - return min(ctrl->queue_count - 1, num_online_cpus()); + unsigned int nr_io_queues; + + nr_io_queues = min(ctrl->opts->nr_io_queues, num_online_cpus()); + nr_io_queues += min(ctrl->opts->nr_write_queues, num_online_cpus()); + + return nr_io_queues; } static int nvme_alloc_io_queues(struct nvme_ctrl *ctrl) @@ -2052,6 +2059,38 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, return BLK_STS_OK; } +static int nvme_tcp_map_queues(struct blk_mq_tag_set *set) +{ + struct nvme_tcp_ctrl *ctrl = set->driver_data; + struct blk_mq_queue_map *map; + + if (ctrl->ctrl.opts->nr_write_queues) { + /* separate read/write queues */ + map = &set->map[HCTX_TYPE_DEFAULT]; + map->queue_offset = 0; + map->nr_queues = ctrl->ctrl.opts->nr_write_queues; + blk_mq_map_queues(map); + + map = &set->map[HCTX_TYPE_READ]; + map->nr_queues = ctrl->ctrl.opts->nr_io_queues; + map->queue_offset = ctrl->ctrl.opts->nr_write_queues; + blk_mq_map_queues(map); + } else { + /* mixed read/write queues */ + map = &set->map[HCTX_TYPE_DEFAULT]; + map->queue_offset = 0; + map->nr_queues = ctrl->ctrl.opts->nr_io_queues; + blk_mq_map_queues(map); + + map = &set->map[HCTX_TYPE_READ]; + map->queue_offset = 0; + map->nr_queues = ctrl->ctrl.opts->nr_io_queues; + blk_mq_map_queues(map); + } + + return 0; +} + static struct blk_mq_ops nvme_tcp_mq_ops = { .queue_rq = nvme_tcp_queue_rq, .complete = nvme_complete_rq, @@ -2059,6 +2098,7 @@ static struct blk_mq_ops nvme_tcp_mq_ops = { .exit_request = nvme_tcp_exit_request, .init_hctx = nvme_tcp_init_hctx, .timeout = nvme_tcp_timeout, + .map_queues = nvme_tcp_map_queues, }; static struct blk_mq_ops nvme_tcp_admin_mq_ops = { @@ -2113,7 +2153,7 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, INIT_LIST_HEAD(&ctrl->list); ctrl->ctrl.opts = opts; - ctrl->ctrl.queue_count = opts->nr_io_queues + 1; /* +1 for admin queue */ + ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues + 1; ctrl->ctrl.sqsize = opts->queue_size - 1; ctrl->ctrl.kato = opts->kato; @@ -2155,7 +2195,7 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, goto out_free_ctrl; } - ctrl->queues = kcalloc(opts->nr_io_queues + 1, sizeof(*ctrl->queues), + ctrl->queues = kcalloc(ctrl->ctrl.queue_count, sizeof(*ctrl->queues), GFP_KERNEL); if (!ctrl->queues) { ret = -ENOMEM; @@ -2206,7 +2246,8 @@ static struct nvmf_transport_ops nvme_tcp_transport = { .required_opts = NVMF_OPT_TRADDR, .allowed_opts = NVMF_OPT_TRSVCID | NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO | - NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST, + NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST | + NVMF_OPT_NR_IO_QUEUES, .create_ctrl = nvme_tcp_create_ctrl, }; From patchwork Tue Dec 11 10:49:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10723535 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 638DE1869 for ; Tue, 11 Dec 2018 10:50:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 54BE02A1DA for ; Tue, 11 Dec 2018 10:50:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 491332A215; Tue, 11 Dec 2018 10:50:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA7B62A1E5 for ; Tue, 11 Dec 2018 10:50:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726242AbeLKKu3 (ORCPT ); Tue, 11 Dec 2018 05:50:29 -0500 Received: from bombadil.infradead.org ([198.137.202.133]:58432 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726467AbeLKKu3 (ORCPT ); Tue, 11 Dec 2018 05:50:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=LFkZyPvbqSYLLS6vBNRWZ3gVK017bJYsIXx6LHwoeTc=; b=qW/EbUwW6Ror9R6WTv6DffZ9f 3ZaXvX8R8BxLBI8dp4p9CPB+XgPfcSKRfbvENuhYzzo7rmPxDobz3pXLa7J68YW+OphGJV7Ur2zxu xJ3HAn6qC4d80AIhn+VdNcBESgRP3ViJ6QvSCj2mCgmw5k1p0O8lZQdZqaSF3jK2Eeg9NUx5LH2Ah eF5m8oZ+4vPq0AhS2xKeu5wvuYSBGMiF2WW3ACPHudY57gl6+IpTCIfF8EsRK80l5ZJpi++cDAyCf TW25GYG9EwFzGSptIV18sg5355rXUL+8CbzcoRlzBjkqzJj2tojAd4mZFLPfUIWdisTrb0Pcw5suS WwqDJZsvg==; Received: from [2601:647:4800:973f:7888:b13c:bff:87b0] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1gWfbe-0004p6-Au; Tue, 11 Dec 2018 10:49:38 +0000 From: Sagi Grimberg To: linux-nvme@lists.infradead.org Cc: Christoph Hellwig , Keith Busch , linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH 5/5] nvme-rdma: support read/write queue separation Date: Tue, 11 Dec 2018 02:49:35 -0800 Message-Id: <20181211104936.25333-6-sagi@grimberg.me> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181211104936.25333-1-sagi@grimberg.me> References: <20181211104936.25333-1-sagi@grimberg.me> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Sagi Grimberg --- drivers/nvme/host/rdma.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 5057d5ab5aaa..cfe823a491f2 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -645,6 +645,8 @@ static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) nr_io_queues = min_t(unsigned int, nr_io_queues, ibdev->num_comp_vectors); + nr_io_queues += min(opts->nr_write_queues, num_online_cpus()); + ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues); if (ret) return ret; @@ -714,6 +716,7 @@ static struct blk_mq_tag_set *nvme_rdma_alloc_tagset(struct nvme_ctrl *nctrl, set->driver_data = ctrl; set->nr_hw_queues = nctrl->queue_count - 1; set->timeout = NVME_IO_TIMEOUT; + set->nr_maps = 2 /* default + read */; } ret = blk_mq_alloc_tag_set(set); @@ -1750,8 +1753,37 @@ static void nvme_rdma_complete_rq(struct request *rq) static int nvme_rdma_map_queues(struct blk_mq_tag_set *set) { struct nvme_rdma_ctrl *ctrl = set->driver_data; + struct blk_mq_queue_map *map; + int offset = 0; + + if (ctrl->ctrl.opts->nr_write_queues) { + /* separate read/write queues */ + map = &set->map[HCTX_TYPE_DEFAULT]; + map->queue_offset = offset; + map->nr_queues = ctrl->ctrl.opts->nr_write_queues; + blk_mq_rdma_map_queues(map, ctrl->device->dev, 0); + offset += map->nr_queues; + + map = &set->map[HCTX_TYPE_READ]; + map->nr_queues = ctrl->ctrl.opts->nr_io_queues; + map->queue_offset = offset; + blk_mq_rdma_map_queues(map, ctrl->device->dev, offset); + offset += map->nr_queues; - return blk_mq_rdma_map_queues(&set->map[0], ctrl->device->dev, 0); + } else { + /* mixed read/write queues */ + map = &set->map[HCTX_TYPE_DEFAULT]; + map->queue_offset = 0; + map->nr_queues = ctrl->ctrl.opts->nr_io_queues; + blk_mq_rdma_map_queues(map, ctrl->device->dev, 0); + + map = &set->map[HCTX_TYPE_READ]; + map->queue_offset = 0; + map->nr_queues = ctrl->ctrl.opts->nr_io_queues; + blk_mq_rdma_map_queues(map, ctrl->device->dev, 0); + } + + return 0; } static const struct blk_mq_ops nvme_rdma_mq_ops = { @@ -1906,7 +1938,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, INIT_WORK(&ctrl->err_work, nvme_rdma_error_recovery_work); INIT_WORK(&ctrl->ctrl.reset_work, nvme_rdma_reset_ctrl_work); - ctrl->ctrl.queue_count = opts->nr_io_queues + 1; /* +1 for admin queue */ + ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues + 1; ctrl->ctrl.sqsize = opts->queue_size - 1; ctrl->ctrl.kato = opts->kato; @@ -1957,7 +1989,8 @@ static struct nvmf_transport_ops nvme_rdma_transport = { .module = THIS_MODULE, .required_opts = NVMF_OPT_TRADDR, .allowed_opts = NVMF_OPT_TRSVCID | NVMF_OPT_RECONNECT_DELAY | - NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO, + NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO | + NVMF_OPT_NR_IO_QUEUES, .create_ctrl = nvme_rdma_create_ctrl, }; From patchwork Tue Dec 11 10:49:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10723531 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4174C91E for ; Tue, 11 Dec 2018 10:50:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2D5E2A04C for ; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B73AC2A051; Tue, 11 Dec 2018 10:50:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A06F12A075 for ; Tue, 11 Dec 2018 10:50:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726157AbeLKKuE (ORCPT ); Tue, 11 Dec 2018 05:50:04 -0500 Received: from bombadil.infradead.org ([198.137.202.133]:58322 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726242AbeLKKuE (ORCPT ); Tue, 11 Dec 2018 05:50:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=E7bYPCWGrph1P1/NkRkn2dznOK+J7KtiEK/gajlmlas=; b=AOYPNv6/A7uS6ho1QrsSE3hGr 4OjJsWqkFrho6y675DyilyN+nfc6t5EpEIbdudSdanu5wMofLXPvpRLqfZuvTqFnn67TMeY6qIBtd YXEe3JQLxqUYecIbzw32ecktAI1fbGD0PBsH1im6ftPaIMcO4HH+vTTzt9FIPSgfwdhn9G8QhqROi bxUxZMzbmM7RZ1CCPfmLhbboWegxGSELZPZOSaUfGsde8FqPj0eyJKhntPLt0YdonyfoyedjBz/dw aawI0QHCCI2z6iu8cgLkrh4YjfwIS20A8Fg4cpC4VU07ctrPwSwpiQMJHqJOeOw1hb4VZKxx329D+ UURT0fprQ==; Received: from [2601:647:4800:973f:7888:b13c:bff:87b0] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1gWfbe-0004p6-Fi; Tue, 11 Dec 2018 10:49:38 +0000 From: Sagi Grimberg To: linux-nvme@lists.infradead.org Cc: Christoph Hellwig , Keith Busch , linux-block@vger.kernel.org, Jens Axboe Subject: [PATCH nvme-cli 6/5] fabrics: pass in nr_write_queues Date: Tue, 11 Dec 2018 02:49:36 -0800 Message-Id: <20181211104936.25333-7-sagi@grimberg.me> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181211104936.25333-1-sagi@grimberg.me> References: <20181211104936.25333-1-sagi@grimberg.me> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP nr_write_queues specifies the number of queues additional to nr_io_queues will be connected. These queues will host write I/O (host to target payload) while nr_io_queues will host read I/O (target to host payload). Signed-off-by: Sagi Grimberg --- fabrics.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/fabrics.c b/fabrics.c index 93f3410a1c72..55ce2ba79c4f 100644 --- a/fabrics.c +++ b/fabrics.c @@ -53,6 +53,7 @@ static struct config { char *hostnqn; char *hostid; int nr_io_queues; + int nr_write_queues; int queue_size; int keep_alive_tmo; int reconnect_delay; @@ -621,6 +622,8 @@ static int build_options(char *argstr, int max_len) add_argument(&argstr, &max_len, "hostid", cfg.hostid)) || add_int_argument(&argstr, &max_len, "nr_io_queues", cfg.nr_io_queues) || + add_int_argument(&argstr, &max_len, "nr_write_queues", + cfg.nr_write_queues) || add_int_argument(&argstr, &max_len, "queue_size", cfg.queue_size) || add_int_argument(&argstr, &max_len, "keep_alive_tmo", cfg.keep_alive_tmo) || @@ -694,6 +697,13 @@ retry: p += len; } + if (cfg.nr_write_queues) { + len = sprintf(p, ",nr_write_queues=%d", cfg.nr_write_queues); + if (len < 0) + return -EINVAL; + p += len; + } + if (cfg.host_traddr) { len = sprintf(p, ",host_traddr=%s", cfg.host_traddr); if (len < 0) @@ -1009,6 +1019,7 @@ int connect(const char *desc, int argc, char **argv) {"hostnqn", 'q', "LIST", CFG_STRING, &cfg.hostnqn, required_argument, "user-defined hostnqn" }, {"hostid", 'I', "LIST", CFG_STRING, &cfg.hostid, required_argument, "user-defined hostid (if default not used)"}, {"nr-io-queues", 'i', "LIST", CFG_INT, &cfg.nr_io_queues, required_argument, "number of io queues to use (default is core count)" }, + {"nr_write_queues", 'W', "LIST", CFG_INT, &cfg.nr_write_queues, required_argument, "number of write queues to use (default 0)" }, {"queue-size", 'Q', "LIST", CFG_INT, &cfg.queue_size, required_argument, "number of io queue elements to use (default 128)" }, {"keep-alive-tmo", 'k', "LIST", CFG_INT, &cfg.keep_alive_tmo, required_argument, "keep alive timeout period in seconds" }, {"reconnect-delay", 'c', "LIST", CFG_INT, &cfg.reconnect_delay, required_argument, "reconnect timeout period in seconds" },