From patchwork Mon Jul 16 06:51:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10525577 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 374EF6020A for ; Mon, 16 Jul 2018 06:51:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1578328ABF for ; Mon, 16 Jul 2018 06:51:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1319228AC1; Mon, 16 Jul 2018 06:51:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97C6528ABC for ; Mon, 16 Jul 2018 06:51:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727682AbeGPHRC (ORCPT ); Mon, 16 Jul 2018 03:17:02 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:46111 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727649AbeGPHRC (ORCPT ); Mon, 16 Jul 2018 03:17:02 -0400 Received: by mail-wr1-f65.google.com with SMTP id s11-v6so30640299wra.13 for ; Sun, 15 Jul 2018 23:51:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=hjpM2nHc8QIVQ261uCg+TboWerZxv+ZyFHG6o1TSu4U=; b=CgvLo6r/xd5oOOVV0wyWBChnp3OAZJ0gqu8HTw/+KEzdJ0Vdmtdt1HQr0IT4rnTpYi ehRd4ICnUdXFQ3txuQBmDqo1YN1WG6364JbnJkGkjx5yvvey8By59KeyvkMTZZeS8HO5 OHBXmk2Nhrn7dlzyaczcK5CT6laRzZDTnm+S3Y3bdsCTlM6//QLEXiYM+E72Y9+DDSZ6 7QCpkxqIvUBfYIiIW+jQyFQYLvW9j/O24+qNaHzH17fHyJISiujuPVoUxnqjGbl6d1pW pWGTjLj2UlqWrxf0bJVaRE62BbR3FAz6CPjdINQ3Gh1lDBNuegRfv7bVRKEVLMF5NZKO ZVLQ== X-Gm-Message-State: AOUpUlEmpt9T5HIilRjapjb5WmOnYmnm2FVTdQcegqUH6TI5RKJotsJa X5AwI3yLv/l/iDbZVHGlxNw= X-Google-Smtp-Source: AAOMgpeBhKOPc/QdDqCzp8JvfN3tbr6zmNOEhtGxQ3/RYnai15RDWW7qE3ABudGm61TwPdMsvJ8BoQ== X-Received: by 2002:adf:e992:: with SMTP id h18-v6mr4776252wrm.238.1531723869506; Sun, 15 Jul 2018 23:51:09 -0700 (PDT) Received: from [192.168.64.169] (bzq-219-42-90.isdn.bezeqint.net. [62.219.42.90]) by smtp.gmail.com with ESMTPSA id t9-v6sm5008175wra.62.2018.07.15.23.51.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 15 Jul 2018 23:51:08 -0700 (PDT) Subject: Re: nvme-rdma and rdma comp vector affinity problem To: Steve Wise Cc: RDMA mailing list , "hch@lst.de" , "linux-nvme@lists.infradead.org" References: <984d4c8e-16b9-d459-e4b9-bf1703b4c226@opengridcomputing.com> From: Sagi Grimberg Message-ID: Date: Mon, 16 Jul 2018 09:51:07 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <984d4c8e-16b9-d459-e4b9-bf1703b4c226@opengridcomputing.com> Content-Language: en-US Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP > Hey Sagi and Christoph, > > Do you all have any thoughts on this?  It seems like a bug in nvme-rdma > or the blk-mq code.   I can debug it further, if we agree this does look > like a bug... It is a bug... blk-mq tells expects us to skip unmapped queues but we fail the controller altogether... I assume managed affinity would have take care of linearization for us.. Does this quick untested patch work? --- static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) -- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 8023054ec83e..766d10acb1b9 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -604,20 +604,33 @@ static int nvme_rdma_start_queue(struct nvme_rdma_ctrl *ctrl, int idx) static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl) { - int i, ret = 0; + int i, ret = 0, count = 0; for (i = 1; i < ctrl->ctrl.queue_count; i++) { ret = nvme_rdma_start_queue(ctrl, i); - if (ret) + if (ret) { + if (ret == -EXDEV) { + /* unmapped queue, skip ... */ + nvme_rdma_free_queue(&ctrl->queues[i]); + continue; + } goto out_stop_queues; + } + count++; } + if (!count) + /* no started queues, fail */ + goto out_stop_queues; + + dev_info(ctrl->ctrl.device, "connected %d I/O queues.\n", count); + return 0; out_stop_queues: for (i--; i >= 1; i--) nvme_rdma_stop_queue(&ctrl->queues[i]); - return ret; + return -EIO; }