From patchwork Tue Dec 4 01:52:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10711045 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 109A515A6 for ; Tue, 4 Dec 2018 01:54:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05A3B2AA76 for ; Tue, 4 Dec 2018 01:54:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED6062AB2C; Tue, 4 Dec 2018 01:54:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F3BB2AA76 for ; Tue, 4 Dec 2018 01:54:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726034AbeLDBwV (ORCPT ); Mon, 3 Dec 2018 20:52:21 -0500 Received: from bombadil.infradead.org ([198.137.202.133]:39010 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725937AbeLDBwU (ORCPT ); Mon, 3 Dec 2018 20:52:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=hJd6EBluiw4U3ZnUhV8q7Tb+wHuvITbItn/5jn0A6YA=; b=o1UvByhmBRDGGfdisIAvRFdvq G6d6Hz2kG/Q1QrmQihJvqIxEe3NW5YZiMchFAQ/Fmr2mfWYEF6iw4rbmvl4vD8VJTSSeiz3zM2mjV b+giR5guXI99F+vj4nz3fkY5/3wXxuHHZENElfc4JGObHF+3CYrVn2bby9P83xv0TXI6aNNOzlgNU ZskqDIF4S9440YXzLUbyRiXKkeGJWrvlPiiN4vfIAtiEkX07pL54+y7C2BK5wxzVbStc4sBPpjYiR H1UpXQwvh7THmhl2q8e9yHqfaTCkgOkSnGJXiHjryyaXuwE4O5tODyXszccMbpHyfsUAmcjinCZ20 mQqNuA58g==; Received: from [2600:1700:65a0:78e0:514:7862:1503:8e4d] (helo=sagi-Latitude-E7470.lbits) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1gTzsp-0003Zq-Fq; Tue, 04 Dec 2018 01:52:19 +0000 From: Sagi Grimberg To: linux-nvme@lists.infradead.org Cc: linux-block@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Keith Busch , "David S. Miller" Subject: [PATCH v5 00/13] TCP transport binding for NVMe over Fabrics Date: Mon, 3 Dec 2018 17:52:04 -0800 Message-Id: <20181204015217.16613-1-sagi@grimberg.me> X-Mailer: git-send-email 2.17.1 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch set implements the NVMe over Fabrics TCP host and the target drivers. Now NVMe over Fabrics can run on every Ethernet port in the world. The implementation conforms to NVMe over Fabrics 1.1 specification (which will include already publicly available NVMe/TCP transport binding, TP 8000). The host driver hooks into the NVMe host stack and implements the TCP transport binding for NVMe over Fabrics. The NVMe over Fabrics TCP host driver is responsible for establishing a NVMe/TCP connection, TCP event and error handling and data-plane messaging and stream processing. The target driver hooks into the NVMe target core stack and implements the TCP transport binding. The NVMe over Fabrics target driver is responsible for accepting and establishing NVMe/TCP connections, TCP event and error handling, and data-plane messaging and stream processing. The implementation of both the host and target are fairly simple and straight-forward. Every NVMe queue is backed by a TCP socket that provides us reliable, in-order delivery of fabrics capsules and/or data. All NVMe queues are sharded over a private bound workqueue such that we always have a single context handling the byte stream and we don't need to worry about any locking/serialization. In addition, close attention was paid to a completely non-blocking data plane to minimize context switching and/or unforced scheduling. Also, @netdev mailing list is cc'd as this patch set contains generic helpers for online digest calculation (patches 1-3). The patchset structure: - patches 1-6 are prep to add a helper for digest calculation online with data placement - patches 7-9 are preparatory patches for NVMe/TCP - patches 10-13 implements NVMe/TCP Thanks to the members of the Fabrics Linux Driver team that helped development, testing and benchmarking this work. Gitweb code is available at: git://git.infradead.org/nvme.git nvme-tcp Changes from v4: - Added acks from Dave Miller for relevant patches - Fixed possible memory leak in nvmet-tcp error flow Changes from v3: - various changes based on comments from christoph - removed unused variables - united send/recv iter initialization - removed unneeded void * casting - fixed long lines - removed unneeded wrappers (nvme_tcp_free_tagset and friends) - remove null sgl setting - fixed socket callbacks naming - reworked nvmet-tcp send_list processing - omitted nvme-cli patches as no changes were made to them and no negative feedback was accepted since v3 Changes from v2: - fixed stupid missing symbol export for skb_copy_and_hash_datagram_iter - dropped patch that moved err_work and connect_work to nvme_ctrl - fixed maxr2t icreq validation - got rid of host and target send/recv context structures by moving the members directly to their parent structure along with some struct documentation - removed bh disable when locking the queue lock - moved definition in nvme-tcp.h to appropriate patch - added patch to rework nvme-cli trtype handling for discovery log entries a bit - rebased on top of nvme-4.21 branch - cleaned up some checkpatch warnings - collected review tags Changes from v1: - unified skb_copy_datagram_iter and skb_copy_and_csum_datagram (and the new skb_hash_and_copy_datagram_iter) to a single code path - removed nvmet modparam budgets (made them a define set to their default values) - fixed nvme-tcp host chained r2t transfers reported off-list - made .install_queue callout return nvme status code - Added some review tags - rebased on top of nvme-4.21 branch (nvme tree) + sqflow disable patches Sagi Grimberg (13): ath6kl: add ath6kl_ prefix to crypto_type datagram: open-code copy_page_to_iter iov_iter: pass void csum pointer to csum_and_copy_to_iter datagram: consolidate datagram copy to iter helpers iov_iter: introduce hash_and_copy_to_iter helper datagram: introduce skb_copy_and_hash_datagram_iter helper nvmet: Add install_queue callout nvme-fabrics: allow user passing header digest nvme-fabrics: allow user passing data digest nvme-tcp: Add protocol header nvmet-tcp: add NVMe over TCP target driver nvmet: allow configfs tcp trtype configuration nvme-tcp: add NVMe over TCP host driver drivers/net/wireless/ath/ath6kl/cfg80211.c | 2 +- drivers/net/wireless/ath/ath6kl/common.h | 2 +- drivers/net/wireless/ath/ath6kl/wmi.c | 6 +- drivers/net/wireless/ath/ath6kl/wmi.h | 6 +- drivers/nvme/host/Kconfig | 15 + drivers/nvme/host/Makefile | 3 + drivers/nvme/host/fabrics.c | 10 + drivers/nvme/host/fabrics.h | 4 + drivers/nvme/host/tcp.c | 2242 ++++++++++++++++++++ drivers/nvme/target/Kconfig | 10 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/configfs.c | 1 + drivers/nvme/target/fabrics-cmd.c | 10 + drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/tcp.c | 1737 +++++++++++++++ include/linux/nvme-tcp.h | 189 ++ include/linux/nvme.h | 1 + include/linux/skbuff.h | 3 + include/linux/uio.h | 5 +- lib/iov_iter.c | 19 +- net/core/datagram.c | 159 +- 21 files changed, 4322 insertions(+), 105 deletions(-) create mode 100644 drivers/nvme/host/tcp.c create mode 100644 drivers/nvme/target/tcp.c create mode 100644 include/linux/nvme-tcp.h