From patchwork Thu Feb 21 15:33:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gal Pressman X-Patchwork-Id: 10824195 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AFCE86C2 for ; Thu, 21 Feb 2019 15:33:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FD0D315B3 for ; Thu, 21 Feb 2019 15:33:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D88F31844; Thu, 21 Feb 2019 15:33:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D7F4431752 for ; Thu, 21 Feb 2019 15:33:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726697AbfBUPdb (ORCPT ); Thu, 21 Feb 2019 10:33:31 -0500 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:24080 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725866AbfBUPda (ORCPT ); Thu, 21 Feb 2019 10:33:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1550763209; x=1582299209; h=from:to:cc:subject:date:message-id:mime-version; bh=UTw+GWwAEQU5jKXy4YRrcwNbxGeVruI3a26L1vb8l8M=; b=Vz03HbOm1BtrhDZD2hmJD3CYB+U9X/uH8c9X8S+2FllYYgUECBYS4zGY /fT/mv89ZGaVyCV+5n8XV4lPka65+gEyVJTMX8pqcbjMhA16PlfAjQBiZ mAPIkjoOBxWUYX+9aHv/Pwpbi9aY+VPwYilSClzO28VsZgRiYwtjk8KDJ E=; X-IronPort-AV: E=Sophos;i="5.58,395,1544486400"; d="scan'208";a="660972911" Received: from sea3-co-svc-lb6-vlan3.sea.amazon.com (HELO email-inbound-relay-1d-2c665b5d.us-east-1.amazon.com) ([10.47.22.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 21 Feb 2019 15:33:25 +0000 Received: from EX13MTAUWC001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-2c665b5d.us-east-1.amazon.com (8.14.7/8.14.7) with ESMTP id x1LFXHbO072469 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Thu, 21 Feb 2019 15:33:22 GMT Received: from EX13D19UWC002.ant.amazon.com (10.43.162.179) by EX13MTAUWC001.ant.amazon.com (10.43.162.135) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Thu, 21 Feb 2019 15:33:21 +0000 Received: from EX13MTAUWC001.ant.amazon.com (10.43.162.135) by EX13D19UWC002.ant.amazon.com (10.43.162.179) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Thu, 21 Feb 2019 15:33:21 +0000 Received: from galpress-VirtualBox.hfa16.amazon.com (10.218.62.29) by mail-relay.amazon.com (10.43.162.232) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Thu, 21 Feb 2019 15:33:17 +0000 From: Gal Pressman To: Jason Gunthorpe , Doug Ledford CC: Yossi Leybovich , Alexander Matushevsky , Leah Shalev , Dave Goodell , Brian Barrett , , Sean Hefty , "Dennis Dalessandro" , Leon Romanovsky , Christoph Hellwig , Parav Pandit , Sagi Grimberg , Gal Pressman Subject: [PATCH rdma-next v2 00/11] RDMA/efa: Elastic Fabric Adapter (EFA) driver Date: Thu, 21 Feb 2019 17:33:02 +0200 Message-ID: <1550763193-14128-1-git-send-email-galpress@amazon.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hello all, The following v2 patchset introduces the Elastic Fabric Adapter (EFA) driver, that was pre-announced by Amazon. EFA is a networking adapter designed to support user space network communication, initially offered in the Amazon EC2 environment. First release of EFA supports datagram send/receive operations and does not support connection-oriented or read/write operations. EFA supports Unreliable Datagrams (UD) as well as a new unordered, Scalable Reliable Datagram protocol (SRD). SRD provides support for reliable datagrams and more complete error handling than typical RD, but, unlike RD, it does not support ordering nor segmentation. EFA reliable datagram transport provides reliable out-of-order delivery, transparently utilizing multiple network paths to reduce network tail latency. Its interface is similar to UD, in particular it supports message size up to MTU, with error handling extended to support reliable communication. More information regarding SRD can be found at [1]. Kernel verbs and in-kernel services are initially not supported but are planned for future releases. EFA enabled EC2 instances have two different devices allocated, one for ENA (netdev) and one for EFA, the two are separate pci devices with no in-kernel communication between them. PR for rdma-core provider was sent: https://github.com/linux-rdma/rdma-core/pull/475 Thanks to everyone who took the time to review our last submission (Jason, Doug, Sean, Dennis, Leon, Christoph, Parav, Sagi), it is very appreciated. Major issues addressed in this v2: * Userspace libibverbs provider is implemented and attached for review. * Respect the atomic requirement of create/destroy AH flows using the new sleepable flag [2]. * Change link layer from Ethernet to Unspecified (Proprietary EC2 link layer). * Use RDMA mmap API. * Coherent DMA memory is no longer mapped to the userspace, streaming DMA mappings are used instead. * Introduce alloc/dealloc PD admin commands, PDs are now backed by an object on the device. This removes the bitmap used for PD number allocations. * Addressed the mmap lifetime issues: Each ucontext now uses a new User Access Region (UAR) abstraction. Objects which are tied to a specific UAR will not be allocated to a different user until the UAR is deallocated (on application exit). DMA memory will be unmapped when the QP/CQ is destroyed, but the buffers will remain allocated until application exit. The mmap entries now remain valid until application exit and allow for reuse of the same mmap key more than once. * SRD QP type is now a driver QP type (previously was IB_QPT_SRD). * Match UD QP Infiniband semantics, including 40 bytes offset, state transitions, QKey validation, etc. * Move AH reference counts to the device (previously was in the driver). When creating more than one AH with the same GID, the same device resource is used internally. Instead of keeping the reference count in the driver (and issue one create AH command only), each AH creation is now passed on to the device (accompanied with the PD number). This allows for future optimizations for AHs that are no longer used by a specific PD. * Removed all stub functions, which will mark EFA driver as a non-kverbs provider [3]. * Replace all pr_* prints with dev_* prints [1] https://github.com/amzn/rdma-core/wiki/SRD [2] https://patchwork.kernel.org/cover/10725727/ [3] https://patchwork.kernel.org/cover/10775039/ Thanks, Gal Gal Pressman (11): RDMA: Add EFA related definitions RDMA/efa: Add EFA device definitions RDMA/efa: Add the efa.h header file RDMA/efa: Add the efa_com.h file RDMA/efa: Add the com service API definitions RDMA/efa: Add the ABI definitions RDMA/efa: Implement functions that submit and complete admin commands RDMA/efa: Add common command handlers RDMA/efa: Add EFA verbs implementation RDMA/efa: Add the efa module RDMA/efa: Add driver to Kconfig/Makefile MAINTAINERS | 8 + drivers/infiniband/Kconfig | 1 + drivers/infiniband/core/sysfs.c | 1 + drivers/infiniband/core/verbs.c | 2 + drivers/infiniband/hw/Makefile | 1 + drivers/infiniband/hw/efa/Kconfig | 15 + drivers/infiniband/hw/efa/Makefile | 9 + drivers/infiniband/hw/efa/efa.h | 202 +++ drivers/infiniband/hw/efa/efa_admin_cmds_defs.h | 830 ++++++++++ drivers/infiniband/hw/efa/efa_admin_defs.h | 136 ++ drivers/infiniband/hw/efa/efa_com.c | 1184 ++++++++++++++ drivers/infiniband/hw/efa/efa_com.h | 141 ++ drivers/infiniband/hw/efa/efa_com_cmd.c | 726 +++++++++ drivers/infiniband/hw/efa/efa_com_cmd.h | 270 ++++ drivers/infiniband/hw/efa/efa_common_defs.h | 18 + drivers/infiniband/hw/efa/efa_main.c | 577 +++++++ drivers/infiniband/hw/efa/efa_regs_defs.h | 113 ++ drivers/infiniband/hw/efa/efa_verbs.c | 1891 +++++++++++++++++++++++ include/rdma/ib_verbs.h | 8 +- include/uapi/rdma/efa-abi.h | 129 ++ include/uapi/rdma/rdma_user_ioctl_cmds.h | 1 + 21 files changed, 6261 insertions(+), 2 deletions(-) create mode 100644 drivers/infiniband/hw/efa/Kconfig create mode 100644 drivers/infiniband/hw/efa/Makefile create mode 100644 drivers/infiniband/hw/efa/efa.h create mode 100644 drivers/infiniband/hw/efa/efa_admin_cmds_defs.h create mode 100644 drivers/infiniband/hw/efa/efa_admin_defs.h create mode 100644 drivers/infiniband/hw/efa/efa_com.c create mode 100644 drivers/infiniband/hw/efa/efa_com.h create mode 100644 drivers/infiniband/hw/efa/efa_com_cmd.c create mode 100644 drivers/infiniband/hw/efa/efa_com_cmd.h create mode 100644 drivers/infiniband/hw/efa/efa_common_defs.h create mode 100644 drivers/infiniband/hw/efa/efa_main.c create mode 100644 drivers/infiniband/hw/efa/efa_regs_defs.h create mode 100644 drivers/infiniband/hw/efa/efa_verbs.c create mode 100644 include/uapi/rdma/efa-abi.h