From patchwork Thu Mar 30 12:08:01 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wu, Hao" X-Patchwork-Id: 9654017 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 446006034C for ; Thu, 30 Mar 2017 12:15:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 310142858C for ; Thu, 30 Mar 2017 12:15:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2213E2858F; Thu, 30 Mar 2017 12:15:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E6F92858C for ; Thu, 30 Mar 2017 12:15:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933474AbdC3MP2 (ORCPT ); Thu, 30 Mar 2017 08:15:28 -0400 Received: from mga02.intel.com ([134.134.136.20]:27696 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933466AbdC3MP1 (ORCPT ); Thu, 30 Mar 2017 08:15:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=intel; t=1490876126; x=1522412126; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Y6uVBmAeNOtPCIQ0OgPFzJaESp5DTsiGZ4HsbFin9pU=; b=GNFJjWWF3+SRpISByhnA9q+dNx0y5z6fDQUW5ObzioPyJa45BtUEMxp/ u7X0N5oVm1rIyumeLCrVTNM2KhpHng==; Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Mar 2017 05:15:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.36,246,1486454400"; d="scan'208";a="840018042" Received: from hao-dev.bj.intel.com ([10.238.157.61]) by FMSMGA003.fm.intel.com with ESMTP; 30 Mar 2017 05:15:23 -0700 From: Wu Hao To: atull@kernel.org, moritz.fischer@ettus.com, linux-fpga@vger.kernel.org, linux-kernel@vger.kernel.org Cc: luwei.kang@intel.com, yi.z.zhang@intel.com, hao.wu@intel.com, Enno Luebbers , Xiao Guangrong Subject: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Date: Thu, 30 Mar 2017 20:08:01 +0800 Message-Id: <1490875696-15145-2-git-send-email-hao.wu@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1490875696-15145-1-git-send-email-hao.wu@intel.com> References: <1490875696-15145-1-git-send-email-hao.wu@intel.com> Sender: linux-fpga-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fpga@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add a document for Intel FPGA driver overview. Signed-off-by: Enno Luebbers Signed-off-by: Xiao Guangrong Signed-off-by: Wu Hao --- Documentation/fpga/intel-fpga.txt | 259 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 Documentation/fpga/intel-fpga.txt diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt new file mode 100644 index 0000000..9396cea --- /dev/null +++ b/Documentation/fpga/intel-fpga.txt @@ -0,0 +1,259 @@ +=============================================================================== + Intel FPGA driver Overview +------------------------------------------------------------------------------- + Enno Luebbers + Xiao Guangrong + Wu Hao + +The Intel FPGA driver provides interfaces for userspace applications to +configure, enumerate, open, and access FPGA accelerators on platforms equipped +with Intel(R) FPGA solutions and enables system level management functions such +as FPGA reconfiguration, power management, and virtualization. + +HW Architecture +=============== +From the OS's point of view, the FPGA hardware appears as a regular PCIe device. +The FPGA device memory is organized using a predefined data structure (Device +Feature List). Features supported by the particular FPGA device are exposed +through these data structures, as illustrated below: + + +-------------------------------+ +-------------+ + | PF | | VF | + +-------------------------------+ +-------------+ + ^ ^ ^ ^ + | | | | ++-----|------------|---------|--------------|-------+ +| | | | | | +| +-----+ +-------+ +-------+ +-------+ | +| | FME | | Port0 | | Port1 | | Port2 | | +| +-----+ +-------+ +-------+ +-------+ | +| ^ ^ ^ | +| | | | | +| +-------+ +------+ +-------+ | +| | AFU | | AFU | | AFU | | +| +-------+ +------+ +-------+ | +| | +| FPGA PCIe Device | ++---------------------------------------------------+ + +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be +used to assign individual accelerators to virtual machines . + +FME (FPGA Management Engine) +============================ +The FPGA Management Enging performs power and thermal management, error +reporting, reconfiguration, performance reporting, and other infrastructure +functions. Each FPGA has one FME, which is always accessed through the physical +function (PF). + +User-space applications can acquire exclusive access to the FME using open(), +and release it using close(). + +The following functions are exposed through ioctls: + + Get driver API version (FPGA_GET_API_VERSION) + Check for extensions (FPGA_CHECK_EXTENSION) + Assign port to PF (FPGA_FME_PORT_ASSIGN) + Release port from PF (FPGA_FME_PORT_RELEASE) + Program bitstream (FPGA_FME_PORT_PR) + +More functions are exposed through sysfs +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/): + + Read bitstream ID (bitstream_id) + Read bitstream metadata (bitstream_metadata) + Read number of ports (ports_num) + Read socket ID (socket_id) + Read performance counters (perf/) + Power management (power_mgmt/) + Thermal management (thermal_mgmt/) + Error reporting (errors/) + +PORT +==== +A port represents the interface between the static FPGA fabric (the "blue +bitstream") and a partially reconfigurable region containing an AFU (the "green +bitstream"). It controls the communication from SW to the accelerator and +exposes features such as reset and debug. + +A PCIe device may have several ports and each port can be released from PF by +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov +sysfs interface. + +AFU +=== +An AFU is attached to a port and exposes a 256k MMIO region to be used for +accelerator-specific control registers. + +User-space applications can acquire exclusive access to an AFU attached to a +port by using open() on the port device node, and release it using close(). + +The following functions are exposed through ioctls: + + Get driver API version (FPGA_GET_API_VERSION) + Check for extensions (FPGA_CHECK_EXTENSION) + Get port info (FPGA_PORT_GET_INFO) + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) + Map DMA buffer (FPGA_PORT_DMA_MAP) + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) + Reset AFU (FPGA_PORT_RESET) + Enable UMsg (FPGA_PORT_UMSG_ENABLE) + Disable UMsg (FPGA_PORT_UMSG_DISABLE) + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) + +User-space applications can also mmap() accelerator MMIO regions. + +More functions are exposed through sysfs: +(/sys/class/fpga/fpga.n/intel-fpga-port.m/): + + Read Accelerator GUID (afu_id) + Error reporting (errors/) + +Partial Reconfiguration +======================= +As mentioned above, accelerators can be reconfigured through partial +reconfiguration of a green bitstream file (GBS). The green bitstream must have +been generated for the exact blue bitstream and targeted reconfigurable region +(port) of the FPGA; otherwise, the reconfiguration operation will fail and +possibly cause system instability. This compatibility can be checked by +comparing the interface ID noted in the GBS header against the interface ID +exposed by the FME through sysfs (see above). This check is usually done by +user-space before calling the reconfiguration IOCTL. + +FPGA virtualization +=================== +To enable accessing an accelerator from applications running in a VM, the +respective AFU's port needs to be assigned to a VF using the following steps: + + a) The PF owns all AFU ports by default. Any port that needs to be reassigned + to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE + ioctl on the FME device. + + b) Once N ports are released from PF, then user can use below command to + enable SRIOV and VFs. Each VF owns only one Port with AFU. + + echo N > $PCI_DEVICE_PATH/sriov_numvfs + + c) Pass through the VFs to VMs + + d) The AFU under VF is accessiable from applications in VM (using the same + driver inside the VF). + +Note the an FME can't be assigned to a VF, thus PR and other management +functions are only available via the PF. + + +Driver organization +=================== + + +------------------+ +---------+ | +---------+ + | +-------+ | | | | | | + | | FPGA | FME | | AFU | | | AFU | + | |Manager| Module | | Module | | | Module | + | +-------+ | | | | | | + +------------------+ +---------+ | +---------+ + +-----------------------+ | +-----------------------+ + | FPGA Container Device | | | FPGA Container Device | + +-----------------------+ | +-----------------------+ + +------------------+ | +------------------+ + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | + +------------------+ Host | Machine +------------------+ + ------------------------------------ | ------------------------------ + +---------------+ | +---------------+ + | PCI PF Device | | | PCI VF Device | + +---------------+ | +---------------+ + +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This +driver plays an infrastructural role in the driver architecuture. It: + + a) creates FPGA container device as parent of the feature devices. + b) walks through the Device Feature List, which is implemented in PCIE + device BAR memory, to discover feature devices and their sub features + and create platform device for them under the container device. + c) supports SRIOV. + d) introduces the feature device infrastructure, which abstracts + operations for sub features and exposes common functions to feature + device drivers. + +The FPGA Management Engine (FME) driver is a platform driver which is loaded +automatically after FME platform device creation from the PCIE driver. It +provides the key features for FPGA management, including: + + a) Power and thermal management, error reporting, performance reporting + and other infrastructure functions. Users can access these functions + via sysfs interfaces exposed by FME driver. + b) Paritial Reconfiguration. The FME driver registers a FPGA Manager + during PR sub feature initialization; once it receives an + FPGA_FME_PORT_PR ioctl from user, it invokes the common interface + function from FPGA Manager to complete the partial reconfiguration of + the bitstream to the given port. + c) Port management for virtualization. The FME driver introduces two + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is + released from the PF, it can be assigned to the VF through the SRIOV + interfaces provided by PCIE driver. (Refer to "FPGA virtualization" + for more details). + +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver +is probed once the AFU platform device is created. The main function of this +module is to provide an interface for userspace applications to access the +individual accelerators, including basic reset control on port, AFU MMIO region +export, dma buffer mapping service, UMsg notification, and remote debug +functions (see above). + + +Device enumeration +================== +This section introduces how applications enumerate the fpga device from +the sysfs hierarchy under /sys/class/fpga. + +In the example below, two Intel(R) FPGA devices are installed in the host. Each +fpga device has one FME and two ports (AFUs). + +For each FPGA device, a device director is created under /sys/class/fpga/: + + /sys/class/fpga/fpga.0 + /sys/class/fpga/fpga.1 + +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name. +Application can retrieve name information via the sysfs interface: + + /sys/class/fpga/fpga.0/name + +Each node has one FME and two ports (AFUs) as child devices: + + /sys/class/fpga/fpga.0/intel-fpga-fme.0 + /sys/class/fpga/fpga.0/intel-fpga-port.0 + /sys/class/fpga/fpga.0/intel-fpga-port.1 + + /sys/class/fpga/fpga.1/intel-fpga-fme.1 + /sys/class/fpga/fpga.1/intel-fpga-port.2 + /sys/class/fpga/fpga.1/intel-fpga-port.3 + +In general, the FME/AFU sysfs interfaces are named as follows: + + /sys/class/fpga/// + /sys/class/fpga/// + +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all +ports. + +The device nodes used for ioctl() or mmap() can be referenced through: + + /sys/class/fpga///dev + /sys/class/fpga///dev + + +Open discussions +================ +The current FME driver does not provide user space access to the FME MMIO +region, but exposes access through sysfs and ioctls. It also provides an FPGA +manger interface for partial reconfiguration (PR), but does not make use of +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled inside +the FME, and fpga-region depends on device tree which is not used at all. There +are patches from Alan Tull to separate the device tree specific code and +introduce a sysfs interface for PR. We plan to add fpga-regions support in the +driver once the related patches get merged. Then the FME driver should create +one fpga-region for each Port/AFU.