[v3,01/21] docs: fpga: add a document for Intel FPGA driver overview

Commit Message

Wu, Hao Nov. 27, 2017, 6:42 a.m. UTC

Add a document for Intel FPGA driver overview.

Signed-off-by: Enno Luebbers <[e-mail]>
Signed-off-by: Xiao Guangrong <[e-mail]>
Signed-off-by: Wu Hao <[e-mail]>
----
v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
    updated open discussion per current implementation.
    fixed some typos.
v3: use FPGA base region as container device instead of fpga-dev class.
    split common enumeration code from pcie driver to functions exposed by
    device feature list framework.
    update FME performance reporting which supports both integrated (iperf/)
    and discrete (dperf/) FPGA solutions.
---
 Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 261 insertions(+)
 create mode 100644 Documentation/fpga/intel-fpga.txt

Comments

Alan Tull Dec. 4, 2017, 7:55 p.m. UTC | #1

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <[e-mail]> wrote:
> Add a document for Intel FPGA driver overview.
>
> Signed-off-by: Enno Luebbers <[e-mail]>
> Signed-off-by: Xiao Guangrong <[e-mail]>
> Signed-off-by: Wu Hao <[e-mail]>
> ----
> v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
>     updated open discussion per current implementation.
>     fixed some typos.
> v3: use FPGA base region as container device instead of fpga-dev class.
>     split common enumeration code from pcie driver to functions exposed by
>     device feature list framework.
>     update FME performance reporting which supports both integrated (iperf/)
>     and discrete (dperf/) FPGA solutions.
> ---
>  Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 261 insertions(+)
>  create mode 100644 Documentation/fpga/intel-fpga.txt
>
> diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> new file mode 100644
> index 0000000..0754733
> --- /dev/null
> +++ b/Documentation/fpga/intel-fpga.txt
> @@ -0,0 +1,261 @@
> +===============================================================================
> +                    Intel FPGA driver Overview

This doesn't look Intel specific to me.  This could all be 'DFL FPGA Framework'

> +-------------------------------------------------------------------------------
> +                Enno Luebbers <[e-mail]>
> +                Xiao Guangrong <[e-mail]>
> +                Wu Hao <[e-mail]>
> +
> +The Intel FPGA driver provides interfaces for userspace applications to
> +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> +with Intel(R) FPGA PCIe based solutions and enables system level management
> +functions such as FPGA reconfiguration, power management, and virtualization.
> +
> +HW Architecture
> +===============
> +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> +The FPGA device memory is organized using a predefined data structure (Device
> +Feature List). Features supported by the particular FPGA device are exposed
> +through these data structures, as illustrated below:
> +
> +  +-------------------------------+  +-------------+
> +  |              PF               |  |     VF      |
> +  +-------------------------------+  +-------------+
> +      ^            ^         ^              ^
> +      |            |         |              |
> ++-----|------------|---------|--------------|-------+
> +|     |            |         |              |       |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|                  ^         ^              ^       |
> +|                  |         |              |       |
> +|              +-------+ +------+       +-------+   |
> +|              |  AFU  | |  AFU |       |  AFU  |   |
> +|              +-------+ +------+       +-------+   |
> +|                                                   |
> +|                 FPGA PCIe Device                  |
> ++---------------------------------------------------+
> +
> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> +used to assign individual accelerators to virtual machines.
> +
> +FME (FPGA Management Engine)
> +============================
> +The FPGA Management Engine performs power and thermal management, error
> +reporting, reconfiguration, performance reporting for integrated and discrete
> +solution, and other infrastructure functions. Each FPGA has one FME, which is
> +always accessed through the physical function (PF).
> +
> +User-space applications can acquire exclusive access to the FME using open(),
> +and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> +       Program bitstream (FPGA_FME_PORT_PR)
> +
> +More functions are exposed through sysfs
> +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):

I see that /sys/class/fpga/* has changed to /sys/class/fpga_region/*
now as requested (thanks!).  It looks like it ended up being pretty
straightforward (so far, just diffing this doc with the previous v2).

> +
> +       Read bitstream ID (bitstream_id)
> +       Read bitstream metadata (bitstream_metadata)
> +       Read number of ports (ports_num)
> +       Read socket ID (socket_id)
> +       Read performance counters (iperf/ and dperf/)
> +       Power management (power_mgmt/)
> +       Thermal management (thermal_mgmt/)
> +       Error reporting (errors/)
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.
> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
> +AFU
> +===
> +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> +accelerator-specific control registers.
> +
> +User-space applications can acquire exclusive access to an AFU attached to a
> +port by using open() on the port device node, and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Get port info (FPGA_PORT_GET_INFO)
> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> +       Reset AFU (FPGA_PORT_RESET)
> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> +
> +User-space applications can also mmap() accelerator MMIO regions.
> +
> +More functions are exposed through sysfs:
> +(/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/):
> +
> +       Read Accelerator GUID (afu_id)
> +       Error reporting (errors/)
> +
> +Partial Reconfiguration
> +=======================
> +As mentioned above, accelerators can be reconfigured through partial
> +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> +been generated for the exact blue bitstream and targeted reconfigurable region
> +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> +possibly cause system instability. This compatibility can be checked by
> +comparing the interface ID noted in the GBS header against the interface ID
> +exposed by the FME through sysfs (see above). This check is usually done by
> +user-space before calling the reconfiguration IOCTL.
> +
> +FPGA virtualization
> +===================
> +To enable accessing an accelerator from applications running in a VM, the
> +respective AFU's port needs to be assigned to a VF using the following steps:
> +
> + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> + to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the
> + FME device.
> +
> + b) Once N ports are released from PF, then user can use command below to
> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> +
> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> +
> + c) Pass through the VFs to VMs
> +
> + d) The AFU under VF is accessible from applications in VM (using the same
> + driver inside the VF).
> +
> +Note that an FME can't be assigned to a VF, thus PR and other management
> +functions are only available via the PF.
> +
> +
> +Driver organization
> +===================
> +
> +  +-------++------++------+             |
> +  | FME   || FME  || FME  |             |
> +  | FPGA  || FPGA || FPGA |             |
> +  |Manager||Bridge||Region|             |
> +  +-------++------++------+             |
> +  +-----------------------+  +--------+ |             +--------+
> +  |          FME          |  |  AFU   | |             |  AFU   |
> +  |         Module        |  | Module | |             | Module |
> +  +-----------------------+  +--------+ |             +--------+
> +        +-----------------------+       |       +-----------------------+
> +        | FPGA Container Device |       |       | FPGA Container Device |
> +        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
> +        +-----------------------+       |       +-----------------------+
> +          +------------------+          |         +------------------+
> +          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
> +          +------------------+   Host   | Machine +------------------+
> + -------------------------------------- | ------------------------------
> +           +---------------+            |          +---------------+
> +           | PCI PF Device |            |          | PCI VF Device |
> +           +---------------+            |          +---------------+
> +
> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> +driver plays an infrastructural role in the driver architecture.  It:
> +
> +       a) locates the Device Feature Lists in PCIE device BAR memory, handles
> +          them and related resources to common interfaces from DFL framework
> +          for enumeration.
> +       b) supports SRIOV.
> +
> +The feature device infrastructure provides common interfaces to create container
> +device (FPGA base region), discover feature devices and their sub features from
> +the given Device Feature Lists, and create platform devices for feature devices
> +with related resources under the container device. It also abstracts operations
> +for sub features and exposes common interfaces to feature device drivers.
> +
> +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> +automatically after FME platform device creation from the PCIE driver. It
> +provides the key features for FPGA management, including:
> +
> +       a) Power and thermal management, error reporting, performance reporting
> +          and other infrastructure functions. Users can access these functions
> +          via sysfs interfaces exposed by FME driver.
> +       b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
> +          bridges and FPGA regions during PR sub feature initialization; Once
> +          it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
> +          common interface function from FPGA Region to complete the partial
> +          reconfiguration of the bitstream to the given port.
> +       c) Port management for virtualization. The FME driver introduces two
> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> +          released from the PF, it can be assigned to the VF through the SRIOV
> +          interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> +          for more details).
> +
> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> +is probed once the AFU platform device is created. The main function of this
> +module is to provide an interface for userspace applications to access the
> +individual accelerators, including basic reset control on port, AFU MMIO region
> +export, dma buffer mapping service, UMsg notification, and remote debug
> +functions (see above).
> +
> +
> +Device enumeration
> +==================
> +This section introduces how applications enumerate the fpga device from
> +the sysfs hierarchy under /sys/class/fpga_region.
> +
> +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> +fpga device has one FME and two ports (AFUs).
> +
> +FPGA regions are created under /sys/class/fpga_region/
> +
> +       /sys/class/fpga_region/region0
> +       /sys/class/fpga_region/region1
> +       /sys/class/fpga_region/region2
> +       ...
> +
> +Application needs to search each regionX folder, if feature device is found,
> +(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base
> +fpga region which represents the FPGA device.
> +
> +Each base region has one FME and two ports (AFUs) as child devices:
> +
> +       /sys/class/fpga_region/region0/fpga-dfl-fme.0
> +       /sys/class/fpga_region/region0/fpga-dfl-port.0
> +       /sys/class/fpga_region/region0/fpga-dfl-port.1
> +       ...
> +
> +       /sys/class/fpga_region/region3/fpga-dfl-fme.1
> +       /sys/class/fpga_region/region3/fpga-dfl-port.2
> +       /sys/class/fpga_region/region3/fpga-dfl-port.3
> +       ...
> +
> +In general, the FME/AFU sysfs interfaces are named as follows:
> +
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/
> +
> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> +ports.
> +
> +The device nodes used for ioctl() or mmap() can be referenced through:
> +
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/dev
> +       /sys/class/fpga_region/<regionX>/<fpga-dfl-port.n>/dev
> +
> +Open discussion
> +===============
> +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
> +user now. In the future, if unified user interfaces for reconfiguration are
> +added, FME driver should switch to them from ioctl interface.
> --
> 1.8.3.1
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao Dec. 5, 2017, 3:57 a.m. UTC | #2

On Mon, Dec 04, 2017 at 01:55:37PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <[e-mail]> wrote:
> > Add a document for Intel FPGA driver overview.
> >
> > Signed-off-by: Enno Luebbers <[e-mail]>
> > Signed-off-by: Xiao Guangrong <[e-mail]>
> > Signed-off-by: Wu Hao <[e-mail]>
> > ----
> > v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
> >     updated open discussion per current implementation.
> >     fixed some typos.
> > v3: use FPGA base region as container device instead of fpga-dev class.
> >     split common enumeration code from pcie driver to functions exposed by
> >     device feature list framework.
> >     update FME performance reporting which supports both integrated (iperf/)
> >     and discrete (dperf/) FPGA solutions.
> > ---
> >  Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 261 insertions(+)
> >  create mode 100644 Documentation/fpga/intel-fpga.txt
> >
> > diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> > new file mode 100644
> > index 0000000..0754733
> > --- /dev/null
> > +++ b/Documentation/fpga/intel-fpga.txt
> > @@ -0,0 +1,261 @@
> > +===============================================================================
> > +                    Intel FPGA driver Overview
> 
> This doesn't look Intel specific to me.  This could all be 'DFL FPGA Framework'

Sure, will rename this doc to dfl-fpga.txt in the next version as we plan to rename
the pcie driver to dfl-pci per your comments on the patch #8, there is no reason to
keep it in this doc as all drivers will be dfl-* in the next version. :)

> 
> > +-------------------------------------------------------------------------------
> > +                Enno Luebbers <[e-mail]>
> > +                Xiao Guangrong <[e-mail]>
> > +                Wu Hao <[e-mail]>
> > +
> > +The Intel FPGA driver provides interfaces for userspace applications to
> > +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> > +with Intel(R) FPGA PCIe based solutions and enables system level management
> > +functions such as FPGA reconfiguration, power management, and virtualization.
> > +
> > +HW Architecture
> > +===============
> > +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> > +The FPGA device memory is organized using a predefined data structure (Device
> > +Feature List). Features supported by the particular FPGA device are exposed
> > +through these data structures, as illustrated below:
> > +
> > +  +-------------------------------+  +-------------+
> > +  |              PF               |  |     VF      |
> > +  +-------------------------------+  +-------------+
> > +      ^            ^         ^              ^
> > +      |            |         |              |
> > ++-----|------------|---------|--------------|-------+
> > +|     |            |         |              |       |
> > +|  +-----+     +-------+ +-------+      +-------+   |
> > +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> > +|  +-----+     +-------+ +-------+      +-------+   |
> > +|                  ^         ^              ^       |
> > +|                  |         |              |       |
> > +|              +-------+ +------+       +-------+   |
> > +|              |  AFU  | |  AFU |       |  AFU  |   |
> > +|              +-------+ +------+       +-------+   |
> > +|                                                   |
> > +|                 FPGA PCIe Device                  |
> > ++---------------------------------------------------+
> > +
> > +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> > +used to assign individual accelerators to virtual machines.
> > +
> > +FME (FPGA Management Engine)
> > +============================
> > +The FPGA Management Engine performs power and thermal management, error
> > +reporting, reconfiguration, performance reporting for integrated and discrete
> > +solution, and other infrastructure functions. Each FPGA has one FME, which is
> > +always accessed through the physical function (PF).
> > +
> > +User-space applications can acquire exclusive access to the FME using open(),
> > +and release it using close().
> > +
> > +The following functions are exposed through ioctls:
> > +
> > +       Get driver API version (FPGA_GET_API_VERSION)
> > +       Check for extensions (FPGA_CHECK_EXTENSION)
> > +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> > +       Release port from PF (FPGA_FME_PORT_RELEASE)
> > +       Program bitstream (FPGA_FME_PORT_PR)
> > +
> > +More functions are exposed through sysfs
> > +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):
> 
> I see that /sys/class/fpga/* has changed to /sys/class/fpga_region/*
> now as requested (thanks!).  It looks like it ended up being pretty
> straightforward (so far, just diffing this doc with the previous v2).

Thanks for the suggestion on using fpga base region. :)

Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

David Laight Dec. 6, 2017, 10:04 a.m. UTC | #3

From: Alan Tull

> Sent: 04 December 2017 19:56

> > +===============================================================================

> > +                    Intel FPGA driver Overview

> 

> This doesn't look Intel specific to me.  This could all be 'DFL FPGA Framework'


I've not read the details, but I susect that it is 'an Intel DFL Framework'.
(It might not even be 'the Intel DFL Framework').

	David

Alan Tull Dec. 20, 2017, 10:31 p.m. UTC | #4

On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <[e-mail]> wrote:
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.

Hi Hao,

If I remember correctly, reset means that the accelerator gets reset
and this is something that is desirable to do between jobs.  I've
asked for some documentation about the port reset function, partly
because the idea of being able to reset hardware from userspace
somehow scares me.  So please find a good logical place to explain
what a port reset does and how it is safe for userspace to request it
at some arbitrary time and how it won't crash the kernel.  We
discussed this in v2, I grepped v3 for it, maybe I missed it, but I
don't see it in v3.  My understanding is that disabling and reenabling
the port bridge causes the accelerator in its FPGA region to get
reset.

Alan

> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao Dec. 21, 2017, 6:02 a.m. UTC | #5

On Wed, Dec 20, 2017 at 04:31:15PM -0600, Alan Tull wrote:
> On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <[e-mail]> wrote:
> > +
> > +PORT
> > +====
> > +A port represents the interface between the static FPGA fabric (the "blue
> > +bitstream") and a partially reconfigurable region containing an AFU (the "green
> > +bitstream"). It controls the communication from SW to the accelerator and
> > +exposes features such as reset and debug.
> 
> Hi Hao,
> 
> If I remember correctly, reset means that the accelerator gets reset
> and this is something that is desirable to do between jobs.  I've
> asked for some documentation about the port reset function, partly
> because the idea of being able to reset hardware from userspace
> somehow scares me.  So please find a good logical place to explain
> what a port reset does and how it is safe for userspace to request it
> at some arbitrary time and how it won't crash the kernel.  We
> discussed this in v2, I grepped v3 for it, maybe I missed it, but I
> don't see it in v3.  My understanding is that disabling and reenabling
> the port bridge causes the accelerator in its FPGA region to get
> reset.

Hi Alan

Yes, that's correct.

Some descriptions are added in Patch#18[1] when introduced the reset ioctl.
I will add some descriptions in the doc as well.

[1]https://marc.info/?l=linux-fpga&m=151176566714744&w=2

@@ -50,6 +53,20 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for AFU file descriptor */
+
+/**
+ * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
+ *
+ * Reset the FPGA Port and its AFU. No parameters are supported.
+ * Userspace can do Port reset at any time, e.g during DMA or PR. But
+ * it should never cause any system level issue, only functional failure
+ * (e.g DMA or PR operation failure) and be recoverable from the failure.
+ * Return: 0 on success, -errno of failure
+ */
+
+#define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
+

Thanks
Hao

> 
> Alan
> 
> > +
> > +A PCIe device may have several ports and each port can be released from PF by
> > +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> > +sysfs interface.
> > +
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to [e-mail]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
new file mode 100644
index 0000000..0754733
--- /dev/null
+++ b/Documentation/fpga/intel-fpga.txt
@@ -0,0 +1,261 @@ 
+===============================================================================
+                    Intel FPGA driver Overview
+-------------------------------------------------------------------------------
+                Enno Luebbers <[e-mail]>
+                Xiao Guangrong <[e-mail]>
+                Wu Hao <[e-mail]>
+
+The Intel FPGA driver provides interfaces for userspace applications to
+configure, enumerate, open, and access FPGA accelerators on platforms equipped
+with Intel(R) FPGA PCIe based solutions and enables system level management
+functions such as FPGA reconfiguration, power management, and virtualization.
+
+HW Architecture
+===============
+From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
+The FPGA device memory is organized using a predefined data structure (Device
+Feature List). Features supported by the particular FPGA device are exposed
+through these data structures, as illustrated below:
+
+  +-------------------------------+  +-------------+
+  |              PF               |  |     VF      |
+  +-------------------------------+  +-------------+
+      ^            ^         ^              ^
+      |            |         |              |
++-----|------------|---------|--------------|-------+
+|     |            |         |              |       |
+|  +-----+     +-------+ +-------+      +-------+   |
+|  | FME |     | Port0 | | Port1 |      | Port2 |   |
+|  +-----+     +-------+ +-------+      +-------+   |
+|                  ^         ^              ^       |
+|                  |         |              |       |
+|              +-------+ +------+       +-------+   |
+|              |  AFU  | |  AFU |       |  AFU  |   |
+|              +-------+ +------+       +-------+   |
+|                                                   |
+|                 FPGA PCIe Device                  |
++---------------------------------------------------+
+
+The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
+used to assign individual accelerators to virtual machines.
+
+FME (FPGA Management Engine)
+============================
+The FPGA Management Engine performs power and thermal management, error
+reporting, reconfiguration, performance reporting for integrated and discrete
+solution, and other infrastructure functions. Each FPGA has one FME, which is
+always accessed through the physical function (PF).
+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Assign port to PF (FPGA_FME_PORT_ASSIGN)
+	Release port from PF (FPGA_FME_PORT_RELEASE)
+	Program bitstream (FPGA_FME_PORT_PR)
+
+More functions are exposed through sysfs
+(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/):
+
+	Read bitstream ID (bitstream_id)
+	Read bitstream metadata (bitstream_metadata)
+	Read number of ports (ports_num)
+	Read socket ID (socket_id)
+	Read performance counters (iperf/ and dperf/)
+	Power management (power_mgmt/)
+	Thermal management (thermal_mgmt/)
+	Error reporting (errors/)
+
+PORT
+====
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug.
+
+A PCIe device may have several ports and each port can be released from PF by
+FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
+sysfs interface.
+
+AFU
+===
+An AFU is attached to a port and exposes a 256k MMIO region to be used for
+accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Get port info (FPGA_PORT_GET_INFO)
+	Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+	Map DMA buffer (FPGA_PORT_DMA_MAP)
+	Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+	Reset AFU (FPGA_PORT_RESET)
+	Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+	Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+	Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+	Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/):
+
+	Read Accelerator GUID (afu_id)
+	Error reporting (errors/)
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above). This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization
+===================
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+ a) The PF owns all AFU ports by default. Any port that needs to be reassigned
+ to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the
+ FME device.
+
+ b) Once N ports are released from PF, then user can use command below to
+ enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+ c) Pass through the VFs to VMs
+
+ d) The AFU under VF is accessible from applications in VM (using the same
+ driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+
+Driver organization
+===================
+
+  +-------++------++------+             |
+  | FME   || FME  || FME  |             |
+  | FPGA  || FPGA || FPGA |             |
+  |Manager||Bridge||Region|             |
+  +-------++------++------+             |
+  +-----------------------+  +--------+ |             +--------+
+  |          FME          |  |  AFU   | |             |  AFU   |
+  |         Module        |  | Module | |             | Module |
+  +-----------------------+  +--------+ |             +--------+
+        +-----------------------+       |       +-----------------------+
+        | FPGA Container Device |       |       | FPGA Container Device |
+        |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
+        +-----------------------+       |       +-----------------------+
+          +------------------+          |         +------------------+
+          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
+          +------------------+   Host   | Machine +------------------+
+ -------------------------------------- | ------------------------------
+           +---------------+            |          +---------------+
+           | PCI PF Device |            |          | PCI VF Device |
+           +---------------+            |          +---------------+
+
+The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
+driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
+driver plays an infrastructural role in the driver architecture.  It:
+
+	a) locates the Device Feature Lists in PCIE device BAR memory, handles
+	   them and related resources to common interfaces from DFL framework
+	   for enumeration.
+	b) supports SRIOV.
+
+The feature device infrastructure provides common interfaces to create container
+device (FPGA base region), discover feature devices and their sub features from
+the given Device Feature Lists, and create platform devices for feature devices
+with related resources under the container device. It also abstracts operations
+for sub features and exposes common interfaces to feature device drivers.
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver. It
+provides the key features for FPGA management, including:
+
+	a) Power and thermal management, error reporting, performance reporting
+	   and other infrastructure functions. Users can access these functions
+	   via sysfs interfaces exposed by FME driver.
+	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
+	   bridges and FPGA regions during PR sub feature initialization; Once
+	   it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the
+	   common interface function from FPGA Region to complete the partial
+	   reconfiguration of the bitstream to the given port.
+	c) Port management for virtualization. The FME driver introduces two
+	   ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+	   FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+	   released from the PF, it can be assigned to the VF through the SRIOV
+	   interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+	   for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga_region.
+
+In the example below, two Intel(R) FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+FPGA regions are created under /sys/class/fpga_region/
+
+	/sys/class/fpga_region/region0
+	/sys/class/fpga_region/region1
+	/sys/class/fpga_region/region2
+	...
+
+Application needs to search each regionX folder, if feature device is found,
+(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base
+fpga region which represents the FPGA device.
+
+Each base region has one FME and two ports (AFUs) as child devices:
+
+	/sys/class/fpga_region/region0/fpga-dfl-fme.0
+	/sys/class/fpga_region/region0/fpga-dfl-port.0
+	/sys/class/fpga_region/region0/fpga-dfl-port.1
+	...
+
+	/sys/class/fpga_region/region3/fpga-dfl-fme.1
+	/sys/class/fpga_region/region3/fpga-dfl-port.2
+	/sys/class/fpga_region/region3/fpga-dfl-port.3
+	...
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/dev
+	/sys/class/fpga_region/<regionX>/<fpga-dfl-port.n>/dev
+
+Open discussion
+===============
+FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
+user now. In the future, if unified user interfaces for reconfiguration are
+added, FME driver should switch to them from ioctl interface.