[05/15] Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
diff mbox series

Message ID 20190628004951.6202-6-mdf@kernel.org
State Awaiting Upstream
Headers show
Series
  • FPGA DFL updates
Related show

Commit Message

Moritz Fischer June 28, 2019, 12:49 a.m. UTC
From: Wu Hao <hao.wu@intel.com>

This patch adds virtualization support description for DFL based
FPGA devices (based on PCIe SRIOV), and introductions to new
interfaces added by new dfl private feature drivers.

[mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
Signed-off-by: Xu Yilun <yilun.xu@intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>
Signed-off-by: Moritz Fischer <mdf@kernel.org>
---
 Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)

Comments

Greg Kroah-Hartman July 3, 2019, 5:59 p.m. UTC | #1
On Thu, Jun 27, 2019 at 05:49:41PM -0700, Moritz Fischer wrote:
> From: Wu Hao <hao.wu@intel.com>
> 
> This patch adds virtualization support description for DFL based
> FPGA devices (based on PCIe SRIOV), and introductions to new
> interfaces added by new dfl private feature drivers.
> 
> [mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
> Signed-off-by: Xu Yilun <yilun.xu@intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> Acked-by: Alan Tull <atull@kernel.org>
> Signed-off-by: Moritz Fischer <mdf@kernel.org>
> ---
>  Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 100 insertions(+)

This doesn't apply to my tree, where is this file created?

thanks,

greg k-h
Wu, Hao July 3, 2019, 11:38 p.m. UTC | #2
On Wed, Jul 03, 2019 at 07:59:26PM +0200, Greg KH wrote:
> On Thu, Jun 27, 2019 at 05:49:41PM -0700, Moritz Fischer wrote:
> > From: Wu Hao <hao.wu@intel.com>
> > 
> > This patch adds virtualization support description for DFL based
> > FPGA devices (based on PCIe SRIOV), and introductions to new
> > interfaces added by new dfl private feature drivers.
> > 
> > [mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
> > Signed-off-by: Xu Yilun <yilun.xu@intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > Acked-by: Alan Tull <atull@kernel.org>
> > Signed-off-by: Moritz Fischer <mdf@kernel.org>
> > ---
> >  Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
> >  1 file changed, 100 insertions(+)
> 
> This doesn't apply to my tree, where is this file created?

Hi Greg,

From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
in linux-next. I think this patch is created on top of that by Moritz.

"Note: I've seen that Mauro touched Documentation/fpga/dfl.rst in linux-next
commit c220a1fae6c5d ("docs: fpga: convert docs to ReST and rename to *.rst")"

Thanks
Hao

> 
> thanks,
> 
> greg k-h
Greg Kroah-Hartman July 4, 2019, 5:37 a.m. UTC | #3
On Thu, Jul 04, 2019 at 07:38:22AM +0800, Wu Hao wrote:
> On Wed, Jul 03, 2019 at 07:59:26PM +0200, Greg KH wrote:
> > On Thu, Jun 27, 2019 at 05:49:41PM -0700, Moritz Fischer wrote:
> > > From: Wu Hao <hao.wu@intel.com>
> > > 
> > > This patch adds virtualization support description for DFL based
> > > FPGA devices (based on PCIe SRIOV), and introductions to new
> > > interfaces added by new dfl private feature drivers.
> > > 
> > > [mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
> > > Signed-off-by: Xu Yilun <yilun.xu@intel.com>
> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > Acked-by: Alan Tull <atull@kernel.org>
> > > Signed-off-by: Moritz Fischer <mdf@kernel.org>
> > > ---
> > >  Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 100 insertions(+)
> > 
> > This doesn't apply to my tree, where is this file created?
> 
> Hi Greg,
> 
> >From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
> in linux-next. I think this patch is created on top of that by Moritz.
> 
> "Note: I've seen that Mauro touched Documentation/fpga/dfl.rst in linux-next
> commit c220a1fae6c5d ("docs: fpga: convert docs to ReST and rename to *.rst")"

Ok, but I can't take this patch just because the file is converted in
someone else's tree :(
Wu, Hao July 4, 2019, 6:42 a.m. UTC | #4
On Thu, Jul 04, 2019 at 07:37:19AM +0200, Greg KH wrote:
> On Thu, Jul 04, 2019 at 07:38:22AM +0800, Wu Hao wrote:
> > On Wed, Jul 03, 2019 at 07:59:26PM +0200, Greg KH wrote:
> > > On Thu, Jun 27, 2019 at 05:49:41PM -0700, Moritz Fischer wrote:
> > > > From: Wu Hao <hao.wu@intel.com>
> > > > 
> > > > This patch adds virtualization support description for DFL based
> > > > FPGA devices (based on PCIe SRIOV), and introductions to new
> > > > interfaces added by new dfl private feature drivers.
> > > > 
> > > > [mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
> > > > Signed-off-by: Xu Yilun <yilun.xu@intel.com>
> > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > Acked-by: Alan Tull <atull@kernel.org>
> > > > Signed-off-by: Moritz Fischer <mdf@kernel.org>
> > > > ---
> > > >  Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 100 insertions(+)
> > > 
> > > This doesn't apply to my tree, where is this file created?
> > 
> > Hi Greg,
> > 
> > >From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
> > in linux-next. I think this patch is created on top of that by Moritz.
> > 
> > "Note: I've seen that Mauro touched Documentation/fpga/dfl.rst in linux-next
> > commit c220a1fae6c5d ("docs: fpga: convert docs to ReST and rename to *.rst")"
> 
> Ok, but I can't take this patch just because the file is converted in
> someone else's tree :(

Oh.. if that patch is merged from fpga tree, then we wont have this problem. :(

So do you have any suggestions on what should i do now?
wait that patch goes to offical release, and then resubmit this patch?

Hao
Greg Kroah-Hartman July 4, 2019, 8:17 a.m. UTC | #5
On Thu, Jul 04, 2019 at 02:42:08PM +0800, Wu Hao wrote:
> On Thu, Jul 04, 2019 at 07:37:19AM +0200, Greg KH wrote:
> > On Thu, Jul 04, 2019 at 07:38:22AM +0800, Wu Hao wrote:
> > > On Wed, Jul 03, 2019 at 07:59:26PM +0200, Greg KH wrote:
> > > > On Thu, Jun 27, 2019 at 05:49:41PM -0700, Moritz Fischer wrote:
> > > > > From: Wu Hao <hao.wu@intel.com>
> > > > > 
> > > > > This patch adds virtualization support description for DFL based
> > > > > FPGA devices (based on PCIe SRIOV), and introductions to new
> > > > > interfaces added by new dfl private feature drivers.
> > > > > 
> > > > > [mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
> > > > > Signed-off-by: Xu Yilun <yilun.xu@intel.com>
> > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > > Acked-by: Alan Tull <atull@kernel.org>
> > > > > Signed-off-by: Moritz Fischer <mdf@kernel.org>
> > > > > ---
> > > > >  Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
> > > > >  1 file changed, 100 insertions(+)
> > > > 
> > > > This doesn't apply to my tree, where is this file created?
> > > 
> > > Hi Greg,
> > > 
> > > >From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
> > > in linux-next. I think this patch is created on top of that by Moritz.
> > > 
> > > "Note: I've seen that Mauro touched Documentation/fpga/dfl.rst in linux-next
> > > commit c220a1fae6c5d ("docs: fpga: convert docs to ReST and rename to *.rst")"
> > 
> > Ok, but I can't take this patch just because the file is converted in
> > someone else's tree :(
> 
> Oh.. if that patch is merged from fpga tree, then we wont have this problem. :(

You better not be basing your tree on linux-next :)

> So do you have any suggestions on what should i do now?
> wait that patch goes to offical release, and then resubmit this patch?

Wait until the change hits Linus's tree and then resend it to me.

thanks,

greg k-h
Wu, Hao July 4, 2019, 8:19 a.m. UTC | #6
On Thu, Jul 04, 2019 at 10:17:13AM +0200, Greg KH wrote:
> On Thu, Jul 04, 2019 at 02:42:08PM +0800, Wu Hao wrote:
> > On Thu, Jul 04, 2019 at 07:37:19AM +0200, Greg KH wrote:
> > > On Thu, Jul 04, 2019 at 07:38:22AM +0800, Wu Hao wrote:
> > > > On Wed, Jul 03, 2019 at 07:59:26PM +0200, Greg KH wrote:
> > > > > On Thu, Jun 27, 2019 at 05:49:41PM -0700, Moritz Fischer wrote:
> > > > > > From: Wu Hao <hao.wu@intel.com>
> > > > > > 
> > > > > > This patch adds virtualization support description for DFL based
> > > > > > FPGA devices (based on PCIe SRIOV), and introductions to new
> > > > > > interfaces added by new dfl private feature drivers.
> > > > > > 
> > > > > > [mdf@kernel.org: Fixed up to make it work with new reStructuredText docs]
> > > > > > Signed-off-by: Xu Yilun <yilun.xu@intel.com>
> > > > > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > > > > Acked-by: Alan Tull <atull@kernel.org>
> > > > > > Signed-off-by: Moritz Fischer <mdf@kernel.org>
> > > > > > ---
> > > > > >  Documentation/fpga/dfl.rst | 100 +++++++++++++++++++++++++++++++++++++
> > > > > >  1 file changed, 100 insertions(+)
> > > > > 
> > > > > This doesn't apply to my tree, where is this file created?
> > > > 
> > > > Hi Greg,
> > > > 
> > > > >From the cover-letter, Moritz mentioned, dfl.txt has been converted to .rst
> > > > in linux-next. I think this patch is created on top of that by Moritz.
> > > > 
> > > > "Note: I've seen that Mauro touched Documentation/fpga/dfl.rst in linux-next
> > > > commit c220a1fae6c5d ("docs: fpga: convert docs to ReST and rename to *.rst")"
> > > 
> > > Ok, but I can't take this patch just because the file is converted in
> > > someone else's tree :(
> > 
> > Oh.. if that patch is merged from fpga tree, then we wont have this problem. :(
> 
> You better not be basing your tree on linux-next :)
> 
> > So do you have any suggestions on what should i do now?
> > wait that patch goes to offical release, and then resubmit this patch?
> 
> Wait until the change hits Linus's tree and then resend it to me.

OK, get it! Thanks!

Hao

> 
> thanks,
> 
> greg k-h

Patch
diff mbox series

diff --git a/Documentation/fpga/dfl.rst b/Documentation/fpga/dfl.rst
index 2f125abd777f..be9929dd7251 100644
--- a/Documentation/fpga/dfl.rst
+++ b/Documentation/fpga/dfl.rst
@@ -87,6 +87,8 @@  The following functions are exposed through ioctls:
 - Get driver API version (DFL_FPGA_GET_API_VERSION)
 - Check for extensions (DFL_FPGA_CHECK_EXTENSION)
 - Program bitstream (DFL_FPGA_FME_PORT_PR)
+- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
+- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
 
 More functions are exposed through sysfs
 (/sys/class/fpga_region/regionX/dfl-fme.n/):
@@ -143,6 +145,9 @@  More functions are exposed through sysfs:
  Read Accelerator GUID (afu_id)
      afu_id indicates which PR bitstream is programmed to this AFU.
 
+ Global error reporting management (errors/)
+     error reporting sysfs interfaces allow user to read errors detected by the
+     hardware, and clear the logged errors.
 
 DFL Framework Overview
 ======================
@@ -218,6 +223,101 @@  the compat_id exposed by the target FPGA region. This check is usually done by
 userspace before calling the reconfiguration IOCTL.
 
 
+FPGA virtualization - PCIe SRIOV
+================================
+This section describes the virtualization support on DFL based FPGA device to
+enable accessing an accelerator from applications running in a virtual machine
+(VM). This section only describes the PCIe based FPGA device with SRIOV support.
+
+Features supported by the particular FPGA device are exposed through Device
+Feature Lists, as illustrated below:
+
+::
+
+    +-------------------------------+  +-------------+
+    |              PF               |  |     VF      |
+    +-------------------------------+  +-------------+
+        ^            ^         ^              ^
+        |            |         |              |
+  +-----|------------|---------|--------------|-------+
+  |     |            |         |              |       |
+  |  +-----+     +-------+ +-------+      +-------+   |
+  |  | FME |     | Port0 | | Port1 |      | Port2 |   |
+  |  +-----+     +-------+ +-------+      +-------+   |
+  |                  ^         ^              ^       |
+  |                  |         |              |       |
+  |              +-------+ +------+       +-------+   |
+  |              |  AFU  | |  AFU |       |  AFU  |   |
+  |              +-------+ +------+       +-------+   |
+  |                                                   |
+  |            DFL based FPGA PCIe Device             |
+  +---------------------------------------------------+
+
+FME is always accessed through the physical function (PF).
+
+Ports (and related AFUs) are accessed via PF by default, but could be exposed
+through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
+1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
+created via PCIe SRIOV interface, to virtual machines.
+
+The driver organization in virtualization case is illustrated below:
+::
+
+    +-------++------++------+             |
+    | FME   || FME  || FME  |             |
+    | FPGA  || FPGA || FPGA |             |
+    |Manager||Bridge||Region|             |
+    +-------++------++------+             |
+    +-----------------------+  +--------+ |             +--------+
+    |          FME          |  |  AFU   | |             |  AFU   |
+    |         Module        |  | Module | |             | Module |
+    +-----------------------+  +--------+ |             +--------+
+          +-----------------------+       |       +-----------------------+
+          | FPGA Container Device |       |       | FPGA Container Device |
+          |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
+          +-----------------------+       |       +-----------------------+
+            +------------------+          |         +------------------+
+            | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
+            +------------------+   Host   | Machine +------------------+
+   -------------------------------------- | ------------------------------
+             +---------------+            |          +---------------+
+             | PCI PF Device |            |          | PCI VF Device |
+             +---------------+            |          +---------------+
+
+FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
+is detected. It:
+
+* Finishes enumeration on both FPGA PCIe PF and VF device using common
+	   interfaces from DFL framework.
+* Supports SRIOV.
+
+The FME device driver plays a management role in this driver architecture, it
+provides ioctls to release Port from PF and assign Port to PF. After release
+a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
+sysfs interface.
+
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+#. The PF owns all AFU ports by default. Any port that needs to be
+   reassigned to a VF must first be released through the
+   DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
+
+#. Once N ports are released from PF, then user can use command below
+   to enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+   ::
+
+      echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+#. Pass through the VFs to VMs
+
+#. The AFU under VF is accessible from applications in VM (using the
+   same driver inside the VF).
+
+Note that an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
 Device enumeration
 ==================
 This section introduces how applications enumerate the fpga device from