| Message ID | 20200904075052.8911-18-kishon@ti.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Delegated to: | Lorenzo Pieralisi |
| Headers | show |
| Series | Implement NTB Controller using multiple PCI EP | expand |
On 9/4/20 12:50 AM, Kishon Vijay Abraham I wrote: > Add documentation to help users use pci-epf-ntb function driver and > existing host side NTB infrastructure for NTB functionality. > > Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > --- > Documentation/PCI/endpoint/index.rst | 1 + > Documentation/PCI/endpoint/pci-ntb-howto.rst | 160 +++++++++++++++++++ > 2 files changed, 161 insertions(+) > create mode 100644 Documentation/PCI/endpoint/pci-ntb-howto.rst Hi, There are a few edits below: > diff --git a/Documentation/PCI/endpoint/pci-ntb-howto.rst b/Documentation/PCI/endpoint/pci-ntb-howto.rst > new file mode 100644 > index 000000000000..2fbb0a051c3b > --- /dev/null > +++ b/Documentation/PCI/endpoint/pci-ntb-howto.rst > @@ -0,0 +1,160 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +====================== > +PCI NTB EPF User Guide > +====================== > + > +:Author: Kishon Vijay Abraham I <kishon@ti.com> > + > +This document is a guide to help users use pci-epf-ntb function driver > +and ntb_hw_epf host driver for NTB functionality. The list of steps to > +be followed in the host side and EP side is given below. For the hardware > +configuration and internals of NTB using configurable endpoints see > +Documentation/PCI/endpoint/pci-ntb-function.rst > + > +Endpoint Device > +=============== > + > +Endpoint Controller Devices > +--------------------------- > + > +For implementing NTB functionality atleast two endpoint controller devices at least > +are required. > +To find the list of endpoint controller devices in the system:: > + > + # ls /sys/class/pci_epc/ > + 2900000.pcie-ep 2910000.pcie-ep > + > +If PCI_ENDPOINT_CONFIGFS is enabled:: > + > + # ls /sys/kernel/config/pci_ep/controllers > + 2900000.pcie-ep 2910000.pcie-ep > + > + > +Endpoint Function Drivers > +------------------------- > + > +To find the list of endpoint function drivers in the system:: > + > + # ls /sys/bus/pci-epf/drivers > + pci_epf_ntb pci_epf_ntb > + > +If PCI_ENDPOINT_CONFIGFS is enabled:: > + > + # ls /sys/kernel/config/pci_ep/functions > + pci_epf_ntb pci_epf_ntb > + > + > +Creating pci-epf-ntb Device > +---------------------------- > + > +PCI endpoint function device can be created using the configfs. To create > +pci-epf-ntb device, the following commands can be used:: > + > + # mount -t configfs none /sys/kernel/config > + # cd /sys/kernel/config/pci_ep/ > + # mkdir functions/pci_epf_ntb/func1 > + > +The "mkdir func1" above creates the pci-epf-ntb function device that will > +be probed by pci_epf_ntb driver. > + > +The PCI endpoint framework populates the directory with the following > +configurable fields:: > + > + # ls functions/pci_epf_ntb/func1 > + baseclass_code deviceid msi_interrupts pci-epf-ntb.0 > + progif_code secondary subsys_id vendorid > + cache_line_size interrupt_pin msix_interrupts primary > + revid subclass_code subsys_vendor_id > + > +The PCI endpoint function driver populates these entries with default values > +when the device is bound to the driver. The pci-epf-ntb driver populates > +vendorid with 0xffff and interrupt_pin with 0x0001:: > + > + # cat functions/pci_epf_ntb/func1/vendorid > + 0xffff > + # cat functions/pci_epf_ntb/func1/interrupt_pin > + 0x0001 > + > + > +Configuring pci-epf-ntb Device > +------------------------------- > + > +The user can configure the pci-epf-ntb device using configfs entry. In order using its configfs entry. > +to change the vendorid and the deviceid, the following > +commands can be used:: > + > + # echo 0x104c > functions/pci_epf_ntb/func1/vendorid > + # echo 0xb00d > functions/pci_epf_ntb/func1/deviceid > + > +In-order to configure NTB specific attributes, a new sub-directory to func1 In order > +should be created:: > + > + # mkdir functions/pci_epf_ntb/func1/pci_epf_ntb.0/ > + > +The NTB function driver will populate this directory with various attributes > +that can be configured by the user:: > + > + # ls functions/pci_epf_ntb/func1/pci_epf_ntb.0/ > + db_count mw1 mw2 mw3 mw4 num_mws > + spad_count > + > +A sample configuration for NTB function is given below:: > + > + # echo 4 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/db_count > + # echo 128 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/spad_count > + # echo 2 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/num_mws > + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw1 > + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw2 > + > +Binding pci-epf-ntb Device to EP Controller > +-------------------------------------------- > + > +NTB function device should be attached to two PCIe endpoint controllers > +connected to the two hosts. Use the 'primary' and 'secondary' entries > +inside NTB function device to attach one PCIe endpoint controller to > +primary interface and the other PCIe endpoint controller to the secondary > +interface. :: > + > + # ln -s controllers/2900000.pcie-ep/ functions/pci-epf-ntb/func1/primary > + # ln -s controllers/2910000.pcie-ep/ functions/pci-epf-ntb/func1/secondary > + > +Once the above step is completed, both the PCI endpoint controllers is ready to are ready > +establish a link with the host. > + > + > +Start the Link > +-------------- > + > +In order for the endpoint device to establish a link with the host, the _start_ > +field should be populated with '1'. For NTB, both the PCIe endpoint controllers > +should establish link with the host:: > + > + #echo 1 > controllers/2900000.pcie-ep/start > + #echo 1 > controllers/2910000.pcie-ep/start > + > + > +RootComplex Device > +================== > + > +lspci Output > +------------ > + > +Note that the devices listed here correspond to the value populated in 1.4 Can you use a section name (or reference) here instead of "1.4"? I can't see 1.4 when reading with an editor. > +above:: > + > + # lspci > + 0000:00:00.0 PCI bridge: Texas Instruments Device b00d > + 0000:01:00.0 RAM memory: Texas Instruments Device b00d > + > + > +Using ntb_hw_epf Device > +----------------------- > + > +The host side software follows the standard NTB software architecture in Linux. > +All the existing client side NTB utilities like NTB Transport Client and NTB > +Netdev, NTB Ping Pong Test Client and NTB Tool Test Clientcan be used with NTB Client can be > +function device. > + > +For more information on NTB see > +Documentation/driver-api/ntb.rst thanks.
Hi Randy, On 06/09/20 3:38 am, Randy Dunlap wrote: > On 9/4/20 12:50 AM, Kishon Vijay Abraham I wrote: >> Add documentation to help users use pci-epf-ntb function driver and >> existing host side NTB infrastructure for NTB functionality. >> >> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> >> --- >> Documentation/PCI/endpoint/index.rst | 1 + >> Documentation/PCI/endpoint/pci-ntb-howto.rst | 160 +++++++++++++++++++ >> 2 files changed, 161 insertions(+) >> create mode 100644 Documentation/PCI/endpoint/pci-ntb-howto.rst > > Hi, > There are a few edits below: Thanks for reviewing. I'll fix and post a new revision. Thanks Kishon > > >> diff --git a/Documentation/PCI/endpoint/pci-ntb-howto.rst b/Documentation/PCI/endpoint/pci-ntb-howto.rst >> new file mode 100644 >> index 000000000000..2fbb0a051c3b >> --- /dev/null >> +++ b/Documentation/PCI/endpoint/pci-ntb-howto.rst >> @@ -0,0 +1,160 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +====================== >> +PCI NTB EPF User Guide >> +====================== >> + >> +:Author: Kishon Vijay Abraham I <kishon@ti.com> >> + >> +This document is a guide to help users use pci-epf-ntb function driver >> +and ntb_hw_epf host driver for NTB functionality. The list of steps to >> +be followed in the host side and EP side is given below. For the hardware >> +configuration and internals of NTB using configurable endpoints see >> +Documentation/PCI/endpoint/pci-ntb-function.rst >> + >> +Endpoint Device >> +=============== >> + >> +Endpoint Controller Devices >> +--------------------------- >> + >> +For implementing NTB functionality atleast two endpoint controller devices > > at least > >> +are required. >> +To find the list of endpoint controller devices in the system:: >> + >> + # ls /sys/class/pci_epc/ >> + 2900000.pcie-ep 2910000.pcie-ep >> + >> +If PCI_ENDPOINT_CONFIGFS is enabled:: >> + >> + # ls /sys/kernel/config/pci_ep/controllers >> + 2900000.pcie-ep 2910000.pcie-ep >> + >> + >> +Endpoint Function Drivers >> +------------------------- >> + >> +To find the list of endpoint function drivers in the system:: >> + >> + # ls /sys/bus/pci-epf/drivers >> + pci_epf_ntb pci_epf_ntb >> + >> +If PCI_ENDPOINT_CONFIGFS is enabled:: >> + >> + # ls /sys/kernel/config/pci_ep/functions >> + pci_epf_ntb pci_epf_ntb >> + >> + >> +Creating pci-epf-ntb Device >> +---------------------------- >> + >> +PCI endpoint function device can be created using the configfs. To create >> +pci-epf-ntb device, the following commands can be used:: >> + >> + # mount -t configfs none /sys/kernel/config >> + # cd /sys/kernel/config/pci_ep/ >> + # mkdir functions/pci_epf_ntb/func1 >> + >> +The "mkdir func1" above creates the pci-epf-ntb function device that will >> +be probed by pci_epf_ntb driver. >> + >> +The PCI endpoint framework populates the directory with the following >> +configurable fields:: >> + >> + # ls functions/pci_epf_ntb/func1 >> + baseclass_code deviceid msi_interrupts pci-epf-ntb.0 >> + progif_code secondary subsys_id vendorid >> + cache_line_size interrupt_pin msix_interrupts primary >> + revid subclass_code subsys_vendor_id >> + >> +The PCI endpoint function driver populates these entries with default values >> +when the device is bound to the driver. The pci-epf-ntb driver populates >> +vendorid with 0xffff and interrupt_pin with 0x0001:: >> + >> + # cat functions/pci_epf_ntb/func1/vendorid >> + 0xffff >> + # cat functions/pci_epf_ntb/func1/interrupt_pin >> + 0x0001 >> + >> + >> +Configuring pci-epf-ntb Device >> +------------------------------- >> + >> +The user can configure the pci-epf-ntb device using configfs entry. In order > > using its configfs entry. > >> +to change the vendorid and the deviceid, the following >> +commands can be used:: >> + >> + # echo 0x104c > functions/pci_epf_ntb/func1/vendorid >> + # echo 0xb00d > functions/pci_epf_ntb/func1/deviceid >> + >> +In-order to configure NTB specific attributes, a new sub-directory to func1 > > In order > >> +should be created:: >> + >> + # mkdir functions/pci_epf_ntb/func1/pci_epf_ntb.0/ >> + >> +The NTB function driver will populate this directory with various attributes >> +that can be configured by the user:: >> + >> + # ls functions/pci_epf_ntb/func1/pci_epf_ntb.0/ >> + db_count mw1 mw2 mw3 mw4 num_mws >> + spad_count >> + >> +A sample configuration for NTB function is given below:: >> + >> + # echo 4 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/db_count >> + # echo 128 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/spad_count >> + # echo 2 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/num_mws >> + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw1 >> + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw2 >> + >> +Binding pci-epf-ntb Device to EP Controller >> +-------------------------------------------- >> + >> +NTB function device should be attached to two PCIe endpoint controllers >> +connected to the two hosts. Use the 'primary' and 'secondary' entries >> +inside NTB function device to attach one PCIe endpoint controller to >> +primary interface and the other PCIe endpoint controller to the secondary >> +interface. :: >> + >> + # ln -s controllers/2900000.pcie-ep/ functions/pci-epf-ntb/func1/primary >> + # ln -s controllers/2910000.pcie-ep/ functions/pci-epf-ntb/func1/secondary >> + >> +Once the above step is completed, both the PCI endpoint controllers is ready to > > are ready > >> +establish a link with the host. >> + >> + >> +Start the Link >> +-------------- >> + >> +In order for the endpoint device to establish a link with the host, the _start_ >> +field should be populated with '1'. For NTB, both the PCIe endpoint controllers >> +should establish link with the host:: >> + >> + #echo 1 > controllers/2900000.pcie-ep/start >> + #echo 1 > controllers/2910000.pcie-ep/start >> + >> + >> +RootComplex Device >> +================== >> + >> +lspci Output >> +------------ >> + >> +Note that the devices listed here correspond to the value populated in 1.4 > > Can you use a section name (or reference) here instead of "1.4"? I can't see 1.4 > when reading with an editor. > >> +above:: >> + >> + # lspci >> + 0000:00:00.0 PCI bridge: Texas Instruments Device b00d >> + 0000:01:00.0 RAM memory: Texas Instruments Device b00d >> + >> + >> +Using ntb_hw_epf Device >> +----------------------- >> + >> +The host side software follows the standard NTB software architecture in Linux. >> +All the existing client side NTB utilities like NTB Transport Client and NTB >> +Netdev, NTB Ping Pong Test Client and NTB Tool Test Clientcan be used with NTB > > Client can be > >> +function device. >> + >> +For more information on NTB see >> +Documentation/driver-api/ntb.rst > > > thanks. >
diff --git a/Documentation/PCI/endpoint/index.rst b/Documentation/PCI/endpoint/index.rst index 9cb6e5f3c4d5..38ea1f604b6d 100644 --- a/Documentation/PCI/endpoint/index.rst +++ b/Documentation/PCI/endpoint/index.rst @@ -12,6 +12,7 @@ PCI Endpoint Framework pci-test-function pci-test-howto pci-ntb-function + pci-ntb-howto function/binding/pci-test function/binding/pci-ntb diff --git a/Documentation/PCI/endpoint/pci-ntb-howto.rst b/Documentation/PCI/endpoint/pci-ntb-howto.rst new file mode 100644 index 000000000000..2fbb0a051c3b --- /dev/null +++ b/Documentation/PCI/endpoint/pci-ntb-howto.rst @@ -0,0 +1,160 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====================== +PCI NTB EPF User Guide +====================== + +:Author: Kishon Vijay Abraham I <kishon@ti.com> + +This document is a guide to help users use pci-epf-ntb function driver +and ntb_hw_epf host driver for NTB functionality. The list of steps to +be followed in the host side and EP side is given below. For the hardware +configuration and internals of NTB using configurable endpoints see +Documentation/PCI/endpoint/pci-ntb-function.rst + +Endpoint Device +=============== + +Endpoint Controller Devices +--------------------------- + +For implementing NTB functionality atleast two endpoint controller devices +are required. +To find the list of endpoint controller devices in the system:: + + # ls /sys/class/pci_epc/ + 2900000.pcie-ep 2910000.pcie-ep + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/controllers + 2900000.pcie-ep 2910000.pcie-ep + + +Endpoint Function Drivers +------------------------- + +To find the list of endpoint function drivers in the system:: + + # ls /sys/bus/pci-epf/drivers + pci_epf_ntb pci_epf_ntb + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/functions + pci_epf_ntb pci_epf_ntb + + +Creating pci-epf-ntb Device +---------------------------- + +PCI endpoint function device can be created using the configfs. To create +pci-epf-ntb device, the following commands can be used:: + + # mount -t configfs none /sys/kernel/config + # cd /sys/kernel/config/pci_ep/ + # mkdir functions/pci_epf_ntb/func1 + +The "mkdir func1" above creates the pci-epf-ntb function device that will +be probed by pci_epf_ntb driver. + +The PCI endpoint framework populates the directory with the following +configurable fields:: + + # ls functions/pci_epf_ntb/func1 + baseclass_code deviceid msi_interrupts pci-epf-ntb.0 + progif_code secondary subsys_id vendorid + cache_line_size interrupt_pin msix_interrupts primary + revid subclass_code subsys_vendor_id + +The PCI endpoint function driver populates these entries with default values +when the device is bound to the driver. The pci-epf-ntb driver populates +vendorid with 0xffff and interrupt_pin with 0x0001:: + + # cat functions/pci_epf_ntb/func1/vendorid + 0xffff + # cat functions/pci_epf_ntb/func1/interrupt_pin + 0x0001 + + +Configuring pci-epf-ntb Device +------------------------------- + +The user can configure the pci-epf-ntb device using configfs entry. In order +to change the vendorid and the deviceid, the following +commands can be used:: + + # echo 0x104c > functions/pci_epf_ntb/func1/vendorid + # echo 0xb00d > functions/pci_epf_ntb/func1/deviceid + +In-order to configure NTB specific attributes, a new sub-directory to func1 +should be created:: + + # mkdir functions/pci_epf_ntb/func1/pci_epf_ntb.0/ + +The NTB function driver will populate this directory with various attributes +that can be configured by the user:: + + # ls functions/pci_epf_ntb/func1/pci_epf_ntb.0/ + db_count mw1 mw2 mw3 mw4 num_mws + spad_count + +A sample configuration for NTB function is given below:: + + # echo 4 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/db_count + # echo 128 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/spad_count + # echo 2 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/num_mws + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw1 + # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw2 + +Binding pci-epf-ntb Device to EP Controller +-------------------------------------------- + +NTB function device should be attached to two PCIe endpoint controllers +connected to the two hosts. Use the 'primary' and 'secondary' entries +inside NTB function device to attach one PCIe endpoint controller to +primary interface and the other PCIe endpoint controller to the secondary +interface. :: + + # ln -s controllers/2900000.pcie-ep/ functions/pci-epf-ntb/func1/primary + # ln -s controllers/2910000.pcie-ep/ functions/pci-epf-ntb/func1/secondary + +Once the above step is completed, both the PCI endpoint controllers is ready to +establish a link with the host. + + +Start the Link +-------------- + +In order for the endpoint device to establish a link with the host, the _start_ +field should be populated with '1'. For NTB, both the PCIe endpoint controllers +should establish link with the host:: + + #echo 1 > controllers/2900000.pcie-ep/start + #echo 1 > controllers/2910000.pcie-ep/start + + +RootComplex Device +================== + +lspci Output +------------ + +Note that the devices listed here correspond to the value populated in 1.4 +above:: + + # lspci + 0000:00:00.0 PCI bridge: Texas Instruments Device b00d + 0000:01:00.0 RAM memory: Texas Instruments Device b00d + + +Using ntb_hw_epf Device +----------------------- + +The host side software follows the standard NTB software architecture in Linux. +All the existing client side NTB utilities like NTB Transport Client and NTB +Netdev, NTB Ping Pong Test Client and NTB Tool Test Clientcan be used with NTB +function device. + +For more information on NTB see +Documentation/driver-api/ntb.rst
Add documentation to help users use pci-epf-ntb function driver and existing host side NTB infrastructure for NTB functionality. Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> --- Documentation/PCI/endpoint/index.rst | 1 + Documentation/PCI/endpoint/pci-ntb-howto.rst | 160 +++++++++++++++++++ 2 files changed, 161 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-ntb-howto.rst