| Message ID | 20220222162355.32369-1-Frank.Li@nxp.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | NTB function for PCIe RC to EP connection | expand |
On Tue, Feb 22, 2022 at 10:24 AM Frank Li <Frank.Li@nxp.com> wrote: > > This implement NTB function for PCIe EP to RC connections. > The existed ntb epf need two PCI EPs and two PCI Host. > > This just need EP to RC connections. > > ┌────────────┐ ┌─────────────────────────────────────┐ > │ │ │ │ > ├────────────┤ │ ┌──────────────┤ > │ NTB │ │ │ NTB │ > │ NetDev │ │ │ NetDev │ > ├────────────┤ │ ├──────────────┤ > │ NTB │ │ │ NTB │ > │ Transfer │ │ │ Transfer │ > ├────────────┤ │ ├──────────────┤ > │ │ │ │ │ > │ PCI NTB │ │ │ │ > │ EPF │ │ │ │ > │ Driver │ │ │ PCI Virtual │ > │ │ ├───────────────┐ │ NTB Driver │ > │ │ │ PCI EP NTB │◄────►│ │ > │ │ │ FN Driver │ │ │ > ├────────────┤ ├───────────────┤ ├──────────────┤ > │ │ │ │ │ │ > │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > │ │ PCI │ │ │ BUS │ > └────────────┘ └───────────────┴──────┴──────────────┘ > PCI RC PCI EP > > > > Frank Li (4): > PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > NTB: epf: Allow more flexibility in the memory BAR map method > PCI: endpoint: Support NTB transfer between RC and EP > Documentation: PCI: Add specification for the PCI vNTB function device > Ping > Documentation/PCI/endpoint/index.rst | 2 + > .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > drivers/pci/endpoint/functions/Kconfig | 11 + > drivers/pci/endpoint/functions/Makefile | 1 + > drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > 8 files changed, 1775 insertions(+), 14 deletions(-) > create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > > -- > 2.24.0.rc1 >
On Thu, Mar 10, 2022 at 4:01 PM Zhi Li <lznuaa@gmail.com> wrote: > > On Tue, Feb 22, 2022 at 10:24 AM Frank Li <Frank.Li@nxp.com> wrote: > > > > This implement NTB function for PCIe EP to RC connections. > > The existed ntb epf need two PCI EPs and two PCI Host. > > > > This just need EP to RC connections. > > > > ┌────────────┐ ┌─────────────────────────────────────┐ > > │ │ │ │ > > ├────────────┤ │ ┌──────────────┤ > > │ NTB │ │ │ NTB │ > > │ NetDev │ │ │ NetDev │ > > ├────────────┤ │ ├──────────────┤ > > │ NTB │ │ │ NTB │ > > │ Transfer │ │ │ Transfer │ > > ├────────────┤ │ ├──────────────┤ > > │ │ │ │ │ > > │ PCI NTB │ │ │ │ > > │ EPF │ │ │ │ > > │ Driver │ │ │ PCI Virtual │ > > │ │ ├───────────────┐ │ NTB Driver │ > > │ │ │ PCI EP NTB │◄────►│ │ > > │ │ │ FN Driver │ │ │ > > ├────────────┤ ├───────────────┤ ├──────────────┤ > > │ │ │ │ │ │ > > │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > > │ │ PCI │ │ │ BUS │ > > └────────────┘ └───────────────┴──────┴──────────────┘ > > PCI RC PCI EP > > > > > > > > Frank Li (4): > > PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > > NTB: epf: Allow more flexibility in the memory BAR map method > > PCI: endpoint: Support NTB transfer between RC and EP > > Documentation: PCI: Add specification for the PCI vNTB function device > > > Update ntb mail list > > > Documentation/PCI/endpoint/index.rst | 2 + > > .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > > Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > > drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > > .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > > drivers/pci/endpoint/functions/Kconfig | 11 + > > drivers/pci/endpoint/functions/Makefile | 1 + > > drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > > 8 files changed, 1775 insertions(+), 14 deletions(-) > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > > create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > > > > -- > > 2.24.0.rc1 > >
On Thu, Mar 10, 2022 at 4:07 PM Zhi Li <lznuaa@gmail.com> wrote: > > On Thu, Mar 10, 2022 at 4:01 PM Zhi Li <lznuaa@gmail.com> wrote: > > > > On Tue, Feb 22, 2022 at 10:24 AM Frank Li <Frank.Li@nxp.com> wrote: > > > > > > This implement NTB function for PCIe EP to RC connections. > > > The existed ntb epf need two PCI EPs and two PCI Host. > > > > > > This just need EP to RC connections. > > > > > > ┌────────────┐ ┌─────────────────────────────────────┐ > > > │ │ │ │ > > > ├────────────┤ │ ┌──────────────┤ > > > │ NTB │ │ │ NTB │ > > > │ NetDev │ │ │ NetDev │ > > > ├────────────┤ │ ├──────────────┤ > > > │ NTB │ │ │ NTB │ > > > │ Transfer │ │ │ Transfer │ > > > ├────────────┤ │ ├──────────────┤ > > > │ │ │ │ │ > > > │ PCI NTB │ │ │ │ > > > │ EPF │ │ │ │ > > > │ Driver │ │ │ PCI Virtual │ > > > │ │ ├───────────────┐ │ NTB Driver │ > > > │ │ │ PCI EP NTB │◄────►│ │ > > > │ │ │ FN Driver │ │ │ > > > ├────────────┤ ├───────────────┤ ├──────────────┤ > > > │ │ │ │ │ │ > > > │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > > > │ │ PCI │ │ │ BUS │ > > > └────────────┘ └───────────────┴──────┴──────────────┘ > > > PCI RC PCI EP > > > > > > > > > > > > Frank Li (4): > > > PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > > > NTB: epf: Allow more flexibility in the memory BAR map method > > > PCI: endpoint: Support NTB transfer between RC and EP > > > Documentation: PCI: Add specification for the PCI vNTB function device > > > > > > > Update ntb mail list Friendly ping! Frank Li > > > > > > Documentation/PCI/endpoint/index.rst | 2 + > > > .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > > > Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > > > drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > > > .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > > > drivers/pci/endpoint/functions/Kconfig | 11 + > > > drivers/pci/endpoint/functions/Makefile | 1 + > > > drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > > > 8 files changed, 1775 insertions(+), 14 deletions(-) > > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > > > create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > > > > > > -- > > > 2.24.0.rc1 > > >
Hi Frank Li, On 22/02/22 9:53 pm, Frank Li wrote: > This implement NTB function for PCIe EP to RC connections. > The existed ntb epf need two PCI EPs and two PCI Host. As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio layer instead of trying to build on NTB layer (which is specific to RC<->RC communication). Are there any specific reasons for not taking that path? Thanks, Kishon [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com > > This just need EP to RC connections. > > ┌────────────┐ ┌─────────────────────────────────────┐ > │ │ │ │ > ├────────────┤ │ ┌──────────────┤ > │ NTB │ │ │ NTB │ > │ NetDev │ │ │ NetDev │ > ├────────────┤ │ ├──────────────┤ > │ NTB │ │ │ NTB │ > │ Transfer │ │ │ Transfer │ > ├────────────┤ │ ├──────────────┤ > │ │ │ │ │ > │ PCI NTB │ │ │ │ > │ EPF │ │ │ │ > │ Driver │ │ │ PCI Virtual │ > │ │ ├───────────────┐ │ NTB Driver │ > │ │ │ PCI EP NTB │◄────►│ │ > │ │ │ FN Driver │ │ │ > ├────────────┤ ├───────────────┤ ├──────────────┤ > │ │ │ │ │ │ > │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > │ │ PCI │ │ │ BUS │ > └────────────┘ └───────────────┴──────┴──────────────┘ > PCI RC PCI EP > > > > Frank Li (4): > PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > NTB: epf: Allow more flexibility in the memory BAR map method > PCI: endpoint: Support NTB transfer between RC and EP > Documentation: PCI: Add specification for the PCI vNTB function device > > Documentation/PCI/endpoint/index.rst | 2 + > .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > drivers/pci/endpoint/functions/Kconfig | 11 + > drivers/pci/endpoint/functions/Makefile | 1 + > drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > 8 files changed, 1775 insertions(+), 14 deletions(-) > create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c >
On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: > > Hi Frank Li, > > On 22/02/22 9:53 pm, Frank Li wrote: > > This implement NTB function for PCIe EP to RC connections. > > The existed ntb epf need two PCI EPs and two PCI Host. > > As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio > layer instead of trying to build on NTB layer (which is specific to RC<->RC > communication). > > Are there any specific reasons for not taking that path? 1. EP side work as vHOST mode. vHost suppose access all memory of virtual io. But there are only map windows on the EP side to access RC side memory. You have to move map windows for each access. It is quite low efficiency. 2. So far as I know, virtio is still not DMA yet. CPU access PCI can't generate longer PCI TLP, So the speed is quite slow. NTB already has DMA support. If you use system level DMA, no change is needed at NTB level. If we want to use a PCI controller embedded DMA, some small changes need if based on my other Designware PCI eDMA patches, which are under review. 3. All the major data transfer of NTB is using write. Because TLP write needn't wait for complete, write performance is better than reading. On our platform, write performance is about 10% better than read. Frank > > Thanks, > Kishon > > [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com > > > > This just need EP to RC connections. > > > > ┌────────────┐ ┌─────────────────────────────────────┐ > > │ │ │ │ > > ├────────────┤ │ ┌──────────────┤ > > │ NTB │ │ │ NTB │ > > │ NetDev │ │ │ NetDev │ > > ├────────────┤ │ ├──────────────┤ > > │ NTB │ │ │ NTB │ > > │ Transfer │ │ │ Transfer │ > > ├────────────┤ │ ├──────────────┤ > > │ │ │ │ │ > > │ PCI NTB │ │ │ │ > > │ EPF │ │ │ │ > > │ Driver │ │ │ PCI Virtual │ > > │ │ ├───────────────┐ │ NTB Driver │ > > │ │ │ PCI EP NTB │◄────►│ │ > > │ │ │ FN Driver │ │ │ > > ├────────────┤ ├───────────────┤ ├──────────────┤ > > │ │ │ │ │ │ > > │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > > │ │ PCI │ │ │ BUS │ > > └────────────┘ └───────────────┴──────┴──────────────┘ > > PCI RC PCI EP > > > > > > > > Frank Li (4): > > PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > > NTB: epf: Allow more flexibility in the memory BAR map method > > PCI: endpoint: Support NTB transfer between RC and EP > > Documentation: PCI: Add specification for the PCI vNTB function device > > > > Documentation/PCI/endpoint/index.rst | 2 + > > .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > > Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > > drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > > .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > > drivers/pci/endpoint/functions/Kconfig | 11 + > > drivers/pci/endpoint/functions/Makefile | 1 + > > drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > > 8 files changed, 1775 insertions(+), 14 deletions(-) > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > > create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > >
On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@gmail.com> wrote: > > On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: > > > > Hi Frank Li, > > > > On 22/02/22 9:53 pm, Frank Li wrote: > > > This implement NTB function for PCIe EP to RC connections. > > > The existed ntb epf need two PCI EPs and two PCI Host. > > > > As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio > > layer instead of trying to build on NTB layer (which is specific to RC<->RC > > communication). > > > > Are there any specific reasons for not taking that path? > > 1. EP side work as vHOST mode. vHost suppose access all memory of virtual io. > But there are only map windows on the EP side to access RC side > memory. You have to move > map windows for each access. It is quite low efficiency. > > 2. So far as I know, virtio is still not DMA yet. CPU access PCI > can't generate longer PCI TLP, > So the speed is quite slow. NTB already has DMA support. If you use > system level DMA, > no change is needed at NTB level. If we want to use a PCI controller > embedded DMA, some small > changes need if based on my other Designware PCI eDMA patches, which > are under review. > > 3. All the major data transfer of NTB is using write. Because TLP > write needn't wait for complete, write > performance is better than reading. On our platform, write > performance is about 10% better than read. > > Frank Any Comments or rejection? @Kishon Vijay Abraham I best regards Frank Li > > > > > Thanks, > > Kishon > > > > [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com > > > > > > This just need EP to RC connections. > > > > > > ┌────────────┐ ┌─────────────────────────────────────┐ > > > │ │ │ │ > > > ├────────────┤ │ ┌──────────────┤ > > > │ NTB │ │ │ NTB │ > > > │ NetDev │ │ │ NetDev │ > > > ├────────────┤ │ ├──────────────┤ > > > │ NTB │ │ │ NTB │ > > > │ Transfer │ │ │ Transfer │ > > > ├────────────┤ │ ├──────────────┤ > > > │ │ │ │ │ > > > │ PCI NTB │ │ │ │ > > > │ EPF │ │ │ │ > > > │ Driver │ │ │ PCI Virtual │ > > > │ │ ├───────────────┐ │ NTB Driver │ > > > │ │ │ PCI EP NTB │◄────►│ │ > > > │ │ │ FN Driver │ │ │ > > > ├────────────┤ ├───────────────┤ ├──────────────┤ > > > │ │ │ │ │ │ > > > │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > > > │ │ PCI │ │ │ BUS │ > > > └────────────┘ └───────────────┴──────┴──────────────┘ > > > PCI RC PCI EP > > > > > > > > > > > > Frank Li (4): > > > PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > > > NTB: epf: Allow more flexibility in the memory BAR map method > > > PCI: endpoint: Support NTB transfer between RC and EP > > > Documentation: PCI: Add specification for the PCI vNTB function device > > > > > > Documentation/PCI/endpoint/index.rst | 2 + > > > .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > > > Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > > > drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > > > .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > > > drivers/pci/endpoint/functions/Kconfig | 11 + > > > drivers/pci/endpoint/functions/Makefile | 1 + > > > drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > > > 8 files changed, 1775 insertions(+), 14 deletions(-) > > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > > > create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > > > create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > > >
Hi Frank, On 21/04/22 1:52 am, Zhi Li wrote: > On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@gmail.com> wrote: >> >> On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: >>> >>> Hi Frank Li, >>> >>> On 22/02/22 9:53 pm, Frank Li wrote: >>>> This implement NTB function for PCIe EP to RC connections. >>>> The existed ntb epf need two PCI EPs and two PCI Host. >>> >>> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio >>> layer instead of trying to build on NTB layer (which is specific to RC<->RC >>> communication). >>> >>> Are there any specific reasons for not taking that path? >> >> 1. EP side work as vHOST mode. vHost suppose access all memory of virtual io. >> But there are only map windows on the EP side to access RC side >> memory. You have to move >> map windows for each access. It is quite low efficiency. I'm not sure I quite get this. EP HW has limited outbound memory to access RC memory irrespective of how we implement it. This is not a SW framework limitation AFAICS. >> >> 2. So far as I know, virtio is still not DMA yet. CPU access PCI >> can't generate longer PCI TLP, >> So the speed is quite slow. NTB already has DMA support. If you use >> system level DMA, >> no change is needed at NTB level. If we want to use a PCI controller >> embedded DMA, some small >> changes need if based on my other Designware PCI eDMA patches, which >> are under review. Adding dmaengine API to do memcopy should be simple to add in vhost/virtio interface. >> >> 3. All the major data transfer of NTB is using write. Because TLP >> write needn't wait for complete, write >> performance is better than reading. On our platform, write >> performance is about 10% better than read. >> >> Frank > > Any Comments or rejection? @Kishon Vijay Abraham I I'd strongly recommend going with virtio/vhost based approach and standardizing it IMO. Thanks, Kishon > > best regards > Frank Li > >> >>> >>> Thanks, >>> Kishon >>> >>> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com >>>> >>>> This just need EP to RC connections. >>>> >>>> ┌────────────┐ ┌─────────────────────────────────────┐ >>>> │ │ │ │ >>>> ├────────────┤ │ ┌──────────────┤ >>>> │ NTB │ │ │ NTB │ >>>> │ NetDev │ │ │ NetDev │ >>>> ├────────────┤ │ ├──────────────┤ >>>> │ NTB │ │ │ NTB │ >>>> │ Transfer │ │ │ Transfer │ >>>> ├────────────┤ │ ├──────────────┤ >>>> │ │ │ │ │ >>>> │ PCI NTB │ │ │ │ >>>> │ EPF │ │ │ │ >>>> │ Driver │ │ │ PCI Virtual │ >>>> │ │ ├───────────────┐ │ NTB Driver │ >>>> │ │ │ PCI EP NTB │◄────►│ │ >>>> │ │ │ FN Driver │ │ │ >>>> ├────────────┤ ├───────────────┤ ├──────────────┤ >>>> │ │ │ │ │ │ >>>> │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ >>>> │ │ PCI │ │ │ BUS │ >>>> └────────────┘ └───────────────┴──────┴──────────────┘ >>>> PCI RC PCI EP >>>> >>>> >>>> >>>> Frank Li (4): >>>> PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address >>>> NTB: epf: Allow more flexibility in the memory BAR map method >>>> PCI: endpoint: Support NTB transfer between RC and EP >>>> Documentation: PCI: Add specification for the PCI vNTB function device >>>> >>>> Documentation/PCI/endpoint/index.rst | 2 + >>>> .../PCI/endpoint/pci-vntb-function.rst | 126 ++ >>>> Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ >>>> drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- >>>> .../pci/controller/dwc/pcie-designware-ep.c | 10 +- >>>> drivers/pci/endpoint/functions/Kconfig | 11 + >>>> drivers/pci/endpoint/functions/Makefile | 1 + >>>> drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ >>>> 8 files changed, 1775 insertions(+), 14 deletions(-) >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst >>>> create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c >>>>
On Fri, Apr 22, 2022 at 10:15 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: > > Hi Frank, > > On 21/04/22 1:52 am, Zhi Li wrote: > > On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@gmail.com> wrote: > >> > >> On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: > >>> > >>> Hi Frank Li, > >>> > >>> On 22/02/22 9:53 pm, Frank Li wrote: > >>>> This implement NTB function for PCIe EP to RC connections. > >>>> The existed ntb epf need two PCI EPs and two PCI Host. > >>> > >>> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio > >>> layer instead of trying to build on NTB layer (which is specific to RC<->RC > >>> communication). > >>> > >>> Are there any specific reasons for not taking that path? > >> > >> 1. EP side work as vHOST mode. vHost suppose access all memory of virtual io. > >> But there are only map windows on the EP side to access RC side > >> memory. You have to move > >> map windows for each access. It is quite low efficiency. > > I'm not sure I quite get this. EP HW has limited outbound memory to access RC > memory irrespective of how we implement it. This is not a SW framework > limitation AFAICS. Almost all EP HW have limited outbound memory windows to access RC. We face transfer efficiency problems if we stick into vhost. > >> > >> 2. So far as I know, virtio is still not DMA yet. CPU access PCI > >> can't generate longer PCI TLP, > >> So the speed is quite slow. NTB already has DMA support. If you use > >> system level DMA, > >> no change is needed at NTB level. If we want to use a PCI controller > >> embedded DMA, some small > >> changes need if based on my other Designware PCI eDMA patches, which > >> are under review. > > Adding dmaengine API to do memcopy should be simple to add in vhost/virtio > interface. > >> > >> 3. All the major data transfer of NTB is using write. Because TLP > >> write needn't wait for complete, write > >> performance is better than reading. On our platform, write > >> performance is about 10% better than read. > >> > >> Frank > > > > Any Comments or rejection? @Kishon Vijay Abraham I > > I'd strongly recommend going with virtio/vhost based approach and standardizing > it IMO. But No progress in recent years on this path. At least, my patches make PCIe EP work as enet with minimized change. And NTB don't conflict with virtio/vhost solution. Frank > > Thanks, > Kishon > > > > > best regards > > Frank Li > > > >> > >>> > >>> Thanks, > >>> Kishon > >>> > >>> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com > >>>> > >>>> This just need EP to RC connections. > >>>> > >>>> ┌────────────┐ ┌─────────────────────────────────────┐ > >>>> │ │ │ │ > >>>> ├────────────┤ │ ┌──────────────┤ > >>>> │ NTB │ │ │ NTB │ > >>>> │ NetDev │ │ │ NetDev │ > >>>> ├────────────┤ │ ├──────────────┤ > >>>> │ NTB │ │ │ NTB │ > >>>> │ Transfer │ │ │ Transfer │ > >>>> ├────────────┤ │ ├──────────────┤ > >>>> │ │ │ │ │ > >>>> │ PCI NTB │ │ │ │ > >>>> │ EPF │ │ │ │ > >>>> │ Driver │ │ │ PCI Virtual │ > >>>> │ │ ├───────────────┐ │ NTB Driver │ > >>>> │ │ │ PCI EP NTB │◄────►│ │ > >>>> │ │ │ FN Driver │ │ │ > >>>> ├────────────┤ ├───────────────┤ ├──────────────┤ > >>>> │ │ │ │ │ │ > >>>> │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > >>>> │ │ PCI │ │ │ BUS │ > >>>> └────────────┘ └───────────────┴──────┴──────────────┘ > >>>> PCI RC PCI EP > >>>> > >>>> > >>>> > >>>> Frank Li (4): > >>>> PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > >>>> NTB: epf: Allow more flexibility in the memory BAR map method > >>>> PCI: endpoint: Support NTB transfer between RC and EP > >>>> Documentation: PCI: Add specification for the PCI vNTB function device > >>>> > >>>> Documentation/PCI/endpoint/index.rst | 2 + > >>>> .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > >>>> Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > >>>> drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > >>>> .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > >>>> drivers/pci/endpoint/functions/Kconfig | 11 + > >>>> drivers/pci/endpoint/functions/Makefile | 1 + > >>>> drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > >>>> 8 files changed, 1775 insertions(+), 14 deletions(-) > >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > >>>> create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > >>>> create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > >>>>
On Tue, Feb 22, 2022 at 10:23:51AM -0600, Frank Li wrote: > This implement NTB function for PCIe EP to RC connections. > The existed ntb epf need two PCI EPs and two PCI Host. > > This just need EP to RC connections. > > ┌────────────┐ ┌─────────────────────────────────────┐ > │ │ │ │ > ├────────────┤ │ ┌──────────────┤ > │ NTB │ │ │ NTB │ > │ NetDev │ │ │ NetDev │ > ├────────────┤ │ ├──────────────┤ > │ NTB │ │ │ NTB │ > │ Transfer │ │ │ Transfer │ > ├────────────┤ │ ├──────────────┤ > │ │ │ │ │ > │ PCI NTB │ │ │ │ > │ EPF │ │ │ │ > │ Driver │ │ │ PCI Virtual │ > │ │ ├───────────────┐ │ NTB Driver │ > │ │ │ PCI EP NTB │◄────►│ │ > │ │ │ FN Driver │ │ │ > ├────────────┤ ├───────────────┤ ├──────────────┤ > │ │ │ │ │ │ > │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ > │ │ PCI │ │ │ BUS │ > └────────────┘ └───────────────┴──────┴──────────────┘ > PCI RC PCI EP > > > > Frank Li (4): > PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address > NTB: epf: Allow more flexibility in the memory BAR map method > PCI: endpoint: Support NTB transfer between RC and EP > Documentation: PCI: Add specification for the PCI vNTB function device > > Documentation/PCI/endpoint/index.rst | 2 + > .../PCI/endpoint/pci-vntb-function.rst | 126 ++ > Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ > drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- > .../pci/controller/dwc/pcie-designware-ep.c | 10 +- > drivers/pci/endpoint/functions/Kconfig | 11 + > drivers/pci/endpoint/functions/Makefile | 1 + > drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ > 8 files changed, 1775 insertions(+), 14 deletions(-) > create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst > create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst > create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c > > -- > 2.24.0.rc1 > Sorry for the extremely long delay in response. This series has been in my ntb-next branch for some time and will be in my pull request for v5.20 which should be going out later today. Thanks, Jon
This implement NTB function for PCIe EP to RC connections. The existed ntb epf need two PCI EPs and two PCI Host. This just need EP to RC connections. ┌────────────┐ ┌─────────────────────────────────────┐ │ │ │ │ ├────────────┤ │ ┌──────────────┤ │ NTB │ │ │ NTB │ │ NetDev │ │ │ NetDev │ ├────────────┤ │ ├──────────────┤ │ NTB │ │ │ NTB │ │ Transfer │ │ │ Transfer │ ├────────────┤ │ ├──────────────┤ │ │ │ │ │ │ PCI NTB │ │ │ │ │ EPF │ │ │ │ │ Driver │ │ │ PCI Virtual │ │ │ ├───────────────┐ │ NTB Driver │ │ │ │ PCI EP NTB │◄────►│ │ │ │ │ FN Driver │ │ │ ├────────────┤ ├───────────────┤ ├──────────────┤ │ │ │ │ │ │ │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ │ │ PCI │ │ │ BUS │ └────────────┘ └───────────────┴──────┴──────────────┘ PCI RC PCI EP Frank Li (4): PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address NTB: epf: Allow more flexibility in the memory BAR map method PCI: endpoint: Support NTB transfer between RC and EP Documentation: PCI: Add specification for the PCI vNTB function device Documentation/PCI/endpoint/index.rst | 2 + .../PCI/endpoint/pci-vntb-function.rst | 126 ++ Documentation/PCI/endpoint/pci-vntb-howto.rst | 167 ++ drivers/ntb/hw/epf/ntb_hw_epf.c | 48 +- .../pci/controller/dwc/pcie-designware-ep.c | 10 +- drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++ 8 files changed, 1775 insertions(+), 14 deletions(-) create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c