mbox series

[V2,0/4] NTB function for PCIe RC to EP connection

Message ID 20220222162355.32369-1-Frank.Li@nxp.com (mailing list archive)
Headers show
Series NTB function for PCIe RC to EP connection | expand

Message

Frank Li Feb. 22, 2022, 4:23 p.m. UTC
This implement NTB function for PCIe EP to RC connections.
The existed ntb epf need two PCI EPs and two PCI Host.

This just need EP to RC connections.

    ┌────────────┐         ┌─────────────────────────────────────┐
    │            │         │                                     │
    ├────────────┤         │                      ┌──────────────┤
    │ NTB        │         │                      │ NTB          │
    │ NetDev     │         │                      │ NetDev       │
    ├────────────┤         │                      ├──────────────┤
    │ NTB        │         │                      │ NTB          │
    │ Transfer   │         │                      │ Transfer     │
    ├────────────┤         │                      ├──────────────┤
    │            │         │                      │              │
    │  PCI NTB   │         │                      │              │
    │    EPF     │         │                      │              │
    │   Driver   │         │                      │ PCI Virtual  │
    │            │         ├───────────────┐      │ NTB Driver   │
    │            │         │ PCI EP NTB    │◄────►│              │
    │            │         │  FN Driver    │      │              │
    ├────────────┤         ├───────────────┤      ├──────────────┤
    │            │         │               │      │              │
    │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
    │            │  PCI    │               │      │     BUS      │
    └────────────┘         └───────────────┴──────┴──────────────┘
        PCI RC                        PCI EP



Frank Li (4):
  PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
  NTB: epf: Allow more flexibility in the memory BAR map method
  PCI: endpoint: Support NTB transfer between RC and EP
  Documentation: PCI: Add specification for the PCI vNTB function device

 Documentation/PCI/endpoint/index.rst          |    2 +
 .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
 Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
 drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
 .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
 drivers/pci/endpoint/functions/Kconfig        |   11 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
 8 files changed, 1775 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
 create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c

Comments

Zhi Li March 10, 2022, 10:01 p.m. UTC | #1
On Tue, Feb 22, 2022 at 10:24 AM Frank Li <Frank.Li@nxp.com> wrote:
>
> This implement NTB function for PCIe EP to RC connections.
> The existed ntb epf need two PCI EPs and two PCI Host.
>
> This just need EP to RC connections.
>
>     ┌────────────┐         ┌─────────────────────────────────────┐
>     │            │         │                                     │
>     ├────────────┤         │                      ┌──────────────┤
>     │ NTB        │         │                      │ NTB          │
>     │ NetDev     │         │                      │ NetDev       │
>     ├────────────┤         │                      ├──────────────┤
>     │ NTB        │         │                      │ NTB          │
>     │ Transfer   │         │                      │ Transfer     │
>     ├────────────┤         │                      ├──────────────┤
>     │            │         │                      │              │
>     │  PCI NTB   │         │                      │              │
>     │    EPF     │         │                      │              │
>     │   Driver   │         │                      │ PCI Virtual  │
>     │            │         ├───────────────┐      │ NTB Driver   │
>     │            │         │ PCI EP NTB    │◄────►│              │
>     │            │         │  FN Driver    │      │              │
>     ├────────────┤         ├───────────────┤      ├──────────────┤
>     │            │         │               │      │              │
>     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
>     │            │  PCI    │               │      │     BUS      │
>     └────────────┘         └───────────────┴──────┴──────────────┘
>         PCI RC                        PCI EP
>
>
>
> Frank Li (4):
>   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
>   NTB: epf: Allow more flexibility in the memory BAR map method
>   PCI: endpoint: Support NTB transfer between RC and EP
>   Documentation: PCI: Add specification for the PCI vNTB function device
>

Ping

>  Documentation/PCI/endpoint/index.rst          |    2 +
>  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
>  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
>  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
>  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
>  drivers/pci/endpoint/functions/Kconfig        |   11 +
>  drivers/pci/endpoint/functions/Makefile       |    1 +
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
>  8 files changed, 1775 insertions(+), 14 deletions(-)
>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
>
> --
> 2.24.0.rc1
>
Zhi Li March 10, 2022, 10:07 p.m. UTC | #2
On Thu, Mar 10, 2022 at 4:01 PM Zhi Li <lznuaa@gmail.com> wrote:
>
> On Tue, Feb 22, 2022 at 10:24 AM Frank Li <Frank.Li@nxp.com> wrote:
> >
> > This implement NTB function for PCIe EP to RC connections.
> > The existed ntb epf need two PCI EPs and two PCI Host.
> >
> > This just need EP to RC connections.
> >
> >     ┌────────────┐         ┌─────────────────────────────────────┐
> >     │            │         │                                     │
> >     ├────────────┤         │                      ┌──────────────┤
> >     │ NTB        │         │                      │ NTB          │
> >     │ NetDev     │         │                      │ NetDev       │
> >     ├────────────┤         │                      ├──────────────┤
> >     │ NTB        │         │                      │ NTB          │
> >     │ Transfer   │         │                      │ Transfer     │
> >     ├────────────┤         │                      ├──────────────┤
> >     │            │         │                      │              │
> >     │  PCI NTB   │         │                      │              │
> >     │    EPF     │         │                      │              │
> >     │   Driver   │         │                      │ PCI Virtual  │
> >     │            │         ├───────────────┐      │ NTB Driver   │
> >     │            │         │ PCI EP NTB    │◄────►│              │
> >     │            │         │  FN Driver    │      │              │
> >     ├────────────┤         ├───────────────┤      ├──────────────┤
> >     │            │         │               │      │              │
> >     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
> >     │            │  PCI    │               │      │     BUS      │
> >     └────────────┘         └───────────────┴──────┴──────────────┘
> >         PCI RC                        PCI EP
> >
> >
> >
> > Frank Li (4):
> >   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
> >   NTB: epf: Allow more flexibility in the memory BAR map method
> >   PCI: endpoint: Support NTB transfer between RC and EP
> >   Documentation: PCI: Add specification for the PCI vNTB function device
> >
>

Update ntb mail list

>
> >  Documentation/PCI/endpoint/index.rst          |    2 +
> >  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
> >  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
> >  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
> >  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
> >  drivers/pci/endpoint/functions/Kconfig        |   11 +
> >  drivers/pci/endpoint/functions/Makefile       |    1 +
> >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
> >  8 files changed, 1775 insertions(+), 14 deletions(-)
> >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
> >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
> >  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
> >
> > --
> > 2.24.0.rc1
> >
Zhi Li April 4, 2022, 8:12 p.m. UTC | #3
On Thu, Mar 10, 2022 at 4:07 PM Zhi Li <lznuaa@gmail.com> wrote:
>
> On Thu, Mar 10, 2022 at 4:01 PM Zhi Li <lznuaa@gmail.com> wrote:
> >
> > On Tue, Feb 22, 2022 at 10:24 AM Frank Li <Frank.Li@nxp.com> wrote:
> > >
> > > This implement NTB function for PCIe EP to RC connections.
> > > The existed ntb epf need two PCI EPs and two PCI Host.
> > >
> > > This just need EP to RC connections.
> > >
> > >     ┌────────────┐         ┌─────────────────────────────────────┐
> > >     │            │         │                                     │
> > >     ├────────────┤         │                      ┌──────────────┤
> > >     │ NTB        │         │                      │ NTB          │
> > >     │ NetDev     │         │                      │ NetDev       │
> > >     ├────────────┤         │                      ├──────────────┤
> > >     │ NTB        │         │                      │ NTB          │
> > >     │ Transfer   │         │                      │ Transfer     │
> > >     ├────────────┤         │                      ├──────────────┤
> > >     │            │         │                      │              │
> > >     │  PCI NTB   │         │                      │              │
> > >     │    EPF     │         │                      │              │
> > >     │   Driver   │         │                      │ PCI Virtual  │
> > >     │            │         ├───────────────┐      │ NTB Driver   │
> > >     │            │         │ PCI EP NTB    │◄────►│              │
> > >     │            │         │  FN Driver    │      │              │
> > >     ├────────────┤         ├───────────────┤      ├──────────────┤
> > >     │            │         │               │      │              │
> > >     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
> > >     │            │  PCI    │               │      │     BUS      │
> > >     └────────────┘         └───────────────┴──────┴──────────────┘
> > >         PCI RC                        PCI EP
> > >
> > >
> > >
> > > Frank Li (4):
> > >   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
> > >   NTB: epf: Allow more flexibility in the memory BAR map method
> > >   PCI: endpoint: Support NTB transfer between RC and EP
> > >   Documentation: PCI: Add specification for the PCI vNTB function device
> > >
> >
>
> Update ntb mail list

Friendly ping!

Frank Li

>
> >
> > >  Documentation/PCI/endpoint/index.rst          |    2 +
> > >  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
> > >  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
> > >  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
> > >  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
> > >  drivers/pci/endpoint/functions/Kconfig        |   11 +
> > >  drivers/pci/endpoint/functions/Makefile       |    1 +
> > >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
> > >  8 files changed, 1775 insertions(+), 14 deletions(-)
> > >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
> > >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
> > >  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
> > >
> > > --
> > > 2.24.0.rc1
> > >
Kishon Vijay Abraham I April 5, 2022, 10:34 a.m. UTC | #4
Hi Frank Li,

On 22/02/22 9:53 pm, Frank Li wrote:
> This implement NTB function for PCIe EP to RC connections.
> The existed ntb epf need two PCI EPs and two PCI Host.

As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio
layer instead of trying to build on NTB layer (which is specific to RC<->RC
communication).

Are there any specific reasons for not taking that path?

Thanks,
Kishon

[1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com
> 
> This just need EP to RC connections.
> 
>     ┌────────────┐         ┌─────────────────────────────────────┐
>     │            │         │                                     │
>     ├────────────┤         │                      ┌──────────────┤
>     │ NTB        │         │                      │ NTB          │
>     │ NetDev     │         │                      │ NetDev       │
>     ├────────────┤         │                      ├──────────────┤
>     │ NTB        │         │                      │ NTB          │
>     │ Transfer   │         │                      │ Transfer     │
>     ├────────────┤         │                      ├──────────────┤
>     │            │         │                      │              │
>     │  PCI NTB   │         │                      │              │
>     │    EPF     │         │                      │              │
>     │   Driver   │         │                      │ PCI Virtual  │
>     │            │         ├───────────────┐      │ NTB Driver   │
>     │            │         │ PCI EP NTB    │◄────►│              │
>     │            │         │  FN Driver    │      │              │
>     ├────────────┤         ├───────────────┤      ├──────────────┤
>     │            │         │               │      │              │
>     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
>     │            │  PCI    │               │      │     BUS      │
>     └────────────┘         └───────────────┴──────┴──────────────┘
>         PCI RC                        PCI EP
> 
> 
> 
> Frank Li (4):
>   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
>   NTB: epf: Allow more flexibility in the memory BAR map method
>   PCI: endpoint: Support NTB transfer between RC and EP
>   Documentation: PCI: Add specification for the PCI vNTB function device
> 
>  Documentation/PCI/endpoint/index.rst          |    2 +
>  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
>  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
>  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
>  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
>  drivers/pci/endpoint/functions/Kconfig        |   11 +
>  drivers/pci/endpoint/functions/Makefile       |    1 +
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
>  8 files changed, 1775 insertions(+), 14 deletions(-)
>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
>
Zhi Li April 5, 2022, 3:35 p.m. UTC | #5
On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Hi Frank Li,
>
> On 22/02/22 9:53 pm, Frank Li wrote:
> > This implement NTB function for PCIe EP to RC connections.
> > The existed ntb epf need two PCI EPs and two PCI Host.
>
> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio
> layer instead of trying to build on NTB layer (which is specific to RC<->RC
> communication).
>
> Are there any specific reasons for not taking that path?

1. EP side work as vHOST mode.  vHost suppose access all memory of virtual io.
But there are only map windows on the EP side to access RC side
memory. You have to move
map windows for each access.  It is quite low efficiency.

2. So far as I know, virtio is still not DMA yet.  CPU access PCI
can't generate longer PCI TLP,
So the speed is quite slow.  NTB already has DMA support.  If you use
system level DMA,
no change is needed at NTB level.  If we want to use a PCI controller
embedded DMA,  some small
changes need if based on my other Designware PCI eDMA patches, which
are under review.

3. All the major data transfer of NTB is using write.  Because TLP
write needn't wait for complete,  write
performance is better than reading.  On our platform,  write
performance is about 10% better than  read.

Frank

>
> Thanks,
> Kishon
>
> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com
> >
> > This just need EP to RC connections.
> >
> >     ┌────────────┐         ┌─────────────────────────────────────┐
> >     │            │         │                                     │
> >     ├────────────┤         │                      ┌──────────────┤
> >     │ NTB        │         │                      │ NTB          │
> >     │ NetDev     │         │                      │ NetDev       │
> >     ├────────────┤         │                      ├──────────────┤
> >     │ NTB        │         │                      │ NTB          │
> >     │ Transfer   │         │                      │ Transfer     │
> >     ├────────────┤         │                      ├──────────────┤
> >     │            │         │                      │              │
> >     │  PCI NTB   │         │                      │              │
> >     │    EPF     │         │                      │              │
> >     │   Driver   │         │                      │ PCI Virtual  │
> >     │            │         ├───────────────┐      │ NTB Driver   │
> >     │            │         │ PCI EP NTB    │◄────►│              │
> >     │            │         │  FN Driver    │      │              │
> >     ├────────────┤         ├───────────────┤      ├──────────────┤
> >     │            │         │               │      │              │
> >     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
> >     │            │  PCI    │               │      │     BUS      │
> >     └────────────┘         └───────────────┴──────┴──────────────┘
> >         PCI RC                        PCI EP
> >
> >
> >
> > Frank Li (4):
> >   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
> >   NTB: epf: Allow more flexibility in the memory BAR map method
> >   PCI: endpoint: Support NTB transfer between RC and EP
> >   Documentation: PCI: Add specification for the PCI vNTB function device
> >
> >  Documentation/PCI/endpoint/index.rst          |    2 +
> >  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
> >  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
> >  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
> >  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
> >  drivers/pci/endpoint/functions/Kconfig        |   11 +
> >  drivers/pci/endpoint/functions/Makefile       |    1 +
> >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
> >  8 files changed, 1775 insertions(+), 14 deletions(-)
> >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
> >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
> >  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
> >
Zhi Li April 20, 2022, 8:22 p.m. UTC | #6
On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@gmail.com> wrote:
>
> On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >
> > Hi Frank Li,
> >
> > On 22/02/22 9:53 pm, Frank Li wrote:
> > > This implement NTB function for PCIe EP to RC connections.
> > > The existed ntb epf need two PCI EPs and two PCI Host.
> >
> > As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio
> > layer instead of trying to build on NTB layer (which is specific to RC<->RC
> > communication).
> >
> > Are there any specific reasons for not taking that path?
>
> 1. EP side work as vHOST mode.  vHost suppose access all memory of virtual io.
> But there are only map windows on the EP side to access RC side
> memory. You have to move
> map windows for each access.  It is quite low efficiency.
>
> 2. So far as I know, virtio is still not DMA yet.  CPU access PCI
> can't generate longer PCI TLP,
> So the speed is quite slow.  NTB already has DMA support.  If you use
> system level DMA,
> no change is needed at NTB level.  If we want to use a PCI controller
> embedded DMA,  some small
> changes need if based on my other Designware PCI eDMA patches, which
> are under review.
>
> 3. All the major data transfer of NTB is using write.  Because TLP
> write needn't wait for complete,  write
> performance is better than reading.  On our platform,  write
> performance is about 10% better than  read.
>
> Frank

Any Comments or rejection? @Kishon Vijay Abraham I

best regards
Frank Li

>
> >
> > Thanks,
> > Kishon
> >
> > [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com
> > >
> > > This just need EP to RC connections.
> > >
> > >     ┌────────────┐         ┌─────────────────────────────────────┐
> > >     │            │         │                                     │
> > >     ├────────────┤         │                      ┌──────────────┤
> > >     │ NTB        │         │                      │ NTB          │
> > >     │ NetDev     │         │                      │ NetDev       │
> > >     ├────────────┤         │                      ├──────────────┤
> > >     │ NTB        │         │                      │ NTB          │
> > >     │ Transfer   │         │                      │ Transfer     │
> > >     ├────────────┤         │                      ├──────────────┤
> > >     │            │         │                      │              │
> > >     │  PCI NTB   │         │                      │              │
> > >     │    EPF     │         │                      │              │
> > >     │   Driver   │         │                      │ PCI Virtual  │
> > >     │            │         ├───────────────┐      │ NTB Driver   │
> > >     │            │         │ PCI EP NTB    │◄────►│              │
> > >     │            │         │  FN Driver    │      │              │
> > >     ├────────────┤         ├───────────────┤      ├──────────────┤
> > >     │            │         │               │      │              │
> > >     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
> > >     │            │  PCI    │               │      │     BUS      │
> > >     └────────────┘         └───────────────┴──────┴──────────────┘
> > >         PCI RC                        PCI EP
> > >
> > >
> > >
> > > Frank Li (4):
> > >   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
> > >   NTB: epf: Allow more flexibility in the memory BAR map method
> > >   PCI: endpoint: Support NTB transfer between RC and EP
> > >   Documentation: PCI: Add specification for the PCI vNTB function device
> > >
> > >  Documentation/PCI/endpoint/index.rst          |    2 +
> > >  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
> > >  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
> > >  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
> > >  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
> > >  drivers/pci/endpoint/functions/Kconfig        |   11 +
> > >  drivers/pci/endpoint/functions/Makefile       |    1 +
> > >  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
> > >  8 files changed, 1775 insertions(+), 14 deletions(-)
> > >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
> > >  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
> > >  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
> > >
Kishon Vijay Abraham I April 22, 2022, 3:15 p.m. UTC | #7
Hi Frank,

On 21/04/22 1:52 am, Zhi Li wrote:
> On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@gmail.com> wrote:
>>
>> On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>>>
>>> Hi Frank Li,
>>>
>>> On 22/02/22 9:53 pm, Frank Li wrote:
>>>> This implement NTB function for PCIe EP to RC connections.
>>>> The existed ntb epf need two PCI EPs and two PCI Host.
>>>
>>> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio
>>> layer instead of trying to build on NTB layer (which is specific to RC<->RC
>>> communication).
>>>
>>> Are there any specific reasons for not taking that path?
>>
>> 1. EP side work as vHOST mode.  vHost suppose access all memory of virtual io.
>> But there are only map windows on the EP side to access RC side
>> memory. You have to move
>> map windows for each access.  It is quite low efficiency.

I'm not sure I quite get this. EP HW has limited outbound memory to access RC
memory irrespective of how we implement it. This is not a SW framework
limitation AFAICS.
>>
>> 2. So far as I know, virtio is still not DMA yet.  CPU access PCI
>> can't generate longer PCI TLP,
>> So the speed is quite slow.  NTB already has DMA support.  If you use
>> system level DMA,
>> no change is needed at NTB level.  If we want to use a PCI controller
>> embedded DMA,  some small
>> changes need if based on my other Designware PCI eDMA patches, which
>> are under review.

Adding dmaengine API to do memcopy should be simple to add in vhost/virtio
interface.
>>
>> 3. All the major data transfer of NTB is using write.  Because TLP
>> write needn't wait for complete,  write
>> performance is better than reading.  On our platform,  write
>> performance is about 10% better than  read.
>>
>> Frank
> 
> Any Comments or rejection? @Kishon Vijay Abraham I

I'd strongly recommend going with virtio/vhost based approach and standardizing
it IMO.

Thanks,
Kishon

> 
> best regards
> Frank Li
> 
>>
>>>
>>> Thanks,
>>> Kishon
>>>
>>> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com
>>>>
>>>> This just need EP to RC connections.
>>>>
>>>>     ┌────────────┐         ┌─────────────────────────────────────┐
>>>>     │            │         │                                     │
>>>>     ├────────────┤         │                      ┌──────────────┤
>>>>     │ NTB        │         │                      │ NTB          │
>>>>     │ NetDev     │         │                      │ NetDev       │
>>>>     ├────────────┤         │                      ├──────────────┤
>>>>     │ NTB        │         │                      │ NTB          │
>>>>     │ Transfer   │         │                      │ Transfer     │
>>>>     ├────────────┤         │                      ├──────────────┤
>>>>     │            │         │                      │              │
>>>>     │  PCI NTB   │         │                      │              │
>>>>     │    EPF     │         │                      │              │
>>>>     │   Driver   │         │                      │ PCI Virtual  │
>>>>     │            │         ├───────────────┐      │ NTB Driver   │
>>>>     │            │         │ PCI EP NTB    │◄────►│              │
>>>>     │            │         │  FN Driver    │      │              │
>>>>     ├────────────┤         ├───────────────┤      ├──────────────┤
>>>>     │            │         │               │      │              │
>>>>     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
>>>>     │            │  PCI    │               │      │     BUS      │
>>>>     └────────────┘         └───────────────┴──────┴──────────────┘
>>>>         PCI RC                        PCI EP
>>>>
>>>>
>>>>
>>>> Frank Li (4):
>>>>   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
>>>>   NTB: epf: Allow more flexibility in the memory BAR map method
>>>>   PCI: endpoint: Support NTB transfer between RC and EP
>>>>   Documentation: PCI: Add specification for the PCI vNTB function device
>>>>
>>>>  Documentation/PCI/endpoint/index.rst          |    2 +
>>>>  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
>>>>  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
>>>>  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
>>>>  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
>>>>  drivers/pci/endpoint/functions/Kconfig        |   11 +
>>>>  drivers/pci/endpoint/functions/Makefile       |    1 +
>>>>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
>>>>  8 files changed, 1775 insertions(+), 14 deletions(-)
>>>>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
>>>>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
>>>>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
>>>>
Zhi Li April 22, 2022, 3:36 p.m. UTC | #8
On Fri, Apr 22, 2022 at 10:15 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
>
> Hi Frank,
>
> On 21/04/22 1:52 am, Zhi Li wrote:
> > On Tue, Apr 5, 2022 at 10:35 AM Zhi Li <lznuaa@gmail.com> wrote:
> >>
> >> On Tue, Apr 5, 2022 at 5:34 AM Kishon Vijay Abraham I <kishon@ti.com> wrote:
> >>>
> >>> Hi Frank Li,
> >>>
> >>> On 22/02/22 9:53 pm, Frank Li wrote:
> >>>> This implement NTB function for PCIe EP to RC connections.
> >>>> The existed ntb epf need two PCI EPs and two PCI Host.
> >>>
> >>> As I had earlier mentioned in [1], IMHO ideal solution would be build on virtio
> >>> layer instead of trying to build on NTB layer (which is specific to RC<->RC
> >>> communication).
> >>>
> >>> Are there any specific reasons for not taking that path?
> >>
> >> 1. EP side work as vHOST mode.  vHost suppose access all memory of virtual io.
> >> But there are only map windows on the EP side to access RC side
> >> memory. You have to move
> >> map windows for each access.  It is quite low efficiency.
>
> I'm not sure I quite get this. EP HW has limited outbound memory to access RC
> memory irrespective of how we implement it. This is not a SW framework
> limitation AFAICS.

Almost all EP HW have limited outbound memory windows to access RC.
We face transfer efficiency problems if we stick into vhost.

> >>
> >> 2. So far as I know, virtio is still not DMA yet.  CPU access PCI
> >> can't generate longer PCI TLP,
> >> So the speed is quite slow.  NTB already has DMA support.  If you use
> >> system level DMA,
> >> no change is needed at NTB level.  If we want to use a PCI controller
> >> embedded DMA,  some small
> >> changes need if based on my other Designware PCI eDMA patches, which
> >> are under review.
>
> Adding dmaengine API to do memcopy should be simple to add in vhost/virtio
> interface.
> >>
> >> 3. All the major data transfer of NTB is using write.  Because TLP
> >> write needn't wait for complete,  write
> >> performance is better than reading.  On our platform,  write
> >> performance is about 10% better than  read.
> >>
> >> Frank
> >
> > Any Comments or rejection? @Kishon Vijay Abraham I
>
> I'd strongly recommend going with virtio/vhost based approach and standardizing
> it IMO.

But No progress in recent years on this path.  At least, my patches make PCIe EP
work as enet with minimized change.
And NTB don't conflict with virtio/vhost solution.

Frank
>
> Thanks,
> Kishon
>
> >
> > best regards
> > Frank Li
> >
> >>
> >>>
> >>> Thanks,
> >>> Kishon
> >>>
> >>> [1] -> https://lore.kernel.org/r/459745d1-9fe7-e792-3532-33ee9552bc4d@ti.com
> >>>>
> >>>> This just need EP to RC connections.
> >>>>
> >>>>     ┌────────────┐         ┌─────────────────────────────────────┐
> >>>>     │            │         │                                     │
> >>>>     ├────────────┤         │                      ┌──────────────┤
> >>>>     │ NTB        │         │                      │ NTB          │
> >>>>     │ NetDev     │         │                      │ NetDev       │
> >>>>     ├────────────┤         │                      ├──────────────┤
> >>>>     │ NTB        │         │                      │ NTB          │
> >>>>     │ Transfer   │         │                      │ Transfer     │
> >>>>     ├────────────┤         │                      ├──────────────┤
> >>>>     │            │         │                      │              │
> >>>>     │  PCI NTB   │         │                      │              │
> >>>>     │    EPF     │         │                      │              │
> >>>>     │   Driver   │         │                      │ PCI Virtual  │
> >>>>     │            │         ├───────────────┐      │ NTB Driver   │
> >>>>     │            │         │ PCI EP NTB    │◄────►│              │
> >>>>     │            │         │  FN Driver    │      │              │
> >>>>     ├────────────┤         ├───────────────┤      ├──────────────┤
> >>>>     │            │         │               │      │              │
> >>>>     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
> >>>>     │            │  PCI    │               │      │     BUS      │
> >>>>     └────────────┘         └───────────────┴──────┴──────────────┘
> >>>>         PCI RC                        PCI EP
> >>>>
> >>>>
> >>>>
> >>>> Frank Li (4):
> >>>>   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
> >>>>   NTB: epf: Allow more flexibility in the memory BAR map method
> >>>>   PCI: endpoint: Support NTB transfer between RC and EP
> >>>>   Documentation: PCI: Add specification for the PCI vNTB function device
> >>>>
> >>>>  Documentation/PCI/endpoint/index.rst          |    2 +
> >>>>  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
> >>>>  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
> >>>>  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
> >>>>  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
> >>>>  drivers/pci/endpoint/functions/Kconfig        |   11 +
> >>>>  drivers/pci/endpoint/functions/Makefile       |    1 +
> >>>>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
> >>>>  8 files changed, 1775 insertions(+), 14 deletions(-)
> >>>>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
> >>>>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
> >>>>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
> >>>>
Jon Mason Aug. 12, 2022, 2:02 p.m. UTC | #9
On Tue, Feb 22, 2022 at 10:23:51AM -0600, Frank Li wrote:
> This implement NTB function for PCIe EP to RC connections.
> The existed ntb epf need two PCI EPs and two PCI Host.
> 
> This just need EP to RC connections.
> 
>     ┌────────────┐         ┌─────────────────────────────────────┐
>     │            │         │                                     │
>     ├────────────┤         │                      ┌──────────────┤
>     │ NTB        │         │                      │ NTB          │
>     │ NetDev     │         │                      │ NetDev       │
>     ├────────────┤         │                      ├──────────────┤
>     │ NTB        │         │                      │ NTB          │
>     │ Transfer   │         │                      │ Transfer     │
>     ├────────────┤         │                      ├──────────────┤
>     │            │         │                      │              │
>     │  PCI NTB   │         │                      │              │
>     │    EPF     │         │                      │              │
>     │   Driver   │         │                      │ PCI Virtual  │
>     │            │         ├───────────────┐      │ NTB Driver   │
>     │            │         │ PCI EP NTB    │◄────►│              │
>     │            │         │  FN Driver    │      │              │
>     ├────────────┤         ├───────────────┤      ├──────────────┤
>     │            │         │               │      │              │
>     │  PCI BUS   │ ◄─────► │  PCI EP BUS   │      │  Virtual PCI │
>     │            │  PCI    │               │      │     BUS      │
>     └────────────┘         └───────────────┴──────┴──────────────┘
>         PCI RC                        PCI EP
> 
> 
> 
> Frank Li (4):
>   PCI: designware-ep: Allow pci_epc_set_bar() update inbound map address
>   NTB: epf: Allow more flexibility in the memory BAR map method
>   PCI: endpoint: Support NTB transfer between RC and EP
>   Documentation: PCI: Add specification for the PCI vNTB function device
> 
>  Documentation/PCI/endpoint/index.rst          |    2 +
>  .../PCI/endpoint/pci-vntb-function.rst        |  126 ++
>  Documentation/PCI/endpoint/pci-vntb-howto.rst |  167 ++
>  drivers/ntb/hw/epf/ntb_hw_epf.c               |   48 +-
>  .../pci/controller/dwc/pcie-designware-ep.c   |   10 +-
>  drivers/pci/endpoint/functions/Kconfig        |   11 +
>  drivers/pci/endpoint/functions/Makefile       |    1 +
>  drivers/pci/endpoint/functions/pci-epf-vntb.c | 1424 +++++++++++++++++
>  8 files changed, 1775 insertions(+), 14 deletions(-)
>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst
>  create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c
> 
> -- 
> 2.24.0.rc1
> 

Sorry for the extremely long delay in response.  This series has been
in my ntb-next branch for some time and will be in my pull request for
v5.20 which should be going out later today.

Thanks,
Jon