[RFC,0/3] Introduce a PCIe endpoint virtio console

Message

Shunsuke Mie April 14, 2023, 12:39 p.m. UTC

PCIe endpoint framework provides APIs to implement PCIe endpoint function.
This framework allows defining various PCIe endpoint function behaviors in
software. This patch extend the framework for virtio pci device. The
virtio is defined to communicate guest on virtual machine and host side.
Advantage of the virtio is the efficiency of data transfer and the conciseness
of implementation device using software. It also be applied to PCIe
endpoint function.

We designed and implemented a PCIe EP virtio console function driver using
the extended PCIe endpoint framework for virtio. It can be communicate
host and endpoint over virtio as console.

An architecture of the function driver is following:

 ┌────────────┐         ┌──────────────────────┬────────────┐
 │virtio      │         │                      │virtio      │
 │console drv │         ├───────────────┐      │console drv │
 ├────────────┤         │(virtio console│      ├────────────┤
 │ virtio bus │         │ device)       │◄────►│ virtio bus │
 ├────────────┤         ├---------------┤      └────────────┤
 │            │         │ pci ep virtio │                   │
 │  pci bus   │         │  console drv  │                   │
 │            │  pcie   ├───────────────┤                   │
 │            │ ◄─────► │  pci ep Bus   │                   │
 └────────────┘         └───────────────┴───────────────────┘
   PCIe Root              PCIe Endpoint

Introduced driver is `pci ep virtio console drv` in the figure. It works
as ep function for PCIe root and virtual virtio console device for PCIe
endpoint. Each side of virtio console driver has virtqueue, and
introduced driver transfers data on the virtqueue to each other. A data
on root tx queue is transfered to endpoint rx queue and vice versa.

This patchset is depend follwing patches which are under discussion.

- [RFC PATCH 0/3] Deal with alignment restriction on EP side
link: https://lore.kernel.org/linux-pci/[e-mail]/
- [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
link: https://lore.kernel.org/virtualization/[e-mail]/

First of this patchset is introduce a helper function to realize pci
virtio function using PCIe endpoint framework. The second one is adding
a missing definition for virtio pci header. The last one is for PCIe
endpoint virtio console driver.

This is tested on linux-20230406 and RCar S4 board as PCIe endpoint.

Shunsuke Mie (3):
  PCI: endpoint: introduce a helper to implement pci ep virtio function
  virtio_pci: add a definition of queue flag in ISR
  PCI: endpoint: Add EP function driver to provide virtio-console
    functionality

 drivers/pci/endpoint/functions/Kconfig        |  19 +
 drivers/pci/endpoint/functions/Makefile       |   2 +
 drivers/pci/endpoint/functions/pci-epf-vcon.c | 554 ++++++++++++++++++
 .../pci/endpoint/functions/pci-epf-virtio.c   | 469 +++++++++++++++
 .../pci/endpoint/functions/pci-epf-virtio.h   | 123 ++++
 include/uapi/linux/virtio_pci.h               |   3 +
 6 files changed, 1170 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h

Comments

Frank Li April 14, 2023, 2:39 p.m. UTC | #1

> -----Original Message-----
> From: Shunsuke Mie <[e-mail]>
> Sent: Friday, April 14, 2023 7:39 AM
> To: Lorenzo Pieralisi <[e-mail]>
> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
> Jason Wang <[e-mail]>; Shunsuke Mie <[e-mail]>;
> Frank Li <[e-mail]>; Jon Mason <[e-mail]>; Randy
> Dunlap <[e-mail]>; Ren Zhijie <[e-mail]>;
> [e-mail]; [e-mail];
> [e-mail]
> Subject: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
>
> Caution: EXT Email
>
> PCIe endpoint framework provides APIs to implement PCIe endpoint
> function.
> This framework allows defining various PCIe endpoint function behaviors in
> software. This patch extend the framework for virtio pci device. The
> virtio is defined to communicate guest on virtual machine and host side.
> Advantage of the virtio is the efficiency of data transfer and the conciseness
> of implementation device using software. It also be applied to PCIe
> endpoint function.
>
> We designed and implemented a PCIe EP virtio console function driver using
> the extended PCIe endpoint framework for virtio. It can be communicate
> host and endpoint over virtio as console.
>
> An architecture of the function driver is following:
>
>  ┌────────────┐         ┌──────────────
> ────────┬────────────┐
>  │virtio      │         │                      │virtio      │
>  │console drv │         ├───────────────┐      │console
> drv │
>  ├────────────┤         │(virtio console│      ├─────
> ───────┤
>  │ virtio bus │         │ device)       │◄────►│ virtio bus │
>  ├────────────┤         ├---------------┤      └──────
> ──────┤
>  │            │         │ pci ep virtio │                   │
>  │  pci bus   │         │  console drv  │                   │
>  │            │  pcie   ├───────────────┤                   │
>  │            │ ◄─────► │  pci ep Bus   │                   │
>  └────────────┘         └──────────────
> ─┴───────────────────┘
>    PCIe Root              PCIe Endpoint
>

[Frank Li] Some basic question,
I see you call register_virtio_device at epf_vcon_setup_vdev,
Why call it as virtio console?  I suppose it should be virtiobus directly?

Previous you use virtio-net, why change to virtio-console here?  Does it matter?
All virtio-XXX should work?

You removed EDMA support this version?


> Introduced driver is `pci ep virtio console drv` in the figure. It works
> as ep function for PCIe root and virtual virtio console device for PCIe
> endpoint. Each side of virtio console driver has virtqueue, and
> introduced driver transfers data on the virtqueue to each other. A data
> on root tx queue is transfered to endpoint rx queue and vice versa.
>
> This patchset is depend follwing patches which are under discussion.
>
> - [RFC PATCH 0/3] Deal with alignment restriction on EP side
> link:
> https://lore.k/
> ernel.org%2Flinux-pci%2F20230113090350.1103494-1-
> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=jYgy%2Bxk84ZXZRVfqm0GCXoRnCTLMrX4zTfV%2Bs5Mmsvo
> %3D&reserved=0
> - [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
> link:
> https://lore.k/
> ernel.org%2Fvirtualization%2F20230202090934.549556-1-
> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> %7C%7C&sdata=K4El76GSAGtsWkNBXJK5%2Fn7flCN20eEMZpZYTX2WIZ0%3
> D&reserved=0
>
> First of this patchset is introduce a helper function to realize pci
> virtio function using PCIe endpoint framework. The second one is adding
> a missing definition for virtio pci header. The last one is for PCIe
> endpoint virtio console driver.
>
> This is tested on linux-20230406 and RCar S4 board as PCIe endpoint.
>
> Shunsuke Mie (3):
>   PCI: endpoint: introduce a helper to implement pci ep virtio function
>   virtio_pci: add a definition of queue flag in ISR
>   PCI: endpoint: Add EP function driver to provide virtio-console
>     functionality
>
>  drivers/pci/endpoint/functions/Kconfig        |  19 +
>  drivers/pci/endpoint/functions/Makefile       |   2 +
>  drivers/pci/endpoint/functions/pci-epf-vcon.c | 554 ++++++++++++++++++
>  .../pci/endpoint/functions/pci-epf-virtio.c   | 469 +++++++++++++++
>  .../pci/endpoint/functions/pci-epf-virtio.h   | 123 ++++
>  include/uapi/linux/virtio_pci.h               |   3 +
>  6 files changed, 1170 insertions(+)
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
>
> --
> 2.25.1

Shunsuke Mie April 17, 2023, 2:11 a.m. UTC | #2

On 2023/04/14 23:39, Frank Li wrote:
>
>> -----Original Message-----
>> From: Shunsuke Mie <[e-mail]>
>> Sent: Friday, April 14, 2023 7:39 AM
>> To: Lorenzo Pieralisi <[e-mail]>
>> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
>> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
>> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
>> Jason Wang <[e-mail]>; Shunsuke Mie <[e-mail]>;
>> Frank Li <[e-mail]>; Jon Mason <[e-mail]>; Randy
>> Dunlap <[e-mail]>; Ren Zhijie <[e-mail]>;
>> [e-mail]; [e-mail];
>> [e-mail]
>> Subject: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
>>
>> Caution: EXT Email
>>
>> PCIe endpoint framework provides APIs to implement PCIe endpoint
>> function.
>> This framework allows defining various PCIe endpoint function behaviors in
>> software. This patch extend the framework for virtio pci device. The
>> virtio is defined to communicate guest on virtual machine and host side.
>> Advantage of the virtio is the efficiency of data transfer and the conciseness
>> of implementation device using software. It also be applied to PCIe
>> endpoint function.
>>
>> We designed and implemented a PCIe EP virtio console function driver using
>> the extended PCIe endpoint framework for virtio. It can be communicate
>> host and endpoint over virtio as console.
>>
>> An architecture of the function driver is following:
>>
>>   ┌────────────┐         ┌──────────────
>> ────────┬────────────┐
>>   │virtio      │         │                      │virtio      │
>>   │console drv │         ├───────────────┐      │console
>> drv │
>>   ├────────────┤         │(virtio console│      ├─────
>> ───────┤
>>   │ virtio bus │         │ device)       │◄────►│ virtio bus │
>>   ├────────────┤         ├---------------┤      └──────
>> ──────┤
>>   │            │         │ pci ep virtio │                   │
>>   │  pci bus   │         │  console drv  │                   │
>>   │            │  pcie   ├───────────────┤                   │
>>   │            │ ◄─────► │  pci ep Bus   │                   │
>>   └────────────┘         └──────────────
>> ─┴───────────────────┘
>>     PCIe Root              PCIe Endpoint
>>
> [Frank Li] Some basic question,
> I see you call register_virtio_device at epf_vcon_setup_vdev,
> Why call it as virtio console?  I suppose it should be virtiobus directly?

I'm sorry I didn't understand your question. What do you mean the 
virtiobus directly?

>
> Previous you use virtio-net, why change to virtio-console here?  Does it matter?

No, it doesn't. Just I'd like to break down the changes into smaller 
steps to make it easier to review and merge the changes.

As a first step, I propose adding a simplest virtio function driver with 
the extension defined in pci-epf-virtio.{h,c}.

> All virtio-XXX should work?
Yes, the extension is designed to use any type of virtio device.
>
> You removed EDMA support this version?
I planed the support will be added with epf virtio-net patches.
>
>> Introduced driver is `pci ep virtio console drv` in the figure. It works
>> as ep function for PCIe root and virtual virtio console device for PCIe
>> endpoint. Each side of virtio console driver has virtqueue, and
>> introduced driver transfers data on the virtqueue to each other. A data
>> on root tx queue is transfered to endpoint rx queue and vice versa.
>>
>> This patchset is depend follwing patches which are under discussion.
>>
>> - [RFC PATCH 0/3] Deal with alignment restriction on EP side
>> link:
>> https://lore.k/
>> ernel.org%2Flinux-pci%2F20230113090350.1103494-1-
>> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
>> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
>> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
>> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>> %7C%7C&sdata=jYgy%2Bxk84ZXZRVfqm0GCXoRnCTLMrX4zTfV%2Bs5Mmsvo
>> %3D&reserved=0
>> - [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
>> link:
>> https://lore.k/
>> ernel.org%2Fvirtualization%2F20230202090934.549556-1-
>> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
>> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
>> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
>> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>> %7C%7C&sdata=K4El76GSAGtsWkNBXJK5%2Fn7flCN20eEMZpZYTX2WIZ0%3
>> D&reserved=0
>>
>> First of this patchset is introduce a helper function to realize pci
>> virtio function using PCIe endpoint framework. The second one is adding
>> a missing definition for virtio pci header. The last one is for PCIe
>> endpoint virtio console driver.
>>
>> This is tested on linux-20230406 and RCar S4 board as PCIe endpoint.
>>
>> Shunsuke Mie (3):
>>    PCI: endpoint: introduce a helper to implement pci ep virtio function
>>    virtio_pci: add a definition of queue flag in ISR
>>    PCI: endpoint: Add EP function driver to provide virtio-console
>>      functionality
>>
>>   drivers/pci/endpoint/functions/Kconfig        |  19 +
>>   drivers/pci/endpoint/functions/Makefile       |   2 +
>>   drivers/pci/endpoint/functions/pci-epf-vcon.c | 554 ++++++++++++++++++
>>   .../pci/endpoint/functions/pci-epf-virtio.c   | 469 +++++++++++++++
>>   .../pci/endpoint/functions/pci-epf-virtio.h   | 123 ++++
>>   include/uapi/linux/virtio_pci.h               |   3 +
>>   6 files changed, 1170 insertions(+)
>>   create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
>>   create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
>>   create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
>>
>> --
>> 2.25.1

Frank Li April 17, 2023, 3:19 p.m. UTC | #3

> -----Original Message-----
> From: Shunsuke Mie <[e-mail]>
> Sent: Sunday, April 16, 2023 9:12 PM
> To: Frank Li <[e-mail]>; Lorenzo Pieralisi <[e-mail]>
> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
> Jason Wang <[e-mail]>; Jon Mason <[e-mail]>;
> Randy Dunlap <[e-mail]>; Ren Zhijie
> <[e-mail]>; [e-mail]; linux-
> [e-mail]; [e-mail]
> Subject: Re: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
>
> Caution: EXT Email
>
> On 2023/04/14 23:39, Frank Li wrote:
> >
> >> -----Original Message-----
> >> From: Shunsuke Mie <[e-mail]>
> >> Sent: Friday, April 14, 2023 7:39 AM
> >> To: Lorenzo Pieralisi <[e-mail]>
> >> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> >> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
> >> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
> >> Jason Wang <[e-mail]>; Shunsuke Mie <[e-mail]>;
> >> Frank Li <[e-mail]>; Jon Mason <[e-mail]>; Randy
> >> Dunlap <[e-mail]>; Ren Zhijie <[e-mail]>;
> >> [e-mail]; [e-mail];
> >> [e-mail]
> >> Subject: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
> >>
> >> Caution: EXT Email
> >>
> >> PCIe endpoint framework provides APIs to implement PCIe endpoint
> >> function.
> >> This framework allows defining various PCIe endpoint function behaviors
> in
> >> software. This patch extend the framework for virtio pci device. The
> >> virtio is defined to communicate guest on virtual machine and host side.
> >> Advantage of the virtio is the efficiency of data transfer and the
> conciseness
> >> of implementation device using software. It also be applied to PCIe
> >> endpoint function.
> >>
> >> We designed and implemented a PCIe EP virtio console function driver
> using
> >> the extended PCIe endpoint framework for virtio. It can be communicate
> >> host and endpoint over virtio as console.
> >>
> >> An architecture of the function driver is following:
> >>
> >>   ┌────────────┐         ┌────────────
> ──
> >> ────────┬────────────┐
> >>   │virtio      │         │                      │virtio      │
> >>   │console drv │         ├───────────────┐      │
> console
> >> drv │
> >>   ├────────────┤         │(virtio console│      ├───
> ──
> >> ───────┤
> >>   │ virtio bus │         │ device)       │◄────►│ virtio bus │
> >>   ├────────────┤         ├---------------┤      └────
> ──
> >> ──────┤
> >>   │            │         │ pci ep virtio │                   │
> >>   │  pci bus   │         │  console drv  │                   │
> >>   │            │  pcie   ├───────────────┤                   │
> >>   │            │ ◄─────► │  pci ep Bus   │                   │
> >>   └────────────┘         └────────────
> ──
> >> ─┴───────────────────┘
> >>     PCIe Root              PCIe Endpoint
> >>
> > [Frank Li] Some basic question,
> > I see you call register_virtio_device at epf_vcon_setup_vdev,
> > Why call it as virtio console?  I suppose it should be virtiobus directly?
>
> I'm sorry I didn't understand your question. What do you mean the
> virtiobus directly?

I go through your code again.  I think I understand why you need pci-epf-vcon.c.
Actually,  my means is like virtio_mmio_probe.

vm_dev->vdev.id.device = readl(vm_dev->base + VIRTIO_MMIO_DEVICE_ID);
vm_dev->vdev.id.vendor = readl(vm_dev->base + VIRTIO_MMIO_VENDOR_ID);

I am not sure that if VIRTIO_MMIO_VENDOR_ID and VIRTIO_MMIO_DEVICE_ID
reuse PCI's vendor ID and Device ID.  If yes, you can directly get such information
from epf.  If no,  a customer field can been added at epf driver.

So you needn't write pci-epf-vcon  and pci-epf-vnet .....

Of cause it will be wonderful if directly use virtio_mmio_probe by dynmatic create platform
Devices.  It may have some difficult because pci memory map requirement.

>
> >
> > Previous you use virtio-net, why change to virtio-console here?  Does it
> matter?
>
> No, it doesn't. Just I'd like to break down the changes into smaller
> steps to make it easier to review and merge the changes.
>
> As a first step, I propose adding a simplest virtio function driver with
> the extension defined in pci-epf-virtio.{h,c}.
>
> > All virtio-XXX should work?
> Yes, the extension is designed to use any type of virtio device.
> >
> > You removed EDMA support this version?
> I planed the support will be added with epf virtio-net patches.
> >
> >> Introduced driver is `pci ep virtio console drv` in the figure. It works
> >> as ep function for PCIe root and virtual virtio console device for PCIe
> >> endpoint. Each side of virtio console driver has virtqueue, and
> >> introduced driver transfers data on the virtqueue to each other. A data
> >> on root tx queue is transfered to endpoint rx queue and vice versa.
> >>
> >> This patchset is depend follwing patches which are under discussion.
> >>
> >> - [RFC PATCH 0/3] Deal with alignment restriction on EP side
> >> link:
> >>
> https://lore.k/
> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908db3
> ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
> 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
> >> ernel.org%2Flinux-pci%2F20230113090350.1103494-1-
> >>
> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> >>
> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> >>
> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> >>
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> >> %7C%7C&sdata=jYgy%2Bxk84ZXZRVfqm0GCXoRnCTLMrX4zTfV%2Bs5Mm
> svo
> >> %3D&reserved=0
> >> - [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
> >> link:
> >>
> https://lore.k/
> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908db3
> ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
> 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
> >> ernel.org%2Fvirtualization%2F20230202090934.549556-1-
> >>
> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> >>
> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> >>
> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> >>
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> >> %7C%7C&sdata=K4El76GSAGtsWkNBXJK5%2Fn7flCN20eEMZpZYTX2WIZ0
> %3
> >> D&reserved=0
> >>
> >> First of this patchset is introduce a helper function to realize pci
> >> virtio function using PCIe endpoint framework. The second one is adding
> >> a missing definition for virtio pci header. The last one is for PCIe
> >> endpoint virtio console driver.
> >>
> >> This is tested on linux-20230406 and RCar S4 board as PCIe endpoint.
> >>
> >> Shunsuke Mie (3):
> >>    PCI: endpoint: introduce a helper to implement pci ep virtio function
> >>    virtio_pci: add a definition of queue flag in ISR
> >>    PCI: endpoint: Add EP function driver to provide virtio-console
> >>      functionality
> >>
> >>   drivers/pci/endpoint/functions/Kconfig        |  19 +
> >>   drivers/pci/endpoint/functions/Makefile       |   2 +
> >>   drivers/pci/endpoint/functions/pci-epf-vcon.c | 554
> ++++++++++++++++++
> >>   .../pci/endpoint/functions/pci-epf-virtio.c   | 469 +++++++++++++++
> >>   .../pci/endpoint/functions/pci-epf-virtio.h   | 123 ++++
> >>   include/uapi/linux/virtio_pci.h               |   3 +
> >>   6 files changed, 1170 insertions(+)
> >>   create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
> >>   create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
> >>   create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
> >>
> >> --
> >> 2.25.1

Shunsuke Mie April 18, 2023, 10:31 a.m. UTC | #4

On 2023/04/18 0:19, Frank Li wrote:
>
>> -----Original Message-----
>> From: Shunsuke Mie <[e-mail]>
>> Sent: Sunday, April 16, 2023 9:12 PM
>> To: Frank Li <[e-mail]>; Lorenzo Pieralisi <[e-mail]>
>> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
>> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
>> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
>> Jason Wang <[e-mail]>; Jon Mason <[e-mail]>;
>> Randy Dunlap <[e-mail]>; Ren Zhijie
>> <[e-mail]>; [e-mail]; linux-
>> [e-mail]; [e-mail]
>> Subject: Re: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
>>
>> Caution: EXT Email
>>
>> On 2023/04/14 23:39, Frank Li wrote:
>>>> -----Original Message-----
>>>> From: Shunsuke Mie <[e-mail]>
>>>> Sent: Friday, April 14, 2023 7:39 AM
>>>> To: Lorenzo Pieralisi <[e-mail]>
>>>> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
>>>> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
>>>> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
>>>> Jason Wang <[e-mail]>; Shunsuke Mie <[e-mail]>;
>>>> Frank Li <[e-mail]>; Jon Mason <[e-mail]>; Randy
>>>> Dunlap <[e-mail]>; Ren Zhijie <[e-mail]>;
>>>> [e-mail]; [e-mail];
>>>> [e-mail]
>>>> Subject: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
>>>>
>>>> Caution: EXT Email
>>>>
>>>> PCIe endpoint framework provides APIs to implement PCIe endpoint
>>>> function.
>>>> This framework allows defining various PCIe endpoint function behaviors
>> in
>>>> software. This patch extend the framework for virtio pci device. The
>>>> virtio is defined to communicate guest on virtual machine and host side.
>>>> Advantage of the virtio is the efficiency of data transfer and the
>> conciseness
>>>> of implementation device using software. It also be applied to PCIe
>>>> endpoint function.
>>>>
>>>> We designed and implemented a PCIe EP virtio console function driver
>> using
>>>> the extended PCIe endpoint framework for virtio. It can be communicate
>>>> host and endpoint over virtio as console.
>>>>
>>>> An architecture of the function driver is following:
>>>>
>>>>    ┌────────────┐         ┌────────────
>> ──
>>>> ────────┬────────────┐
>>>>    │virtio      │         │                      │virtio      │
>>>>    │console drv │         ├───────────────┐      │
>> console
>>>> drv │
>>>>    ├────────────┤         │(virtio console│      ├───
>> ──
>>>> ───────┤
>>>>    │ virtio bus │         │ device)       │◄────►│ virtio bus │
>>>>    ├────────────┤         ├---------------┤      └────
>> ──
>>>> ──────┤
>>>>    │            │         │ pci ep virtio │                   │
>>>>    │  pci bus   │         │  console drv  │                   │
>>>>    │            │  pcie   ├───────────────┤                   │
>>>>    │            │ ◄─────► │  pci ep Bus   │                   │
>>>>    └────────────┘         └────────────
>> ──
>>>> ─┴───────────────────┘
>>>>      PCIe Root              PCIe Endpoint
>>>>
>>> [Frank Li] Some basic question,
>>> I see you call register_virtio_device at epf_vcon_setup_vdev,
>>> Why call it as virtio console?  I suppose it should be virtiobus directly?
>> I'm sorry I didn't understand your question. What do you mean the
>> virtiobus directly?
> I go through your code again.  I think I understand why you need pci-epf-vcon.c.
> Actually,  my means is like virtio_mmio_probe.
>
> vm_dev->vdev.id.device = readl(vm_dev->base + VIRTIO_MMIO_DEVICE_ID);
> vm_dev->vdev.id.vendor = readl(vm_dev->base + VIRTIO_MMIO_VENDOR_ID);
>
> I am not sure that if VIRTIO_MMIO_VENDOR_ID and VIRTIO_MMIO_DEVICE_ID
> reuse PCI's vendor ID and Device ID.  If yes, you can directly get such information
> from epf.  If no,  a customer field can been added at epf driver.
>
> So you needn't write pci-epf-vcon  and pci-epf-vnet .....
>
> Of cause it will be wonderful if directly use virtio_mmio_probe by dynmatic create platform
> Devices.  It may have some difficult because pci memory map requirement.
I think that some fields are shared between the vdev and epf device.
However, we need to implement device emulation because each virtio device
has its specific set of tasks. For example, the virtio-net device has a
control queue, and the driver can request MAC filters, VLANs, and other
settings via this queue. These requests have to be processed by the virtio
device that we are implementing in pci-epf-vnet.

The simplest virtio-console device doesn’t have these tasks, but the other
virtio devices requireprocessing them.

That's why the current pci-epf-virtio design requires a specific
implementation for each virtio device.

Is this what you meant?

>
>>> Previous you use virtio-net, why change to virtio-console here?  Does it
>> matter?
>>
>> No, it doesn't. Just I'd like to break down the changes into smaller
>> steps to make it easier to review and merge the changes.
>>
>> As a first step, I propose adding a simplest virtio function driver with
>> the extension defined in pci-epf-virtio.{h,c}.
>>
>>> All virtio-XXX should work?
>> Yes, the extension is designed to use any type of virtio device.
>>> You removed EDMA support this version?
>> I planed the support will be added with epf virtio-net patches.
>>>> Introduced driver is `pci ep virtio console drv` in the figure. It works
>>>> as ep function for PCIe root and virtual virtio console device for PCIe
>>>> endpoint. Each side of virtio console driver has virtqueue, and
>>>> introduced driver transfers data on the virtqueue to each other. A data
>>>> on root tx queue is transfered to endpoint rx queue and vice versa.
>>>>
>>>> This patchset is depend follwing patches which are under discussion.
>>>>
>>>> - [RFC PATCH 0/3] Deal with alignment restriction on EP side
>>>> link:
>>>>
>> https://lore.k/
>> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908db3
>> ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
>> 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
>> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
>> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
>>>> ernel.org%2Flinux-pci%2F20230113090350.1103494-1-
>>>>
>> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
>> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
>> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
>> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>>>> %7C%7C&sdata=jYgy%2Bxk84ZXZRVfqm0GCXoRnCTLMrX4zTfV%2Bs5Mm
>> svo
>>>> %3D&reserved=0
>>>> - [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
>>>> link:
>>>>
>> https://lore.k/
>> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908db3
>> ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
>> 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
>> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
>> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
>>>> ernel.org%2Fvirtualization%2F20230202090934.549556-1-
>>>>
>> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
>> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
>> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
>> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
>>>> %7C%7C&sdata=K4El76GSAGtsWkNBXJK5%2Fn7flCN20eEMZpZYTX2WIZ0
>> %3
>>>> D&reserved=0
>>>>
>>>> First of this patchset is introduce a helper function to realize pci
>>>> virtio function using PCIe endpoint framework. The second one is adding
>>>> a missing definition for virtio pci header. The last one is for PCIe
>>>> endpoint virtio console driver.
>>>>
>>>> This is tested on linux-20230406 and RCar S4 board as PCIe endpoint.
>>>>
>>>> Shunsuke Mie (3):
>>>>     PCI: endpoint: introduce a helper to implement pci ep virtio function
>>>>     virtio_pci: add a definition of queue flag in ISR
>>>>     PCI: endpoint: Add EP function driver to provide virtio-console
>>>>       functionality
>>>>
>>>>    drivers/pci/endpoint/functions/Kconfig        |  19 +
>>>>    drivers/pci/endpoint/functions/Makefile       |   2 +
>>>>    drivers/pci/endpoint/functions/pci-epf-vcon.c | 554
>> ++++++++++++++++++
>>>>    .../pci/endpoint/functions/pci-epf-virtio.c   | 469 +++++++++++++++
>>>>    .../pci/endpoint/functions/pci-epf-virtio.h   | 123 ++++
>>>>    include/uapi/linux/virtio_pci.h               |   3 +
>>>>    6 files changed, 1170 insertions(+)
>>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
>>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
>>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
>>>>
>>>> --
>>>> 2.25.1

Frank Li April 18, 2023, 4:42 p.m. UTC | #5

> -----Original Message-----
> From: Shunsuke Mie <[e-mail]>
> Sent: Tuesday, April 18, 2023 5:31 AM
> To: Frank Li <[e-mail]>; Lorenzo Pieralisi <[e-mail]>
> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
> Jason Wang <[e-mail]>; Jon Mason <[e-mail]>;
> Randy Dunlap <[e-mail]>; Ren Zhijie
> <[e-mail]>; [e-mail]; linux-
> [e-mail]; [e-mail]
> Subject: Re: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
>
> Caution: EXT Email
>
> On 2023/04/18 0:19, Frank Li wrote:
> >
> >> -----Original Message-----
> >> From: Shunsuke Mie <[e-mail]>
> >> Sent: Sunday, April 16, 2023 9:12 PM
> >> To: Frank Li <[e-mail]>; Lorenzo Pieralisi <[e-mail]>
> >> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> >> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
> >> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
> >> Jason Wang <[e-mail]>; Jon Mason <[e-mail]>;
> >> Randy Dunlap <[e-mail]>; Ren Zhijie
> >> <[e-mail]>; [e-mail]; linux-
> >> [e-mail]; [e-mail]
> >> Subject: Re: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio
> console
> >>
> >> Caution: EXT Email
> >>
> >> On 2023/04/14 23:39, Frank Li wrote:
> >>>> -----Original Message-----
> >>>> From: Shunsuke Mie <[e-mail]>
> >>>> Sent: Friday, April 14, 2023 7:39 AM
> >>>> To: Lorenzo Pieralisi <[e-mail]>
> >>>> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> >>>> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>;
> Bjorn
> >>>> Helgaas <[e-mail]>; Michael S. Tsirkin
> <[e-mail]>;
> >>>> Jason Wang <[e-mail]>; Shunsuke Mie
> <[e-mail]>;
> >>>> Frank Li <[e-mail]>; Jon Mason <[e-mail]>; Randy
> >>>> Dunlap <[e-mail]>; Ren Zhijie
> <[e-mail]>;
> >>>> [e-mail]; [e-mail];
> >>>> [e-mail]
> >>>> Subject: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
> >>>>
> >>>> Caution: EXT Email
> >>>>
> >>>> PCIe endpoint framework provides APIs to implement PCIe endpoint
> >>>> function.
> >>>> This framework allows defining various PCIe endpoint function
> behaviors
> >> in
> >>>> software. This patch extend the framework for virtio pci device. The
> >>>> virtio is defined to communicate guest on virtual machine and host side.
> >>>> Advantage of the virtio is the efficiency of data transfer and the
> >> conciseness
> >>>> of implementation device using software. It also be applied to PCIe
> >>>> endpoint function.
> >>>>
> >>>> We designed and implemented a PCIe EP virtio console function driver
> >> using
> >>>> the extended PCIe endpoint framework for virtio. It can be
> communicate
> >>>> host and endpoint over virtio as console.
> >>>>
> >>>> An architecture of the function driver is following:
> >>>>
> >>>>    ┌────────────┐         ┌───────────
> ─
> >> ──
> >>>> ────────┬────────────┐
> >>>>    │virtio      │         │                      │virtio      │
> >>>>    │console drv │         ├───────────────┐      │
> >> console
> >>>> drv │
> >>>>    ├────────────┤         │(virtio console│      ├──
> ─
> >> ──
> >>>> ───────┤
> >>>>    │ virtio bus │         │ device)       │◄────►│ virtio bus │
> >>>>    ├────────────┤         ├---------------┤      └───
> ─
> >> ──
> >>>> ──────┤
> >>>>    │            │         │ pci ep virtio │                   │
> >>>>    │  pci bus   │         │  console drv  │                   │
> >>>>    │            │  pcie   ├───────────────┤
> │
> >>>>    │            │ ◄─────► │  pci ep Bus   │                   │
> >>>>    └────────────┘         └───────────
> ─
> >> ──
> >>>> ─┴───────────────────┘
> >>>>      PCIe Root              PCIe Endpoint
> >>>>
> >>> [Frank Li] Some basic question,
> >>> I see you call register_virtio_device at epf_vcon_setup_vdev,
> >>> Why call it as virtio console?  I suppose it should be virtiobus directly?
> >> I'm sorry I didn't understand your question. What do you mean the
> >> virtiobus directly?
> > I go through your code again.  I think I understand why you need pci-epf-
> vcon.c.
> > Actually,  my means is like virtio_mmio_probe.
> >
> > vm_dev->vdev.id.device = readl(vm_dev->base +
> VIRTIO_MMIO_DEVICE_ID);
> > vm_dev->vdev.id.vendor = readl(vm_dev->base +
> VIRTIO_MMIO_VENDOR_ID);
> >
> > I am not sure that if VIRTIO_MMIO_VENDOR_ID and
> VIRTIO_MMIO_DEVICE_ID
> > reuse PCI's vendor ID and Device ID.  If yes, you can directly get such
> information
> > from epf.  If no,  a customer field can been added at epf driver.
> >
> > So you needn't write pci-epf-vcon  and pci-epf-vnet .....
> >
> > Of cause it will be wonderful if directly use virtio_mmio_probe by dynmatic
> create platform
> > Devices.  It may have some difficult because pci memory map requirement.
> I think that some fields are shared between the vdev and epf device.
> However, we need to implement device emulation because each virtio
> device
> has its specific set of tasks. For example, the virtio-net device has a
> control queue, and the driver can request MAC filters, VLANs, and other
> settings via this queue. These requests have to be processed by the virtio
> device that we are implementing in pci-epf-vnet.
>
> The simplest virtio-console device doesn’t have these tasks, but the other
> virtio devices requireprocessing them.
>
> That's why the current pci-epf-virtio design requires a specific
> implementation for each virtio device.
>
> Is this what you meant

I see. Thank you for explain.
Thing may become complex. How to implement composite at EPF driver?
Such as console + net

>
> >
> >>> Previous you use virtio-net, why change to virtio-console here?  Does it
> >> matter?
> >>
> >> No, it doesn't. Just I'd like to break down the changes into smaller
> >> steps to make it easier to review and merge the changes.
> >>
> >> As a first step, I propose adding a simplest virtio function driver with
> >> the extension defined in pci-epf-virtio.{h,c}.
> >>
> >>> All virtio-XXX should work?
> >> Yes, the extension is designed to use any type of virtio device.
> >>> You removed EDMA support this version?
> >> I planed the support will be added with epf virtio-net patches.
> >>>> Introduced driver is `pci ep virtio console drv` in the figure. It works
> >>>> as ep function for PCIe root and virtual virtio console device for PCIe
> >>>> endpoint. Each side of virtio console driver has virtqueue, and
> >>>> introduced driver transfers data on the virtqueue to each other. A data
> >>>> on root tx queue is transfered to endpoint rx queue and vice versa.
> >>>>
> >>>> This patchset is depend follwing patches which are under discussion.
> >>>>
> >>>> - [RFC PATCH 0/3] Deal with alignment restriction on EP side
> >>>> link:
> >>>>
> >>
> https://lore.k/
> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7C80a237184ff4420a96de08db
> 3ff80d78%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817410683
> 2489955%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LSP
> LeFzTYpUVTZpymoM00Es9kOpxlN90%2BDM6ceDJMFE%3D&reserved=0
> >> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908
> db3
> >>
> ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
> >>
> 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> >>
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
> >> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
> >>>> ernel.org%2Flinux-pci%2F20230113090350.1103494-1-
> >>>>
> >>
> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> >>
> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> >>
> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> >>
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> >>>> %7C%7C&sdata=jYgy%2Bxk84ZXZRVfqm0GCXoRnCTLMrX4zTfV%2Bs5
> Mm
> >> svo
> >>>> %3D&reserved=0
> >>>> - [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
> >>>> link:
> >>>>
> >>
> https://lore.k/
> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7C80a237184ff4420a96de08db
> 3ff80d78%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817410683
> 2489955%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LSP
> LeFzTYpUVTZpymoM00Es9kOpxlN90%2BDM6ceDJMFE%3D&reserved=0
> >> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908
> db3
> >>
> ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
> >>
> 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> >>
> 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
> >> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
> >>>> ernel.org%2Fvirtualization%2F20230202090934.549556-1-
> >>>>
> >>
> mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> >>
> 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> >>
> C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> >>
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> >>>> %7C%7C&sdata=K4El76GSAGtsWkNBXJK5%2Fn7flCN20eEMZpZYTX2WI
> Z0
> >> %3
> >>>> D&reserved=0
> >>>>
> >>>> First of this patchset is introduce a helper function to realize pci
> >>>> virtio function using PCIe endpoint framework. The second one is
> adding
> >>>> a missing definition for virtio pci header. The last one is for PCIe
> >>>> endpoint virtio console driver.
> >>>>
> >>>> This is tested on linux-20230406 and RCar S4 board as PCIe endpoint.
> >>>>
> >>>> Shunsuke Mie (3):
> >>>>     PCI: endpoint: introduce a helper to implement pci ep virtio function
> >>>>     virtio_pci: add a definition of queue flag in ISR
> >>>>     PCI: endpoint: Add EP function driver to provide virtio-console
> >>>>       functionality
> >>>>
> >>>>    drivers/pci/endpoint/functions/Kconfig        |  19 +
> >>>>    drivers/pci/endpoint/functions/Makefile       |   2 +
> >>>>    drivers/pci/endpoint/functions/pci-epf-vcon.c | 554
> >> ++++++++++++++++++
> >>>>    .../pci/endpoint/functions/pci-epf-virtio.c   | 469 +++++++++++++++
> >>>>    .../pci/endpoint/functions/pci-epf-virtio.h   | 123 ++++
> >>>>    include/uapi/linux/virtio_pci.h               |   3 +
> >>>>    6 files changed, 1170 insertions(+)
> >>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
> >>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
> >>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
> >>>>
> >>>> --
> >>>> 2.25.1

Shunsuke Mie April 24, 2023, 10:06 a.m. UTC | #6

2023年4月19日(水) 1:42 Frank Li <[e-mail]>:
>
>
>
> > -----Original Message-----
> > From: Shunsuke Mie <[e-mail]>
> > Sent: Tuesday, April 18, 2023 5:31 AM
> > To: Frank Li <[e-mail]>; Lorenzo Pieralisi <[e-mail]>
> > Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> > <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
> > Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
> > Jason Wang <[e-mail]>; Jon Mason <[e-mail]>;
> > Randy Dunlap <[e-mail]>; Ren Zhijie
> > <[e-mail]>; [e-mail]; linux-
> > [e-mail]; [e-mail]
> > Subject: Re: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
> >
> > Caution: EXT Email
> >
> > On 2023/04/18 0:19, Frank Li wrote:
> > >
> > >> -----Original Message-----
> > >> From: Shunsuke Mie <[e-mail]>
> > >> Sent: Sunday, April 16, 2023 9:12 PM
> > >> To: Frank Li <[e-mail]>; Lorenzo Pieralisi <[e-mail]>
> > >> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> > >> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>; Bjorn
> > >> Helgaas <[e-mail]>; Michael S. Tsirkin <[e-mail]>;
> > >> Jason Wang <[e-mail]>; Jon Mason <[e-mail]>;
> > >> Randy Dunlap <[e-mail]>; Ren Zhijie
> > >> <[e-mail]>; [e-mail]; linux-
> > >> [e-mail]; [e-mail]
> > >> Subject: Re: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio
> > console
> > >>
> > >> Caution: EXT Email
> > >>
> > >> On 2023/04/14 23:39, Frank Li wrote:
> > >>>> -----Original Message-----
> > >>>> From: Shunsuke Mie <[e-mail]>
> > >>>> Sent: Friday, April 14, 2023 7:39 AM
> > >>>> To: Lorenzo Pieralisi <[e-mail]>
> > >>>> Cc: Krzysztof Wilczyński <[e-mail]>; Manivannan Sadhasivam
> > >>>> <[e-mail]>; Kishon Vijay Abraham I <[e-mail]>;
> > Bjorn
> > >>>> Helgaas <[e-mail]>; Michael S. Tsirkin
> > <[e-mail]>;
> > >>>> Jason Wang <[e-mail]>; Shunsuke Mie
> > <[e-mail]>;
> > >>>> Frank Li <[e-mail]>; Jon Mason <[e-mail]>; Randy
> > >>>> Dunlap <[e-mail]>; Ren Zhijie
> > <[e-mail]>;
> > >>>> [e-mail]; [e-mail];
> > >>>> [e-mail]
> > >>>> Subject: [EXT] [RFC PATCH 0/3] Introduce a PCIe endpoint virtio console
> > >>>>
> > >>>> Caution: EXT Email
> > >>>>
> > >>>> PCIe endpoint framework provides APIs to implement PCIe endpoint
> > >>>> function.
> > >>>> This framework allows defining various PCIe endpoint function
> > behaviors
> > >> in
> > >>>> software. This patch extend the framework for virtio pci device. The
> > >>>> virtio is defined to communicate guest on virtual machine and host side.
> > >>>> Advantage of the virtio is the efficiency of data transfer and the
> > >> conciseness
> > >>>> of implementation device using software. It also be applied to PCIe
> > >>>> endpoint function.
> > >>>>
> > >>>> We designed and implemented a PCIe EP virtio console function driver
> > >> using
> > >>>> the extended PCIe endpoint framework for virtio. It can be
> > communicate
> > >>>> host and endpoint over virtio as console.
> > >>>>
> > >>>> An architecture of the function driver is following:
> > >>>>
> > >>>>    ┌────────────┐         ┌───────────
> > ─
> > >> ──
> > >>>> ────────┬────────────┐
> > >>>>    │virtio      │         │                      │virtio      │
> > >>>>    │console drv │         ├───────────────┐      │
> > >> console
> > >>>> drv │
> > >>>>    ├────────────┤         │(virtio console│      ├──
> > ─
> > >> ──
> > >>>> ───────┤
> > >>>>    │ virtio bus │         │ device)       │◄────►│ virtio bus │
> > >>>>    ├────────────┤         ├---------------┤      └───
> > ─
> > >> ──
> > >>>> ──────┤
> > >>>>    │            │         │ pci ep virtio │                   │
> > >>>>    │  pci bus   │         │  console drv  │                   │
> > >>>>    │            │  pcie   ├───────────────┤
> > │
> > >>>>    │            │ ◄─────► │  pci ep Bus   │                   │
> > >>>>    └────────────┘         └───────────
> > ─
> > >> ──
> > >>>> ─┴───────────────────┘
> > >>>>      PCIe Root              PCIe Endpoint
> > >>>>
> > >>> [Frank Li] Some basic question,
> > >>> I see you call register_virtio_device at epf_vcon_setup_vdev,
> > >>> Why call it as virtio console?  I suppose it should be virtiobus directly?
> > >> I'm sorry I didn't understand your question. What do you mean the
> > >> virtiobus directly?
> > > I go through your code again.  I think I understand why you need pci-epf-
> > vcon.c.
> > > Actually,  my means is like virtio_mmio_probe.
> > >
> > > vm_dev->vdev.id.device = readl(vm_dev->base +
> > VIRTIO_MMIO_DEVICE_ID);
> > > vm_dev->vdev.id.vendor = readl(vm_dev->base +
> > VIRTIO_MMIO_VENDOR_ID);
> > >
> > > I am not sure that if VIRTIO_MMIO_VENDOR_ID and
> > VIRTIO_MMIO_DEVICE_ID
> > > reuse PCI's vendor ID and Device ID.  If yes, you can directly get such
> > information
> > > from epf.  If no,  a customer field can been added at epf driver.
> > >
> > > So you needn't write pci-epf-vcon  and pci-epf-vnet .....
> > >
> > > Of cause it will be wonderful if directly use virtio_mmio_probe by dynmatic
> > create platform
> > > Devices.  It may have some difficult because pci memory map requirement.
> > I think that some fields are shared between the vdev and epf device.
> > However, we need to implement device emulation because each virtio
> > device
> > has its specific set of tasks. For example, the virtio-net device has a
> > control queue, and the driver can request MAC filters, VLANs, and other
> > settings via this queue. These requests have to be processed by the virtio
> > device that we are implementing in pci-epf-vnet.
> >
> > The simplest virtio-console device doesn’t have these tasks, but the other
> > virtio devices requireprocessing them.
> >
> > That's why the current pci-epf-virtio design requires a specific
> > implementation for each virtio device.
> >
> > Is this what you meant
>
> I see. Thank you for explain.
> Thing may become complex. How to implement composite at EPF driver?
> Such as console + net
I'm not sure your suggestion. Could you tell me the advantages?
> >
> > >
> > >>> Previous you use virtio-net, why change to virtio-console here?  Does it
> > >> matter?
> > >>
> > >> No, it doesn't. Just I'd like to break down the changes into smaller
> > >> steps to make it easier to review and merge the changes.
> > >>
> > >> As a first step, I propose adding a simplest virtio function driver with
> > >> the extension defined in pci-epf-virtio.{h,c}.
> > >>
> > >>> All virtio-XXX should work?
> > >> Yes, the extension is designed to use any type of virtio device.
> > >>> You removed EDMA support this version?
> > >> I planed the support will be added with epf virtio-net patches.
> > >>>> Introduced driver is `pci ep virtio console drv` in the figure. It works
> > >>>> as ep function for PCIe root and virtual virtio console device for PCIe
> > >>>> endpoint. Each side of virtio console driver has virtqueue, and
> > >>>> introduced driver transfers data on the virtqueue to each other. A data
> > >>>> on root tx queue is transfered to endpoint rx queue and vice versa.
> > >>>>
> > >>>> This patchset is depend follwing patches which are under discussion.
> > >>>>
> > >>>> - [RFC PATCH 0/3] Deal with alignment restriction on EP side
> > >>>> link:
> > >>>>
> > >>
> > https://lore.k/
> > %2F&data=05%7C01%7Cfrank.li%40nxp.com%7C80a237184ff4420a96de08db
> > 3ff80d78%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817410683
> > 2489955%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LSP
> > LeFzTYpUVTZpymoM00Es9kOpxlN90%2BDM6ceDJMFE%3D&reserved=0
> > >> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908
> > db3
> > >>
> > ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
> > >>
> > 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> > >>
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
> > >> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
> > >>>> ernel.org%2Flinux-pci%2F20230113090350.1103494-1-
> > >>>>
> > >>
> > mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> > >>
> > 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> > >>
> > C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> > >>
> > LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> > >>>> %7C%7C&sdata=jYgy%2Bxk84ZXZRVfqm0GCXoRnCTLMrX4zTfV%2Bs5
> > Mm
> > >> svo
> > >>>> %3D&reserved=0
> > >>>> - [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
> > >>>> link:
> > >>>>
> > >>
> > https://lore.k/
> > %2F&data=05%7C01%7Cfrank.li%40nxp.com%7C80a237184ff4420a96de08db
> > 3ff80d78%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817410683
> > 2489955%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LSP
> > LeFzTYpUVTZpymoM00Es9kOpxlN90%2BDM6ceDJMFE%3D&reserved=0
> > >> %2F&data=05%7C01%7Cfrank.li%40nxp.com%7Cff59a16f88c643913e3908
> > db3
> > >>
> > ee91ca8%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63817294315
> > >>
> > 3821831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV
> > >>
> > 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hTB
> > >> 5UDaJoJfta9ohMG%2BrxCVJY34ANn10iSLP9iCHX3M%3D&reserved=0
> > >>>> ernel.org%2Fvirtualization%2F20230202090934.549556-1-
> > >>>>
> > >>
> > mie%40igel.co.jp%2F&data=05%7C01%7CFrank.Li%40nxp.com%7Cea6513dbf
> > >>
> > 4084b80ced208db3ce54133%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7
> > >>
> > C0%7C638170727558800720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
> > >>
> > LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C
> > >>>> %7C%7C&sdata=K4El76GSAGtsWkNBXJK5%2Fn7flCN20eEMZpZYTX2WI
> > Z0
> > >> %3
> > >>>> D&reserved=0
> > >>>>
> > >>>> First of this patchset is introduce a helper function to realize pci
> > >>>> virtio function using PCIe endpoint framework. The second one is
> > adding
> > >>>> a missing definition for virtio pci header. The last one is for PCIe
> > >>>> endpoint virtio console driver.
> > >>>>
> > >>>> This is tested on linux-20230406 and RCar S4 board as PCIe endpoint.
> > >>>>
> > >>>> Shunsuke Mie (3):
> > >>>>     PCI: endpoint: introduce a helper to implement pci ep virtio function
> > >>>>     virtio_pci: add a definition of queue flag in ISR
> > >>>>     PCI: endpoint: Add EP function driver to provide virtio-console
> > >>>>       functionality
> > >>>>
> > >>>>    drivers/pci/endpoint/functions/Kconfig        |  19 +
> > >>>>    drivers/pci/endpoint/functions/Makefile       |   2 +
> > >>>>    drivers/pci/endpoint/functions/pci-epf-vcon.c | 554
> > >> ++++++++++++++++++
> > >>>>    .../pci/endpoint/functions/pci-epf-virtio.c   | 469 +++++++++++++++
> > >>>>    .../pci/endpoint/functions/pci-epf-virtio.h   | 123 ++++
> > >>>>    include/uapi/linux/virtio_pci.h               |   3 +
> > >>>>    6 files changed, 1170 insertions(+)
> > >>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
> > >>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
> > >>>>    create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
> > >>>>
> > >>>> --
> > >>>> 2.25.1
Best regards,
Shunsuike