[V8,0/6] mdev based hardware virtio offloading support
mbox series

Message ID 20191105093240.5135-1-jasowang@redhat.com
Headers show
Series
  • mdev based hardware virtio offloading support
Related show

Message

Jason Wang Nov. 5, 2019, 9:32 a.m. UTC
Hi all:

There are hardwares that can do virtio datapath offloading while
having its own control path. This path tries to implement a mdev based
unified API to support using kernel virtio driver to drive those
devices. This is done by introducing a new mdev transport for virtio
(virtio_mdev) and register itself as a new kind of mdev driver. Then
it provides a unified way for kernel virtio driver to talk with mdev
device implementation.

Though the series only contains kernel driver support, the goal is to
make the transport generic enough to support userspace drivers. This
means vhost-mdev[1] could be built on top as well by resuing the
transport.

A sample driver is also implemented which simulate a virito-net
loopback ethernet device on top of vringh + workqueue. This could be
used as a reference implementation for real hardware driver.

Also a real ICF VF driver was also posted here[2] which is a good
reference for vendors who is interested in their own virtio datapath
offloading product.

Consider mdev framework only support VFIO device and driver right now,
this series also extend it to support other types. This is done
through introducing class id to the device and pairing it with
id_talbe claimed by the driver. On top, this seris also decouple
device specific parents ops out of the common ones.

Pktgen test was done with virito-net + mvnet loop back device.

Please review.

[1] https://lkml.org/lkml/2019/10/31/440
[2] https://lkml.org/lkml/2019/10/15/1226

Changes from V7:
- drop {set|get}_mdev_features for virtio
- typo and comment style fixes

Changes from V6:

- rename ops files and compile guard

Changes from V5:

- use dev_warn() instead of WARN(1) when class id is not set
- validate id_table before trying to do matching between device and
  driver
- add wildcard for modpost script
- use unique name for id_table
- move get_mdev_features() to be the first member of virtio_device_ops
  and more comments for it
- typo fixes for the comments above virtio_mdev_ops

Changes from V4:

- keep mdev_set_class() for the device that doesn't use device ops
- use union for device ops pointer in mdev_device
- introduce class specific helper for getting is device ops
- use WARN_ON instead of BUG_ON in mdev_set_virtio_ops
- explain details of get_mdev_features() and get_vendor_id()
- distinguish the optional virito device ops from mandatory ones and
  make get_generation() optional
- rename vfio_mdev.h to vfio_mdev_ops.h, rename virito_mdev.h to
  virtio_mdev_ops.h
- don't abuse version fileds in virtio_mdev structure, use features
  instead
- fix warning during device remove
- style & docs tweaks and typo fixes

Changes from V3:

- document that class id (device ops) must be specified in create()
- add WARN() when trying to set class_id when it has already set
- add WARN() when class_id is not specified in create() and correctly
  return an error in this case
- correct the prototype of mdev_set_class() in the doc
- add documention of mdev_set_class()
- remove the unnecessary "class_id_fail" label when class id is not
  specified in create()
- convert id_table in vfio_mdev to const
- move mdev_set_class and its friends after mdev_uuid()
- suqash the patch of bus uevent into patch of introducing class id
- tweak the words in the docs per Cornelia suggestion
- tie class_id and device ops through class specific initialization
  routine like mdev_set_vfio_ops()
- typos fixes in the docs of virtio-mdev callbacks
- document the usage of virtqueues in struct virtio_mdev_device
- remove the useless vqs array in struct virtio_mdev_device
- rename MDEV_ID_XXX to MDEV_CLASS_ID_XXX

Changes from V2:

- fail when class_id is not specified
- drop the vringh patch
- match the doc to the code
- tweak the commit log
- move device_ops from parent to mdev device
- remove the unused MDEV_ID_VHOST

Changes from V1:

- move virtio_mdev.c to drivers/virtio
- store class_id in mdev_device instead of mdev_parent
- store device_ops in mdev_device instead of mdev_parent
- reorder the patch, vringh fix comes first
- really silent compiling warnings
- really switch to use u16 for class_id
- uevent and modpost support for mdev class_id
- vraious tweaks per comments from Parav

Changes from RFC-V2:

- silent compile warnings on some specific configuration
- use u16 instead u8 for class id
- reseve MDEV_ID_VHOST for future vhost-mdev work
- introduce "virtio" type for mvnet and make "vhost" type for future
  work
- add entries in MAINTAINER
- tweak and typos fixes in commit log

Changes from RFC-V1:

- rename device id to class id
- add docs for class id and device specific ops (device_ops)
- split device_ops into seperate headers
- drop the mdev_set_dma_ops()
- use device_ops to implement the transport API, then it's not a part
  of UAPI any more
- use GFP_ATOMIC in mvnet sample device and other tweaks
- set_vring_base/get_vring_base support for mvnet device

Jason Wang (6):
  mdev: class id support
  modpost: add support for mdev class id
  mdev: introduce device specific ops
  mdev: introduce virtio device and its device ops
  virtio: introduce a mdev based transport
  docs: sample driver to demonstrate how to implement virtio-mdev
    framework

 .../driver-api/vfio-mediated-device.rst       |  38 +-
 MAINTAINERS                                   |   2 +
 drivers/gpu/drm/i915/gvt/kvmgt.c              |  17 +-
 drivers/s390/cio/vfio_ccw_ops.c               |  17 +-
 drivers/s390/crypto/vfio_ap_ops.c             |  13 +-
 drivers/vfio/mdev/mdev_core.c                 |  60 ++
 drivers/vfio/mdev/mdev_driver.c               |  25 +
 drivers/vfio/mdev/mdev_private.h              |   8 +
 drivers/vfio/mdev/vfio_mdev.c                 |  45 +-
 drivers/virtio/Kconfig                        |   7 +
 drivers/virtio/Makefile                       |   1 +
 drivers/virtio/virtio_mdev.c                  | 407 +++++++++++
 include/linux/mdev.h                          |  57 +-
 include/linux/mdev_vfio_ops.h                 |  52 ++
 include/linux/mdev_virtio_ops.h               | 149 ++++
 include/linux/mod_devicetable.h               |   8 +
 samples/Kconfig                               |   7 +
 samples/vfio-mdev/Makefile                    |   1 +
 samples/vfio-mdev/mbochs.c                    |  19 +-
 samples/vfio-mdev/mdpy.c                      |  19 +-
 samples/vfio-mdev/mtty.c                      |  17 +-
 samples/vfio-mdev/mvnet.c                     | 685 ++++++++++++++++++
 scripts/mod/devicetable-offsets.c             |   3 +
 scripts/mod/file2alias.c                      |  11 +
 24 files changed, 1577 insertions(+), 91 deletions(-)
 create mode 100644 drivers/virtio/virtio_mdev.c
 create mode 100644 include/linux/mdev_vfio_ops.h
 create mode 100644 include/linux/mdev_virtio_ops.h
 create mode 100644 samples/vfio-mdev/mvnet.c

Comments

Alex Williamson Nov. 5, 2019, 5:58 p.m. UTC | #1
On Tue,  5 Nov 2019 17:32:34 +0800
Jason Wang <jasowang@redhat.com> wrote:

> Hi all:
> 
> There are hardwares that can do virtio datapath offloading while
> having its own control path. This path tries to implement a mdev based
> unified API to support using kernel virtio driver to drive those
> devices. This is done by introducing a new mdev transport for virtio
> (virtio_mdev) and register itself as a new kind of mdev driver. Then
> it provides a unified way for kernel virtio driver to talk with mdev
> device implementation.
> 
> Though the series only contains kernel driver support, the goal is to
> make the transport generic enough to support userspace drivers. This
> means vhost-mdev[1] could be built on top as well by resuing the
> transport.
> 
> A sample driver is also implemented which simulate a virito-net
> loopback ethernet device on top of vringh + workqueue. This could be
> used as a reference implementation for real hardware driver.
> 
> Also a real ICF VF driver was also posted here[2] which is a good
> reference for vendors who is interested in their own virtio datapath
> offloading product.
> 
> Consider mdev framework only support VFIO device and driver right now,
> this series also extend it to support other types. This is done
> through introducing class id to the device and pairing it with
> id_talbe claimed by the driver. On top, this seris also decouple
> device specific parents ops out of the common ones.
> 
> Pktgen test was done with virito-net + mvnet loop back device.
> 
> Please review.
> 
> [1] https://lkml.org/lkml/2019/10/31/440
> [2] https://lkml.org/lkml/2019/10/15/1226
> 
> Changes from V7:
> - drop {set|get}_mdev_features for virtio
> - typo and comment style fixes


Seems we're nearly there, all the remaining comments are relatively
superficial, though I would appreciate a v9 addressing them as well as
the checkpatch warnings:

https://patchwork.freedesktop.org/series/68977/

Consider this a last call for reviews or acks (or naks) from affected
mdev vendor drivers, mdev-core sub-maintainers (Hi Kirti), virtio
stakeholders, etc.  Thanks,

Alex
Jason Wang Nov. 6, 2019, 3:56 a.m. UTC | #2
On 2019/11/6 上午1:58, Alex Williamson wrote:
> On Tue,  5 Nov 2019 17:32:34 +0800
> Jason Wang <jasowang@redhat.com> wrote:
>
>> Hi all:
>>
>> There are hardwares that can do virtio datapath offloading while
>> having its own control path. This path tries to implement a mdev based
>> unified API to support using kernel virtio driver to drive those
>> devices. This is done by introducing a new mdev transport for virtio
>> (virtio_mdev) and register itself as a new kind of mdev driver. Then
>> it provides a unified way for kernel virtio driver to talk with mdev
>> device implementation.
>>
>> Though the series only contains kernel driver support, the goal is to
>> make the transport generic enough to support userspace drivers. This
>> means vhost-mdev[1] could be built on top as well by resuing the
>> transport.
>>
>> A sample driver is also implemented which simulate a virito-net
>> loopback ethernet device on top of vringh + workqueue. This could be
>> used as a reference implementation for real hardware driver.
>>
>> Also a real ICF VF driver was also posted here[2] which is a good
>> reference for vendors who is interested in their own virtio datapath
>> offloading product.
>>
>> Consider mdev framework only support VFIO device and driver right now,
>> this series also extend it to support other types. This is done
>> through introducing class id to the device and pairing it with
>> id_talbe claimed by the driver. On top, this seris also decouple
>> device specific parents ops out of the common ones.
>>
>> Pktgen test was done with virito-net + mvnet loop back device.
>>
>> Please review.
>>
>> [1] https://lkml.org/lkml/2019/10/31/440
>> [2] https://lkml.org/lkml/2019/10/15/1226
>>
>> Changes from V7:
>> - drop {set|get}_mdev_features for virtio
>> - typo and comment style fixes
>
> Seems we're nearly there, all the remaining comments are relatively
> superficial, though I would appreciate a v9 addressing them as well as
> the checkpatch warnings:
>
> https://patchwork.freedesktop.org/series/68977/


Will do.

Btw, do you plan to merge vhost-mdev patch on top? Or you prefer it to 
go through Michael's vhost tree?

Thanks


>
> Consider this a last call for reviews or acks (or naks) from affected
> mdev vendor drivers, mdev-core sub-maintainers (Hi Kirti), virtio
> stakeholders, etc.  Thanks,
>
> Alex
>
Alex Williamson Nov. 6, 2019, 7:03 p.m. UTC | #3
On Wed, 6 Nov 2019 11:56:46 +0800
Jason Wang <jasowang@redhat.com> wrote:

> On 2019/11/6 上午1:58, Alex Williamson wrote:
> > On Tue,  5 Nov 2019 17:32:34 +0800
> > Jason Wang <jasowang@redhat.com> wrote:
> >  
> >> Hi all:
> >>
> >> There are hardwares that can do virtio datapath offloading while
> >> having its own control path. This path tries to implement a mdev based
> >> unified API to support using kernel virtio driver to drive those
> >> devices. This is done by introducing a new mdev transport for virtio
> >> (virtio_mdev) and register itself as a new kind of mdev driver. Then
> >> it provides a unified way for kernel virtio driver to talk with mdev
> >> device implementation.
> >>
> >> Though the series only contains kernel driver support, the goal is to
> >> make the transport generic enough to support userspace drivers. This
> >> means vhost-mdev[1] could be built on top as well by resuing the
> >> transport.
> >>
> >> A sample driver is also implemented which simulate a virito-net
> >> loopback ethernet device on top of vringh + workqueue. This could be
> >> used as a reference implementation for real hardware driver.
> >>
> >> Also a real ICF VF driver was also posted here[2] which is a good
> >> reference for vendors who is interested in their own virtio datapath
> >> offloading product.
> >>
> >> Consider mdev framework only support VFIO device and driver right now,
> >> this series also extend it to support other types. This is done
> >> through introducing class id to the device and pairing it with
> >> id_talbe claimed by the driver. On top, this seris also decouple
> >> device specific parents ops out of the common ones.
> >>
> >> Pktgen test was done with virito-net + mvnet loop back device.
> >>
> >> Please review.
> >>
> >> [1] https://lkml.org/lkml/2019/10/31/440
> >> [2] https://lkml.org/lkml/2019/10/15/1226
> >>
> >> Changes from V7:
> >> - drop {set|get}_mdev_features for virtio
> >> - typo and comment style fixes  
> >
> > Seems we're nearly there, all the remaining comments are relatively
> > superficial, though I would appreciate a v9 addressing them as well as
> > the checkpatch warnings:
> >
> > https://patchwork.freedesktop.org/series/68977/  
> 
> 
> Will do.
> 
> Btw, do you plan to merge vhost-mdev patch on top? Or you prefer it to 
> go through Michael's vhost tree?

I can include it if you wish.  The mdev changes are isolated enough in
that patch that I wouldn't presume it, but clearly it would require
less merge coordination to drop it in my tree.  Let me know.  Thanks,

Alex
Michael S. Tsirkin Nov. 6, 2019, 7:25 p.m. UTC | #4
On Wed, Nov 06, 2019 at 12:03:12PM -0700, Alex Williamson wrote:
> On Wed, 6 Nov 2019 11:56:46 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
> > On 2019/11/6 上午1:58, Alex Williamson wrote:
> > > On Tue,  5 Nov 2019 17:32:34 +0800
> > > Jason Wang <jasowang@redhat.com> wrote:
> > >  
> > >> Hi all:
> > >>
> > >> There are hardwares that can do virtio datapath offloading while
> > >> having its own control path. This path tries to implement a mdev based
> > >> unified API to support using kernel virtio driver to drive those
> > >> devices. This is done by introducing a new mdev transport for virtio
> > >> (virtio_mdev) and register itself as a new kind of mdev driver. Then
> > >> it provides a unified way for kernel virtio driver to talk with mdev
> > >> device implementation.
> > >>
> > >> Though the series only contains kernel driver support, the goal is to
> > >> make the transport generic enough to support userspace drivers. This
> > >> means vhost-mdev[1] could be built on top as well by resuing the
> > >> transport.
> > >>
> > >> A sample driver is also implemented which simulate a virito-net
> > >> loopback ethernet device on top of vringh + workqueue. This could be
> > >> used as a reference implementation for real hardware driver.
> > >>
> > >> Also a real ICF VF driver was also posted here[2] which is a good
> > >> reference for vendors who is interested in their own virtio datapath
> > >> offloading product.
> > >>
> > >> Consider mdev framework only support VFIO device and driver right now,
> > >> this series also extend it to support other types. This is done
> > >> through introducing class id to the device and pairing it with
> > >> id_talbe claimed by the driver. On top, this seris also decouple
> > >> device specific parents ops out of the common ones.
> > >>
> > >> Pktgen test was done with virito-net + mvnet loop back device.
> > >>
> > >> Please review.
> > >>
> > >> [1] https://lkml.org/lkml/2019/10/31/440
> > >> [2] https://lkml.org/lkml/2019/10/15/1226
> > >>
> > >> Changes from V7:
> > >> - drop {set|get}_mdev_features for virtio
> > >> - typo and comment style fixes  
> > >
> > > Seems we're nearly there, all the remaining comments are relatively
> > > superficial, though I would appreciate a v9 addressing them as well as
> > > the checkpatch warnings:
> > >
> > > https://patchwork.freedesktop.org/series/68977/  
> > 
> > 
> > Will do.
> > 
> > Btw, do you plan to merge vhost-mdev patch on top? Or you prefer it to 
> > go through Michael's vhost tree?
> 
> I can include it if you wish.  The mdev changes are isolated enough in
> that patch that I wouldn't presume it, but clearly it would require
> less merge coordination to drop it in my tree.  Let me know.  Thanks,
> 
> Alex

I'm fine with merging through your tree. If you do, feel free to
include

Acked-by: Michael S. Tsirkin <mst@redhat.com>
Alex Williamson Nov. 6, 2019, 9:13 p.m. UTC | #5
On Wed, 6 Nov 2019 14:25:23 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Nov 06, 2019 at 12:03:12PM -0700, Alex Williamson wrote:
> > On Wed, 6 Nov 2019 11:56:46 +0800
> > Jason Wang <jasowang@redhat.com> wrote:
> >   
> > > On 2019/11/6 上午1:58, Alex Williamson wrote:  
> > > > On Tue,  5 Nov 2019 17:32:34 +0800
> > > > Jason Wang <jasowang@redhat.com> wrote:
> > > >    
> > > >> Hi all:
> > > >>
> > > >> There are hardwares that can do virtio datapath offloading while
> > > >> having its own control path. This path tries to implement a mdev based
> > > >> unified API to support using kernel virtio driver to drive those
> > > >> devices. This is done by introducing a new mdev transport for virtio
> > > >> (virtio_mdev) and register itself as a new kind of mdev driver. Then
> > > >> it provides a unified way for kernel virtio driver to talk with mdev
> > > >> device implementation.
> > > >>
> > > >> Though the series only contains kernel driver support, the goal is to
> > > >> make the transport generic enough to support userspace drivers. This
> > > >> means vhost-mdev[1] could be built on top as well by resuing the
> > > >> transport.
> > > >>
> > > >> A sample driver is also implemented which simulate a virito-net
> > > >> loopback ethernet device on top of vringh + workqueue. This could be
> > > >> used as a reference implementation for real hardware driver.
> > > >>
> > > >> Also a real ICF VF driver was also posted here[2] which is a good
> > > >> reference for vendors who is interested in their own virtio datapath
> > > >> offloading product.
> > > >>
> > > >> Consider mdev framework only support VFIO device and driver right now,
> > > >> this series also extend it to support other types. This is done
> > > >> through introducing class id to the device and pairing it with
> > > >> id_talbe claimed by the driver. On top, this seris also decouple
> > > >> device specific parents ops out of the common ones.
> > > >>
> > > >> Pktgen test was done with virito-net + mvnet loop back device.
> > > >>
> > > >> Please review.
> > > >>
> > > >> [1] https://lkml.org/lkml/2019/10/31/440
> > > >> [2] https://lkml.org/lkml/2019/10/15/1226
> > > >>
> > > >> Changes from V7:
> > > >> - drop {set|get}_mdev_features for virtio
> > > >> - typo and comment style fixes    
> > > >
> > > > Seems we're nearly there, all the remaining comments are relatively
> > > > superficial, though I would appreciate a v9 addressing them as well as
> > > > the checkpatch warnings:
> > > >
> > > > https://patchwork.freedesktop.org/series/68977/    
> > > 
> > > 
> > > Will do.
> > > 
> > > Btw, do you plan to merge vhost-mdev patch on top? Or you prefer it to 
> > > go through Michael's vhost tree?  
> > 
> > I can include it if you wish.  The mdev changes are isolated enough in
> > that patch that I wouldn't presume it, but clearly it would require
> > less merge coordination to drop it in my tree.  Let me know.  Thanks,
> > 
> > Alex  
> 
> I'm fine with merging through your tree. If you do, feel free to
> include
> 
> Acked-by: Michael S. Tsirkin <mst@redhat.com>

AFAICT, it looks like we're expecting at least one more version of
Tiwei's patch after V5, so it'd probably be best to provide the ack and
go-ahead on that next version so there's no confusion.  Thanks,

Alex
Jason Wang Nov. 7, 2019, 4:11 a.m. UTC | #6
On 2019/11/7 上午5:13, Alex Williamson wrote:
> On Wed, 6 Nov 2019 14:25:23 -0500
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
>> On Wed, Nov 06, 2019 at 12:03:12PM -0700, Alex Williamson wrote:
>>> On Wed, 6 Nov 2019 11:56:46 +0800
>>> Jason Wang <jasowang@redhat.com> wrote:
>>>    
>>>> On 2019/11/6 上午1:58, Alex Williamson wrote:
>>>>> On Tue,  5 Nov 2019 17:32:34 +0800
>>>>> Jason Wang <jasowang@redhat.com> wrote:
>>>>>     
>>>>>> Hi all:
>>>>>>
>>>>>> There are hardwares that can do virtio datapath offloading while
>>>>>> having its own control path. This path tries to implement a mdev based
>>>>>> unified API to support using kernel virtio driver to drive those
>>>>>> devices. This is done by introducing a new mdev transport for virtio
>>>>>> (virtio_mdev) and register itself as a new kind of mdev driver. Then
>>>>>> it provides a unified way for kernel virtio driver to talk with mdev
>>>>>> device implementation.
>>>>>>
>>>>>> Though the series only contains kernel driver support, the goal is to
>>>>>> make the transport generic enough to support userspace drivers. This
>>>>>> means vhost-mdev[1] could be built on top as well by resuing the
>>>>>> transport.
>>>>>>
>>>>>> A sample driver is also implemented which simulate a virito-net
>>>>>> loopback ethernet device on top of vringh + workqueue. This could be
>>>>>> used as a reference implementation for real hardware driver.
>>>>>>
>>>>>> Also a real ICF VF driver was also posted here[2] which is a good
>>>>>> reference for vendors who is interested in their own virtio datapath
>>>>>> offloading product.
>>>>>>
>>>>>> Consider mdev framework only support VFIO device and driver right now,
>>>>>> this series also extend it to support other types. This is done
>>>>>> through introducing class id to the device and pairing it with
>>>>>> id_talbe claimed by the driver. On top, this seris also decouple
>>>>>> device specific parents ops out of the common ones.
>>>>>>
>>>>>> Pktgen test was done with virito-net + mvnet loop back device.
>>>>>>
>>>>>> Please review.
>>>>>>
>>>>>> [1] https://lkml.org/lkml/2019/10/31/440
>>>>>> [2] https://lkml.org/lkml/2019/10/15/1226
>>>>>>
>>>>>> Changes from V7:
>>>>>> - drop {set|get}_mdev_features for virtio
>>>>>> - typo and comment style fixes
>>>>> Seems we're nearly there, all the remaining comments are relatively
>>>>> superficial, though I would appreciate a v9 addressing them as well as
>>>>> the checkpatch warnings:
>>>>>
>>>>> https://patchwork.freedesktop.org/series/68977/
>>>>
>>>> Will do.
>>>>
>>>> Btw, do you plan to merge vhost-mdev patch on top? Or you prefer it to
>>>> go through Michael's vhost tree?
>>> I can include it if you wish.  The mdev changes are isolated enough in
>>> that patch that I wouldn't presume it, but clearly it would require
>>> less merge coordination to drop it in my tree.  Let me know.  Thanks,
>>>
>>> Alex
>> I'm fine with merging through your tree. If you do, feel free to
>> include
>>
>> Acked-by: Michael S. Tsirkin <mst@redhat.com>
> AFAICT, it looks like we're expecting at least one more version of
> Tiwei's patch after V5, so it'd probably be best to provide the ack and
> go-ahead on that next version so there's no confusion.  Thanks,
>
> Alex


Yes, it's probably need a V6. Will give ack there.

Thanks