Message ID | 0-v4-0de2f6c78ed0+9d1-iommufd_jgg@nvidia.com (mailing list archive) |
---|---|
Headers | show |
Series | IOMMUFD Generic interface | expand |
On Mon, Nov 07, 2022 at 08:49:08PM -0400, Jason Gunthorpe wrote: > Cover the essential functionality of the iommufd with a directed > test. This aims to achieve reasonable functional coverage using the > in-kernel self test framework. > > It provides a mock kernel module for the iommu_domain that allows it to > run without any HW and the mocking provides a way to directly validate > that the PFNs loaded into the iommu_domain are correct. > > The mock also simulates the rare case of PAGE_SIZE > iommu page size as > the mock will operate at a 2K iommu page size. This allows exercising all > of the calculations to support this mismatch. > > This allows achieving high coverage of the corner cases in the iopt_pages. > > However, it is an unusually invasive config option to enable all of > this. The config option should not be enabled in a production kernel. This patch crossed 100k so it was bounced from the vger lists. If anyone didn't get it and would like to see it lore has it: https://lore.kernel.org/linux-iommu/15-v4-0de2f6c78ed0+9d1-iommufd_jgg@nvidia.com/ For the eventual v5 I will split the nth test into its own patch Jason
On Mon, Nov 07, 2022 at 08:48:53PM -0400, Jason Gunthorpe wrote: > [ > This has been in linux-next for a little while now, and we've completed > the syzkaller run. 1300 hours of CPU time have been invested since the > last report with no improvement in coverage or new detections. syzkaller > coverage reached 69%(75%), and review of the misses show substantial > amounts are WARN_ON's and other debugging which are not expected to be > covered. > ] > > iommufd is the user API to control the IOMMU subsystem as it relates to > managing IO page tables that point at user space memory. [chop cc list] s390 mdev maintainers, Can I ask your help to test this with the two S390 mdev drivers? Now that gvt is passing and we've covered alot of the QA ground it is a good time to run it. Take the branch from here: https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next And build the kernel with CONFIG_VFIO_CONTAINER=n CONFIG_IOMMUFD=y CONFIG_IOMMUFD_VFIO_CONTAINER=y And your existing stuff should work with iommufd providing the iommu support to vfio. There will be a dmesg confirming this. Let me know if there are any problems! If I recall there was some desire from the S390 platform team to start building on iommufd to create some vIOMMU acceleration for S390 guests, this is a necessary first step. Thanks, Jason
On Mon, Nov 07, 2022 at 08:49:08PM -0400, Jason Gunthorpe wrote: > diff --git a/tools/testing/selftests/iommu/iommufd.c b/tools/testing/selftests/iommu/iommufd.c > +TEST_F(iommufd, cmd_length) > +{ > +#define TEST_LENGTH(_struct, _ioctl) \ > + { \ > + struct { \ > + struct _struct cmd; \ > + uint8_t extra; \ > + } cmd = { .cmd = { .size = sizeof(struct _struct) - 1 }, \ > + .extra = UINT8_MAX }; \ > + int old_errno; \ > + int rc; \ > + \ > + EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, _ioctl, &cmd)); \ I guess it should be EINVAL corresponding to updated kernel code? > +TEST_F(iommufd, cmd_ex_fail) > +{ > + struct { > + struct iommu_destroy cmd; > + __u64 future; > + } cmd = { .cmd = { .size = sizeof(cmd), .id = 0 } }; > + > + /* object id is invalid and command is longer */ > + EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd)); > + /* future area is non-zero */ > + cmd.future = 1; > + EXPECT_ERRNO(E2BIG, ioctl(self->fd, IOMMU_DESTROY, &cmd)); > + /* Original command "works" */ > + cmd.cmd.size = sizeof(cmd.cmd); > + EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd)); > + /* Short command fails */ > + cmd.cmd.size = sizeof(cmd.cmd) - 1; > + EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, IOMMU_DESTROY, &cmd)); Ditto > +TEST_HARNESS_MAIN > diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c > +static void fail_nth_first(struct __test_metadata *_metadata, > + struct fail_nth_state *nth_state) > +{ > + char buf[300]; > + > + snprintf(buf, sizeof(buf), "/proc/self/task/%u/fail-nth", gettid()); Not sure what's missing, I have a build error at gettid. Copying a solution from tools/perf/jvmti/jvmti_agent.c file, can fix with: ------------------------------ diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c index 99eaa9f32e0b..7704b3a754d3 100644 --- a/tools/testing/selftests/iommu/iommufd_fail_nth.c +++ b/tools/testing/selftests/iommu/iommufd_fail_nth.c @@ -19,6 +19,7 @@ #define __EXPORTED_HEADERS__ #include <linux/vfio.h> +#include <syscall.h> /* for gettid() */ #include "iommufd_utils.h" @@ -84,6 +85,13 @@ struct fail_nth_state { unsigned int iteration; }; +#ifndef HAVE_GETTID +static inline pid_t gettid(void) +{ + return (pid_t)syscall(__NR_gettid); +} +#endif + static void fail_nth_first(struct __test_metadata *_metadata, struct fail_nth_state *nth_state) { ------------------------------
Am 08.11.22 um 02:09 schrieb Jason Gunthorpe: > On Mon, Nov 07, 2022 at 08:48:53PM -0400, Jason Gunthorpe wrote: >> [ >> This has been in linux-next for a little while now, and we've completed >> the syzkaller run. 1300 hours of CPU time have been invested since the >> last report with no improvement in coverage or new detections. syzkaller >> coverage reached 69%(75%), and review of the misses show substantial >> amounts are WARN_ON's and other debugging which are not expected to be >> covered. >> ] >> >> iommufd is the user API to control the IOMMU subsystem as it relates to >> managing IO page tables that point at user space memory. > > [chop cc list] > > s390 mdev maintainers, > > Can I ask your help to test this with the two S390 mdev drivers? Now > that gvt is passing and we've covered alot of the QA ground it is a > good time to run it. > > Take the branch from here: > > https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next > > And build the kernel with > > CONFIG_VFIO_CONTAINER=n > CONFIG_IOMMUFD=y > CONFIG_IOMMUFD_VFIO_CONTAINER=y > > And your existing stuff should work with iommufd providing the iommu > support to vfio. There will be a dmesg confirming this. Gave it a quick spin with vfio_ap: [ 401.679199] vfio_ap_mdev b01a7c33-9696-48b2-9a98-050e8e17c69a: Adding to iommu group 1 [ 402.085386] iommufd: IOMMUFD is providing /dev/vfio/vfio, not VFIO. Some tests seem to work, but others dont (running into timeouts). I need to look into that (or ideally Tony will have a look, FWIW tests.test_vfio_ap.VfioAPAssignMdevToGuestTest fails for me. The same kernel tree with defconfig (instead of CONFIG_IOMMUFD_VFIO_CONTAINER=y) works fine. > > Let me know if there are any problems! > > If I recall there was some desire from the S390 platform team to start > building on iommufd to create some vIOMMU acceleration for S390 > guests, this is a necessary first step. > > Thanks, > Jason
On Mon, Nov 07, 2022 at 09:48:32PM -0800, Nicolin Chen wrote: > On Mon, Nov 07, 2022 at 08:49:08PM -0400, Jason Gunthorpe wrote: > > > diff --git a/tools/testing/selftests/iommu/iommufd.c b/tools/testing/selftests/iommu/iommufd.c > > > +TEST_F(iommufd, cmd_length) > > +{ > > +#define TEST_LENGTH(_struct, _ioctl) \ > > + { \ > > + struct { \ > > + struct _struct cmd; \ > > + uint8_t extra; \ > > + } cmd = { .cmd = { .size = sizeof(struct _struct) - 1 }, \ > > + .extra = UINT8_MAX }; \ > > + int old_errno; \ > > + int rc; \ > > + \ > > + EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, _ioctl, &cmd)); \ > > I guess it should be EINVAL corresponding to updated kernel code? > > > +TEST_F(iommufd, cmd_ex_fail) > > +{ > > + struct { > > + struct iommu_destroy cmd; > > + __u64 future; > > + } cmd = { .cmd = { .size = sizeof(cmd), .id = 0 } }; > > + > > + /* object id is invalid and command is longer */ > > + EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd)); > > + /* future area is non-zero */ > > + cmd.future = 1; > > + EXPECT_ERRNO(E2BIG, ioctl(self->fd, IOMMU_DESTROY, &cmd)); > > + /* Original command "works" */ > > + cmd.cmd.size = sizeof(cmd.cmd); > > + EXPECT_ERRNO(ENOENT, ioctl(self->fd, IOMMU_DESTROY, &cmd)); > > + /* Short command fails */ > > + cmd.cmd.size = sizeof(cmd.cmd) - 1; > > + EXPECT_ERRNO(EOPNOTSUPP, ioctl(self->fd, IOMMU_DESTROY, &cmd)); > > Ditto Oops, yes, I fixed these > > > +TEST_HARNESS_MAIN > > diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c > > > +static void fail_nth_first(struct __test_metadata *_metadata, > > + struct fail_nth_state *nth_state) > > +{ > > + char buf[300]; > > + > > + snprintf(buf, sizeof(buf), "/proc/self/task/%u/fail-nth", gettid()); > > Not sure what's missing, I have a build error at gettid. Copying > a solution from tools/perf/jvmti/jvmti_agent.c file, can fix with: I think your glibc is probably old > ------------------------------ > diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testing/selftests/iommu/iommufd_fail_nth.c > index 99eaa9f32e0b..7704b3a754d3 100644 > --- a/tools/testing/selftests/iommu/iommufd_fail_nth.c > +++ b/tools/testing/selftests/iommu/iommufd_fail_nth.c > @@ -19,6 +19,7 @@ > > #define __EXPORTED_HEADERS__ > #include <linux/vfio.h> > +#include <syscall.h> /* for gettid() */ > > #include "iommufd_utils.h" > > @@ -84,6 +85,13 @@ struct fail_nth_state { > unsigned int iteration; > }; > > +#ifndef HAVE_GETTID > +static inline pid_t gettid(void) > +{ > + return (pid_t)syscall(__NR_gettid); > +} > +#endif Ah, there is a lot of complicated makefile stuff to make this work, and it only works for perf/bpf not selftests It looks like there is no reason for this to need gettid, we don't use threads. So this can just be getpid and that is portable. Thanks, Jason
On 11/7/22 8:09 PM, Jason Gunthorpe wrote: > On Mon, Nov 07, 2022 at 08:48:53PM -0400, Jason Gunthorpe wrote: >> [ >> This has been in linux-next for a little while now, and we've completed >> the syzkaller run. 1300 hours of CPU time have been invested since the >> last report with no improvement in coverage or new detections. syzkaller >> coverage reached 69%(75%), and review of the misses show substantial >> amounts are WARN_ON's and other debugging which are not expected to be >> covered. >> ] >> >> iommufd is the user API to control the IOMMU subsystem as it relates to >> managing IO page tables that point at user space memory. > > [chop cc list] > > s390 mdev maintainers, > > Can I ask your help to test this with the two S390 mdev drivers? Now > that gvt is passing and we've covered alot of the QA ground it is a > good time to run it. > > Take the branch from here: > > https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next > > And build the kernel with > > CONFIG_VFIO_CONTAINER=n > CONFIG_IOMMUFD=y > CONFIG_IOMMUFD_VFIO_CONTAINER=y > > And your existing stuff should work with iommufd providing the iommu > support to vfio. There will be a dmesg confirming this. > > Let me know if there are any problems! FWIW, vfio-pci via s390 is working fine so far, though I'll put it through more paces over the next few weeks and report if I find anything. As far as mdev drivers... -ccw: Sounds like Eric is already aware there is an issue and is investigating (I see errors as well). -ap: I see the exact same issue that Christian mentioned... I'll talk to Tony & Jason about it. > > If I recall there was some desire from the S390 platform team to start > building on iommufd to create some vIOMMU acceleration for S390 > guests, this is a necessary first step. There's probably something here for -ccw in the future, but you might be thinking of s390 vfio-pci e.g. to implement the in-kernel handling of nested mappings on s390 -- yep, work in in progress here, not ready for sharing yet but I have been most recently basing my work on top of the nesting series https://github.com/yiliu1765/iommufd/tree/iommufd-v6.0-rc3-nesting Matt
On Tue, Nov 08, 2022 at 08:50:53AM -0500, Matthew Rosato wrote: > FWIW, vfio-pci via s390 is working fine so far, though I'll put it > through more paces over the next few weeks and report if I find > anything. OK great > As far as mdev drivers... > > -ccw: Sounds like Eric is already aware there is an issue and is investigating (I see errors as well). > > -ap: I see the exact same issue that Christian mentioned... I'll talk to Tony & Jason about it. A clue what is going wrong might get a quick realization on the problem? I'm guessing something in the vfio side more than the access 'pinning' part? > > If I recall there was some desire from the S390 platform team to start > > building on iommufd to create some vIOMMU acceleration for S390 > > guests, this is a necessary first step. > > There's probably something here for -ccw in the future, but you > might be thinking of s390 vfio-pci e.g. to implement the in-kernel > handling of nested mappings on s390 -- yep, work in in progress > here, not ready for sharing yet but I have been most recently basing > my work on top of the nesting series > https://github.com/yiliu1765/iommufd/tree/iommufd-v6.0-rc3-nesting The relation is that if vfio-pci wants to do the above then vfio-ccw/ap will have to run on iommufd when used in the same VM as vfio-pci. So we need it to work :) Jason
On 11/8/22 5:12 AM, Christian Borntraeger wrote: > > > Am 08.11.22 um 02:09 schrieb Jason Gunthorpe: >> On Mon, Nov 07, 2022 at 08:48:53PM -0400, Jason Gunthorpe wrote: >>> [ >>> This has been in linux-next for a little while now, and we've completed >>> the syzkaller run. 1300 hours of CPU time have been invested since the >>> last report with no improvement in coverage or new detections. >>> syzkaller >>> coverage reached 69%(75%), and review of the misses show substantial >>> amounts are WARN_ON's and other debugging which are not expected to be >>> covered. >>> ] >>> >>> iommufd is the user API to control the IOMMU subsystem as it relates to >>> managing IO page tables that point at user space memory. >> >> [chop cc list] >> >> s390 mdev maintainers, >> >> Can I ask your help to test this with the two S390 mdev drivers? Now >> that gvt is passing and we've covered alot of the QA ground it is a >> good time to run it. >> >> Take the branch from here: >> >> https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next >> > >> >> And build the kernel with >> >> CONFIG_VFIO_CONTAINER=n >> CONFIG_IOMMUFD=y >> CONFIG_IOMMUFD_VFIO_CONTAINER=y >> >> And your existing stuff should work with iommufd providing the iommu >> support to vfio. There will be a dmesg confirming this. > > Gave it a quick spin with vfio_ap: > [ 401.679199] vfio_ap_mdev b01a7c33-9696-48b2-9a98-050e8e17c69a: > Adding to iommu group 1 > [ 402.085386] iommufd: IOMMUFD is providing /dev/vfio/vfio, not VFIO. > > Some tests seem to work, but others dont (running into timeouts). I > need to look > into that (or ideally Tony will have a look, FWIW > tests.test_vfio_ap.VfioAPAssignMdevToGuestTest > fails for me. I'm looking into it. > > > The same kernel tree with defconfig (instead of > CONFIG_IOMMUFD_VFIO_CONTAINER=y) works fine. >> >> Let me know if there are any problems! >> >> If I recall there was some desire from the S390 platform team to start >> building on iommufd to create some vIOMMU acceleration for S390 >> guests, this is a necessary first step. >> >> Thanks, >> Jason
On Tue, 2022-11-08 at 09:54 -0400, Jason Gunthorpe wrote: > On Tue, Nov 08, 2022 at 08:50:53AM -0500, Matthew Rosato wrote: > > > FWIW, vfio-pci via s390 is working fine so far, though I'll put it > > through more paces over the next few weeks and report if I find > > anything. > > OK great > > > As far as mdev drivers... > > > > -ccw: Sounds like Eric is already aware there is an issue and is > > investigating (I see errors as well). I -think- the problem for -ccw is that the new vfio_pin_pages requires the input addresses to be page-aligned, and while most of ours are, the first one in any given transaction may not be. We never bothered to mask off the addresses since it was handled for us, and we needed to keep the offsets anyway. By happenstance, I had some code that would do the masking ourselves (for an unrelated reason); I'll see if I can get that fit on top and if it helps matters. After coffee. Eric > > > > -ap: I see the exact same issue that Christian mentioned... I'll > > talk to Tony & Jason about it. > > A clue what is going wrong might get a quick realization on the > problem? > > I'm guessing something in the vfio side more than the access > 'pinning' > part? > > > > If I recall there was some desire from the S390 platform team to > > > start > > > building on iommufd to create some vIOMMU acceleration for S390 > > > guests, this is a necessary first step. > > > > There's probably something here for -ccw in the future, but you > > might be thinking of s390 vfio-pci e.g. to implement the in-kernel > > handling of nested mappings on s390 -- yep, work in in progress > > here, not ready for sharing yet but I have been most recently > > basing > > my work on top of the nesting series > > https://github.com/yiliu1765/iommufd/tree/iommufd-v6.0-rc3-nesting > > The relation is that if vfio-pci wants to do the above then > vfio-ccw/ap will have to run on iommufd when used in the same VM as > vfio-pci. > > So we need it to work :) > > Jason
On Tue, Nov 08, 2022 at 09:19:17AM -0500, Eric Farman wrote: > On Tue, 2022-11-08 at 09:54 -0400, Jason Gunthorpe wrote: > > On Tue, Nov 08, 2022 at 08:50:53AM -0500, Matthew Rosato wrote: > > > > > FWIW, vfio-pci via s390 is working fine so far, though I'll put it > > > through more paces over the next few weeks and report if I find > > > anything. > > > > OK great > > > > > As far as mdev drivers... > > > > > > -ccw: Sounds like Eric is already aware there is an issue and is > > > investigating (I see errors as well). > > I -think- the problem for -ccw is that the new vfio_pin_pages requires > the input addresses to be page-aligned, and while most of ours are, the > first one in any given transaction may not be. We never bothered to > mask off the addresses since it was handled for us, and we needed to > keep the offsets anyway. > > By happenstance, I had some code that would do the masking ourselves > (for an unrelated reason); I'll see if I can get that fit on top and if > it helps matters. After coffee. Oh, yes, that makes alot of sense. Ah, if that is how VFIO worked we could match it like below: EXPORT_SYMBOL_NS_GPL(iommufd_access_unpin_pages, IOMMUFD); static bool iopt_area_contig_is_aligned(struct iopt_area_contig_iter *iter, - bool first) + bool first, unsigned long first_iova) { - if (iopt_area_start_byte(iter->area, iter->cur_iova) % PAGE_SIZE) + unsigned long start_offset = first ? (first_iova % PAGE_SIZE) : 0; + + if ((iopt_area_start_byte(iter->area, iter->cur_iova) % PAGE_SIZE) != + start_offset) return false; if (!iopt_area_contig_done(iter) && @@ -607,7 +610,7 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, iopt_area_iova_to_index(area, iter.cur_iova); if (area->prevent_access || - !iopt_area_contig_is_aligned(&iter, first)) { + !iopt_area_contig_is_aligned(&iter, first, iova)) { rc = -EINVAL; goto err_remove; } Jason
On Tue, 2022-11-08 at 10:37 -0400, Jason Gunthorpe wrote: > On Tue, Nov 08, 2022 at 09:19:17AM -0500, Eric Farman wrote: > > On Tue, 2022-11-08 at 09:54 -0400, Jason Gunthorpe wrote: > > > On Tue, Nov 08, 2022 at 08:50:53AM -0500, Matthew Rosato wrote: > > > > > > > FWIW, vfio-pci via s390 is working fine so far, though I'll put > > > > it > > > > through more paces over the next few weeks and report if I find > > > > anything. > > > > > > OK great > > > > > > > As far as mdev drivers... > > > > > > > > -ccw: Sounds like Eric is already aware there is an issue and > > > > is > > > > investigating (I see errors as well). > > > > I -think- the problem for -ccw is that the new vfio_pin_pages > > requires > > the input addresses to be page-aligned, and while most of ours are, > > the > > first one in any given transaction may not be. We never bothered to > > mask off the addresses since it was handled for us, and we needed > > to > > keep the offsets anyway. > > > > By happenstance, I had some code that would do the masking > > ourselves > > (for an unrelated reason); I'll see if I can get that fit on top > > and if > > it helps matters. After coffee. > > Oh, yes, that makes alot of sense. > > Ah, if that is how VFIO worked we could match it like below: That's a start. The pin appears to have worked, but the unpin fails at the bottom of iommufd_access_unpin_pages: WARN_ON(!iopt_area_contig_done(&iter)); > > EXPORT_SYMBOL_NS_GPL(iommufd_access_unpin_pages, IOMMUFD); > > static bool iopt_area_contig_is_aligned(struct iopt_area_contig_iter > *iter, > - bool first) > + bool first, unsigned long > first_iova) > { > - if (iopt_area_start_byte(iter->area, iter->cur_iova) % > PAGE_SIZE) > + unsigned long start_offset = first ? (first_iova % PAGE_SIZE) > : 0; > + > + if ((iopt_area_start_byte(iter->area, iter->cur_iova) % > PAGE_SIZE) != > + start_offset) > return false; > > if (!iopt_area_contig_done(iter) && > @@ -607,7 +610,7 @@ int iommufd_access_pin_pages(struct > iommufd_access *access, unsigned long iova, > iopt_area_iova_to_index(area, iter.cur_iova); > > if (area->prevent_access || > - !iopt_area_contig_is_aligned(&iter, first)) { > + !iopt_area_contig_is_aligned(&iter, first, iova)) > { > rc = -EINVAL; > goto err_remove; > } > > Jason
On 11/8/22 10:29 AM, Eric Farman wrote: > On Tue, 2022-11-08 at 10:37 -0400, Jason Gunthorpe wrote: >> On Tue, Nov 08, 2022 at 09:19:17AM -0500, Eric Farman wrote: >>> On Tue, 2022-11-08 at 09:54 -0400, Jason Gunthorpe wrote: >>>> On Tue, Nov 08, 2022 at 08:50:53AM -0500, Matthew Rosato wrote: >>>> >>>>> FWIW, vfio-pci via s390 is working fine so far, though I'll put >>>>> it >>>>> through more paces over the next few weeks and report if I find >>>>> anything. >>>> >>>> OK great >>>> >>>>> As far as mdev drivers... >>>>> >>>>> -ccw: Sounds like Eric is already aware there is an issue and >>>>> is >>>>> investigating (I see errors as well). >>> >>> I -think- the problem for -ccw is that the new vfio_pin_pages >>> requires >>> the input addresses to be page-aligned, and while most of ours are, >>> the >>> first one in any given transaction may not be. We never bothered to >>> mask off the addresses since it was handled for us, and we needed >>> to >>> keep the offsets anyway. >>> >>> By happenstance, I had some code that would do the masking >>> ourselves >>> (for an unrelated reason); I'll see if I can get that fit on top >>> and if >>> it helps matters. After coffee. >> >> Oh, yes, that makes alot of sense. >> >> Ah, if that is how VFIO worked we could match it like below: > > That's a start. The pin appears to have worked, but the unpin fails at > the bottom of iommufd_access_unpin_pages: > > WARN_ON(!iopt_area_contig_done(&iter)); > Update on why -ap is failing -- I see vfio_pin_pages requests from vfio_ap_irq_enable that are failing on -EINVAL -- input is not page-aligned, just like what vfio-ccw was hitting. I just tried a quick hack to force these to page-aligned requests and with that the vfio-ap tests I'm running start passing again. So I think a proper fix in the iommufd code for this will also fix vfio-ap (we will test of course) >> >> EXPORT_SYMBOL_NS_GPL(iommufd_access_unpin_pages, IOMMUFD); >> >> static bool iopt_area_contig_is_aligned(struct iopt_area_contig_iter >> *iter, >> - bool first) >> + bool first, unsigned long >> first_iova) >> { >> - if (iopt_area_start_byte(iter->area, iter->cur_iova) % >> PAGE_SIZE) >> + unsigned long start_offset = first ? (first_iova % PAGE_SIZE) >> : 0; >> + >> + if ((iopt_area_start_byte(iter->area, iter->cur_iova) % >> PAGE_SIZE) != >> + start_offset) >> return false; >> >> if (!iopt_area_contig_done(iter) && >> @@ -607,7 +610,7 @@ int iommufd_access_pin_pages(struct >> iommufd_access *access, unsigned long iova, >> iopt_area_iova_to_index(area, iter.cur_iova); >> >> if (area->prevent_access || >> - !iopt_area_contig_is_aligned(&iter, first)) { >> + !iopt_area_contig_is_aligned(&iter, first, iova)) >> { >> rc = -EINVAL; >> goto err_remove; >> } >> >> Jason >
On Tue, Nov 08, 2022 at 10:29:33AM -0500, Eric Farman wrote: > On Tue, 2022-11-08 at 10:37 -0400, Jason Gunthorpe wrote: > > On Tue, Nov 08, 2022 at 09:19:17AM -0500, Eric Farman wrote: > > > On Tue, 2022-11-08 at 09:54 -0400, Jason Gunthorpe wrote: > > > > On Tue, Nov 08, 2022 at 08:50:53AM -0500, Matthew Rosato wrote: > > > > > > > > > FWIW, vfio-pci via s390 is working fine so far, though I'll put > > > > > it > > > > > through more paces over the next few weeks and report if I find > > > > > anything. > > > > > > > > OK great > > > > > > > > > As far as mdev drivers... > > > > > > > > > > -ccw: Sounds like Eric is already aware there is an issue and > > > > > is > > > > > investigating (I see errors as well). > > > > > > I -think- the problem for -ccw is that the new vfio_pin_pages > > > requires > > > the input addresses to be page-aligned, and while most of ours are, > > > the > > > first one in any given transaction may not be. We never bothered to > > > mask off the addresses since it was handled for us, and we needed > > > to > > > keep the offsets anyway. > > > > > > By happenstance, I had some code that would do the masking > > > ourselves > > > (for an unrelated reason); I'll see if I can get that fit on top > > > and if > > > it helps matters. After coffee. > > > > Oh, yes, that makes alot of sense. > > > > Ah, if that is how VFIO worked we could match it like below: > > That's a start. The pin appears to have worked, but the unpin fails at > the bottom of iommufd_access_unpin_pages: > > WARN_ON(!iopt_area_contig_done(&iter)); This seems like a different bug, probably a ccw driver bug. The WARN_ON is designed to detect cases where the driver is unpinning an IOVA range that is not exactly what it pinned. The pin side already does this validation, so if it fails it means pin/unpin did not have identical iova ranges. Some debugging prints should confirm this. I looked at CCW and came up with the following two things, can you look at them and finish them off? It will probably help. Thanks, Jason
On Tue, Nov 08, 2022 at 02:18:12PM -0500, Matthew Rosato wrote: > Update on why -ap is failing -- I see vfio_pin_pages requests from > vfio_ap_irq_enable that are failing on -EINVAL -- input is not > page-aligned, just like what vfio-ccw was hitting. > > I just tried a quick hack to force these to page-aligned requests > and with that the vfio-ap tests I'm running start passing again. So > I think a proper fix in the iommufd code for this will also fix > vfio-ap (we will test of course) Right, so my first fix isn't the right thing. The APIs are mismatched too much. The length gets all messed up in the process. So how about this? (drop the prior attempt) diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index d835a77aaf26d9..b590ca3c186396 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1906,8 +1906,13 @@ int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova, if (iova > ULONG_MAX) return -EINVAL; + /* + * VFIO ignores the sub page offset, npages is from the start of + * a PAGE_SIZE chunk of IOVA. + */ ret = iommufd_access_pin_pages( - device->iommufd_access, iova, npage * PAGE_SIZE, pages, + device->iommufd_access, ALIGN_DOWN(iova, PAGE_SIZE), + npage * PAGE_SIZE, pages, (prot & IOMMU_WRITE) ? IOMMUFD_ACCESS_RW_WRITE : 0); if (ret) return ret; @@ -1937,7 +1942,8 @@ void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, int npage) if (device->iommufd_access) { if (WARN_ON(iova > ULONG_MAX)) return; - iommufd_access_unpin_pages(device->iommufd_access, iova, + iommufd_access_unpin_pages(device->iommufd_access, + ALIGN_DOWN(iova, PAGE_SIZE), npage * PAGE_SIZE); return; } Thanks, Jason
On Tue, 2022-11-08 at 15:34 -0400, Jason Gunthorpe wrote: > On Tue, Nov 08, 2022 at 10:29:33AM -0500, Eric Farman wrote: > > On Tue, 2022-11-08 at 10:37 -0400, Jason Gunthorpe wrote: > > > On Tue, Nov 08, 2022 at 09:19:17AM -0500, Eric Farman wrote: > > > > On Tue, 2022-11-08 at 09:54 -0400, Jason Gunthorpe wrote: > > > > > On Tue, Nov 08, 2022 at 08:50:53AM -0500, Matthew Rosato > > > > > wrote: > > > > > > > > > > > FWIW, vfio-pci via s390 is working fine so far, though I'll > > > > > > put > > > > > > it > > > > > > through more paces over the next few weeks and report if I > > > > > > find > > > > > > anything. > > > > > > > > > > OK great > > > > > > > > > > > As far as mdev drivers... > > > > > > > > > > > > -ccw: Sounds like Eric is already aware there is an issue > > > > > > and > > > > > > is > > > > > > investigating (I see errors as well). > > > > > > > > I -think- the problem for -ccw is that the new vfio_pin_pages > > > > requires > > > > the input addresses to be page-aligned, and while most of ours > > > > are, > > > > the > > > > first one in any given transaction may not be. We never > > > > bothered to > > > > mask off the addresses since it was handled for us, and we > > > > needed > > > > to > > > > keep the offsets anyway. > > > > > > > > By happenstance, I had some code that would do the masking > > > > ourselves > > > > (for an unrelated reason); I'll see if I can get that fit on > > > > top > > > > and if > > > > it helps matters. After coffee. > > > > > > Oh, yes, that makes alot of sense. > > > > > > Ah, if that is how VFIO worked we could match it like below: > > > > That's a start. The pin appears to have worked, but the unpin fails > > at > > the bottom of iommufd_access_unpin_pages: > > > > WARN_ON(!iopt_area_contig_done(&iter)); > > This seems like a different bug, probably a ccw driver bug. The > WARN_ON is designed to detect cases where the driver is unpinning an > IOVA range that is not exactly what it pinned. The pin side already > does this validation, so if it fails it means pin/unpin did not have > identical iova ranges. Some debugging prints should confirm this. > > I looked at CCW and came up with the following two things, can you > look at them and finish them off? It will probably help. I happen to already have patch 1 in a series I've been working on in parallel with the private/parent split. I haven't forgotten it. :) Patch 2 doesn't address the above symptoms, but a lot of that code is getting reworked by the aforementioned series so I didn't spend a lot of time studying your suggestion. And as I type this I see you just sent a new patch, let me go try that...
On Tue, Nov 08, 2022 at 03:07:19PM -0500, Eric Farman wrote: > Patch 2 doesn't address the above symptoms, but a lot of that code is > getting reworked by the aforementioned series so I didn't spend a lot > of time studying your suggestion. And as I type this I see you just > sent a new patch, let me go try that... Patch2 is following the assumption the WARN_ON is triggered by a failure path that isn't work right. Eg trying to unpin a 0 IOVA because the failure flow is wrong. Removing all the calls to unpin on failure paths make that impossible. Jason
On Tue, 2022-11-08 at 16:04 -0400, Jason Gunthorpe wrote: > On Tue, Nov 08, 2022 at 02:18:12PM -0500, Matthew Rosato wrote: > > > Update on why -ap is failing -- I see vfio_pin_pages requests from > > vfio_ap_irq_enable that are failing on -EINVAL -- input is not > > page-aligned, just like what vfio-ccw was hitting. > > > > I just tried a quick hack to force these to page-aligned requests > > and with that the vfio-ap tests I'm running start passing again. > > So > > I think a proper fix in the iommufd code for this will also fix > > vfio-ap (we will test of course) > > Right, so my first fix isn't the right thing. The APIs are mismatched > too much. The length gets all messed up in the process. > > So how about this? (drop the prior attempt) That seems to get the sniff tests for both -ccw and -ap working. I'll keep playing with it for -ccw; Tony and Jason can do more validation on the -ap side. > > diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c > index d835a77aaf26d9..b590ca3c186396 100644 > --- a/drivers/vfio/vfio_main.c > +++ b/drivers/vfio/vfio_main.c > @@ -1906,8 +1906,13 @@ int vfio_pin_pages(struct vfio_device *device, > dma_addr_t iova, > > if (iova > ULONG_MAX) > return -EINVAL; > + /* > + * VFIO ignores the sub page offset, npages is from > the start of > + * a PAGE_SIZE chunk of IOVA. > + */ > ret = iommufd_access_pin_pages( > - device->iommufd_access, iova, npage * > PAGE_SIZE, pages, > + device->iommufd_access, ALIGN_DOWN(iova, > PAGE_SIZE), > + npage * PAGE_SIZE, pages, > (prot & IOMMU_WRITE) ? > IOMMUFD_ACCESS_RW_WRITE : 0); > if (ret) > return ret; > @@ -1937,7 +1942,8 @@ void vfio_unpin_pages(struct vfio_device > *device, dma_addr_t iova, int npage) > if (device->iommufd_access) { > if (WARN_ON(iova > ULONG_MAX)) > return; > - iommufd_access_unpin_pages(device->iommufd_access, > iova, > + iommufd_access_unpin_pages(device->iommufd_access, > + ALIGN_DOWN(iova, > PAGE_SIZE), > npage * PAGE_SIZE); > return; > } > > Thanks, > Jason
On 11/8/22 9:04 AM, Anthony Krowiak wrote: > > On 11/8/22 5:12 AM, Christian Borntraeger wrote: >> >> >> Am 08.11.22 um 02:09 schrieb Jason Gunthorpe: >>> On Mon, Nov 07, 2022 at 08:48:53PM -0400, Jason Gunthorpe wrote: >>>> [ >>>> This has been in linux-next for a little while now, and we've >>>> completed >>>> the syzkaller run. 1300 hours of CPU time have been invested since the >>>> last report with no improvement in coverage or new detections. >>>> syzkaller >>>> coverage reached 69%(75%), and review of the misses show substantial >>>> amounts are WARN_ON's and other debugging which are not expected to be >>>> covered. >>>> ] >>>> >>>> iommufd is the user API to control the IOMMU subsystem as it >>>> relates to >>>> managing IO page tables that point at user space memory. >>> >>> [chop cc list] >>> >>> s390 mdev maintainers, >>> >>> Can I ask your help to test this with the two S390 mdev drivers? Now >>> that gvt is passing and we've covered alot of the QA ground it is a >>> good time to run it. >>> >>> Take the branch from here: >>> >>> https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next >>> >> >>> >>> And build the kernel with >>> >>> CONFIG_VFIO_CONTAINER=n >>> CONFIG_IOMMUFD=y >>> CONFIG_IOMMUFD_VFIO_CONTAINER=y >>> >>> And your existing stuff should work with iommufd providing the iommu >>> support to vfio. There will be a dmesg confirming this. >> >> Gave it a quick spin with vfio_ap: >> [ 401.679199] vfio_ap_mdev b01a7c33-9696-48b2-9a98-050e8e17c69a: >> Adding to iommu group 1 >> [ 402.085386] iommufd: IOMMUFD is providing /dev/vfio/vfio, not VFIO. >> >> Some tests seem to work, but others dont (running into timeouts). I >> need to look >> into that (or ideally Tony will have a look, FWIW >> tests.test_vfio_ap.VfioAPAssignMdevToGuestTest >> fails for me. > > > I'm looking into it. I cloned the https://lore.kernel.org/kvm/Y2q3nFXwOk9jul5u@nvidia.com/T/#m76a9c609c5ccd1494c05c6f598f9c8e75b7c9888 repo and ran the vfio_ap test cases. The tests ran without encountering the errors related to the vfio_pin_pages() function, but I did see two tests fail attempting to run crypto tests on the guest. I also saw a WARN_ON stack trace in the dmesg output indicating a timeout occurred trying to verify the completion of a queue reset. The reset problem has reared its ugly head in our CI, so this may be a good thing as it will allow me to debug why its happening. > > >> >> >> The same kernel tree with defconfig (instead of >> CONFIG_IOMMUFD_VFIO_CONTAINER=y) works fine. >>> >>> Let me know if there are any problems! >>> >>> If I recall there was some desire from the S390 platform team to start >>> building on iommufd to create some vIOMMU acceleration for S390 >>> guests, this is a necessary first step. >>> >>> Thanks, >>> Jason
On Wed, Nov 09, 2022 at 09:49:01AM -0500, Anthony Krowiak wrote: > I cloned the https://lore.kernel.org/kvm/Y2q3nFXwOk9jul5u@nvidia.com/T/#m76a9c609c5ccd1494c05c6f598f9c8e75b7c9888 > repo and ran the vfio_ap test cases. The tests ran without encountering the > errors related to the vfio_pin_pages() function I updated the git repos with this change now > but I did see two tests fail attempting to run crypto tests on the > guest. I also saw a WARN_ON stack trace in the dmesg output > indicating a timeout occurred trying to verify the completion of a > queue reset. The reset problem has reared its ugly head in our CI, > so this may be a good thing as it will allow me to debug why its > happening. Please let me know if you think this is iommufd related, from your description it sounds like an existing latent bug? Thanks, Jason
On 11/9/22 9:49 AM, Anthony Krowiak wrote: > > On 11/8/22 9:04 AM, Anthony Krowiak wrote: >> >> On 11/8/22 5:12 AM, Christian Borntraeger wrote: >>> >>> >>> Am 08.11.22 um 02:09 schrieb Jason Gunthorpe: >>>> On Mon, Nov 07, 2022 at 08:48:53PM -0400, Jason Gunthorpe wrote: >>>>> [ >>>>> This has been in linux-next for a little while now, and we've >>>>> completed >>>>> the syzkaller run. 1300 hours of CPU time have been invested since >>>>> the >>>>> last report with no improvement in coverage or new detections. >>>>> syzkaller >>>>> coverage reached 69%(75%), and review of the misses show substantial >>>>> amounts are WARN_ON's and other debugging which are not expected >>>>> to be >>>>> covered. >>>>> ] >>>>> >>>>> iommufd is the user API to control the IOMMU subsystem as it >>>>> relates to >>>>> managing IO page tables that point at user space memory. >>>> >>>> [chop cc list] >>>> >>>> s390 mdev maintainers, >>>> >>>> Can I ask your help to test this with the two S390 mdev drivers? Now >>>> that gvt is passing and we've covered alot of the QA ground it is a >>>> good time to run it. >>>> >>>> Take the branch from here: >>>> >>>> https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git/log/?h=for-next >>>> >>> >>>> >>>> And build the kernel with >>>> >>>> CONFIG_VFIO_CONTAINER=n >>>> CONFIG_IOMMUFD=y >>>> CONFIG_IOMMUFD_VFIO_CONTAINER=y >>>> >>>> And your existing stuff should work with iommufd providing the iommu >>>> support to vfio. There will be a dmesg confirming this. >>> >>> Gave it a quick spin with vfio_ap: >>> [ 401.679199] vfio_ap_mdev b01a7c33-9696-48b2-9a98-050e8e17c69a: >>> Adding to iommu group 1 >>> [ 402.085386] iommufd: IOMMUFD is providing /dev/vfio/vfio, not VFIO. >>> >>> Some tests seem to work, but others dont (running into timeouts). I >>> need to look >>> into that (or ideally Tony will have a look, FWIW >>> tests.test_vfio_ap.VfioAPAssignMdevToGuestTest >>> fails for me. >> >> >> I'm looking into it. > > > I cloned the > https://lore.kernel.org/kvm/Y2q3nFXwOk9jul5u@nvidia.com/T/#m76a9c609c5ccd1494c05c6f598f9c8e75b7c9888 > repo and ran the vfio_ap test cases. The tests ran without > encountering the errors related to the vfio_pin_pages() function, but > I did see two tests fail attempting to run crypto tests on the guest. > I also saw a WARN_ON stack trace in the dmesg output indicating a > timeout occurred trying to verify the completion of a queue reset. The > reset problem has reared its ugly head in our CI, so this may be a > good thing as it will allow me to debug why its happening. The problems I encountered were due to using a set of regression tests that were not vanilla. They contained some changes I made to try to improve performance of the tests. After restoring the vanilla regression tests, I was able to successfully execute the tests without any problems or errors with Jason's vfio_ap_(un)pin_pages patch installed, so that looks good to me. > > >> >> >>> >>> >>> The same kernel tree with defconfig (instead of >>> CONFIG_IOMMUFD_VFIO_CONTAINER=y) works fine. >>>> >>>> Let me know if there are any problems! >>>> >>>> If I recall there was some desire from the S390 platform team to start >>>> building on iommufd to create some vIOMMU acceleration for S390 >>>> guests, this is a necessary first step. >>>> >>>> Thanks, >>>> Jason
On 11/9/22 11:12 AM, Jason Gunthorpe wrote: > On Wed, Nov 09, 2022 at 09:49:01AM -0500, Anthony Krowiak wrote: >> I cloned the https://lore.kernel.org/kvm/Y2q3nFXwOk9jul5u@nvidia.com/T/#m76a9c609c5ccd1494c05c6f598f9c8e75b7c9888 >> repo and ran the vfio_ap test cases. The tests ran without encountering the >> errors related to the vfio_pin_pages() function > I updated the git repos with this change now > >> but I did see two tests fail attempting to run crypto tests on the >> guest. I also saw a WARN_ON stack trace in the dmesg output >> indicating a timeout occurred trying to verify the completion of a >> queue reset. The reset problem has reared its ugly head in our CI, >> so this may be a good thing as it will allow me to debug why its >> happening. > Please let me know if you think this is iommufd related, from your > description it sounds like an existing latent bug? Just in case you missed my response to my previous email, the problems I was seeing were due to using a set regression tests that I patched to try to improve the tests performance. When I ran the vanilla tests they ran successfully without any problems with your patch. I will continue testing but as of now, it looks good to me. > > Thanks, > Jason
On Wed, Nov 09, 2022 at 02:13:22PM -0500, Anthony Krowiak wrote: > > On 11/9/22 11:12 AM, Jason Gunthorpe wrote: > > On Wed, Nov 09, 2022 at 09:49:01AM -0500, Anthony Krowiak wrote: > > > I cloned the https://lore.kernel.org/kvm/Y2q3nFXwOk9jul5u@nvidia.com/T/#m76a9c609c5ccd1494c05c6f598f9c8e75b7c9888 > > > repo and ran the vfio_ap test cases. The tests ran without encountering the > > > errors related to the vfio_pin_pages() function > > I updated the git repos with this change now > > > > > but I did see two tests fail attempting to run crypto tests on the > > > guest. I also saw a WARN_ON stack trace in the dmesg output > > > indicating a timeout occurred trying to verify the completion of a > > > queue reset. The reset problem has reared its ugly head in our CI, > > > so this may be a good thing as it will allow me to debug why its > > > happening. > > Please let me know if you think this is iommufd related, from your > > description it sounds like an existing latent bug? > > > Just in case you missed my response to my previous email, the problems I was > seeing were due to using a set regression tests that I patched to try to > improve the tests performance. When I ran the vanilla tests they ran > successfully without any problems with your patch. I will continue testing > but as of now, it looks good to me. Great, can all of you provide some Tested-by's for the various things you've run? Thanks, Jason
> -----Original Message----- > From: Jason Gunthorpe [mailto:jgg@nvidia.com] > Sent: 08 November 2022 00:49 > To: bpf@vger.kernel.org; Jonathan Corbet <corbet@lwn.net>; David > Woodhouse <dwmw2@infradead.org>; iommu@lists.linux.dev; Joerg Roedel > <joro@8bytes.org>; Kevin Tian <kevin.tian@intel.com>; > linux-doc@vger.kernel.org; linux-kselftest@vger.kernel.org; > llvm@lists.linux.dev; Nathan Chancellor <nathan@kernel.org>; Nick > Desaulniers <ndesaulniers@google.com>; Miguel Ojeda <ojeda@kernel.org>; > Robin Murphy <robin.murphy@arm.com>; Shuah Khan <shuah@kernel.org>; > Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>; Tom Rix > <trix@redhat.com>; Will Deacon <will@kernel.org> > Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu > <baolu.lu@linux.intel.com>; Chaitanya Kulkarni <chaitanyak@nvidia.com>; > Cornelia Huck <cohuck@redhat.com>; Daniel Jordan > <daniel.m.jordan@oracle.com>; David Gibson > <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric > Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>; > Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins > <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato > <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>; Nicolin > Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>; > Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi > Liu <yi.l.liu@intel.com>; zhukeqian <zhukeqian1@huawei.com> > Subject: [PATCH v4 00/17] IOMMUFD Generic interface [...] > > - Userspace page tables aka 'nested translation' for ARM and Intel iommu > drivers: > https://github.com/nicolinc/iommufd/commits/iommufd_nesting Hi Eric/Yi/Nicolin, Could you please provide a latest Kernel/Qemu branch for the ARM nesting support? The above link points to Yi's git, but not sure which one is latest/stable to have a play. Thanks, Shameer
Hi Shameer, On 2022/11/11 23:51, Shameerali Kolothum Thodi wrote: > > >> -----Original Message----- >> From: Jason Gunthorpe [mailto:jgg@nvidia.com] >> Sent: 08 November 2022 00:49 >> To: bpf@vger.kernel.org; Jonathan Corbet <corbet@lwn.net>; David >> Woodhouse <dwmw2@infradead.org>; iommu@lists.linux.dev; Joerg Roedel >> <joro@8bytes.org>; Kevin Tian <kevin.tian@intel.com>; >> linux-doc@vger.kernel.org; linux-kselftest@vger.kernel.org; >> llvm@lists.linux.dev; Nathan Chancellor <nathan@kernel.org>; Nick >> Desaulniers <ndesaulniers@google.com>; Miguel Ojeda <ojeda@kernel.org>; >> Robin Murphy <robin.murphy@arm.com>; Shuah Khan <shuah@kernel.org>; >> Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>; Tom Rix >> <trix@redhat.com>; Will Deacon <will@kernel.org> >> Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu >> <baolu.lu@linux.intel.com>; Chaitanya Kulkarni <chaitanyak@nvidia.com>; >> Cornelia Huck <cohuck@redhat.com>; Daniel Jordan >> <daniel.m.jordan@oracle.com>; David Gibson >> <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric >> Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>; >> Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins >> <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato >> <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>; Nicolin >> Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>; >> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi >> Liu <yi.l.liu@intel.com>; zhukeqian <zhukeqian1@huawei.com> >> Subject: [PATCH v4 00/17] IOMMUFD Generic interface > [...] >> >> - Userspace page tables aka 'nested translation' for ARM and Intel iommu >> drivers: >> https://github.com/nicolinc/iommufd/commits/iommufd_nesting > > Hi Eric/Yi/Nicolin, > > Could you please provide a latest Kernel/Qemu branch for the ARM nesting support? > The above link points to Yi's git, but not sure which one is latest/stable to > have a play. Nicolin and I are working on the new version for nesting support. Below kernl branch is our latest progress so far. As the naming, it's still wip. We also need to workout a Qemu version, so still need some time before sharing with you. https://github.com/yiliu1765/iommufd/tree/wip/iommufd-v6.1-rc3-nesting
> -----Original Message----- > From: Yi Liu [mailto:yi.l.liu@intel.com] > Sent: 12 November 2022 12:45 > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; > Jason Gunthorpe <jgg@nvidia.com>; bpf@vger.kernel.org; Jonathan Corbet > <corbet@lwn.net>; David Woodhouse <dwmw2@infradead.org>; > iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; Kevin Tian > <kevin.tian@intel.com>; linux-doc@vger.kernel.org; > linux-kselftest@vger.kernel.org; llvm@lists.linux.dev; Nathan Chancellor > <nathan@kernel.org>; Nick Desaulniers <ndesaulniers@google.com>; Miguel > Ojeda <ojeda@kernel.org>; Robin Murphy <robin.murphy@arm.com>; Shuah > Khan <shuah@kernel.org>; Suravee Suthikulpanit > <suravee.suthikulpanit@amd.com>; Tom Rix <trix@redhat.com>; Will > Deacon <will@kernel.org> > Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu > <baolu.lu@linux.intel.com>; Chaitanya Kulkarni <chaitanyak@nvidia.com>; > Cornelia Huck <cohuck@redhat.com>; Daniel Jordan > <daniel.m.jordan@oracle.com>; David Gibson > <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric > Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>; > Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins > <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato > <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>; Nicolin > Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>; > zhukeqian <zhukeqian1@huawei.com> > Subject: Re: [PATCH v4 00/17] IOMMUFD Generic interface > > Hi Shameer, > > On 2022/11/11 23:51, Shameerali Kolothum Thodi wrote: > > > > > >> -----Original Message----- > >> From: Jason Gunthorpe [mailto:jgg@nvidia.com] > >> Sent: 08 November 2022 00:49 > >> To: bpf@vger.kernel.org; Jonathan Corbet <corbet@lwn.net>; David > >> Woodhouse <dwmw2@infradead.org>; iommu@lists.linux.dev; Joerg > Roedel > >> <joro@8bytes.org>; Kevin Tian <kevin.tian@intel.com>; > >> linux-doc@vger.kernel.org; linux-kselftest@vger.kernel.org; > >> llvm@lists.linux.dev; Nathan Chancellor <nathan@kernel.org>; Nick > >> Desaulniers <ndesaulniers@google.com>; Miguel Ojeda > <ojeda@kernel.org>; > >> Robin Murphy <robin.murphy@arm.com>; Shuah Khan > <shuah@kernel.org>; > >> Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>; Tom Rix > >> <trix@redhat.com>; Will Deacon <will@kernel.org> > >> Cc: Alex Williamson <alex.williamson@redhat.com>; Lu Baolu > >> <baolu.lu@linux.intel.com>; Chaitanya Kulkarni > <chaitanyak@nvidia.com>; > >> Cornelia Huck <cohuck@redhat.com>; Daniel Jordan > >> <daniel.m.jordan@oracle.com>; David Gibson > >> <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; > Eric > >> Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>; > >> Jean-Philippe Brucker <jean-philippe@linaro.org>; Joao Martins > >> <joao.m.martins@oracle.com>; kvm@vger.kernel.org; Matthew Rosato > >> <mjrosato@linux.ibm.com>; Michael S. Tsirkin <mst@redhat.com>; > Nicolin > >> Chen <nicolinc@nvidia.com>; Niklas Schnelle <schnelle@linux.ibm.com>; > >> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; > Yi > >> Liu <yi.l.liu@intel.com>; zhukeqian <zhukeqian1@huawei.com> > >> Subject: [PATCH v4 00/17] IOMMUFD Generic interface > > [...] > >> > >> - Userspace page tables aka 'nested translation' for ARM and Intel iommu > >> drivers: > >> https://github.com/nicolinc/iommufd/commits/iommufd_nesting > > > > Hi Eric/Yi/Nicolin, > > > > Could you please provide a latest Kernel/Qemu branch for the ARM nesting > support? > > The above link points to Yi's git, but not sure which one is latest/stable to > > have a play. > > Nicolin and I are working on the new version for nesting support. Below > kernl branch is our latest progress so far. As the naming, it's still > wip. We also need to workout a Qemu version, so still need some time > before sharing with you. > > https://github.com/yiliu1765/iommufd/tree/wip/iommufd-v6.1-rc3-nesting Hi Yi, Thanks for that. I attempted ARM vSVA support based on your above branch and related Qemu branch. With few hacks and additional patches the prototype code works well on HiSilicon ARM platform. Please find the corresponding branches ere, https://github.com/hisilicon/kernel-dev/tree/iommufd-v6.1-rc3-nesting-arm-vSVA https://github.com/hisilicon/qemu/tree/qemu-iommufd-6.1-rc3-arm-vSVA Please let me know if there are any recent branches for ARM support. Thanks, Shameer > > -- > Regards, > Yi Liu
On Tue, Jan 10, 2023 at 11:35:59AM +0000, Shameerali Kolothum Thodi wrote: > Thanks for that. I attempted ARM vSVA support based on your above branch > and related Qemu branch. With few hacks and additional patches the prototype > code works well on HiSilicon ARM platform. Nice! > Please find the corresponding branches ere, > https://github.com/hisilicon/kernel-dev/tree/iommufd-v6.1-rc3-nesting-arm-vSVA We need to have a big think about how the PASID/PRI caps should be working in VFIO.. The whole PRI thing needs to be its own series and need a careful look Can you also look at the dirty tracking stuff? I'd really like to see that done for the huawei vfio live migration driver Thanks, Jason
On 10/01/2023 13:49, Jason Gunthorpe wrote: > Can you also look at the dirty tracking stuff? I'd really like to see > that done for the huawei vfio live migration driver > He did look and provides comments based on his testing of the initial series (IIUC from what we spoke at LPC). v2 should be simpler, though I am still working it out the merging of unit tests into iommufd. My plan once I post the v2 was to handover the SMMUv3 part back to Shameerali given the fact he has hardware that has the feature and can iterate more meaningfully than me. Joao
On Tue, Jan 10, 2023 at 03:16:23PM +0000, Joao Martins wrote: > On 10/01/2023 13:49, Jason Gunthorpe wrote: > > Can you also look at the dirty tracking stuff? I'd really like to see > > that done for the huawei vfio live migration driver > > > > He did look and provides comments based on his testing of the initial series > (IIUC from what we spoke at LPC). v2 should be simpler, though I am still > working it out the merging of unit tests into iommufd. > > My plan once I post the v2 was to handover the SMMUv3 part back to Shameerali > given the fact he has hardware that has the feature and can iterate more > meaningfully than me. Sounds good Thanks, Jason
> -----Original Message----- > From: Joao Martins [mailto:joao.m.martins@oracle.com] > Sent: 10 January 2023 15:16 > To: Jason Gunthorpe <jgg@nvidia.com>; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi@huawei.com> > Cc: Yi Liu <yi.l.liu@intel.com>; bpf@vger.kernel.org; Jonathan Corbet > <corbet@lwn.net>; David Woodhouse <dwmw2@infradead.org>; > iommu@lists.linux.dev; Joerg Roedel <joro@8bytes.org>; Kevin Tian > <kevin.tian@intel.com>; linux-doc@vger.kernel.org; > linux-kselftest@vger.kernel.org; llvm@lists.linux.dev; Nathan Chancellor > <nathan@kernel.org>; Nick Desaulniers <ndesaulniers@google.com>; Miguel > Ojeda <ojeda@kernel.org>; Robin Murphy <robin.murphy@arm.com>; Shuah > Khan <shuah@kernel.org>; Suravee Suthikulpanit > <suravee.suthikulpanit@amd.com>; Tom Rix <trix@redhat.com>; Will > Deacon <will@kernel.org>; Alex Williamson <alex.williamson@redhat.com>; > Lu Baolu <baolu.lu@linux.intel.com>; Chaitanya Kulkarni > <chaitanyak@nvidia.com>; Cornelia Huck <cohuck@redhat.com>; Daniel > Jordan <daniel.m.jordan@oracle.com>; David Gibson > <david@gibson.dropbear.id.au>; Eric Auger <eric.auger@redhat.com>; Eric > Farman <farman@linux.ibm.com>; Jason Wang <jasowang@redhat.com>; > Jean-Philippe Brucker <jean-philippe@linaro.org>; kvm@vger.kernel.org; > Matthew Rosato <mjrosato@linux.ibm.com>; Michael S. Tsirkin > <mst@redhat.com>; Nicolin Chen <nicolinc@nvidia.com>; Niklas Schnelle > <schnelle@linux.ibm.com>; zhukeqian <zhukeqian1@huawei.com> > Subject: Re: [PATCH v4 00/17] IOMMUFD Generic interface > > On 10/01/2023 13:49, Jason Gunthorpe wrote: > > Can you also look at the dirty tracking stuff? I'd really like to see > > that done for the huawei vfio live migration driver > > > > He did look and provides comments based on his testing of the initial series > (IIUC from what we spoke at LPC). v2 should be simpler, though I am still > working it out the merging of unit tests into iommufd. > > My plan once I post the v2 was to handover the SMMUv3 part back to > Shameerali > given the fact he has hardware that has the feature and can iterate more > meaningfully than me. No problem. Happy to help. Thanks, Shameer