Message ID | 1596478888-23030-8-git-send-email-olekstysh@gmail.com (mailing list archive) |
---|---|
State | Superseded |
Headers | show |
Series | IOREQ feature (+ virtio-mmio) on Arm | expand |
On 03.08.2020 20:21, Oleksandr Tyshchenko wrote: > From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> > > Trying to run emulator in driver domain I ran into various issues > mostly policy-related. So this patch tries to resolve all them > plobably in a hackish way. I would like to get feedback how > to implement them properly as having an emulator in driver domain > is a completely valid use-case. From going over the comments I can only derive you want to run an emulator in a driver domain, which doesn't really make sense to me. A driver domain has a different purpose after all. If instead you mean it to be run in just some other domain (which also isn't the domain controlling the target), then there may be more infrastructure changes needed. Paul - was/is your standalone ioreq server (demu?) able to run in other than the domain controlling a guest? Jan
> -----Original Message----- > From: Jan Beulich <jbeulich@suse.com> > Sent: 05 August 2020 17:20 > To: Oleksandr Tyshchenko <olekstysh@gmail.com>; Paul Durrant <paul@xen.org> > Cc: xen-devel@lists.xenproject.org; Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>; Andrew > Cooper <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>; Ian Jackson > <ian.jackson@eu.citrix.com>; Julien Grall <julien@xen.org>; Stefano Stabellini > <sstabellini@kernel.org>; Wei Liu <wl@xen.org>; Daniel De Graaf <dgdegra@tycho.nsa.gov> > Subject: Re: [RFC PATCH V1 07/12] A collection of tweaks to be able to run emulator in driver domain > > On 03.08.2020 20:21, Oleksandr Tyshchenko wrote: > > From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> > > > > Trying to run emulator in driver domain I ran into various issues > > mostly policy-related. So this patch tries to resolve all them > > plobably in a hackish way. I would like to get feedback how > > to implement them properly as having an emulator in driver domain > > is a completely valid use-case. > > From going over the comments I can only derive you want to run > an emulator in a driver domain, which doesn't really make sense > to me. A driver domain has a different purpose after all. If > instead you mean it to be run in just some other domain (which > also isn't the domain controlling the target), then there may > be more infrastructure changes needed. > > Paul - was/is your standalone ioreq server (demu?) able to run > in other than the domain controlling a guest? > Not something I've done yet, but it was always part of the idea so that we could e.g. pass through a device to a dedicated domain and then run multiple demu instances there to virtualize it for many domUs. (I'm thinking here of a device that is not SR-IOV and hence would need some bespoke emulation code to share it out). That dedicated domain would be termed the 'driver domain' simply because it is running the device driver for the h/w that underpins the emulation. Paul
On 05.08.20 19:40, Paul Durrant wrote: Hi Jan, Paul >> -----Original Message----- >> From: Jan Beulich <jbeulich@suse.com> >> Sent: 05 August 2020 17:20 >> To: Oleksandr Tyshchenko <olekstysh@gmail.com>; Paul Durrant <paul@xen.org> >> Cc: xen-devel@lists.xenproject.org; Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>; Andrew >> Cooper <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>; Ian Jackson >> <ian.jackson@eu.citrix.com>; Julien Grall <julien@xen.org>; Stefano Stabellini >> <sstabellini@kernel.org>; Wei Liu <wl@xen.org>; Daniel De Graaf <dgdegra@tycho.nsa.gov> >> Subject: Re: [RFC PATCH V1 07/12] A collection of tweaks to be able to run emulator in driver domain >> >> On 03.08.2020 20:21, Oleksandr Tyshchenko wrote: >>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> >>> >>> Trying to run emulator in driver domain I ran into various issues >>> mostly policy-related. So this patch tries to resolve all them >>> plobably in a hackish way. I would like to get feedback how >>> to implement them properly as having an emulator in driver domain >>> is a completely valid use-case. >> From going over the comments I can only derive you want to run >> an emulator in a driver domain, which doesn't really make sense >> to me. A driver domain has a different purpose after all. If >> instead you mean it to be run in just some other domain (which >> also isn't the domain controlling the target), then there may >> be more infrastructure changes needed. >> >> Paul - was/is your standalone ioreq server (demu?) able to run >> in other than the domain controlling a guest? >> > Not something I've done yet, but it was always part of the idea so that we could e.g. pass through a device to a dedicated domain and then run multiple demu instances there to virtualize it for many domUs. (I'm thinking here of a device that is not SR-IOV and hence would need some bespoke emulation code to share it out).That dedicated domain would be termed the 'driver domain' simply because it is running the device driver for the h/w that underpins the emulation. I may abuse "driver domain" terminology, but indeed in our use-case we pass through a set of H/W devices to a dedicated domain which is running the device drivers for that H/Ws. Our target system comprises a thin Dom0 (without H/W devices at all), DomD (which owns most of the H/W devices) and DomU which runs on virtual devices. This patch tries to make changes at Xen side to be able run standalone ioreq server (emulator) in that dedicated (driver?) domain. Actually the virtio-mmio PoC is based on IOREQ/DM features and emulator (based on demu) acting as a virtio-mmio backend. But it may be various use-cases for that (some mediator for sharing specific H/W resource between Guests or custom PCI emulator for example). If this is valid from Xen PoV I would be happy to get feedback how to transform tweaks (hacks) in current patch into the proper support.
On 06.08.2020 11:22, Oleksandr wrote: > > On 05.08.20 19:40, Paul Durrant wrote: > > Hi Jan, Paul > >>> -----Original Message----- >>> From: Jan Beulich <jbeulich@suse.com> >>> Sent: 05 August 2020 17:20 >>> To: Oleksandr Tyshchenko <olekstysh@gmail.com>; Paul Durrant <paul@xen.org> >>> Cc: xen-devel@lists.xenproject.org; Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>; Andrew >>> Cooper <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>; Ian Jackson >>> <ian.jackson@eu.citrix.com>; Julien Grall <julien@xen.org>; Stefano Stabellini >>> <sstabellini@kernel.org>; Wei Liu <wl@xen.org>; Daniel De Graaf <dgdegra@tycho.nsa.gov> >>> Subject: Re: [RFC PATCH V1 07/12] A collection of tweaks to be able to run emulator in driver domain >>> >>> On 03.08.2020 20:21, Oleksandr Tyshchenko wrote: >>>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> >>>> >>>> Trying to run emulator in driver domain I ran into various issues >>>> mostly policy-related. So this patch tries to resolve all them >>>> plobably in a hackish way. I would like to get feedback how >>>> to implement them properly as having an emulator in driver domain >>>> is a completely valid use-case. >>> From going over the comments I can only derive you want to run >>> an emulator in a driver domain, which doesn't really make sense >>> to me. A driver domain has a different purpose after all. If >>> instead you mean it to be run in just some other domain (which >>> also isn't the domain controlling the target), then there may >>> be more infrastructure changes needed. >>> >>> Paul - was/is your standalone ioreq server (demu?) able to run >>> in other than the domain controlling a guest? >>> >> Not something I've done yet, but it was always part of the idea so that we could e.g. pass through a device to a dedicated domain and then run multiple demu instances there to virtualize it for many domUs. (I'm thinking here of a device that is not SR-IOV and hence would need some bespoke emulation code to share it out).That dedicated domain would be termed the 'driver domain' simply because it is running the device driver for the h/w that underpins the emulation. > > I may abuse "driver domain" terminology, but indeed in our use-case we > pass through a set of H/W devices to a dedicated domain which is running > the device drivers for that H/Ws. Our target system comprises a thin > Dom0 (without H/W devices at all), DomD (which owns most of the H/W > devices) and DomU which runs on virtual devices. This patch tries to > make changes at Xen side to be able run standalone ioreq server > (emulator) in that dedicated (driver?) domain. Okay, in which case I'm fine with the term. I simply wasn't aware of the targeted scenario, sorry. Jan
Hello all. >>>> -----Original Message----- >>>> From: Jan Beulich <jbeulich@suse.com> >>>> Sent: 05 August 2020 17:20 >>>> To: Oleksandr Tyshchenko <olekstysh@gmail.com>; Paul Durrant <paul@xen.org> >>>> Cc: xen-devel@lists.xenproject.org; Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>; Andrew >>>> Cooper <andrew.cooper3@citrix.com>; George Dunlap <george.dunlap@citrix.com>; Ian Jackson >>>> <ian.jackson@eu.citrix.com>; Julien Grall <julien@xen.org>; Stefano Stabellini >>>> <sstabellini@kernel.org>; Wei Liu <wl@xen.org>; Daniel De Graaf <dgdegra@tycho.nsa.gov> >>>> Subject: Re: [RFC PATCH V1 07/12] A collection of tweaks to be able to run emulator in driver domain >>>> >>>> On 03.08.2020 20:21, Oleksandr Tyshchenko wrote: >>>>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> >>>>> >>>>> Trying to run emulator in driver domain I ran into various issues >>>>> mostly policy-related. So this patch tries to resolve all them >>>>> plobably in a hackish way. I would like to get feedback how >>>>> to implement them properly as having an emulator in driver domain >>>>> is a completely valid use-case. >>>> From going over the comments I can only derive you want to run >>>> an emulator in a driver domain, which doesn't really make sense >>>> to me. A driver domain has a different purpose after all. If >>>> instead you mean it to be run in just some other domain (which >>>> also isn't the domain controlling the target), then there may >>>> be more infrastructure changes needed. >>>> >>>> Paul - was/is your standalone ioreq server (demu?) able to run >>>> in other than the domain controlling a guest? >>>> >>> Not something I've done yet, but it was always part of the idea so that we could e.g. pass through a device to a dedicated domain and then run multiple demu instances there to virtualize it for many domUs. (I'm thinking here of a device that is not SR-IOV and hence would need some bespoke emulation code to share it out).That dedicated domain would be termed the 'driver domain' simply because it is running the device driver for the h/w that underpins the emulation. >> I may abuse "driver domain" terminology, but indeed in our use-case we >> pass through a set of H/W devices to a dedicated domain which is running >> the device drivers for that H/Ws. Our target system comprises a thin >> Dom0 (without H/W devices at all), DomD (which owns most of the H/W >> devices) and DomU which runs on virtual devices. This patch tries to >> make changes at Xen side to be able run standalone ioreq server >> (emulator) in that dedicated (driver?) domain. > Okay, in which case I'm fine with the term. I simply wasn't aware of the > targeted scenario, sorry. May I kindly ask to suggest me the pointers how to *properly* resolve various policy related issues described in that patch? Without having them resolved it wouldn't be able to run standalone IOREQ server in driver domain.
On 14/08/2020 17:30, Oleksandr wrote: > > Hello all. > > >>>>> -----Original Message----- >>>>> From: Jan Beulich <jbeulich@suse.com> >>>>> Sent: 05 August 2020 17:20 >>>>> To: Oleksandr Tyshchenko <olekstysh@gmail.com>; Paul Durrant >>>>> <paul@xen.org> >>>>> Cc: xen-devel@lists.xenproject.org; Oleksandr Tyshchenko >>>>> <oleksandr_tyshchenko@epam.com>; Andrew >>>>> Cooper <andrew.cooper3@citrix.com>; George Dunlap >>>>> <george.dunlap@citrix.com>; Ian Jackson >>>>> <ian.jackson@eu.citrix.com>; Julien Grall <julien@xen.org>; Stefano >>>>> Stabellini >>>>> <sstabellini@kernel.org>; Wei Liu <wl@xen.org>; Daniel De Graaf >>>>> <dgdegra@tycho.nsa.gov> >>>>> Subject: Re: [RFC PATCH V1 07/12] A collection of tweaks to be able >>>>> to run emulator in driver domain >>>>> >>>>> On 03.08.2020 20:21, Oleksandr Tyshchenko wrote: >>>>>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> >>>>>> >>>>>> Trying to run emulator in driver domain I ran into various issues >>>>>> mostly policy-related. So this patch tries to resolve all them >>>>>> plobably in a hackish way. I would like to get feedback how >>>>>> to implement them properly as having an emulator in driver domain >>>>>> is a completely valid use-case. >>>>> From going over the comments I can only derive you want to run >>>>> an emulator in a driver domain, which doesn't really make sense >>>>> to me. A driver domain has a different purpose after all. If >>>>> instead you mean it to be run in just some other domain (which >>>>> also isn't the domain controlling the target), then there may >>>>> be more infrastructure changes needed. >>>>> >>>>> Paul - was/is your standalone ioreq server (demu?) able to run >>>>> in other than the domain controlling a guest? >>>>> >>>> Not something I've done yet, but it was always part of the idea so >>>> that we could e.g. pass through a device to a dedicated domain and >>>> then run multiple demu instances there to virtualize it for many >>>> domUs. (I'm thinking here of a device that is not SR-IOV and hence >>>> would need some bespoke emulation code to share it out).That >>>> dedicated domain would be termed the 'driver domain' simply because >>>> it is running the device driver for the h/w that underpins the >>>> emulation. >>> I may abuse "driver domain" terminology, but indeed in our use-case we >>> pass through a set of H/W devices to a dedicated domain which is running >>> the device drivers for that H/Ws. Our target system comprises a thin >>> Dom0 (without H/W devices at all), DomD (which owns most of the H/W >>> devices) and DomU which runs on virtual devices. This patch tries to >>> make changes at Xen side to be able run standalone ioreq server >>> (emulator) in that dedicated (driver?) domain. >> Okay, in which case I'm fine with the term. I simply wasn't aware of the >> targeted scenario, sorry. > > > May I kindly ask to suggest me the pointers how to *properly* resolve > various policy related issues described in that patch? Without having > them resolved it wouldn't be able to run standalone IOREQ server in > driver domain. You could already do that by writing your own XSM policy. Did you explore it? If so, may I ask why this wouldn't be suitable? Also, I would like to emphasis that because of XSA-295 (Unlimited Arm Atomics Operations), you can only run emulators in trusted domain on Arm. There would be more work to do if you wanted to run them in non-trusted environment. Cheers,
On 16.08.20 18:36, Julien Grall wrote: Hi Julien. > > > On 14/08/2020 17:30, Oleksandr wrote: >> >> Hello all. >> >> >>>>>> -----Original Message----- >>>>>> From: Jan Beulich <jbeulich@suse.com> >>>>>> Sent: 05 August 2020 17:20 >>>>>> To: Oleksandr Tyshchenko <olekstysh@gmail.com>; Paul Durrant >>>>>> <paul@xen.org> >>>>>> Cc: xen-devel@lists.xenproject.org; Oleksandr Tyshchenko >>>>>> <oleksandr_tyshchenko@epam.com>; Andrew >>>>>> Cooper <andrew.cooper3@citrix.com>; George Dunlap >>>>>> <george.dunlap@citrix.com>; Ian Jackson >>>>>> <ian.jackson@eu.citrix.com>; Julien Grall <julien@xen.org>; >>>>>> Stefano Stabellini >>>>>> <sstabellini@kernel.org>; Wei Liu <wl@xen.org>; Daniel De Graaf >>>>>> <dgdegra@tycho.nsa.gov> >>>>>> Subject: Re: [RFC PATCH V1 07/12] A collection of tweaks to be >>>>>> able to run emulator in driver domain >>>>>> >>>>>> On 03.08.2020 20:21, Oleksandr Tyshchenko wrote: >>>>>>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> >>>>>>> >>>>>>> Trying to run emulator in driver domain I ran into various issues >>>>>>> mostly policy-related. So this patch tries to resolve all them >>>>>>> plobably in a hackish way. I would like to get feedback how >>>>>>> to implement them properly as having an emulator in driver domain >>>>>>> is a completely valid use-case. >>>>>> From going over the comments I can only derive you want to run >>>>>> an emulator in a driver domain, which doesn't really make sense >>>>>> to me. A driver domain has a different purpose after all. If >>>>>> instead you mean it to be run in just some other domain (which >>>>>> also isn't the domain controlling the target), then there may >>>>>> be more infrastructure changes needed. >>>>>> >>>>>> Paul - was/is your standalone ioreq server (demu?) able to run >>>>>> in other than the domain controlling a guest? >>>>>> >>>>> Not something I've done yet, but it was always part of the idea so >>>>> that we could e.g. pass through a device to a dedicated domain and >>>>> then run multiple demu instances there to virtualize it for many >>>>> domUs. (I'm thinking here of a device that is not SR-IOV and hence >>>>> would need some bespoke emulation code to share it out).That >>>>> dedicated domain would be termed the 'driver domain' simply >>>>> because it is running the device driver for the h/w that underpins >>>>> the emulation. >>>> I may abuse "driver domain" terminology, but indeed in our use-case we >>>> pass through a set of H/W devices to a dedicated domain which is >>>> running >>>> the device drivers for that H/Ws. Our target system comprises a thin >>>> Dom0 (without H/W devices at all), DomD (which owns most of the H/W >>>> devices) and DomU which runs on virtual devices. This patch tries to >>>> make changes at Xen side to be able run standalone ioreq server >>>> (emulator) in that dedicated (driver?) domain. >>> Okay, in which case I'm fine with the term. I simply wasn't aware of >>> the >>> targeted scenario, sorry. >> >> >> May I kindly ask to suggest me the pointers how to *properly* resolve >> various policy related issues described in that patch? Without having >> them resolved it wouldn't be able to run standalone IOREQ server in >> driver domain. > > You could already do that by writing your own XSM policy. Did you > explore it? If so, may I ask why this wouldn't be suitable? > > Also, I would like to emphasis that because of XSA-295 (Unlimited Arm > Atomics Operations), you can only run emulators in trusted domain on Arm. > > There would be more work to do if you wanted to run them in > non-trusted environment. Thank you for the explanation. Yes, we consider driver domain as a trusted domain, there is no plan to run emulator in non-trusted domains. Indeed, it worth trying to write our own policy which will cover our use case (with emulator in driver domain) rather than tweak Xen's default one.
diff --git a/xen/common/domain.c b/xen/common/domain.c index e9be05f..5c9fef2 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -695,6 +695,7 @@ int domain_kill(struct domain *d) { int rc = 0; struct vcpu *v; + struct domain *td; if ( d == current->domain ) return -EINVAL; @@ -733,6 +734,20 @@ int domain_kill(struct domain *d) * have to be put before we call put_domain. */ vm_event_cleanup(d); put_domain(d); + /* + * XEN_DOMCTL_set_target implementation holds reference on + * target domain which doesn't allow to completely destroy it. + * Check if the reference are hold by someone and drop it + * when destroying target domain. + */ + for_each_domain ( td ) { + if ( td->target == d ) { + td->target = NULL; + put_domain(d); + break; + } + } + send_global_virq(VIRQ_DOM_EXC); /* fallthrough */ case DOMDYING_dead: diff --git a/xen/common/domctl.c b/xen/common/domctl.c index a69b3b5..079c7b0 100644 --- a/xen/common/domctl.c +++ b/xen/common/domctl.c @@ -871,6 +871,12 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl) if ( (d == e) || (d->target != NULL) ) { put_domain(e); + /* + * Be a little bit more polite here, looks like the emulator + * has just been restarted. + */ + if ( d->target == e ) + ret = 0; break; } @@ -883,7 +889,7 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl) break; } - /* Hold reference on @e until we destroy @d. */ + /* Hold reference on @e until we destroy either @d or @e */ d->target = e; break; } diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c index a8d182b5..2aa497a 100644 --- a/xen/common/event_channel.c +++ b/xen/common/event_channel.c @@ -235,7 +235,12 @@ static long evtchn_alloc_unbound(evtchn_alloc_unbound_t *alloc) ERROR_EXIT_DOM(port, d); chn = evtchn_from_port(d, port); - rc = xsm_evtchn_unbound(XSM_TARGET, d, chn, alloc->remote_dom); + /* + * XXX: XSM_TARGET is not functional for emulator running in driver domain. + * See xsm_default_action for details. Probably XSM_DM_PRIV could work, + * but there is a risk to break other users. + */ + rc = xsm_evtchn_unbound(XSM_HOOK, d, chn, alloc->remote_dom); if ( rc ) goto out; @@ -1218,7 +1223,12 @@ int alloc_unbound_xen_event_channel( port = rc; chn = evtchn_from_port(ld, port); - rc = xsm_evtchn_unbound(XSM_TARGET, ld, chn, remote_domid); + /* + * XXX: XSM_TARGET is not functional for emulator running in driver domain. + * See xsm_default_action for details. Probably XSM_DM_PRIV could work, + * but there is a risk to break other users. + */ + rc = xsm_evtchn_unbound(XSM_HOOK, ld, chn, remote_domid); if ( rc ) goto out; diff --git a/xen/common/memory.c b/xen/common/memory.c index 0000477..8b306f6 100644 --- a/xen/common/memory.c +++ b/xen/common/memory.c @@ -1153,12 +1153,18 @@ static int acquire_resource( unsigned int i; /* + * XXX: Ugly hack for now to let emulator running in driver domain + * to succeeded in acquiring resource. + */ +#if 0 + /* * FIXME: Until foreign pages inserted into the P2M are properly * reference counted, it is unsafe to allow mapping of * resource pages unless the caller is the hardware domain. */ if ( !is_hardware_domain(currd) ) return -EACCES; +#endif if ( copy_from_guest(gfn_list, xmar.frame_list, xmar.nr_frames) ) rc = -EFAULT; diff --git a/xen/include/xsm/dummy.h b/xen/include/xsm/dummy.h index 317455a..c0813c0 100644 --- a/xen/include/xsm/dummy.h +++ b/xen/include/xsm/dummy.h @@ -139,13 +139,23 @@ static XSM_INLINE int xsm_domctl(XSM_DEFAULT_ARG struct domain *d, int cmd) XSM_ASSERT_ACTION(XSM_OTHER); switch ( cmd ) { + /* + * XXX: Emulator running in driver domain tries to get vcpus num. + * Probably we could avoid that change by modifying emulator to not use + * domctl for getting vcpus num. + */ + case XEN_DOMCTL_getdomaininfo: + /* + * XXX: XSM_DM_PRIV is not functional for emulator running in driver domain + * without setting a target in advance. See xsm_default_action for details. + */ + case XEN_DOMCTL_set_target: + return xsm_default_action(XSM_HOOK, current->domain, d); case XEN_DOMCTL_ioport_mapping: case XEN_DOMCTL_memory_mapping: case XEN_DOMCTL_bind_pt_irq: case XEN_DOMCTL_unbind_pt_irq: return xsm_default_action(XSM_DM_PRIV, current->domain, d); - case XEN_DOMCTL_getdomaininfo: - return xsm_default_action(XSM_XS_PRIV, current->domain, d); default: return xsm_default_action(XSM_PRIV, current->domain, d); } @@ -275,7 +285,7 @@ static XSM_INLINE int xsm_claim_pages(XSM_DEFAULT_ARG struct domain *d) static XSM_INLINE int xsm_evtchn_unbound(XSM_DEFAULT_ARG struct domain *d, struct evtchn *chn, domid_t id2) { - XSM_ASSERT_ACTION(XSM_TARGET); + XSM_ASSERT_ACTION(XSM_HOOK); return xsm_default_action(action, current->domain, d); }