| Message ID | 20230922112508.1774352-1-kristina.martsenko@arm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | KVM: arm64: Support for Arm v8.8 memcpy instructions in KVM guests | expand |
Hi Kristina, On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote: > Hi, > > This is v2 of the series to allow using the new Arm memory copy instructions > in KVM guests. See v1 for more information [1]. Thanks for sending out the series. I've been thinking about what the architecture says for MOPS, and I wonder if what's currently in the Arm ARM is clear enough for EL1 software to be written robustly. While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS exceptions from EL1, there's no such control for EL0. So when vCPU migration occurs EL1 could get an unexpected MOPS exception, even for a process that was pinned to a single (virtual) CPU implementation. Additionally, the wording of I_NXHPS seems to suggest that EL2 handling of MOPS exceptions is only expected in certain circumstances where EL1 is incapable of handling an exception. Is the unwritten expectation then that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1 and EL0, even if EL1 did not migrate the PE context? Perhaps I'm being pedantic, but I'd really like for there to be some documentation that suggests MOPS exceptions can happen due to context migration done by a higher EL as that is the only option in the context of virtualization.
On 27/09/2023 07:00, Oliver Upton wrote: > Hi Kristina, Hi Oliver, > > On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote: >> Hi, >> >> This is v2 of the series to allow using the new Arm memory copy instructions >> in KVM guests. See v1 for more information [1]. > > > Thanks for sending out the series. I've been thinking about what the > architecture says for MOPS, and I wonder if what's currently in the > Arm ARM is clear enough for EL1 software to be written robustly. > > While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS > exceptions from EL1, there's no such control for EL0. So when vCPU > migration occurs EL1 could get an unexpected MOPS exception, even for a > process that was pinned to a single (virtual) CPU implementation. > > Additionally, the wording of I_NXHPS seems to suggest that EL2 handling > of MOPS exceptions is only expected in certain circumstances where EL1 is > incapable of handling an exception. Is the unwritten expectation then > that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1 > and EL0, even if EL1 did not migrate the PE context? > > Perhaps I'm being pedantic, but I'd really like for there to be some > documentation that suggests MOPS exceptions can happen due to context > migration done by a higher EL as that is the only option in the context > of virtualization. That's a good point. This shouldn't affect Linux guests as Linux is always able to handle a MOPS exception coming from EL0. But it would affect any non-Linux guest that pins all its EL0 tasks and doesn't implement a handler. It's not clear to me what the expectation for guests is, I'll ask the architects to clarify and get back to you. Thanks for the feedback! Kristina
On Thu, Sep 28, 2023 at 05:55:39PM +0100, Kristina Martsenko wrote: [...] > > Perhaps I'm being pedantic, but I'd really like for there to be some > > documentation that suggests MOPS exceptions can happen due to context > > migration done by a higher EL as that is the only option in the context > > of virtualization. > > That's a good point. This shouldn't affect Linux guests as Linux is > always able to handle a MOPS exception coming from EL0. But it would > affect any non-Linux guest that pins all its EL0 tasks and doesn't > implement a handler. It's not clear to me what the expectation for > guests is, I'll ask the architects to clarify and get back to you. That'd be excellent, thanks! All I'm looking for is something to point folks at if/when they complain about MOPS behavior in KVM guests.
On Thu, 28 Sep 2023 17:55:39 +0100, Kristina Martsenko <kristina.martsenko@arm.com> wrote: > > On 27/09/2023 07:00, Oliver Upton wrote: > > Hi Kristina, > > Hi Oliver, > > > > > On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote: > >> Hi, > >> > >> This is v2 of the series to allow using the new Arm memory copy instructions > >> in KVM guests. See v1 for more information [1]. > > > > > > Thanks for sending out the series. I've been thinking about what the > > architecture says for MOPS, and I wonder if what's currently in the > > Arm ARM is clear enough for EL1 software to be written robustly. > > > > While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS > > exceptions from EL1, there's no such control for EL0. So when vCPU > > migration occurs EL1 could get an unexpected MOPS exception, even for a > > process that was pinned to a single (virtual) CPU implementation. > > > > Additionally, the wording of I_NXHPS seems to suggest that EL2 handling > > of MOPS exceptions is only expected in certain circumstances where EL1 is > > incapable of handling an exception. Is the unwritten expectation then > > that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1 > > and EL0, even if EL1 did not migrate the PE context? > > > > Perhaps I'm being pedantic, but I'd really like for there to be some > > documentation that suggests MOPS exceptions can happen due to context > > migration done by a higher EL as that is the only option in the context > > of virtualization. > > That's a good point. This shouldn't affect Linux guests as Linux is > always able to handle a MOPS exception coming from EL0. But it would > affect any non-Linux guest that pins all its EL0 tasks and doesn't > implement a handler. It's not clear to me what the expectation for > guests is, I'll ask the architects to clarify and get back to you. My understanding is that MCE2 should always be set if the hypervisor can migrate vcpus across implementations behind EL1's back, and that in this context, EL1 never sees such an exception. I guess the only case where we could let EL1 handle such exception is by only setting MCE2 on the first entry into the guest after a vcpu migration (and clear it after that). Is it worth the effort? Absolutely not. M.
On 29/09/2023 10:29, Marc Zyngier wrote: > On Thu, 28 Sep 2023 17:55:39 +0100, > Kristina Martsenko <kristina.martsenko@arm.com> wrote: >> >> On 27/09/2023 07:00, Oliver Upton wrote: >>> >>> On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote: >>>> Hi, >>>> >>>> This is v2 of the series to allow using the new Arm memory copy instructions >>>> in KVM guests. See v1 for more information [1]. >>> >>> >>> Thanks for sending out the series. I've been thinking about what the >>> architecture says for MOPS, and I wonder if what's currently in the >>> Arm ARM is clear enough for EL1 software to be written robustly. >>> >>> While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS >>> exceptions from EL1, there's no such control for EL0. So when vCPU >>> migration occurs EL1 could get an unexpected MOPS exception, even for a >>> process that was pinned to a single (virtual) CPU implementation. >>> >>> Additionally, the wording of I_NXHPS seems to suggest that EL2 handling >>> of MOPS exceptions is only expected in certain circumstances where EL1 is >>> incapable of handling an exception. Is the unwritten expectation then >>> that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1 >>> and EL0, even if EL1 did not migrate the PE context? >>> >>> Perhaps I'm being pedantic, but I'd really like for there to be some >>> documentation that suggests MOPS exceptions can happen due to context >>> migration done by a higher EL as that is the only option in the context >>> of virtualization. >> >> That's a good point. This shouldn't affect Linux guests as Linux is >> always able to handle a MOPS exception coming from EL0. But it would >> affect any non-Linux guest that pins all its EL0 tasks and doesn't >> implement a handler. It's not clear to me what the expectation for >> guests is, I'll ask the architects to clarify and get back to you. > > My understanding is that MCE2 should always be set if the hypervisor > can migrate vcpus across implementations behind EL1's back, and that > in this context, EL1 never sees such an exception. Notice that MCE2 only traps exceptions from EL1, not from EL0. Exceptions from EL0 always go to EL1. Even if MCE2 is always set, EL1 will see the exception when the hypervisor migrates the vcpu while the vcpu is executing a MOPS instruction in EL0. Thanks, Kristina
On Fri, 29 Sep 2023 15:51:32 +0100, Kristina Martsenko <kristina.martsenko@arm.com> wrote: > > On 29/09/2023 10:29, Marc Zyngier wrote: > > On Thu, 28 Sep 2023 17:55:39 +0100, > > Kristina Martsenko <kristina.martsenko@arm.com> wrote: > >> > >> On 27/09/2023 07:00, Oliver Upton wrote: > >>> > >>> On Fri, Sep 22, 2023 at 12:25:06PM +0100, Kristina Martsenko wrote: > >>>> Hi, > >>>> > >>>> This is v2 of the series to allow using the new Arm memory copy instructions > >>>> in KVM guests. See v1 for more information [1]. > >>> > >>> > >>> Thanks for sending out the series. I've been thinking about what the > >>> architecture says for MOPS, and I wonder if what's currently in the > >>> Arm ARM is clear enough for EL1 software to be written robustly. > >>> > >>> While HCRX_EL2.MCE2 allows the hypervisor to intervene on MOPS > >>> exceptions from EL1, there's no such control for EL0. So when vCPU > >>> migration occurs EL1 could get an unexpected MOPS exception, even for a > >>> process that was pinned to a single (virtual) CPU implementation. > >>> > >>> Additionally, the wording of I_NXHPS seems to suggest that EL2 handling > >>> of MOPS exceptions is only expected in certain circumstances where EL1 is > >>> incapable of handling an exception. Is the unwritten expectation then > >>> that EL1 software should tolerate 'unexpected' MOPS exceptions from EL1 > >>> and EL0, even if EL1 did not migrate the PE context? > >>> > >>> Perhaps I'm being pedantic, but I'd really like for there to be some > >>> documentation that suggests MOPS exceptions can happen due to context > >>> migration done by a higher EL as that is the only option in the context > >>> of virtualization. > >> > >> That's a good point. This shouldn't affect Linux guests as Linux is > >> always able to handle a MOPS exception coming from EL0. But it would > >> affect any non-Linux guest that pins all its EL0 tasks and doesn't > >> implement a handler. It's not clear to me what the expectation for > >> guests is, I'll ask the architects to clarify and get back to you. > > > > My understanding is that MCE2 should always be set if the hypervisor > > can migrate vcpus across implementations behind EL1's back, and that > > in this context, EL1 never sees such an exception. > > Notice that MCE2 only traps exceptions from EL1, not from EL0. > Exceptions from EL0 always go to EL1. Even if MCE2 is always set, EL1 > will see the exception when the hypervisor migrates the vcpu while the > vcpu is executing a MOPS instruction in EL0. Ah, good point. I stand corrected. M.
On Fri, 22 Sep 2023 12:25:06 +0100, Kristina Martsenko <kristina.martsenko@arm.com> wrote: > > Hi, > > This is v2 of the series to allow using the new Arm memory copy instructions > in KVM guests. See v1 for more information [1]. > > Changes in v2: > - Dropped HCRX_EL2 vcpu field > - Rebased onto v6.6-rc2 FWIW, and despite my misgivings about the architecture: Reviewed-by: Marc Zyngier <maz@kernel.org> M.
On Fri, 22 Sep 2023 12:25:06 +0100, Kristina Martsenko wrote: > This is v2 of the series to allow using the new Arm memory copy instructions > in KVM guests. See v1 for more information [1]. > > Changes in v2: > - Dropped HCRX_EL2 vcpu field > - Rebased onto v6.6-rc2 > > [...] Applied to kvmarm/next, thanks! [1/2] KVM: arm64: Add handler for MOPS exceptions https://git.kernel.org/kvmarm/kvmarm/c/17b8ac23488b [2/2] KVM: arm64: Expose MOPS instructions to guests https://git.kernel.org/kvmarm/kvmarm/c/a24015b6cc66 -- Best, Oliver