| Message ID | 20240918152807.25135-1-lilitj@amazon.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | *** RFC: ARM KVM dirty tracking device *** | expand |
Hi Lilit, +cc kvmarm mailing list, get_maintainer is your friend :) On Wed, Sep 18, 2024 at 03:27:59PM +0000, Lilit Janpoladyan wrote: > An example of a device that tracks accesses to stage-2 translations and will > implement page_tracking_device interface is AWS Graviton Page Tracking Agent > (PTA). We'll be posting code for the Graviton PTA device driver in a separate > series of patches. In order to actually review these patches, we need to see an implementation of such a page tracking device. Otherwise it's hard to tell that the interface accomplishes the right abstractions. Beyond that, I have some reservations about maintaining support for features that cannot actually be tested outside of your own environment. > When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to > use it via the same interface most likely with adaptations. Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS in hardware? I think the best way forward here is to implement the architecture, and hopefully after that your legacy driver can be made to fit the interface. The FVP implements FEAT_HDBSS, so there's some (slow) reference hardware to test against. This is a very interesting feature, so hopefully we can move towards something workable.
Hi Oliver, On 19.09.24, 11:12, "Oliver Upton" <oliver.upton@linux.dev <mailto:oliver.upton@linux.dev>> wrote: > Hi Lilit, > +cc kvmarm mailing list, get_maintainer is your friend :) > On Wed, Sep 18, 2024 at 03:27:59PM +0000, Lilit Janpoladyan wrote: > > An example of a device that tracks accesses to stage-2 translations and will > > implement page_tracking_device interface is AWS Graviton Page Tracking Agent > > (PTA). We'll be posting code for the Graviton PTA device driver in a separate > > series of patches. > In order to actually review these patches, we need to see an > implementation of such a page tracking device. Otherwise it's hard to > tell that the interface accomplishes the right abstractions. We'll be posting driver patches in the coming weeks, they should explain device functionality. > Beyond that, I have some reservations about maintaining support for > features that cannot actually be tested outside of your own environment. I understand, we'll see how we can emulate the functionality and make interface testable. > > When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to > > use it via the same interface most likely with adaptations. > Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS > in hardware? We'd need to keep the interface for as long as hardware without FEAT_HDBSS but with PTA is in use, hence the attempt of generalisation. > I think the best way forward here is to implement the architecture, and > hopefully after that your legacy driver can be made to fit the > interface. The FVP implements FEAT_HDBSS, so there's some (slow) > reference hardware to test against. Thanks for the idea, we'll test with FVP, but we'd need FEAT_HDBSS documentation for that. I don't think it's available yet, is it? > This is a very interesting feature, so hopefully we can move towards > something workable. > -- > Thanks, > Oliver Thanks for the feedback, we'll be working on the discussed points, Lilit Amazon Web Services Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B Sitz: Berlin Ust-ID: DE 365 538 597
On Thu, 2024-09-19 at 02:11 -0700, Oliver Upton wrote: > Hi Lilit, > > +cc kvmarm mailing list, get_maintainer is your friend :) > > On Wed, Sep 18, 2024 at 03:27:59PM +0000, Lilit Janpoladyan wrote: > > An example of a device that tracks accesses to stage-2 translations and will > > implement page_tracking_device interface is AWS Graviton Page Tracking Agent > > (PTA). We'll be posting code for the Graviton PTA device driver in a separate > > series of patches. > > In order to actually review these patches, we need to see an > implementation of such a page tracking device. Otherwise it's hard to > tell that the interface accomplishes the right abstractions. Absolutely. That one is coming soon, but I was chasing the team to post the API and KVM glue parts as early as possible to kick-start the discussion, especially about the upcoming architectural solution. > Beyond that, I have some reservations about maintaining support for > features that cannot actually be tested outside of your own environment. That's more about the hardware driver itself which will follow, than the core API posted here. I understand the reservation, but I think it's fine. In general, Linux does support esoteric hardware that not everyone can test every time. We do sweeping changes across all Ethernet drivers, for example, and some of those barely even exist any more. This particular device should be available on bare metal EC2 instances, of course, but perhaps we should also implement it in QEMU. That would actually be beneficial for our internal testing anyway, as it would allow us to catch regressions much earlier in our own development process. > > When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to > > use it via the same interface most likely with adaptations. > > Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS > in hardware? I don't think there is a definitive answer to that which is ready to tape out, but it certainly seems possible that future generations will eventually move to FEAT_HDBSS, maybe even reaching production by the end of the decade, at the earliest? And then a decade or two later, the existing hardware generations might even get retired, yes¹. ¹ #include <forward-looking statement.disclaimer> > I think the best way forward here is to implement the architecture, and > hopefully after that your legacy driver can be made to fit the > interface. The FVP implements FEAT_HDBSS, so there's some (slow) > reference hardware to test against. Is there actually any documentation available about FEAT_HDBSS? We've been asking, but haven't received it. I can find one or two mentions e.g. https://arm.jonpalmisc.com/2023_09_sysreg/AArch64-hdbssbr_el2 but nothing particularly useful. The main reason for posting this series early is to make sure we do all we can to accommodate FEAT_HDBSS. It's not the *end* of the world if the kernel-internal API has to be tweaked slightly when FEAT_HDBSS actually becomes reality in future, but obviously we'd prefer to support it right from the start.
On Thu, Sep 26, 2024 at 11:00:39AM +0100, David Woodhouse wrote: > > Beyond that, I have some reservations about maintaining support for > > features that cannot actually be tested outside of your own environment. > > That's more about the hardware driver itself which will follow, than > the core API posted here. > > I understand the reservation, but I think it's fine. In general, Linux > does support esoteric hardware that not everyone can test every > time. We do sweeping changes across all Ethernet drivers, for example, > and some of those barely even exist any more. Of course, but I think it is also reasonable to say that ethernet support in the kernel is rather mature with a good variety of hardware. By comparison, what we have here is a brand new driver interface with architecture / KVM code, which is pretty rare, with a single implementation. I'm perfectly happy to tinker on page tracking interface(s) in the future w/o testing everything, but I must insist that we have *some* way of testing the initial infrastructure before even considering taking it. > This particular device should be available on bare metal EC2 instances, > of course, but perhaps we should also implement it in QEMU. That would > actually be beneficial for our internal testing anyway, as it would > allow us to catch regressions much earlier in our own development > process. QEMU would be interesting, but hardware is always welcome too ;-) > > > When ARM architectural solution (FEAT_HDBSS feature) is available, we intend to > > > use it via the same interface most likely with adaptations. > > > > Will the PTA stuff eventually get retired once you get support for FEAT_HDBSS > > in hardware? > > I don't think there is a definitive answer to that which is ready to > tape out, but it certainly seems possible that future generations will > eventually move to FEAT_HDBSS, maybe even reaching production by the > end of the decade, at the earliest? And then a decade or two later, the > existing hardware generations might even get retired, yes¹. > > ¹ #include <forward-looking statement.disclaimer> Well, hopefully that means you folks will look after it then :) > > I think the best way forward here is to implement the architecture, and > > hopefully after that your legacy driver can be made to fit the > > interface. The FVP implements FEAT_HDBSS, so there's some (slow) > > reference hardware to test against. > > Is there actually any documentation available about FEAT_HDBSS? We've > been asking, but haven't received it. I can find one or two mentions > e.g. https://arm.jonpalmisc.com/2023_09_sysreg/AArch64-hdbssbr_el2 but > nothing particularly useful. Annoyingly no, the Arm ARM tends to lag the architecture by quite a bit. The sysreg XML (from which I think this website is derived) gets updated much more frequently. > The main reason for posting this series early is to make sure we do all > we can to accommodate FEAT_HDBSS. It's not the *end* of the world if > the kernel-internal API has to be tweaked slightly when FEAT_HDBSS > actually becomes reality in future, but obviously we'd prefer to > support it right from the start. Jury is still out on how FEAT_HDBSS is gonna fit with this PTA stuff. I'm guessing your hardware has some way of disambiguating dirtied addresses by VMID. The architected solution, OTOH, is tied to a particular stage-2 MMU configuration. KVM proper might need to manage the dirty tracking hardware in that case as it'll need to be context switched on the vcpu_load() / vcpu_put() boundary.