| Message ID | cover.1588278317.git.hongyxia@amazon.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Remove the direct map | expand |
On Thu, Apr 30, 2020 at 09:44:09PM +0100, Hongyan Xia wrote: > From: Hongyan Xia <hongyxia@amazon.com> > > This series depends on Xen page table domheap conversion: > https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg01374.html. > > After breaking the reliance on the direct map to manipulate Xen page > tables, we can now finally remove the direct map altogether. > > This series: > - fixes many places that use the direct map incorrectly or assume the > presence of an always-mapped direct map in a wrong way. > - includes the early vmap patches for global mappings. > - initialises the mapcache for all domains, disables the fast path that > uses the direct map for mappings. > - maps and unmaps xenheap on-demand. > - adds a boot command line switch to enable or disable the direct map. > > This previous version was in RFC state and can be found here: > https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02647.html, > which has since been broken into small series. OOI have you done any performance measurements? Seeing that now even guest table needs mapping / unmapping during restore, I'm curious to know how that would impact performance. Wei. > > Hongyan Xia (12): > acpi: vmap pages in acpi_os_alloc_memory > x86/numa: vmap the pages for memnodemap > x86/srat: vmap the pages for acpi_slit > x86: map/unmap pages in restore_all_guests. > x86/pv: rewrite how building PV dom0 handles domheap mappings > x86/mapcache: initialise the mapcache for the idle domain > x86: add a boot option to enable and disable the direct map > x86/domain_page: remove the fast paths when mfn is not in the > directmap > xen/page_alloc: add a path for xenheap when there is no direct map > x86/setup: leave early boot slightly earlier > x86/setup: vmap heap nodes when they are outside the direct map > x86/setup: do not create valid mappings when directmap=no > > Wei Liu (4): > x86/setup: move vm_init() before acpi calls > x86/pv: domheap pages should be mapped while relocating initrd > x86: add Persistent Map (PMAP) infrastructure > x86: lift mapcache variable to the arch level > > docs/misc/xen-command-line.pandoc | 12 +++ > xen/arch/arm/setup.c | 4 +- > xen/arch/x86/Makefile | 1 + > xen/arch/x86/domain.c | 4 +- > xen/arch/x86/domain_page.c | 53 ++++++++----- > xen/arch/x86/mm.c | 8 +- > xen/arch/x86/numa.c | 8 +- > xen/arch/x86/pmap.c | 87 +++++++++++++++++++++ > xen/arch/x86/pv/dom0_build.c | 75 ++++++++++++++---- > xen/arch/x86/setup.c | 125 +++++++++++++++++++++++++----- > xen/arch/x86/srat.c | 3 +- > xen/arch/x86/x86_64/entry.S | 27 ++++++- > xen/common/page_alloc.c | 85 +++++++++++++++++--- > xen/common/vmap.c | 37 +++++++-- > xen/drivers/acpi/osl.c | 9 ++- > xen/include/asm-arm/mm.h | 5 ++ > xen/include/asm-x86/domain.h | 12 +-- > xen/include/asm-x86/fixmap.h | 3 + > xen/include/asm-x86/mm.h | 17 +++- > xen/include/asm-x86/pmap.h | 10 +++ > xen/include/xen/vmap.h | 5 ++ > 21 files changed, 495 insertions(+), 95 deletions(-) > create mode 100644 xen/arch/x86/pmap.c > create mode 100644 xen/include/asm-x86/pmap.h > > -- > 2.24.1.AMZN >
On Fri, 2020-05-01 at 12:07 +0000, Wei Liu wrote: > On Thu, Apr 30, 2020 at 09:44:09PM +0100, Hongyan Xia wrote: > > From: Hongyan Xia <hongyxia@amazon.com> > > > > This series depends on Xen page table domheap conversion: > > https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg01374.html > > . > > > > After breaking the reliance on the direct map to manipulate Xen > > page > > tables, we can now finally remove the direct map altogether. > > > > This series: > > - fixes many places that use the direct map incorrectly or assume > > the > > presence of an always-mapped direct map in a wrong way. > > - includes the early vmap patches for global mappings. > > - initialises the mapcache for all domains, disables the fast path > > that > > uses the direct map for mappings. > > - maps and unmaps xenheap on-demand. > > - adds a boot command line switch to enable or disable the direct > > map. > > > > This previous version was in RFC state and can be found here: > > https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02647.html > > , > > which has since been broken into small series. > > OOI have you done any performance measurements? > > Seeing that now even guest table needs mapping / unmapping during > restore, I'm curious to know how that would impact performance. I actually have a lot of performance numbers but unfortunately on an older version of Xen, not staging. I need to evaluate it again before coming back to you. As you suspected, one strong signal from the performance results is definitely the impact of walking guest tables. For EPT, mapping and unmapping 20 times is no fun. This shows up in micro-benchmarks, although larger benchmarks tend to be fine. My question is, do we care about hiding EPT? I think it is fine to just use xenheap pages (or any other form which does the job) so that we go down from 20 mappings to only 4. I have done this hack with EPT and saw significantly reduced impact for HVM guests in micro-benchmarks. Hongyan
On Fri, May 01, 2020 at 02:53:08PM +0100, Hongyan Xia wrote: > On Fri, 2020-05-01 at 12:07 +0000, Wei Liu wrote: > > On Thu, Apr 30, 2020 at 09:44:09PM +0100, Hongyan Xia wrote: > > > From: Hongyan Xia <hongyxia@amazon.com> > > > > > > This series depends on Xen page table domheap conversion: > > > > https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg01374.html > > > . > > > > > > After breaking the reliance on the direct map to manipulate Xen > > > page > > > tables, we can now finally remove the direct map altogether. > > > > > > This series: > > > - fixes many places that use the direct map incorrectly or assume > > > the > > > presence of an always-mapped direct map in a wrong way. > > > - includes the early vmap patches for global mappings. > > > - initialises the mapcache for all domains, disables the fast path > > > that > > > uses the direct map for mappings. > > > - maps and unmaps xenheap on-demand. > > > - adds a boot command line switch to enable or disable the direct > > > map. > > > > > > This previous version was in RFC state and can be found here: > > > > https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02647.html > > > , > > > which has since been broken into small series. > > > > OOI have you done any performance measurements? > > > > Seeing that now even guest table needs mapping / unmapping during > > restore, I'm curious to know how that would impact performance. > > I actually have a lot of performance numbers but unfortunately on an > older version of Xen, not staging. I need to evaluate it again before > coming back to you. As you suspected, one strong signal from the > performance results is definitely the impact of walking guest tables. > For EPT, mapping and unmapping 20 times is no fun. This shows up in > micro-benchmarks, although larger benchmarks tend to be fine. > > My question is, do we care about hiding EPT? I think it is fine to just > use xenheap pages (or any other form which does the job) so that we go > down from 20 mappings to only 4. I have done this hack with EPT and saw > significantly reduced impact for HVM guests in micro-benchmarks. Not sure about hiding EPT. I will leave this question to Jan and Andrew... Wei. > > Hongyan >
On Tue, 2020-06-02 at 09:08 +0000, Wei Liu wrote: > On Fri, May 01, 2020 at 02:53:08PM +0100, Hongyan Xia wrote: [...] > Not sure about hiding EPT. I will leave this question to Jan and > Andrew... Quick update on performance numbers. I have seen noticeable performance drop if we need to map and unmap EPT. This translates to up to 10% degredation in networking and database applications. With EPT always mapped (not considering it as secrets), I have not seen any noticeable performance drop in any real-world applications. The only performance drop is from a micro-benchmark that programs HPET in a tight loop (abusing MMIO page table walks), and the worst case is around 10% to 15%. Note that all above results are based on the mapcache rewrite patch 4058e92ce21627731c49b588a95809dc0affd83a.1581015491.git.hongyxia@amazon.com to fix the mapcache lock contention first. Hongyan
From: Hongyan Xia <hongyxia@amazon.com> This series depends on Xen page table domheap conversion: https://lists.xenproject.org/archives/html/xen-devel/2020-04/msg01374.html. After breaking the reliance on the direct map to manipulate Xen page tables, we can now finally remove the direct map altogether. This series: - fixes many places that use the direct map incorrectly or assume the presence of an always-mapped direct map in a wrong way. - includes the early vmap patches for global mappings. - initialises the mapcache for all domains, disables the fast path that uses the direct map for mappings. - maps and unmaps xenheap on-demand. - adds a boot command line switch to enable or disable the direct map. This previous version was in RFC state and can be found here: https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02647.html, which has since been broken into small series. Hongyan Xia (12): acpi: vmap pages in acpi_os_alloc_memory x86/numa: vmap the pages for memnodemap x86/srat: vmap the pages for acpi_slit x86: map/unmap pages in restore_all_guests. x86/pv: rewrite how building PV dom0 handles domheap mappings x86/mapcache: initialise the mapcache for the idle domain x86: add a boot option to enable and disable the direct map x86/domain_page: remove the fast paths when mfn is not in the directmap xen/page_alloc: add a path for xenheap when there is no direct map x86/setup: leave early boot slightly earlier x86/setup: vmap heap nodes when they are outside the direct map x86/setup: do not create valid mappings when directmap=no Wei Liu (4): x86/setup: move vm_init() before acpi calls x86/pv: domheap pages should be mapped while relocating initrd x86: add Persistent Map (PMAP) infrastructure x86: lift mapcache variable to the arch level docs/misc/xen-command-line.pandoc | 12 +++ xen/arch/arm/setup.c | 4 +- xen/arch/x86/Makefile | 1 + xen/arch/x86/domain.c | 4 +- xen/arch/x86/domain_page.c | 53 ++++++++----- xen/arch/x86/mm.c | 8 +- xen/arch/x86/numa.c | 8 +- xen/arch/x86/pmap.c | 87 +++++++++++++++++++++ xen/arch/x86/pv/dom0_build.c | 75 ++++++++++++++---- xen/arch/x86/setup.c | 125 +++++++++++++++++++++++++----- xen/arch/x86/srat.c | 3 +- xen/arch/x86/x86_64/entry.S | 27 ++++++- xen/common/page_alloc.c | 85 +++++++++++++++++--- xen/common/vmap.c | 37 +++++++-- xen/drivers/acpi/osl.c | 9 ++- xen/include/asm-arm/mm.h | 5 ++ xen/include/asm-x86/domain.h | 12 +-- xen/include/asm-x86/fixmap.h | 3 + xen/include/asm-x86/mm.h | 17 +++- xen/include/asm-x86/pmap.h | 10 +++ xen/include/xen/vmap.h | 5 ++ 21 files changed, 495 insertions(+), 95 deletions(-) create mode 100644 xen/arch/x86/pmap.c create mode 100644 xen/include/asm-x86/pmap.h