| Message ID | 20200417155004.16806-4-andrew.cooper3@citrix.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | x86/pv: Start to trim 32bit support | expand |
On 17.04.2020 17:50, Andrew Cooper wrote: > There is no need for the Compat GDT if there are no 32bit PV guests. This > saves 4k per online CPU > > Bloat-o-meter reports the following savings in Xen itself: > > add/remove: 0/3 grow/shrink: 1/4 up/down: 7/-4612 (-4605) > Function old new delta > cpu_smpboot_free 1249 1256 +7 > per_cpu__compat_gdt_l1e 8 - -8 > per_cpu__compat_gdt 8 - -8 > init_idt_traps 442 420 -22 > load_system_tables 414 364 -50 > trap_init 444 280 -164 > cpu_smpboot_callback 1255 991 -264 > boot_compat_gdt 4096 - -4096 > Total: Before=3062726, After=3058121, chg -0.15% > > Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> > --- > CC: Jan Beulich <JBeulich@suse.com> > CC: Wei Liu <wl@xen.org> > CC: Roger Pau Monné <roger.pau@citrix.com> > > The increase in cpu_smpboot_free() appears to be a consequence of a totally > different layout of basic blocks. > --- > xen/arch/x86/cpu/common.c | 5 +++-- > xen/arch/x86/desc.c | 2 ++ > xen/arch/x86/smpboot.c | 5 ++++- > xen/arch/x86/traps.c | 10 +++++++--- > 4 files changed, 16 insertions(+), 6 deletions(-) > > diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c > index 1b33f1ed71..7b093cb421 100644 > --- a/xen/arch/x86/cpu/common.c > +++ b/xen/arch/x86/cpu/common.c > @@ -752,8 +752,9 @@ void load_system_tables(void) > > _set_tssldt_desc(gdt + TSS_ENTRY, (unsigned long)tss, > sizeof(*tss) - 1, SYS_DESC_tss_avail); > - _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, > - sizeof(*tss) - 1, SYS_DESC_tss_busy); > + if ( IS_ENABLED(CONFIG_PV32) ) > + _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, > + sizeof(*tss) - 1, SYS_DESC_tss_busy); Wouldn't this better be "if ( opt_pv32 )"? Also elsewhere then. Jan
On 20/04/2020 15:12, Jan Beulich wrote: > On 17.04.2020 17:50, Andrew Cooper wrote: >> There is no need for the Compat GDT if there are no 32bit PV guests. This >> saves 4k per online CPU >> >> Bloat-o-meter reports the following savings in Xen itself: >> >> add/remove: 0/3 grow/shrink: 1/4 up/down: 7/-4612 (-4605) >> Function old new delta >> cpu_smpboot_free 1249 1256 +7 >> per_cpu__compat_gdt_l1e 8 - -8 >> per_cpu__compat_gdt 8 - -8 >> init_idt_traps 442 420 -22 >> load_system_tables 414 364 -50 >> trap_init 444 280 -164 >> cpu_smpboot_callback 1255 991 -264 >> boot_compat_gdt 4096 - -4096 >> Total: Before=3062726, After=3058121, chg -0.15% >> >> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> >> --- >> CC: Jan Beulich <JBeulich@suse.com> >> CC: Wei Liu <wl@xen.org> >> CC: Roger Pau Monné <roger.pau@citrix.com> >> >> The increase in cpu_smpboot_free() appears to be a consequence of a totally >> different layout of basic blocks. >> --- >> xen/arch/x86/cpu/common.c | 5 +++-- >> xen/arch/x86/desc.c | 2 ++ >> xen/arch/x86/smpboot.c | 5 ++++- >> xen/arch/x86/traps.c | 10 +++++++--- >> 4 files changed, 16 insertions(+), 6 deletions(-) >> >> diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c >> index 1b33f1ed71..7b093cb421 100644 >> --- a/xen/arch/x86/cpu/common.c >> +++ b/xen/arch/x86/cpu/common.c >> @@ -752,8 +752,9 @@ void load_system_tables(void) >> >> _set_tssldt_desc(gdt + TSS_ENTRY, (unsigned long)tss, >> sizeof(*tss) - 1, SYS_DESC_tss_avail); >> - _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >> - sizeof(*tss) - 1, SYS_DESC_tss_busy); >> + if ( IS_ENABLED(CONFIG_PV32) ) >> + _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >> + sizeof(*tss) - 1, SYS_DESC_tss_busy); > Wouldn't this better be "if ( opt_pv32 )"? Also elsewhere then. Doing it like this specifically ensures that there is never a case where things are half configured. I don't think it is wise to suggest that making opt_pv32 runtime configurable might work. ~Andrew
On 20.04.2020 16:39, Andrew Cooper wrote: > On 20/04/2020 15:12, Jan Beulich wrote: >> On 17.04.2020 17:50, Andrew Cooper wrote: >>> There is no need for the Compat GDT if there are no 32bit PV guests. This >>> saves 4k per online CPU >>> >>> Bloat-o-meter reports the following savings in Xen itself: >>> >>> add/remove: 0/3 grow/shrink: 1/4 up/down: 7/-4612 (-4605) >>> Function old new delta >>> cpu_smpboot_free 1249 1256 +7 >>> per_cpu__compat_gdt_l1e 8 - -8 >>> per_cpu__compat_gdt 8 - -8 >>> init_idt_traps 442 420 -22 >>> load_system_tables 414 364 -50 >>> trap_init 444 280 -164 >>> cpu_smpboot_callback 1255 991 -264 >>> boot_compat_gdt 4096 - -4096 >>> Total: Before=3062726, After=3058121, chg -0.15% >>> >>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> >>> --- >>> CC: Jan Beulich <JBeulich@suse.com> >>> CC: Wei Liu <wl@xen.org> >>> CC: Roger Pau Monné <roger.pau@citrix.com> >>> >>> The increase in cpu_smpboot_free() appears to be a consequence of a totally >>> different layout of basic blocks. >>> --- >>> xen/arch/x86/cpu/common.c | 5 +++-- >>> xen/arch/x86/desc.c | 2 ++ >>> xen/arch/x86/smpboot.c | 5 ++++- >>> xen/arch/x86/traps.c | 10 +++++++--- >>> 4 files changed, 16 insertions(+), 6 deletions(-) >>> >>> diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c >>> index 1b33f1ed71..7b093cb421 100644 >>> --- a/xen/arch/x86/cpu/common.c >>> +++ b/xen/arch/x86/cpu/common.c >>> @@ -752,8 +752,9 @@ void load_system_tables(void) >>> >>> _set_tssldt_desc(gdt + TSS_ENTRY, (unsigned long)tss, >>> sizeof(*tss) - 1, SYS_DESC_tss_avail); >>> - _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >>> - sizeof(*tss) - 1, SYS_DESC_tss_busy); >>> + if ( IS_ENABLED(CONFIG_PV32) ) >>> + _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >>> + sizeof(*tss) - 1, SYS_DESC_tss_busy); >> Wouldn't this better be "if ( opt_pv32 )"? Also elsewhere then. > > Doing it like this specifically ensures that there is never a case where > things are half configured. But this way you set up something in the GDT that's never going to be used when "pv=no-32". Why leave a TSS accessible that we don't need? > I don't think it is wise to suggest that making opt_pv32 runtime > configurable might work. I didn't suggest (nor even consider) runtime changing of this setting. If we wanted, _that_ would be what might require using code as you have it right now (if we wanted to avoid setting this up at the time the setting gets flipped from false to true). Jan
On 20/04/2020 16:47, Jan Beulich wrote: > On 20.04.2020 16:39, Andrew Cooper wrote: >> On 20/04/2020 15:12, Jan Beulich wrote: >>> On 17.04.2020 17:50, Andrew Cooper wrote: >>>> There is no need for the Compat GDT if there are no 32bit PV guests. This >>>> saves 4k per online CPU >>>> >>>> Bloat-o-meter reports the following savings in Xen itself: >>>> >>>> add/remove: 0/3 grow/shrink: 1/4 up/down: 7/-4612 (-4605) >>>> Function old new delta >>>> cpu_smpboot_free 1249 1256 +7 >>>> per_cpu__compat_gdt_l1e 8 - -8 >>>> per_cpu__compat_gdt 8 - -8 >>>> init_idt_traps 442 420 -22 >>>> load_system_tables 414 364 -50 >>>> trap_init 444 280 -164 >>>> cpu_smpboot_callback 1255 991 -264 >>>> boot_compat_gdt 4096 - -4096 >>>> Total: Before=3062726, After=3058121, chg -0.15% >>>> >>>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> >>>> --- >>>> CC: Jan Beulich <JBeulich@suse.com> >>>> CC: Wei Liu <wl@xen.org> >>>> CC: Roger Pau Monné <roger.pau@citrix.com> >>>> >>>> The increase in cpu_smpboot_free() appears to be a consequence of a totally >>>> different layout of basic blocks. >>>> --- >>>> xen/arch/x86/cpu/common.c | 5 +++-- >>>> xen/arch/x86/desc.c | 2 ++ >>>> xen/arch/x86/smpboot.c | 5 ++++- >>>> xen/arch/x86/traps.c | 10 +++++++--- >>>> 4 files changed, 16 insertions(+), 6 deletions(-) >>>> >>>> diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c >>>> index 1b33f1ed71..7b093cb421 100644 >>>> --- a/xen/arch/x86/cpu/common.c >>>> +++ b/xen/arch/x86/cpu/common.c >>>> @@ -752,8 +752,9 @@ void load_system_tables(void) >>>> >>>> _set_tssldt_desc(gdt + TSS_ENTRY, (unsigned long)tss, >>>> sizeof(*tss) - 1, SYS_DESC_tss_avail); >>>> - _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >>>> - sizeof(*tss) - 1, SYS_DESC_tss_busy); >>>> + if ( IS_ENABLED(CONFIG_PV32) ) >>>> + _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >>>> + sizeof(*tss) - 1, SYS_DESC_tss_busy); >>> Wouldn't this better be "if ( opt_pv32 )"? Also elsewhere then. >> Doing it like this specifically ensures that there is never a case where >> things are half configured. > But this way you set up something in the GDT that's never going > to be used when "pv=no-32". Why leave a TSS accessible that we > don't need? Defence in depth. Having it only partially set up is more likely to fail in a security relevant way, than having it fully set up. This particular example is poor. There is no need to have the TSS in either GDT after the `ltr` instruction at boot for 64bit, because we don't task switch, but ISTR you requesting that this stayed as-was for consistency. (For 32bit Xen, it was strictly necessary for the #DF task switch to work.) However, the other logic, particularly the cached l1e pointing at the percpu compat_gdt is more liable to go wrong in interesting ways. ~Andrew
On 20.04.2020 19:08, Andrew Cooper wrote: > On 20/04/2020 16:47, Jan Beulich wrote: >> On 20.04.2020 16:39, Andrew Cooper wrote: >>> On 20/04/2020 15:12, Jan Beulich wrote: >>>> On 17.04.2020 17:50, Andrew Cooper wrote: >>>>> There is no need for the Compat GDT if there are no 32bit PV guests. This >>>>> saves 4k per online CPU >>>>> >>>>> Bloat-o-meter reports the following savings in Xen itself: >>>>> >>>>> add/remove: 0/3 grow/shrink: 1/4 up/down: 7/-4612 (-4605) >>>>> Function old new delta >>>>> cpu_smpboot_free 1249 1256 +7 >>>>> per_cpu__compat_gdt_l1e 8 - -8 >>>>> per_cpu__compat_gdt 8 - -8 >>>>> init_idt_traps 442 420 -22 >>>>> load_system_tables 414 364 -50 >>>>> trap_init 444 280 -164 >>>>> cpu_smpboot_callback 1255 991 -264 >>>>> boot_compat_gdt 4096 - -4096 >>>>> Total: Before=3062726, After=3058121, chg -0.15% >>>>> >>>>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> >>>>> --- >>>>> CC: Jan Beulich <JBeulich@suse.com> >>>>> CC: Wei Liu <wl@xen.org> >>>>> CC: Roger Pau Monné <roger.pau@citrix.com> >>>>> >>>>> The increase in cpu_smpboot_free() appears to be a consequence of a totally >>>>> different layout of basic blocks. >>>>> --- >>>>> xen/arch/x86/cpu/common.c | 5 +++-- >>>>> xen/arch/x86/desc.c | 2 ++ >>>>> xen/arch/x86/smpboot.c | 5 ++++- >>>>> xen/arch/x86/traps.c | 10 +++++++--- >>>>> 4 files changed, 16 insertions(+), 6 deletions(-) >>>>> >>>>> diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c >>>>> index 1b33f1ed71..7b093cb421 100644 >>>>> --- a/xen/arch/x86/cpu/common.c >>>>> +++ b/xen/arch/x86/cpu/common.c >>>>> @@ -752,8 +752,9 @@ void load_system_tables(void) >>>>> >>>>> _set_tssldt_desc(gdt + TSS_ENTRY, (unsigned long)tss, >>>>> sizeof(*tss) - 1, SYS_DESC_tss_avail); >>>>> - _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >>>>> - sizeof(*tss) - 1, SYS_DESC_tss_busy); >>>>> + if ( IS_ENABLED(CONFIG_PV32) ) >>>>> + _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, >>>>> + sizeof(*tss) - 1, SYS_DESC_tss_busy); >>>> Wouldn't this better be "if ( opt_pv32 )"? Also elsewhere then. >>> Doing it like this specifically ensures that there is never a case where >>> things are half configured. >> But this way you set up something in the GDT that's never going >> to be used when "pv=no-32". Why leave a TSS accessible that we >> don't need? > > Defence in depth. > > Having it only partially set up is more likely to fail in a security > relevant way, than having it fully set up. Well, I'm not convinced, but anyway Acked-by: Jan Beulich <jbeulich@suse.com> > This particular example is poor. There is no need to have the TSS in > either GDT after the `ltr` instruction at boot for 64bit, because we > don't task switch, but ISTR you requesting that this stayed as-was for > consistency. (For 32bit Xen, it was strictly necessary for the #DF task > switch to work.) Well, I'm simply of the opinion that what the TR holds should point to something valid in the currently active GDT. (As an alternative we could decide to zap TSS descriptors from _both_ GDTs once we're past the LTR. This wouldn't even conflict with resume from S3, as we re-write the TSS descriptors immediately ahead of the LTR.) > However, the other logic, particularly the cached l1e pointing at the > percpu compat_gdt is more liable to go wrong in interesting ways. Possibly, yes. Jan
diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c index 1b33f1ed71..7b093cb421 100644 --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -752,8 +752,9 @@ void load_system_tables(void) _set_tssldt_desc(gdt + TSS_ENTRY, (unsigned long)tss, sizeof(*tss) - 1, SYS_DESC_tss_avail); - _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, - sizeof(*tss) - 1, SYS_DESC_tss_busy); + if ( IS_ENABLED(CONFIG_PV32) ) + _set_tssldt_desc(compat_gdt + TSS_ENTRY, (unsigned long)tss, + sizeof(*tss) - 1, SYS_DESC_tss_busy); per_cpu(full_gdt_loaded, cpu) = false; lgdt(&gdtr); diff --git a/xen/arch/x86/desc.c b/xen/arch/x86/desc.c index dfeb1beaa8..39080ca672 100644 --- a/xen/arch/x86/desc.c +++ b/xen/arch/x86/desc.c @@ -55,6 +55,7 @@ seg_desc_t boot_gdt[PAGE_SIZE / sizeof(seg_desc_t)] = [SEL2GDT(PER_CPU_SELECTOR)] = { 0x0000910000000000 }, }; +#ifdef CONFIG_PV32 __section(".data.page_aligned") __aligned(PAGE_SIZE) seg_desc_t boot_compat_gdt[PAGE_SIZE / sizeof(seg_desc_t)] = { @@ -83,6 +84,7 @@ seg_desc_t boot_compat_gdt[PAGE_SIZE / sizeof(seg_desc_t)] = /* 0xe060 - per-CPU entry (limit == cpu) */ [SEL2GDT(PER_CPU_SELECTOR)] = { 0x0000910000000000 }, }; +#endif /* * Used by each CPU as it starts up, to enter C with a suitable %cs. diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index 09264b02d1..f9f63e496f 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -959,7 +959,8 @@ static void cpu_smpboot_free(unsigned int cpu, bool remove) free_domheap_page(mfn_to_page(mfn)); } - FREE_XENHEAP_PAGE(per_cpu(compat_gdt, cpu)); + if ( IS_ENABLED(CONFIG_PV32) ) + FREE_XENHEAP_PAGE(per_cpu(compat_gdt, cpu)); if ( remove ) { @@ -1001,6 +1002,7 @@ static int cpu_smpboot_alloc(unsigned int cpu) BUILD_BUG_ON(NR_CPUS > 0x10000); gdt[PER_CPU_GDT_ENTRY - FIRST_RESERVED_GDT_ENTRY].a = cpu; +#ifdef CONFIG_PV32 per_cpu(compat_gdt, cpu) = gdt = alloc_xenheap_pages(0, memflags); if ( gdt == NULL ) goto out; @@ -1008,6 +1010,7 @@ static int cpu_smpboot_alloc(unsigned int cpu) l1e_from_pfn(virt_to_mfn(gdt), __PAGE_HYPERVISOR_RW); memcpy(gdt, boot_compat_gdt, NR_RESERVED_GDT_PAGES * PAGE_SIZE); gdt[PER_CPU_GDT_ENTRY - FIRST_RESERVED_GDT_ENTRY].a = cpu; +#endif if ( idt_tables[cpu] == NULL ) idt_tables[cpu] = alloc_xenheap_pages(0, memflags); diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index e838846c6b..0bcf554e93 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -100,8 +100,10 @@ static DEFINE_PER_CPU(unsigned long, last_extable_addr); DEFINE_PER_CPU_READ_MOSTLY(seg_desc_t *, gdt); DEFINE_PER_CPU_READ_MOSTLY(l1_pgentry_t, gdt_l1e); +#ifdef CONFIG_PV32 DEFINE_PER_CPU_READ_MOSTLY(seg_desc_t *, compat_gdt); DEFINE_PER_CPU_READ_MOSTLY(l1_pgentry_t, compat_gdt_l1e); +#endif /* Master table, used by CPU0. */ idt_entry_t __section(".bss.page_aligned") __aligned(PAGE_SIZE) @@ -1999,7 +2001,8 @@ void __init init_idt_traps(void) idt_tables[0] = idt_table; this_cpu(gdt) = boot_gdt; - this_cpu(compat_gdt) = boot_compat_gdt; + if ( IS_ENABLED(CONFIG_PV32) ) + this_cpu(compat_gdt) = boot_compat_gdt; } extern void (*const autogen_entrypoints[X86_NR_VECTORS])(void); @@ -2030,8 +2033,9 @@ void __init trap_init(void) /* Cache {,compat_}gdt_l1e now that physically relocation is done. */ this_cpu(gdt_l1e) = l1e_from_pfn(virt_to_mfn(boot_gdt), __PAGE_HYPERVISOR_RW); - this_cpu(compat_gdt_l1e) = - l1e_from_pfn(virt_to_mfn(boot_compat_gdt), __PAGE_HYPERVISOR_RW); + if ( IS_ENABLED(CONFIG_PV32) ) + this_cpu(compat_gdt_l1e) = + l1e_from_pfn(virt_to_mfn(boot_compat_gdt), __PAGE_HYPERVISOR_RW); percpu_traps_init();
There is no need for the Compat GDT if there are no 32bit PV guests. This saves 4k per online CPU Bloat-o-meter reports the following savings in Xen itself: add/remove: 0/3 grow/shrink: 1/4 up/down: 7/-4612 (-4605) Function old new delta cpu_smpboot_free 1249 1256 +7 per_cpu__compat_gdt_l1e 8 - -8 per_cpu__compat_gdt 8 - -8 init_idt_traps 442 420 -22 load_system_tables 414 364 -50 trap_init 444 280 -164 cpu_smpboot_callback 1255 991 -264 boot_compat_gdt 4096 - -4096 Total: Before=3062726, After=3058121, chg -0.15% Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com> --- CC: Jan Beulich <JBeulich@suse.com> CC: Wei Liu <wl@xen.org> CC: Roger Pau Monné <roger.pau@citrix.com> The increase in cpu_smpboot_free() appears to be a consequence of a totally different layout of basic blocks. --- xen/arch/x86/cpu/common.c | 5 +++-- xen/arch/x86/desc.c | 2 ++ xen/arch/x86/smpboot.c | 5 ++++- xen/arch/x86/traps.c | 10 +++++++--- 4 files changed, 16 insertions(+), 6 deletions(-)