| Message ID | 20230308171328.1562857-1-usama.arif@bytedance.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Parallel CPU bringup for x86_64 | expand |
On 08.03.23 17:13, Usama Arif wrote: > The main code change over v13 is to enable parallel bringup for SEV-ES guests. > > Thanks, > Usama > > Changes across versions: > v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more > v3: Clean up x2apic patch, add MTRR optimisation, lock topology update > in preparation for more parallelisation. > v4: Fixes to the real mode parallelisation patch spotted by SeanC, to > avoid scribbling on initial_gs in common_cpu_up(), and to allow all > 24 bits of the physical X2APIC ID to be used. That patch still needs > a Signed-off-by from its original author, who once claimed not to > remember writing it at all. But now we've fixed it, hopefully he'll > admit it now :) > v5: rebase to v6.1 and remeasure performance, disable parallel bringup > for AMD CPUs. > v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and > reused timer calibration for secondary CPUs. > v7: [David Woodhouse] iterate over all possible CPUs to find any existing > cluster mask in alloc_clustermask. (patch 1/9) > Keep parallel AMD support enabled in AMD, using APIC ID in CPUID leaf > 0x0B (for x2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient. > Included sanity checks for APIC id from 0x0B. (patch 6/9) > Removed patch for reusing timer calibration for secondary CPUs. > commit message and code improvements. > v8: Fix CPU0 hotplug by setting up the initial_gs, initial_stack and > early_gdt_descr. > Drop trampoline lock and bail if APIC ID not found in find_cpunr. > Code comments improved and debug prints added. > v9: Drop patch to avoid repeated saves of MTRR at boot time. > rebased and retested at v6.2-rc8. > added kernel doc for no_parallel_bringup and made do_parallel_bringup > __ro_after_init. > v10: Fixed suspend/resume not working with parallel smpboot. > rebased and retested to 6.2. > fixed checkpatch errors. > v11: Added patches from Brian Gerst to remove the global variables initial_gs, > initial_stack, and early_gdt_descr from the 64-bit boot code > (https://lore.kernel.org/all/20230222221301.245890-1-brgerst@gmail.com/). > v12: Fixed compilation errors, acquire tr_lock for every stack setup in > trampoline_64.S. > Rearranged commits for a cleaner git history. > v13: Fix build error with CONFIG_FORCE_NR_CPUS. > Commit message improved, typos fixed and extra comments added. > v14: Enable parallel bringup for SEV-ES guests > > Brian Gerst (3): > x86/smpboot: Remove initial_stack on 64-bit > x86/smpboot: Remove early_gdt_descr on 64-bit > x86/smpboot: Remove initial_gs > > David Woodhouse (9): > x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel > cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> > cpu/hotplug: Add dynamic parallel bringup states before > CPUHP_BRINGUP_CPU > x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() > x86/smpboot: Split up native_cpu_up into separate phases and document > them > x86/smpboot: Support parallel startup of secondary CPUs > x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel > x86/smpboot: Serialize topology updates for secondary bringup > x86/smpboot: Allow parallel bringup for SEV-ES > > .../admin-guide/kernel-parameters.txt | 3 + > arch/x86/include/asm/cpu.h | 1 + > arch/x86/include/asm/processor.h | 6 +- > arch/x86/include/asm/realmode.h | 4 +- > arch/x86/include/asm/sev-common.h | 3 + > arch/x86/include/asm/sev.h | 5 + > arch/x86/include/asm/smp.h | 18 +- > arch/x86/include/asm/topology.h | 2 - > arch/x86/kernel/acpi/sleep.c | 30 +- > arch/x86/kernel/apic/apic.c | 2 +- > arch/x86/kernel/apic/x2apic_cluster.c | 126 +++--- > arch/x86/kernel/asm-offsets.c | 1 + > arch/x86/kernel/cpu/common.c | 6 +- > arch/x86/kernel/cpu/topology.c | 2 +- > arch/x86/kernel/head_64.S | 162 ++++++-- > arch/x86/kernel/smpboot.c | 366 +++++++++++++----- > arch/x86/realmode/init.c | 3 + > arch/x86/realmode/rm/trampoline_64.S | 27 +- > arch/x86/xen/smp_pv.c | 4 +- > arch/x86/xen/xen-head.S | 2 +- > include/linux/cpuhotplug.h | 2 + > include/linux/smpboot.h | 7 + > kernel/cpu.c | 31 +- > kernel/smpboot.h | 2 - > 24 files changed, 614 insertions(+), 201 deletions(-) > On Linux 6.2, Zen2 and Skylake, no issues or boot problems: Tested-by: Tor Vic <torvic9@mailbox.org>
Dear Tor, Am 10.03.23 um 20:20 schrieb Tor Vic: > On 08.03.23 17:13, Usama Arif wrote: >> The main code change over v13 is to enable parallel bringup for SEV-ES >> guests. […] >> .../admin-guide/kernel-parameters.txt | 3 + >> arch/x86/include/asm/cpu.h | 1 + >> arch/x86/include/asm/processor.h | 6 +- >> arch/x86/include/asm/realmode.h | 4 +- >> arch/x86/include/asm/sev-common.h | 3 + >> arch/x86/include/asm/sev.h | 5 + >> arch/x86/include/asm/smp.h | 18 +- >> arch/x86/include/asm/topology.h | 2 - >> arch/x86/kernel/acpi/sleep.c | 30 +- >> arch/x86/kernel/apic/apic.c | 2 +- >> arch/x86/kernel/apic/x2apic_cluster.c | 126 +++--- >> arch/x86/kernel/asm-offsets.c | 1 + >> arch/x86/kernel/cpu/common.c | 6 +- >> arch/x86/kernel/cpu/topology.c | 2 +- >> arch/x86/kernel/head_64.S | 162 ++++++-- >> arch/x86/kernel/smpboot.c | 366 +++++++++++++----- >> arch/x86/realmode/init.c | 3 + >> arch/x86/realmode/rm/trampoline_64.S | 27 +- >> arch/x86/xen/smp_pv.c | 4 +- >> arch/x86/xen/xen-head.S | 2 +- >> include/linux/cpuhotplug.h | 2 + >> include/linux/smpboot.h | 7 + >> kernel/cpu.c | 31 +- >> kernel/smpboot.h | 2 - >> 24 files changed, 614 insertions(+), 201 deletions(-) >> > > On Linux 6.2, Zen2 and Skylake, no issues or boot problems: > > Tested-by: Tor Vic <torvic9@mailbox.org> Thank you for testing this. It’d be great if you shared the exact timing numbers too. (Just to be sure, did you also test ACPI S3 suspend/resume?) Kind regards, Paul
On 10.03.23 20:18, Paul Menzel wrote: > Dear Tor, > Hi Paul, > > Am 10.03.23 um 20:20 schrieb Tor Vic: >> On 08.03.23 17:13, Usama Arif wrote: >>> The main code change over v13 is to enable parallel bringup for >>> SEV-ES guests. > > […] > >>> .../admin-guide/kernel-parameters.txt | 3 + >>> arch/x86/include/asm/cpu.h | 1 + >>> arch/x86/include/asm/processor.h | 6 +- >>> arch/x86/include/asm/realmode.h | 4 +- >>> arch/x86/include/asm/sev-common.h | 3 + >>> arch/x86/include/asm/sev.h | 5 + >>> arch/x86/include/asm/smp.h | 18 +- >>> arch/x86/include/asm/topology.h | 2 - >>> arch/x86/kernel/acpi/sleep.c | 30 +- >>> arch/x86/kernel/apic/apic.c | 2 +- >>> arch/x86/kernel/apic/x2apic_cluster.c | 126 +++--- >>> arch/x86/kernel/asm-offsets.c | 1 + >>> arch/x86/kernel/cpu/common.c | 6 +- >>> arch/x86/kernel/cpu/topology.c | 2 +- >>> arch/x86/kernel/head_64.S | 162 ++++++-- >>> arch/x86/kernel/smpboot.c | 366 +++++++++++++----- >>> arch/x86/realmode/init.c | 3 + >>> arch/x86/realmode/rm/trampoline_64.S | 27 +- >>> arch/x86/xen/smp_pv.c | 4 +- >>> arch/x86/xen/xen-head.S | 2 +- >>> include/linux/cpuhotplug.h | 2 + >>> include/linux/smpboot.h | 7 + >>> kernel/cpu.c | 31 +- >>> kernel/smpboot.h | 2 - >>> 24 files changed, 614 insertions(+), 201 deletions(-) >>> >> >> On Linux 6.2, Zen2 and Skylake, no issues or boot problems: >> >> Tested-by: Tor Vic <torvic9@mailbox.org> > > Thank you for testing this. It’d be great if you shared the exact timing > numbers too. (Just to be sure, did you also test ACPI S3 suspend/resume?) > I have just tested suspend/resume on the Zen2 machine, it works. Not yet tested on the Skylake platform. What is the best and simplest way to get these timings numbers? > > Kind regards, > > Paul
The main code change over v13 is to enable parallel bringup for SEV-ES guests. Thanks, Usama Changes across versions: v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more v3: Clean up x2apic patch, add MTRR optimisation, lock topology update in preparation for more parallelisation. v4: Fixes to the real mode parallelisation patch spotted by SeanC, to avoid scribbling on initial_gs in common_cpu_up(), and to allow all 24 bits of the physical X2APIC ID to be used. That patch still needs a Signed-off-by from its original author, who once claimed not to remember writing it at all. But now we've fixed it, hopefully he'll admit it now :) v5: rebase to v6.1 and remeasure performance, disable parallel bringup for AMD CPUs. v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and reused timer calibration for secondary CPUs. v7: [David Woodhouse] iterate over all possible CPUs to find any existing cluster mask in alloc_clustermask. (patch 1/9) Keep parallel AMD support enabled in AMD, using APIC ID in CPUID leaf 0x0B (for x2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient. Included sanity checks for APIC id from 0x0B. (patch 6/9) Removed patch for reusing timer calibration for secondary CPUs. commit message and code improvements. v8: Fix CPU0 hotplug by setting up the initial_gs, initial_stack and early_gdt_descr. Drop trampoline lock and bail if APIC ID not found in find_cpunr. Code comments improved and debug prints added. v9: Drop patch to avoid repeated saves of MTRR at boot time. rebased and retested at v6.2-rc8. added kernel doc for no_parallel_bringup and made do_parallel_bringup __ro_after_init. v10: Fixed suspend/resume not working with parallel smpboot. rebased and retested to 6.2. fixed checkpatch errors. v11: Added patches from Brian Gerst to remove the global variables initial_gs, initial_stack, and early_gdt_descr from the 64-bit boot code (https://lore.kernel.org/all/20230222221301.245890-1-brgerst@gmail.com/). v12: Fixed compilation errors, acquire tr_lock for every stack setup in trampoline_64.S. Rearranged commits for a cleaner git history. v13: Fix build error with CONFIG_FORCE_NR_CPUS. Commit message improved, typos fixed and extra comments added. v14: Enable parallel bringup for SEV-ES guests Brian Gerst (3): x86/smpboot: Remove initial_stack on 64-bit x86/smpboot: Remove early_gdt_descr on 64-bit x86/smpboot: Remove initial_gs David Woodhouse (9): x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h> cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU x86/smpboot: Reference count on smpboot_setup_warm_reset_vector() x86/smpboot: Split up native_cpu_up into separate phases and document them x86/smpboot: Support parallel startup of secondary CPUs x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel x86/smpboot: Serialize topology updates for secondary bringup x86/smpboot: Allow parallel bringup for SEV-ES .../admin-guide/kernel-parameters.txt | 3 + arch/x86/include/asm/cpu.h | 1 + arch/x86/include/asm/processor.h | 6 +- arch/x86/include/asm/realmode.h | 4 +- arch/x86/include/asm/sev-common.h | 3 + arch/x86/include/asm/sev.h | 5 + arch/x86/include/asm/smp.h | 18 +- arch/x86/include/asm/topology.h | 2 - arch/x86/kernel/acpi/sleep.c | 30 +- arch/x86/kernel/apic/apic.c | 2 +- arch/x86/kernel/apic/x2apic_cluster.c | 126 +++--- arch/x86/kernel/asm-offsets.c | 1 + arch/x86/kernel/cpu/common.c | 6 +- arch/x86/kernel/cpu/topology.c | 2 +- arch/x86/kernel/head_64.S | 162 ++++++-- arch/x86/kernel/smpboot.c | 366 +++++++++++++----- arch/x86/realmode/init.c | 3 + arch/x86/realmode/rm/trampoline_64.S | 27 +- arch/x86/xen/smp_pv.c | 4 +- arch/x86/xen/xen-head.S | 2 +- include/linux/cpuhotplug.h | 2 + include/linux/smpboot.h | 7 + kernel/cpu.c | 31 +- kernel/smpboot.h | 2 - 24 files changed, 614 insertions(+), 201 deletions(-)