From patchwork Wed Jun 29 16:19:27 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Kuznetsov X-Patchwork-Id: 9205747 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7DEEA6075A for ; Wed, 29 Jun 2016 16:22:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E7BC28658 for ; Wed, 29 Jun 2016 16:22:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6263328666; Wed, 29 Jun 2016 16:22:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CD09728658 for ; Wed, 29 Jun 2016 16:22:07 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bIIDC-0002rn-3V; Wed, 29 Jun 2016 16:19:38 +0000 Received: from mail6.bemta6.messagelabs.com ([85.158.143.247]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bIIDA-0002rd-Mr for xen-devel@lists.xenproject.org; Wed, 29 Jun 2016 16:19:36 +0000 Received: from [85.158.143.35] by server-3.bemta-6.messagelabs.com id FB/96-22092-715F3775; Wed, 29 Jun 2016 16:19:35 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrEIsWRWlGSWpSXmKPExsVysWW7jK741+J wg65jLBbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8blW3tZCq7YVGz5epStgXGtcRcjF4eQwG4m iS27fjBDOLOYJHZveMfUxcjJwSagI/H96SlmEFtEQFfi2YJnbCBFzAIHmSW+/X3ODpIQFvCVu HrzKhuILSSwjkliY4M3iM0ioCpxauInVhCbU6BUYuHxbrBBvAKGEtcPHmEBsUUFLCXu9d1lg4 gLSpyc+QQsziygLzF7ejdYr4SAtkTDrMksEHYfo8S6DukJjPyzkLTMQtICYUtIHHzxghnClpK Y8ecIkM0BZKdJbJmitICRdRWjenFqUVlqka6xXlJRZnpGSW5iZo6uoYGZXm5qcXFiempOYlKx XnJ+7iZGYNgyAMEOxo5/TocYJTmYlER5Nz0oDhfiS8pPqcxILM6ILyrNSS0+xCjDwaEkwbv/M 1BOsCg1PbUiLTMHGEEwaQkOHiURXt4vQGne4oLE3OLMdIjUKUZFKXHeNyB9AiCJjNI8uDZY1F 5ilJUS5mUEOkSIpyC1KDezBFX+FaM4B6OSMO83kCk8mXklcNNfAS1mAlrMXAq2uCQRISXVwBh xtTOP3VN4q2q+X0pk/qlszw07DfZMcxDhqduZ7SVdckfxyI4HnlLqYf8bVaYyFfNLhavo39nu tcjq0ZHVu30OCG56zeixhr8jWTvQ63umjnfN53A+x+lLoqXe7mBkv3ir1ynHQUdcfC5rn+GSE 2+iJPe9U33g5nw35o79j60nfUpTn8/ZoMRSnJFoqMVcVJwIAHkyvj3VAgAA X-Env-Sender: vkuznets@redhat.com X-Msg-Ref: server-11.tower-21.messagelabs.com!1467217173!21479773!1 X-Originating-IP: [209.132.183.28] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMjA5LjEzMi4xODMuMjggPT4gNTQwNjQ=\n X-StarScan-Received: X-StarScan-Version: 8.46; banners=-,-,- X-VirusChecked: Checked Received: (qmail 15749 invoked from network); 29 Jun 2016 16:19:34 -0000 Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by server-11.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 29 Jun 2016 16:19:34 -0000 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7833A7DCD6; Wed, 29 Jun 2016 16:19:31 +0000 (UTC) Received: from vitty.brq.redhat.com.redhat.com (vitty.brq.redhat.com [10.34.26.3]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u5TGJRYv027745 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 29 Jun 2016 12:19:28 -0400 From: Vitaly Kuznetsov To: Andrew Cooper References: <1467132449-1030-1-git-send-email-vkuznets@redhat.com> <1467132449-1030-3-git-send-email-vkuznets@redhat.com> <87shvwur5p.fsf@vitty.brq.redhat.com> <689743e6-0b0e-9935-58e1-2dfa257c7bf8@citrix.com> <87oa6kupko.fsf@vitty.brq.redhat.com> Date: Wed, 29 Jun 2016 18:19:27 +0200 In-Reply-To: <87oa6kupko.fsf@vitty.brq.redhat.com> (Vitaly Kuznetsov's message of "Wed, 29 Jun 2016 14:50:15 +0200") Message-ID: <87k2h8ufw0.fsf@vitty.brq.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 29 Jun 2016 16:19:32 +0000 (UTC) Cc: Juergen Gross , Stefano Stabellini , x86@kernel.org, linux-kernel@vger.kernel.org, Ingo Molnar , David Vrabel , Jan Beulich , "H. Peter Anvin" , xen-devel@lists.xenproject.org, Boris Ostrovsky , Thomas Gleixner Subject: Re: [Xen-devel] [PATCH linux 2/8] xen: introduce xen_vcpu_id mapping X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Vitaly Kuznetsov writes: > Andrew Cooper writes: > >> On 29/06/16 13:16, Vitaly Kuznetsov wrote: >>> Andrew Cooper writes: >>> >>>> On 28/06/16 17:47, Vitaly Kuznetsov wrote: >>>>> @@ -1808,6 +1822,8 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action, >>>>> int cpu = (long)hcpu; >>>>> switch (action) { >>>>> case CPU_UP_PREPARE: >>>>> + /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */ >>>>> + per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2; >>>> Please do not assume or propagate this brokenness. It is incorrect in >>>> the general case, and I will be fixing in the hypervisor in due course. >>>> >>>> Always read the APIC_ID from the LAPIC, per regular hardware. >>> (I'm probbaly missing something important - please bear with me) >>> >>> The problem here is that I need to get _other_ CPU's id before any code >>> is executed on that CPU (or, at least, this is the current state of >>> affairs if you look at xen_hvm_cpu_up()) so I can't use CPUID/do MSR >>> reads/... The only option I see here is to rely on ACPI (MADT) data >>> which is stored in x86_cpu_to_apicid (and that's what cpu_physical_id() >>> gives us). MADT also has processor id which connects it to DSDT but I'm >>> not sure Linux keeps this data. But this is something fixable I guess. >> >> Hmm yes - that is a tricky issue. >> >> It is not safe or correct to assume that xen_vcpu_id is APICID / 2. >> >> This is currently the case for most modern versions of Xen, but isn't >> the case for older versions, and won't be the case in the future when I >> (or someone else) fixes topology representation for guests. >> >> For this to work, we need one or more of: >> >> 1) to provide the guest a full mapping from APIC_ID to vcpu id at boot >> time. > > So can we rely on ACPI data? Especially on MADT and processor ids there? > I think we can always guarantee that processor ids there match vCPU > ids. If yes I can try saving this data when we parse MADT. > To explain better what I'm trying to suggest here please take a look at the attached patch. If we can guarantee long term that ACPI id always equals to Xen's idea of vCPU id this is probably the easiest way. From 7c9cd25bdcd3e6ee866aa49550c9a4160194c3ba Mon Sep 17 00:00:00 2001 From: Vitaly Kuznetsov Date: Wed, 29 Jun 2016 18:13:01 +0200 Subject: [PATCH RFC/WIP] x86/acpi: store APIC ids for future usage (+use it in xen_hvm_cpu_notify) Signed-off-by: Vitaly Kuznetsov --- arch/x86/include/asm/smp.h | 1 + arch/x86/kernel/acpi/boot.c | 18 ++++++++++++++---- arch/x86/kernel/setup_percpu.c | 3 +++ arch/x86/xen/enlighten.c | 8 ++++++-- 4 files changed, 24 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index 66b0573..c68b56a 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -33,6 +33,7 @@ static inline struct cpumask *cpu_llc_shared_mask(int cpu) } DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_cpu_to_apicid); +DECLARE_EARLY_PER_CPU_READ_MOSTLY(int, x86_cpu_to_acpiid); DECLARE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_bios_cpu_apicid); #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86_32) DECLARE_EARLY_PER_CPU_READ_MOSTLY(int, x86_cpu_to_logical_apicid); diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c index 9414f84..3bbf0ab 100644 --- a/arch/x86/kernel/acpi/boot.c +++ b/arch/x86/kernel/acpi/boot.c @@ -110,6 +110,9 @@ static u32 isa_irq_to_gsi[NR_IRQS_LEGACY] __read_mostly = { #define ACPI_INVALID_GSI INT_MIN +DEFINE_EARLY_PER_CPU_READ_MOSTLY(int, x86_cpu_to_acpiid, -1); +EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_acpiid); + /* * This is just a simple wrapper around early_ioremap(), * with sanity checks for phys == 0 and size == 0. @@ -165,9 +168,10 @@ static int __init acpi_parse_madt(struct acpi_table_header *table) * * Returns the logic cpu number which maps to the local apic */ -static int acpi_register_lapic(int id, u8 enabled) +static int acpi_register_lapic(int id, int acpiid, u8 enabled) { unsigned int ver = 0; + int cpu; if (id >= MAX_LOCAL_APIC) { printk(KERN_INFO PREFIX "skipped apicid that is too big\n"); @@ -182,7 +186,11 @@ static int acpi_register_lapic(int id, u8 enabled) if (boot_cpu_physical_apicid != -1U) ver = apic_version[boot_cpu_physical_apicid]; - return generic_processor_info(id, ver); + cpu = generic_processor_info(id, ver); + if (cpu >= 0) + early_per_cpu(x86_cpu_to_acpiid, cpu) = acpiid; + + return cpu; } static int __init @@ -212,7 +220,7 @@ acpi_parse_x2apic(struct acpi_subtable_header *header, const unsigned long end) if (!apic->apic_id_valid(apic_id) && enabled) printk(KERN_WARNING PREFIX "x2apic entry ignored\n"); else - acpi_register_lapic(apic_id, enabled); + acpi_register_lapic(apic_id, processor->uid, enabled); #else printk(KERN_WARNING PREFIX "x2apic entry ignored\n"); #endif @@ -240,6 +248,7 @@ acpi_parse_lapic(struct acpi_subtable_header * header, const unsigned long end) * when we use CPU hotplug. */ acpi_register_lapic(processor->id, /* APIC ID */ + processor->processor_id, /* ACPI ID */ processor->lapic_flags & ACPI_MADT_ENABLED); return 0; @@ -258,6 +267,7 @@ acpi_parse_sapic(struct acpi_subtable_header *header, const unsigned long end) acpi_table_print_madt_entry(header); acpi_register_lapic((processor->id << 8) | processor->eid,/* APIC ID */ + processor->uid, /* ACPI ID */ processor->lapic_flags & ACPI_MADT_ENABLED); return 0; @@ -714,7 +724,7 @@ int acpi_map_cpu(acpi_handle handle, phys_cpuid_t physid, int *pcpu) { int cpu; - cpu = acpi_register_lapic(physid, ACPI_MADT_ENABLED); + cpu = acpi_register_lapic(physid, -1, ACPI_MADT_ENABLED); if (cpu < 0) { pr_info(PREFIX "Unable to map lapic to logical cpu number\n"); return cpu; diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index e4fcb87..7a40e06 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -236,6 +236,8 @@ void __init setup_per_cpu_areas(void) early_per_cpu_map(x86_cpu_to_apicid, cpu); per_cpu(x86_bios_cpu_apicid, cpu) = early_per_cpu_map(x86_bios_cpu_apicid, cpu); + per_cpu(x86_cpu_to_acpiid, cpu) = + early_per_cpu_map(x86_cpu_to_acpiid, cpu); #endif #ifdef CONFIG_X86_32 per_cpu(x86_cpu_to_logical_apicid, cpu) = @@ -271,6 +273,7 @@ void __init setup_per_cpu_areas(void) #ifdef CONFIG_X86_LOCAL_APIC early_per_cpu_ptr(x86_cpu_to_apicid) = NULL; early_per_cpu_ptr(x86_bios_cpu_apicid) = NULL; + early_per_cpu_ptr(x86_cpu_to_acpiid) = NULL; #endif #ifdef CONFIG_X86_32 early_per_cpu_ptr(x86_cpu_to_logical_apicid) = NULL; diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c index 0be29a0..8ae4871 100644 --- a/arch/x86/xen/enlighten.c +++ b/arch/x86/xen/enlighten.c @@ -1825,10 +1825,14 @@ static int xen_hvm_cpu_notify(struct notifier_block *self, unsigned long action, void *hcpu) { int cpu = (long)hcpu; + int acpiid = per_cpu(x86_cpu_to_acpiid, cpu); + switch (action) { case CPU_UP_PREPARE: - /* vLAPIC_ID == Xen's vCPU_ID * 2 for HVM guests */ - per_cpu(xen_vcpu_id, cpu) = cpu_physical_id(cpu) / 2; + if (acpiid != -1) + per_cpu(xen_vcpu_id, cpu) = acpiid; + else + per_cpu(xen_vcpu_id, cpu) = cpu; xen_vcpu_setup(cpu); if (xen_have_vector_callback) { if (xen_feature(XENFEAT_hvm_safe_pvclock)) -- 2.5.5