From patchwork Fri Feb 28 12:07:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11412147 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D707138D for ; Fri, 28 Feb 2020 12:09:18 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 69BCF246A8 for ; Fri, 28 Feb 2020 12:09:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="AZL/cLOc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69BCF246A8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j7eRD-0006QE-Ow; Fri, 28 Feb 2020 12:08:15 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j7eRC-0006Q3-Ff for xen-devel@lists.xenproject.org; Fri, 28 Feb 2020 12:08:14 +0000 X-Inumbo-ID: fe6b76d4-5a22-11ea-b7e8-bc764e2007e4 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id fe6b76d4-5a22-11ea-b7e8-bc764e2007e4; Fri, 28 Feb 2020 12:08:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1582891693; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wqOEWjoupZqj5EzPUH8qsfVbxUho33yiKRIgGC1ywtE=; b=AZL/cLOcQo5IPfmliDMumyimwm7s/pPHfPpP0/bMJmgQaQcZfmnOIKcX f6DCWqp8mlLzynfolrzTpUnG5E+YBQY+gZh/C0I1DehSZ8dQV3fAIUC6d DyYNGi1wbzf6Q7zMPnQtEcCvpskEVRJzvbp6C+L8vfLZMisoedJcRXTqS U=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: x+zukpqixKaY3v3m7PpWEKOZo5nLTZsS3q9QIp/HnWfOHD3taZ33kNt4p29ZID4cwwJY6yZMZi X20ZOSvqEapyhI9Vf10wF5DyrGbJXGXXD4K2x5r2Vih13JOhJ36nziHZ4pzA5dYeSsKZlvruta VcW0g9/4fldsndKw27yiiBwmW7jIKOBQDmMeT2u7siX3sKt5o5GsBtDkF/YOXdgJMDCGcBj7s8 6adPF1ijrimP65WZz+0YJJy6md/WNiL/cZubHY4uf1c+ITWUhmmbBT9iq8Cf/P7UzKtM+gKhMf PgM= X-SBRS: 2.7 X-MesageID: 13510566 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,496,1574139600"; d="scan'208";a="13510566" From: Roger Pau Monne To: Date: Fri, 28 Feb 2020 13:07:52 +0100 Message-ID: <20200228120753.38036-2-roger.pau@citrix.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200228120753.38036-1-roger.pau@citrix.com> References: <20200228120753.38036-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 1/2] x86/smp: use a dedicated CPU mask in send_IPI_mask X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Some callers of send_IPI_mask pass the scratch cpumask as the mask parameter of send_IPI_mask, so the scratch cpumask cannot be used by the function. The following trace has been obtained with a debug patch and shows one of those callers: (XEN) scratch CPU mask already in use by arch/x86/mm.c#_get_page_type+0x1f9/0x1abf (XEN) Xen BUG at smp.c:45 [...] (XEN) Xen call trace: (XEN) [] R scratch_cpumask+0xd3/0xf9 (XEN) [] F send_IPI_mask+0x72/0x1ca (XEN) [] F flush_area_mask+0x10c/0x16c (XEN) [] F arch/x86/mm.c#_get_page_type+0x3ff/0x1abf (XEN) [] F get_page_type+0xe/0x2c (XEN) [] F pv_set_gdt+0xa1/0x2aa (XEN) [] F arch_set_info_guest+0x1196/0x16ba (XEN) [] F default_initialise_vcpu+0xc7/0xd4 (XEN) [] F arch_initialise_vcpu+0x61/0xcd (XEN) [] F do_vcpu_op+0x219/0x690 (XEN) [] F pv_hypercall+0x2f6/0x593 (XEN) [] F lstar_enter+0x112/0x120 _get_page_type will use the scratch cpumask to call flush_tlb_mask, which in turn calls send_IPI_mask. Fix this by using a dedicated per CPU cpumask in send_IPI_mask. Fixes: 5500d265a2a8 ('x86/smp: use APIC ALLBUT destination shorthand when possible') Signed-off-by: Roger Pau Monné Reviewed-by: Jan Beulich --- Changes since v4: - Place the declaration of send_ipi_cpumask in smp.h. --- xen/arch/x86/smp.c | 2 +- xen/arch/x86/smpboot.c | 9 ++++++++- xen/include/asm-x86/smp.h | 1 + 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/xen/arch/x86/smp.c b/xen/arch/x86/smp.c index 0461812cf6..dd0b49d731 100644 --- a/xen/arch/x86/smp.c +++ b/xen/arch/x86/smp.c @@ -67,7 +67,7 @@ static void send_IPI_shortcut(unsigned int shortcut, int vector, void send_IPI_mask(const cpumask_t *mask, int vector) { bool cpus_locked = false; - cpumask_t *scratch = this_cpu(scratch_cpumask); + cpumask_t *scratch = this_cpu(send_ipi_cpumask); if ( in_irq() || in_mce_handler() || in_nmi_handler() ) { diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index ad49f2dcd7..6c548b0b53 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -57,6 +57,9 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask); DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, scratch_cpumask); static cpumask_t scratch_cpu0mask; +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, send_ipi_cpumask); +static cpumask_t send_ipi_cpu0mask; + cpumask_t cpu_online_map __read_mostly; EXPORT_SYMBOL(cpu_online_map); @@ -930,6 +933,8 @@ static void cpu_smpboot_free(unsigned int cpu, bool remove) FREE_CPUMASK_VAR(per_cpu(cpu_core_mask, cpu)); if ( per_cpu(scratch_cpumask, cpu) != &scratch_cpu0mask ) FREE_CPUMASK_VAR(per_cpu(scratch_cpumask, cpu)); + if ( per_cpu(send_ipi_cpumask, cpu) != &send_ipi_cpu0mask ) + FREE_CPUMASK_VAR(per_cpu(send_ipi_cpumask, cpu)); } cleanup_cpu_root_pgt(cpu); @@ -1034,7 +1039,8 @@ static int cpu_smpboot_alloc(unsigned int cpu) if ( !(cond_zalloc_cpumask_var(&per_cpu(cpu_sibling_mask, cpu)) && cond_zalloc_cpumask_var(&per_cpu(cpu_core_mask, cpu)) && - cond_alloc_cpumask_var(&per_cpu(scratch_cpumask, cpu))) ) + cond_alloc_cpumask_var(&per_cpu(scratch_cpumask, cpu)) && + cond_alloc_cpumask_var(&per_cpu(send_ipi_cpumask, cpu))) ) goto out; rc = 0; @@ -1175,6 +1181,7 @@ void __init smp_prepare_boot_cpu(void) cpumask_set_cpu(cpu, &cpu_present_map); #if NR_CPUS > 2 * BITS_PER_LONG per_cpu(scratch_cpumask, cpu) = &scratch_cpu0mask; + per_cpu(send_ipi_cpumask, cpu) = &send_ipi_cpu0mask; #endif get_cpu_info()->use_pv_cr3 = false; diff --git a/xen/include/asm-x86/smp.h b/xen/include/asm-x86/smp.h index 92d69a5ea0..6150363655 100644 --- a/xen/include/asm-x86/smp.h +++ b/xen/include/asm-x86/smp.h @@ -22,6 +22,7 @@ DECLARE_PER_CPU(cpumask_var_t, cpu_sibling_mask); DECLARE_PER_CPU(cpumask_var_t, cpu_core_mask); DECLARE_PER_CPU(cpumask_var_t, scratch_cpumask); +DECLARE_PER_CPU(cpumask_var_t, send_ipi_cpumask); /* * Do we, for platform reasons, need to actually keep CPUs online when we From patchwork Fri Feb 28 12:07:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11412149 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF86614B7 for ; Fri, 28 Feb 2020 12:09:25 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C098F246A8 for ; Fri, 28 Feb 2020 12:09:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="Cz/LK+Wh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C098F246A8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j7eRI-0006Rc-20; Fri, 28 Feb 2020 12:08:20 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j7eRH-0006RN-EX for xen-devel@lists.xenproject.org; Fri, 28 Feb 2020 12:08:19 +0000 X-Inumbo-ID: ff647fea-5a22-11ea-b7e8-bc764e2007e4 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ff647fea-5a22-11ea-b7e8-bc764e2007e4; Fri, 28 Feb 2020 12:08:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1582891694; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tAdxMRqSMJakseu9bK3FWI2Q8T0cmEEErcFePs+To8A=; b=Cz/LK+WhDM23gq/nFPVg5YKN/EtnMwscRASnTNZpEPG6ImX3VT+wBQOx Kncc3Ec9UjGc8gkh7bA/GrwgQ1Koteol3FZUW8Cwl+t6dJRCqrRKRAFnG Vhov5H5kA7lmDryuv0SuU6grFSkieRUQRNfKzh5B5SGuGuupRGcaVoRkP Q=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: MaVxucS9TVYk+/FnLoHEsKdIUyjp1lpO/YIpRIpWwBSsrUgHcV0TzxMOsKGlJhBnKB35q3xMcO IbOftcFudl5ySTNYNn7tfyWCOo6UPnuejMR/6gSMZk465F/3LwwyduX+Z3Am7BucKcI3rNLUoI wdZqRSNcN8KLk4NKzAnFxyoWuc+1T/C+Jzegy6bNeqv91PTxouliyU+1WP3OYDHHTE/e9jhqph d0jYieVKbHYm9T5k+Lsh3METtBhlHD6bMl6YA41zVcQvTzh/IL+Xht9k0Weu9bf0MVjCyYrFAE zOg= X-SBRS: 2.7 X-MesageID: 13510567 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,496,1574139600"; d="scan'208";a="13510567" From: Roger Pau Monne To: Date: Fri, 28 Feb 2020 13:07:53 +0100 Message-ID: <20200228120753.38036-3-roger.pau@citrix.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200228120753.38036-1-roger.pau@citrix.com> References: <20200228120753.38036-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 2/2] x86: add accessors for scratch cpu mask X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Current usage of the per-CPU scratch cpumask is dangerous since there's no way to figure out if the mask is already being used except for manual code inspection of all the callers and possible call paths. This is unsafe and not reliable, so introduce a minimal get/put infrastructure to prevent nested usage of the scratch mask and usage in interrupt context. Move the declaration of scratch_cpumask to smp.c in order to place the declaration and the accessors as close as possible. Signed-off-by: Roger Pau Monné Reviewed-by: Jan Beulich --- Changes since v4: - Fix get_scratch_cpumask call order in _clear_irq_vector. - Remove double newline in __do_update_va_mapping. - Constify scratch_cpumask_use. - Don't explicitly print the address in the format string, a followup patch will be sent to make %p[sS] do so. Changes since v3: - Fix commit message. - Split the cpumask taken section into two in _clear_irq_vector. - Add an empty statement in do_mmuext_op to avoid a break. - Change the logic used to release the scratch cpumask in __do_update_va_mapping. - Add a %ps print to scratch_cpumask helper. - Remove printing the current IP, as that would be done by BUG anyway. - Pass the cpumask to put_scratch_cpumask and zap the pointer. Changes since v1: - Use __builtin_return_address(0) instead of __func__. - Move declaration of scratch_cpumask and scratch_cpumask accessor to smp.c. - Do not allow usage in #MC or #NMI context. --- xen/arch/x86/io_apic.c | 6 ++++-- xen/arch/x86/irq.c | 14 ++++++++++---- xen/arch/x86/mm.c | 39 +++++++++++++++++++++++++++------------ xen/arch/x86/msi.c | 4 +++- xen/arch/x86/smp.c | 25 +++++++++++++++++++++++++ xen/arch/x86/smpboot.c | 1 - xen/include/asm-x86/smp.h | 14 ++++++++++++++ 7 files changed, 83 insertions(+), 20 deletions(-) diff --git a/xen/arch/x86/io_apic.c b/xen/arch/x86/io_apic.c index e98e08e9c8..0bb994f0ba 100644 --- a/xen/arch/x86/io_apic.c +++ b/xen/arch/x86/io_apic.c @@ -2236,10 +2236,11 @@ int io_apic_set_pci_routing (int ioapic, int pin, int irq, int edge_level, int a entry.vector = vector; if (cpumask_intersects(desc->arch.cpu_mask, TARGET_CPUS)) { - cpumask_t *mask = this_cpu(scratch_cpumask); + cpumask_t *mask = get_scratch_cpumask(); cpumask_and(mask, desc->arch.cpu_mask, TARGET_CPUS); SET_DEST(entry, logical, cpu_mask_to_apicid(mask)); + put_scratch_cpumask(mask); } else { printk(XENLOG_ERR "IRQ%d: no target CPU (%*pb vs %*pb)\n", irq, CPUMASK_PR(desc->arch.cpu_mask), CPUMASK_PR(TARGET_CPUS)); @@ -2433,10 +2434,11 @@ int ioapic_guest_write(unsigned long physbase, unsigned int reg, u32 val) if ( cpumask_intersects(desc->arch.cpu_mask, TARGET_CPUS) ) { - cpumask_t *mask = this_cpu(scratch_cpumask); + cpumask_t *mask = get_scratch_cpumask(); cpumask_and(mask, desc->arch.cpu_mask, TARGET_CPUS); SET_DEST(rte, logical, cpu_mask_to_apicid(mask)); + put_scratch_cpumask(mask); } else { diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c index cc2eb8e925..0a526ee800 100644 --- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -196,7 +196,7 @@ static void _clear_irq_vector(struct irq_desc *desc) { unsigned int cpu, old_vector, irq = desc->irq; unsigned int vector = desc->arch.vector; - cpumask_t *tmp_mask = this_cpu(scratch_cpumask); + cpumask_t *tmp_mask = get_scratch_cpumask(); BUG_ON(!valid_irq_vector(vector)); @@ -208,6 +208,7 @@ static void _clear_irq_vector(struct irq_desc *desc) ASSERT(per_cpu(vector_irq, cpu)[vector] == irq); per_cpu(vector_irq, cpu)[vector] = ~irq; } + put_scratch_cpumask(tmp_mask); desc->arch.vector = IRQ_VECTOR_UNASSIGNED; cpumask_clear(desc->arch.cpu_mask); @@ -227,8 +228,9 @@ static void _clear_irq_vector(struct irq_desc *desc) /* If we were in motion, also clear desc->arch.old_vector */ old_vector = desc->arch.old_vector; - cpumask_and(tmp_mask, desc->arch.old_cpu_mask, &cpu_online_map); + tmp_mask = get_scratch_cpumask(); + cpumask_and(tmp_mask, desc->arch.old_cpu_mask, &cpu_online_map); for_each_cpu(cpu, tmp_mask) { ASSERT(per_cpu(vector_irq, cpu)[old_vector] == irq); @@ -236,6 +238,7 @@ static void _clear_irq_vector(struct irq_desc *desc) per_cpu(vector_irq, cpu)[old_vector] = ~irq; } + put_scratch_cpumask(tmp_mask); release_old_vec(desc); desc->arch.move_in_progress = 0; @@ -1152,10 +1155,11 @@ static void irq_guest_eoi_timer_fn(void *data) break; case ACKTYPE_EOI: - cpu_eoi_map = this_cpu(scratch_cpumask); + cpu_eoi_map = get_scratch_cpumask(); cpumask_copy(cpu_eoi_map, action->cpu_eoi_map); spin_unlock_irq(&desc->lock); on_selected_cpus(cpu_eoi_map, set_eoi_ready, desc, 0); + put_scratch_cpumask(cpu_eoi_map); return; } @@ -2531,12 +2535,12 @@ void fixup_irqs(const cpumask_t *mask, bool verbose) unsigned int irq; static int warned; struct irq_desc *desc; + cpumask_t *affinity = get_scratch_cpumask(); for ( irq = 0; irq < nr_irqs; irq++ ) { bool break_affinity = false, set_affinity = true; unsigned int vector; - cpumask_t *affinity = this_cpu(scratch_cpumask); if ( irq == 2 ) continue; @@ -2640,6 +2644,8 @@ void fixup_irqs(const cpumask_t *mask, bool verbose) irq, CPUMASK_PR(affinity)); } + put_scratch_cpumask(affinity); + /* That doesn't seem sufficient. Give it 1ms. */ local_irq_enable(); mdelay(1); diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index 70b87c4830..22787bbd6c 100644 --- a/xen/arch/x86/mm.c +++ b/xen/arch/x86/mm.c @@ -1262,7 +1262,7 @@ void put_page_from_l1e(l1_pgentry_t l1e, struct domain *l1e_owner) (l1e_owner == pg_owner) ) { struct vcpu *v; - cpumask_t *mask = this_cpu(scratch_cpumask); + cpumask_t *mask = get_scratch_cpumask(); cpumask_clear(mask); @@ -1279,6 +1279,7 @@ void put_page_from_l1e(l1_pgentry_t l1e, struct domain *l1e_owner) if ( !cpumask_empty(mask) ) flush_tlb_mask(mask); + put_scratch_cpumask(mask); } #endif /* CONFIG_PV_LDT_PAGING */ put_page(page); @@ -2903,7 +2904,7 @@ static int _get_page_type(struct page_info *page, unsigned long type, * vital that no other CPUs are left with mappings of a frame * which is about to become writeable to the guest. */ - cpumask_t *mask = this_cpu(scratch_cpumask); + cpumask_t *mask = get_scratch_cpumask(); BUG_ON(in_irq()); cpumask_copy(mask, d->dirty_cpumask); @@ -2919,6 +2920,7 @@ static int _get_page_type(struct page_info *page, unsigned long type, perfc_incr(need_flush_tlb_flush); flush_tlb_mask(mask); } + put_scratch_cpumask(mask); /* We lose existing type and validity. */ nx &= ~(PGT_type_mask | PGT_validated); @@ -3635,7 +3637,7 @@ long do_mmuext_op( case MMUEXT_TLB_FLUSH_MULTI: case MMUEXT_INVLPG_MULTI: { - cpumask_t *mask = this_cpu(scratch_cpumask); + cpumask_t *mask = get_scratch_cpumask(); if ( unlikely(currd != pg_owner) ) rc = -EPERM; @@ -3645,12 +3647,13 @@ long do_mmuext_op( mask)) ) rc = -EINVAL; if ( unlikely(rc) ) - break; - - if ( op.cmd == MMUEXT_TLB_FLUSH_MULTI ) + ; + else if ( op.cmd == MMUEXT_TLB_FLUSH_MULTI ) flush_tlb_mask(mask); else if ( __addr_ok(op.arg1.linear_addr) ) flush_tlb_one_mask(mask, op.arg1.linear_addr); + put_scratch_cpumask(mask); + break; } @@ -3683,7 +3686,7 @@ long do_mmuext_op( else if ( likely(cache_flush_permitted(currd)) ) { unsigned int cpu; - cpumask_t *mask = this_cpu(scratch_cpumask); + cpumask_t *mask = get_scratch_cpumask(); cpumask_clear(mask); for_each_online_cpu(cpu) @@ -3691,6 +3694,7 @@ long do_mmuext_op( per_cpu(cpu_sibling_mask, cpu)) ) __cpumask_set_cpu(cpu, mask); flush_mask(mask, FLUSH_CACHE); + put_scratch_cpumask(mask); } else rc = -EINVAL; @@ -4156,12 +4160,13 @@ long do_mmu_update( * Force other vCPU-s of the affected guest to pick up L4 entry * changes (if any). */ - unsigned int cpu = smp_processor_id(); - cpumask_t *mask = per_cpu(scratch_cpumask, cpu); + cpumask_t *mask = get_scratch_cpumask(); - cpumask_andnot(mask, pt_owner->dirty_cpumask, cpumask_of(cpu)); + cpumask_andnot(mask, pt_owner->dirty_cpumask, + cpumask_of(smp_processor_id())); if ( !cpumask_empty(mask) ) flush_mask(mask, FLUSH_TLB_GLOBAL | FLUSH_ROOT_PGTBL); + put_scratch_cpumask(mask); } perfc_add(num_page_updates, i); @@ -4353,7 +4358,7 @@ static int __do_update_va_mapping( mask = d->dirty_cpumask; break; default: - mask = this_cpu(scratch_cpumask); + mask = get_scratch_cpumask(); rc = vcpumask_to_pcpumask(d, const_guest_handle_from_ptr(bmap_ptr, void), mask); @@ -4373,7 +4378,7 @@ static int __do_update_va_mapping( mask = d->dirty_cpumask; break; default: - mask = this_cpu(scratch_cpumask); + mask = get_scratch_cpumask(); rc = vcpumask_to_pcpumask(d, const_guest_handle_from_ptr(bmap_ptr, void), mask); @@ -4384,6 +4389,16 @@ static int __do_update_va_mapping( break; } + switch ( flags & ~UVMF_FLUSHTYPE_MASK ) + { + case UVMF_LOCAL: + case UVMF_ALL: + break; + + default: + put_scratch_cpumask(mask); + } + return rc; } diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index 161ee60dbe..6d198f8665 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -159,13 +159,15 @@ void msi_compose_msg(unsigned vector, const cpumask_t *cpu_mask, struct msi_msg if ( cpu_mask ) { - cpumask_t *mask = this_cpu(scratch_cpumask); + cpumask_t *mask; if ( !cpumask_intersects(cpu_mask, &cpu_online_map) ) return; + mask = get_scratch_cpumask(); cpumask_and(mask, cpu_mask, &cpu_online_map); msg->dest32 = cpu_mask_to_apicid(mask); + put_scratch_cpumask(mask); } msg->address_hi = MSI_ADDR_BASE_HI; diff --git a/xen/arch/x86/smp.c b/xen/arch/x86/smp.c index dd0b49d731..084ad32653 100644 --- a/xen/arch/x86/smp.c +++ b/xen/arch/x86/smp.c @@ -25,6 +25,31 @@ #include #include +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, scratch_cpumask); + +#ifndef NDEBUG +cpumask_t *scratch_cpumask(bool use) +{ + static DEFINE_PER_CPU(const void *, scratch_cpumask_use); + + /* + * Due to reentrancy scratch cpumask cannot be used in IRQ, #MC or NMI + * context. + */ + BUG_ON(in_irq() || in_mce_handler() || in_nmi_handler()); + + if ( use && unlikely(this_cpu(scratch_cpumask_use)) ) + { + printk("scratch CPU mask already in use by %ps\n", + this_cpu(scratch_cpumask_use)); + BUG(); + } + this_cpu(scratch_cpumask_use) = use ? __builtin_return_address(0) : NULL; + + return use ? this_cpu(scratch_cpumask) : NULL; +} +#endif + /* Helper functions to prepare APIC register values. */ static unsigned int prepare_ICR(unsigned int shortcut, int vector) { diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index 6c548b0b53..e26b61a8b4 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -54,7 +54,6 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask); /* representing HT and core siblings of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask); -DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, scratch_cpumask); static cpumask_t scratch_cpu0mask; DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, send_ipi_cpumask); diff --git a/xen/include/asm-x86/smp.h b/xen/include/asm-x86/smp.h index 6150363655..acce9c24a4 100644 --- a/xen/include/asm-x86/smp.h +++ b/xen/include/asm-x86/smp.h @@ -24,6 +24,20 @@ DECLARE_PER_CPU(cpumask_var_t, cpu_core_mask); DECLARE_PER_CPU(cpumask_var_t, scratch_cpumask); DECLARE_PER_CPU(cpumask_var_t, send_ipi_cpumask); +#ifndef NDEBUG +/* Not to be called directly, use {get/put}_scratch_cpumask(). */ +cpumask_t *scratch_cpumask(bool use); +#define get_scratch_cpumask() scratch_cpumask(true) +#define put_scratch_cpumask(m) do { \ + BUG_ON((m) != this_cpu(scratch_cpumask)); \ + scratch_cpumask(false); \ + (m) = NULL; \ +} while ( false ) +#else +#define get_scratch_cpumask() this_cpu(scratch_cpumask) +#define put_scratch_cpumask(m) +#endif + /* * Do we, for platform reasons, need to actually keep CPUs online when we * would otherwise prefer them to be off?