From patchwork Tue Mar 10 18:06:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Druzhinin X-Patchwork-Id: 11430029 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2EA5714BC for ; Tue, 10 Mar 2020 18:07:28 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 08EDB2146E for ; Tue, 10 Mar 2020 18:07:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="XuW9Kp0u" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 08EDB2146E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jBjGu-0002Pc-39; Tue, 10 Mar 2020 18:06:28 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jBjGs-0002PX-1j for xen-devel@lists.xenproject.org; Tue, 10 Mar 2020 18:06:26 +0000 X-Inumbo-ID: da42efdd-62f9-11ea-ae17-12813bfff9fa Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id da42efdd-62f9-11ea-ae17-12813bfff9fa; Tue, 10 Mar 2020 18:06:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1583863584; h=from:to:cc:subject:date:message-id:mime-version; bh=4NGHL3/oxeGQ2tolVP6w//SV2Xxy4Dx1M6XadSOxs2w=; b=XuW9Kp0uCLefQutHA+oMGUivvUjkKch1jNqYiWp7dyrClRABzo+iVy14 MZKoQJRM/4vKQVCJTsSJ0Qhqpqoqaf2to4Bznxtv7ujt77iAIfQWsMw1k 2M399a/z/r+Jsi8d0DGiY8YX0mTImoPfLGYMcToGvu5cPV8zkylDoe8Mr w=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=igor.druzhinin@citrix.com; spf=Pass smtp.mailfrom=igor.druzhinin@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of igor.druzhinin@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="igor.druzhinin@citrix.com"; x-sender="igor.druzhinin@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa4.hc3370-68.iphmx.com: domain of igor.druzhinin@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="igor.druzhinin@citrix.com"; x-sender="igor.druzhinin@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="igor.druzhinin@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: JghENT/eG6K4rEQNl3eUb6Eb0oAM4Gbe9cfgVmiG43e5iX0a+Po4B52OHqy64hLv+LjdRHR9iJ k1NGcsDTRbE1lGmhuLvvcwOjYiwueoCFQSDRStlAXZzuu1UO41LGKkGpvEytmVsdJkADQYX9ny ZbCOSjdaSwrZQ0O3j7anKMOgeMRNwQi2/p64G2oQB2jF+mnjbvmFLFxBiCZOAXK+okmDjrX5F7 Z34tlT1DmyB1Wfi0N0Pq/OZdFNlrGmSmsZ0b1+hfJ3JfAQNPEAmTR2A2Owjr00WMFEFtTqxh/n N9I= X-SBRS: 2.7 X-MesageID: 14346329 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,537,1574139600"; d="scan'208";a="14346329" From: Igor Druzhinin To: Date: Tue, 10 Mar 2020 18:06:18 +0000 Message-ID: <1583863578-18063-1-git-send-email-igor.druzhinin@citrix.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v4] x86/cpu: Sync any remaining RCU callbacks before CPU up/down X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: jgross@suse.com, Igor Druzhinin , sstabellini@kernel.org, julien@xen.org, wl@xen.org, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, george.dunlap@citrix.com, jbeulich@suse.com, roger.pau@citrix.com Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" During CPU down operation RCU callbacks are scheduled to finish off some actions later as soon as CPU is fully dead (the same applies to CPU up operation in case error path is taken). If in the same grace period another CPU up operation is performed on the same CPU, RCU callback will be called later on a CPU in a potentially wrong (already up again instead of still being down) state leading to eventual state inconsistency and/or crash. In order to avoid it - flush RCU callbacks explicitly before starting the next CPU up/down operation. Reviewed-by: Juergen Gross Signed-off-by: Igor Druzhinin Reviewed-by: Jan Beulich --- This got discovered trying to resume PV shim with multiple vCPUs on AMD machine (where park_offline_cpus == 0). RCU callback responsible for freeing percpu area on CPU offline got finally called after CPU went online again as the guest performed regular vCPU offline/online operations on resume. Note: this patch requires RCU series v4 from Juergen to be applied - https://lists.xenproject.org/archives/html/xen-devel/2020-03/msg00668.html v2: changed rcu_barrier() position, updated description v3: moved rcu_barrier() to common cpu_up/cpu_down code to cover more cases v4: kept existing comments in modified form --- xen/arch/x86/acpi/power.c | 1 - xen/arch/x86/sysctl.c | 10 ++-------- xen/common/cpu.c | 2 ++ 3 files changed, 4 insertions(+), 9 deletions(-) diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c index b5df00b..847c273 100644 --- a/xen/arch/x86/acpi/power.c +++ b/xen/arch/x86/acpi/power.c @@ -305,7 +305,6 @@ static int enter_state(u32 state) cpufreq_add_cpu(0); enable_cpu: - rcu_barrier(); mtrr_aps_sync_begin(); enable_nonboot_cpus(); mtrr_aps_sync_end(); diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c index a95923e..b0cb1b5 100644 --- a/xen/arch/x86/sysctl.c +++ b/xen/arch/x86/sysctl.c @@ -84,12 +84,9 @@ long cpu_up_helper(void *data) unsigned int cpu = (unsigned long)data; int ret = cpu_up(cpu); + /* Have one more go on EBUSY. */ if ( ret == -EBUSY ) - { - /* On EBUSY, flush RCU work and have one more go. */ - rcu_barrier(); ret = cpu_up(cpu); - } if ( !ret && !opt_smt && cpu_data[cpu].compute_unit_id == INVALID_CUID && @@ -109,12 +106,9 @@ long cpu_down_helper(void *data) { int cpu = (unsigned long)data; int ret = cpu_down(cpu); + /* Have one more go on EBUSY. */ if ( ret == -EBUSY ) - { - /* On EBUSY, flush RCU work and have one more go. */ - rcu_barrier(); ret = cpu_down(cpu); - } return ret; } diff --git a/xen/common/cpu.c b/xen/common/cpu.c index 31953f3..1f976db 100644 --- a/xen/common/cpu.c +++ b/xen/common/cpu.c @@ -4,6 +4,7 @@ #include #include #include +#include unsigned int __read_mostly nr_cpu_ids = NR_CPUS; #ifndef nr_cpumask_bits @@ -53,6 +54,7 @@ void put_cpu_maps(void) void cpu_hotplug_begin(void) { + rcu_barrier(); write_lock(&cpu_add_remove_lock); }