From patchwork Tue Oct 15 09:59:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836022 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26758CFC27F for ; Tue, 15 Oct 2024 10:03:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eMr-00087w-1c; Tue, 15 Oct 2024 06:01:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eMp-00087e-Ow; Tue, 15 Oct 2024 06:01:27 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eMo-0000pG-1k; Tue, 15 Oct 2024 06:01:27 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSTyv2qdFz6D9Sl; Tue, 15 Oct 2024 17:56:55 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id B47E9140AA7; Tue, 15 Oct 2024 18:01:23 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:01:03 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 01/30] arm/virt, target/arm: Add new ARMCPU {socket, cluster, core, thread}-id property Date: Tue, 15 Oct 2024 10:59:43 +0100 Message-ID: <20241015100012.254223-2-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This shall be used to store user specified topology{socket,cluster,core,thread} and shall be converted to a unique 'vcpu-id' which is used as slot-index during hot(un)plug of vCPU. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 10 ++++++++++ include/hw/arm/virt.h | 28 ++++++++++++++++++++++++++++ target/arm/cpu.c | 4 ++++ target/arm/cpu.h | 4 ++++ 4 files changed, 46 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 8b2b991d97..8ff7e109dd 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2242,6 +2242,14 @@ static void machvirt_init(MachineState *machine) &error_fatal); aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); + object_property_set_int(cpuobj, "socket-id", + virt_get_socket_id(machine, n), NULL); + object_property_set_int(cpuobj, "cluster-id", + virt_get_cluster_id(machine, n), NULL); + object_property_set_int(cpuobj, "core-id", + virt_get_core_id(machine, n), NULL); + object_property_set_int(cpuobj, "thread-id", + virt_get_thread_id(machine, n), NULL); if (!vms->secure) { object_property_set_bool(cpuobj, "has_el3", false, NULL); @@ -2765,6 +2773,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) { int n; unsigned int max_cpus = ms->smp.max_cpus; + unsigned int smp_threads = ms->smp.threads; VirtMachineState *vms = VIRT_MACHINE(ms); MachineClass *mc = MACHINE_GET_CLASS(vms); @@ -2778,6 +2787,7 @@ static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) ms->possible_cpus->len = max_cpus; for (n = 0; n < ms->possible_cpus->len; n++) { ms->possible_cpus->cpus[n].type = ms->cpu_type; + ms->possible_cpus->cpus[n].vcpus_count = smp_threads; ms->possible_cpus->cpus[n].arch_id = virt_cpu_mp_affinity(vms, n); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index aca4f8061b..db3e2aebb9 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -214,4 +214,32 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) vms->highmem_redists) ? 2 : 1; } +static inline int virt_get_socket_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.socket_id; +} + +static inline int virt_get_cluster_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.cluster_id; +} + +static inline int virt_get_core_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.core_id; +} + +static inline int virt_get_thread_id(const MachineState *ms, int cpu_index) +{ + assert(cpu_index >= 0 && cpu_index < ms->possible_cpus->len); + + return ms->possible_cpus->cpus[cpu_index].props.thread_id; +} + #endif /* QEMU_ARM_VIRT_H */ diff --git a/target/arm/cpu.c b/target/arm/cpu.c index 19191c2391..bda95366d1 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -2622,6 +2622,10 @@ static Property arm_cpu_properties[] = { DEFINE_PROP_UINT64("mp-affinity", ARMCPU, mp_affinity, ARM64_AFFINITY_INVALID), DEFINE_PROP_INT32("node-id", ARMCPU, node_id, CPU_UNSET_NUMA_NODE_ID), + DEFINE_PROP_INT32("socket-id", ARMCPU, socket_id, 0), + DEFINE_PROP_INT32("cluster-id", ARMCPU, cluster_id, 0), + DEFINE_PROP_INT32("core-id", ARMCPU, core_id, 0), + DEFINE_PROP_INT32("thread-id", ARMCPU, thread_id, 0), DEFINE_PROP_INT32("core-count", ARMCPU, core_count, -1), /* True to default to the backward-compat old CNTFRQ rather than 1Ghz */ DEFINE_PROP_BOOL("backcompat-cntfrq", ARMCPU, backcompat_cntfrq, false), diff --git a/target/arm/cpu.h b/target/arm/cpu.h index f065756c5c..1277a0ddfc 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1086,6 +1086,10 @@ struct ArchCPU { QLIST_HEAD(, ARMELChangeHook) el_change_hooks; int32_t node_id; /* NUMA node this CPU belongs to */ + int32_t socket_id; + int32_t cluster_id; + int32_t core_id; + int32_t thread_id; /* Used to synchronize KVM and QEMU in-kernel device levels */ uint8_t device_irq_level; From patchwork Tue Oct 15 09:59:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836019 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 71CC0CFC280 for ; Tue, 15 Oct 2024 10:02:37 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eNC-0008BW-VD; Tue, 15 Oct 2024 06:01:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eNA-0008AO-PK; Tue, 15 Oct 2024 06:01:48 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eN9-0000qH-26; Tue, 15 Oct 2024 06:01:48 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV3n57Yzz6K98d; Tue, 15 Oct 2024 18:01:09 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id D298A140CF4; Tue, 15 Oct 2024 18:01:43 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:01:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 02/30] hw/arm/virt: Disable vCPU hotplug for *unsupported* Accel or GIC Type Date: Tue, 15 Oct 2024 10:59:44 +0100 Message-ID: <20241015100012.254223-3-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For unsupported acceleration types and GIC versions, explicitly disable vCPU hotplug support and limit the number of possible vCPUs to those available at boot time (i.e., SMP CPUs). This flag will be referenced at various points in the code to verify the presence of vCPU hotplug functionality on this machine. Signed-off-by: Salil Mehta --- hw/arm/virt.c | 66 +++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 8ff7e109dd..827a796ed6 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2108,8 +2108,6 @@ static void machvirt_init(MachineState *machine) unsigned int smp_cpus = machine->smp.cpus; unsigned int max_cpus = machine->smp.max_cpus; - possible_cpus = mc->possible_cpu_arch_ids(machine); - /* * In accelerated mode, the memory map is computed earlier in kvm_type() * for Linux, or hvf_get_physical_address_range() for macOS to create a @@ -2125,7 +2123,7 @@ static void machvirt_init(MachineState *machine) * we are about to deal with. Once this is done, get rid of * the object. */ - cpuobj = object_new(possible_cpus->cpus[0].type); + cpuobj = object_new(machine->cpu_type); armcpu = ARM_CPU(cpuobj); pa_bits = arm_pamax(armcpu); @@ -2140,6 +2138,43 @@ static void machvirt_init(MachineState *machine) */ finalize_gic_version(vms); + /* + * The maximum number of CPUs depends on the GIC version, or on how + * many redistributors we can fit into the memory map (which in turn + * depends on whether this is a GICv3 or v4). + */ + if (vms->gic_version == VIRT_GIC_VERSION_2) { + virt_max_cpus = GIC_NCPU; + } else { + virt_max_cpus = virt_redist_capacity(vms, VIRT_GIC_REDIST); + if (vms->highmem_redists) { + virt_max_cpus += virt_redist_capacity(vms, VIRT_HIGH_GIC_REDIST2); + } + } + + if ((tcg_enabled() && !qemu_tcg_mttcg_enabled()) || hvf_enabled() || + qtest_enabled() || (vms->gic_version < VIRT_GIC_VERSION_3)) { + max_cpus = machine->smp.max_cpus = smp_cpus; + mc->has_hotpluggable_cpus = false; + if (vms->gic_version >= VIRT_GIC_VERSION_3) { + warn_report("cpu hotplug feature has been disabled"); + } + } + + if (max_cpus > virt_max_cpus) { + error_report("Number of SMP CPUs requested (%d) exceeds max CPUs " + "supported by machine 'mach-virt' (%d)", + max_cpus, virt_max_cpus); + if (vms->gic_version != VIRT_GIC_VERSION_2 && !vms->highmem_redists) { + error_printf("Try 'highmem-redists=on' for more CPUs\n"); + } + + exit(1); + } + + /* uses smp.max_cpus to initialize all possible vCPUs */ + possible_cpus = mc->possible_cpu_arch_ids(machine); + if (vms->secure) { /* * The Secure view of the world is the same as the NonSecure, @@ -2174,31 +2209,6 @@ static void machvirt_init(MachineState *machine) vms->psci_conduit = QEMU_PSCI_CONDUIT_HVC; } - /* - * The maximum number of CPUs depends on the GIC version, or on how - * many redistributors we can fit into the memory map (which in turn - * depends on whether this is a GICv3 or v4). - */ - if (vms->gic_version == VIRT_GIC_VERSION_2) { - virt_max_cpus = GIC_NCPU; - } else { - virt_max_cpus = virt_redist_capacity(vms, VIRT_GIC_REDIST); - if (vms->highmem_redists) { - virt_max_cpus += virt_redist_capacity(vms, VIRT_HIGH_GIC_REDIST2); - } - } - - if (max_cpus > virt_max_cpus) { - error_report("Number of SMP CPUs requested (%d) exceeds max CPUs " - "supported by machine 'mach-virt' (%d)", - max_cpus, virt_max_cpus); - if (vms->gic_version != VIRT_GIC_VERSION_2 && !vms->highmem_redists) { - error_printf("Try 'highmem-redists=on' for more CPUs\n"); - } - - exit(1); - } - if (vms->secure && (kvm_enabled() || hvf_enabled())) { error_report("mach-virt: %s does not support providing " "Security extensions (TrustZone) to the guest CPU", From patchwork Tue Oct 15 09:59:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72D5CCFC27F for ; Tue, 15 Oct 2024 10:03:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eNi-0008UA-6C; Tue, 15 Oct 2024 06:02:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eNU-0008Em-TE; Tue, 15 Oct 2024 06:02:09 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eNS-0000sG-S8; Tue, 15 Oct 2024 06:02:08 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV2y5Z9Vz6FGxQ; Tue, 15 Oct 2024 18:00:26 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id D6A20140451; Tue, 15 Oct 2024 18:02:03 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:01:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 03/30] hw/arm/virt: Move setting of common vCPU properties in a function Date: Tue, 15 Oct 2024 10:59:45 +0100 Message-ID: <20241015100012.254223-4-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Factor out vCPU properties code common for {hot,cold}-plugged vCPUs in the machvirt_init(). This allows code reuse. Signed-off-by: Salil Mehta --- hw/arm/virt.c | 219 ++++++++++++++++++++++++++---------------- include/hw/arm/virt.h | 4 + 2 files changed, 139 insertions(+), 84 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 827a796ed6..a64d028ec4 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2091,16 +2091,129 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) } } +static void virt_cpu_set_properties(Object *cpuobj, const CPUArchId *cpu_slot, + Error **errp) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + VirtMachineState *vms = VIRT_MACHINE(ms); + Error *local_err = NULL; + VirtMachineClass *vmc; + + vmc = VIRT_MACHINE_GET_CLASS(ms); + + /* now, set the cpu object property values */ + numa_cpu_pre_plug(cpu_slot, DEVICE(cpuobj), &local_err); + if (local_err) { + goto out; + } + + object_property_set_int(cpuobj, "mp-affinity", cpu_slot->arch_id, NULL); + + if (!vms->secure) { + object_property_set_bool(cpuobj, "has_el3", false, NULL); + } + + if (!vms->virt && object_property_find(cpuobj, "has_el2")) { + object_property_set_bool(cpuobj, "has_el2", false, NULL); + } + + if (vmc->kvm_no_adjvtime && + object_property_find(cpuobj, "kvm-no-adjvtime")) { + object_property_set_bool(cpuobj, "kvm-no-adjvtime", true, NULL); + } + + if (vmc->no_kvm_steal_time && + object_property_find(cpuobj, "kvm-steal-time")) { + object_property_set_bool(cpuobj, "kvm-steal-time", false, NULL); + } + + if (vmc->no_pmu && object_property_find(cpuobj, "pmu")) { + object_property_set_bool(cpuobj, "pmu", false, NULL); + } + + if (vmc->no_tcg_lpa2 && object_property_find(cpuobj, "lpa2")) { + object_property_set_bool(cpuobj, "lpa2", false, NULL); + } + + if (object_property_find(cpuobj, "reset-cbar")) { + object_property_set_int(cpuobj, "reset-cbar", + vms->memmap[VIRT_CPUPERIPHS].base, + &local_err); + if (local_err) { + goto out; + } + } + + /* link already initialized {secure,tag}-memory regions to this cpu */ + object_property_set_link(cpuobj, "memory", OBJECT(vms->sysmem), &local_err); + if (local_err) { + goto out; + } + + if (vms->secure) { + object_property_set_link(cpuobj, "secure-memory", + OBJECT(vms->secure_sysmem), &local_err); + if (local_err) { + goto out; + } + } + + if (vms->mte) { + if (!object_property_find(cpuobj, "tag-memory")) { + error_setg(&local_err, "MTE requested, but not supported " + "by the guest CPU"); + if (local_err) { + goto out; + } + } + + object_property_set_link(cpuobj, "tag-memory", OBJECT(vms->tag_sysmem), + &local_err); + if (local_err) { + goto out; + } + + if (vms->secure) { + object_property_set_link(cpuobj, "secure-tag-memory", + OBJECT(vms->secure_tag_sysmem), + &local_err); + if (local_err) { + goto out; + } + } + } + + /* + * RFC: Question: this must only be called for the hotplugged cpus. For the + * cold booted secondary cpus this is being taken care in arm_load_kernel() + * in boot.c. Perhaps we should remove that code now? + */ + if (vms->psci_conduit != QEMU_PSCI_CONDUIT_DISABLED) { + object_property_set_int(cpuobj, "psci-conduit", vms->psci_conduit, + NULL); + + /* Secondary CPUs start in PSCI powered-down state */ + if (CPU(cpuobj)->cpu_index > 0) { + object_property_set_bool(cpuobj, "start-powered-off", true, NULL); + } + } + +out: + if (local_err) { + error_propagate(errp, local_err); + } +} + static void machvirt_init(MachineState *machine) { VirtMachineState *vms = VIRT_MACHINE(machine); VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(machine); MachineClass *mc = MACHINE_GET_CLASS(machine); const CPUArchIdList *possible_cpus; - MemoryRegion *sysmem = get_system_memory(); + MemoryRegion *secure_tag_sysmem = NULL; MemoryRegion *secure_sysmem = NULL; MemoryRegion *tag_sysmem = NULL; - MemoryRegion *secure_tag_sysmem = NULL; + MemoryRegion *sysmem; int n, virt_max_cpus; bool firmware_loaded; bool aarch64 = true; @@ -2175,6 +2288,8 @@ static void machvirt_init(MachineState *machine) /* uses smp.max_cpus to initialize all possible vCPUs */ possible_cpus = mc->possible_cpu_arch_ids(machine); + sysmem = vms->sysmem = get_system_memory(); + if (vms->secure) { /* * The Secure view of the world is the same as the NonSecure, @@ -2182,7 +2297,7 @@ static void machvirt_init(MachineState *machine) * containing the system memory at low priority; any secure-only * devices go in at higher priority and take precedence. */ - secure_sysmem = g_new(MemoryRegion, 1); + secure_sysmem = vms->secure_sysmem = g_new(MemoryRegion, 1); memory_region_init(secure_sysmem, OBJECT(machine), "secure-memory", UINT64_MAX); memory_region_add_subregion_overlap(secure_sysmem, 0, sysmem, -1); @@ -2230,6 +2345,23 @@ static void machvirt_init(MachineState *machine) exit(1); } + if (vms->mte) { + /* Create the memory region only once, but link to all cpus later */ + tag_sysmem = vms->tag_sysmem = g_new(MemoryRegion, 1); + memory_region_init(tag_sysmem, OBJECT(machine), + "tag-memory", UINT64_MAX / 32); + + if (vms->secure) { + secure_tag_sysmem = vms->secure_tag_sysmem = g_new(MemoryRegion, 1); + memory_region_init(secure_tag_sysmem, OBJECT(machine), + "secure-tag-memory", UINT64_MAX / 32); + + /* As with ram, secure-tag takes precedence over tag. */ + memory_region_add_subregion_overlap(secure_tag_sysmem, 0, + tag_sysmem, -1); + } + } + create_fdt(vms); assert(possible_cpus->len == max_cpus); @@ -2242,15 +2374,10 @@ static void machvirt_init(MachineState *machine) } cpuobj = object_new(possible_cpus->cpus[n].type); - object_property_set_int(cpuobj, "mp-affinity", - possible_cpus->cpus[n].arch_id, NULL); cs = CPU(cpuobj); cs->cpu_index = n; - numa_cpu_pre_plug(&possible_cpus->cpus[cs->cpu_index], DEVICE(cpuobj), - &error_fatal); - aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, "socket-id", virt_get_socket_id(machine, n), NULL); @@ -2261,82 +2388,6 @@ static void machvirt_init(MachineState *machine) object_property_set_int(cpuobj, "thread-id", virt_get_thread_id(machine, n), NULL); - if (!vms->secure) { - object_property_set_bool(cpuobj, "has_el3", false, NULL); - } - - if (!vms->virt && object_property_find(cpuobj, "has_el2")) { - object_property_set_bool(cpuobj, "has_el2", false, NULL); - } - - if (vmc->kvm_no_adjvtime && - object_property_find(cpuobj, "kvm-no-adjvtime")) { - object_property_set_bool(cpuobj, "kvm-no-adjvtime", true, NULL); - } - - if (vmc->no_kvm_steal_time && - object_property_find(cpuobj, "kvm-steal-time")) { - object_property_set_bool(cpuobj, "kvm-steal-time", false, NULL); - } - - if (vmc->no_pmu && object_property_find(cpuobj, "pmu")) { - object_property_set_bool(cpuobj, "pmu", false, NULL); - } - - if (vmc->no_tcg_lpa2 && object_property_find(cpuobj, "lpa2")) { - object_property_set_bool(cpuobj, "lpa2", false, NULL); - } - - if (object_property_find(cpuobj, "reset-cbar")) { - object_property_set_int(cpuobj, "reset-cbar", - vms->memmap[VIRT_CPUPERIPHS].base, - &error_abort); - } - - object_property_set_link(cpuobj, "memory", OBJECT(sysmem), - &error_abort); - if (vms->secure) { - object_property_set_link(cpuobj, "secure-memory", - OBJECT(secure_sysmem), &error_abort); - } - - if (vms->mte) { - /* Create the memory region only once, but link to all cpus. */ - if (!tag_sysmem) { - /* - * The property exists only if MemTag is supported. - * If it is, we must allocate the ram to back that up. - */ - if (!object_property_find(cpuobj, "tag-memory")) { - error_report("MTE requested, but not supported " - "by the guest CPU"); - exit(1); - } - - tag_sysmem = g_new(MemoryRegion, 1); - memory_region_init(tag_sysmem, OBJECT(machine), - "tag-memory", UINT64_MAX / 32); - - if (vms->secure) { - secure_tag_sysmem = g_new(MemoryRegion, 1); - memory_region_init(secure_tag_sysmem, OBJECT(machine), - "secure-tag-memory", UINT64_MAX / 32); - - /* As with ram, secure-tag takes precedence over tag. */ - memory_region_add_subregion_overlap(secure_tag_sysmem, 0, - tag_sysmem, -1); - } - } - - object_property_set_link(cpuobj, "tag-memory", OBJECT(tag_sysmem), - &error_abort); - if (vms->secure) { - object_property_set_link(cpuobj, "secure-tag-memory", - OBJECT(secure_tag_sysmem), - &error_abort); - } - } - qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); object_unref(cpuobj); } diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index db3e2aebb9..5300e8d2bc 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -143,6 +143,10 @@ struct VirtMachineState { DeviceState *platform_bus_dev; FWCfgState *fw_cfg; PFlashCFI01 *flash[2]; + MemoryRegion *sysmem; + MemoryRegion *secure_sysmem; + MemoryRegion *tag_sysmem; + MemoryRegion *secure_tag_sysmem; bool secure; bool highmem; bool highmem_compact; From patchwork Tue Oct 15 09:59:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91955CFC280 for ; Tue, 15 Oct 2024 10:03:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eOJ-0000nr-9P; Tue, 15 Oct 2024 06:03:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eNv-0000Xa-3h; Tue, 15 Oct 2024 06:02:38 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eNs-0000vA-8l; Tue, 15 Oct 2024 06:02:34 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV4Y64mfz6K94H; Tue, 15 Oct 2024 18:01:49 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id F3476140A34; Tue, 15 Oct 2024 18:02:23 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:02:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 04/30] arm/virt, target/arm: Machine init time change common to vCPU {cold|hot}-plug Date: Tue, 15 Oct 2024 10:59:46 +0100 Message-ID: <20241015100012.254223-5-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Introduce the common logic required during the initialization of both cold and hot-plugged vCPUs. Additionally, initialize the *disabled* state of the vCPUs, which will be used further during the initialization phases of various other components like GIC, PMU, ACPI, etc., as part of the virtual machine initialization. Reported-by: Gavin Shan [GS: pointed the assertion due to wrong range check in cpu_pre_plug phase] Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 90 ++++++++++++++++++++++++++++++++++++++++++- include/hw/arm/virt.h | 26 +++++++++++++ target/arm/cpu.c | 7 ++++ target/arm/cpu64.c | 2 + 4 files changed, 124 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index a64d028ec4..48102d5a4c 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2830,6 +2830,26 @@ static int64_t virt_get_default_cpu_node_id(const MachineState *ms, int idx) return socket_id % ms->numa_state->num_nodes; } +static int +virt_get_cpu_id_from_cpu_topo(const MachineState *ms, DeviceState *dev) +{ + int cpu_id, sock_vcpu_num, clus_vcpu_num, core_vcpu_num; + ARMCPU *cpu = ARM_CPU(dev); + + /* calculate total logical cpus across socket/cluster/core */ + sock_vcpu_num = cpu->socket_id * (ms->smp.threads * ms->smp.cores * + ms->smp.clusters); + clus_vcpu_num = cpu->cluster_id * (ms->smp.threads * ms->smp.cores); + core_vcpu_num = cpu->core_id * ms->smp.threads; + + /* get vcpu-id(logical cpu index) for this vcpu from this topology */ + cpu_id = (sock_vcpu_num + clus_vcpu_num + core_vcpu_num) + cpu->thread_id; + + assert(cpu_id >= 0 && cpu_id < ms->possible_cpus->len); + + return cpu_id; +} + static const CPUArchIdList *virt_possible_cpu_arch_ids(MachineState *ms) { int n; @@ -2912,6 +2932,68 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, dev, &error_abort); } +static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + MachineState *ms = MACHINE(hotplug_dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + /* sanity check the cpu */ + if (!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) { + error_setg(errp, "Invalid CPU type, expected cpu type: '%s'", + ms->cpu_type); + return; + } + + if ((cpu->thread_id < 0) || (cpu->thread_id >= ms->smp.threads)) { + error_setg(errp, "Invalid thread-id %u specified, correct range 0:%u", + cpu->thread_id, ms->smp.threads - 1); + return; + } + + if ((cpu->core_id < 0) || (cpu->core_id >= ms->smp.cores)) { + error_setg(errp, "Invalid core-id %d specified, correct range 0:%u", + cpu->core_id, ms->smp.cores - 1); + return; + } + + if ((cpu->cluster_id < 0) || (cpu->cluster_id >= ms->smp.clusters)) { + error_setg(errp, "Invalid cluster-id %u specified, correct range 0:%u", + cpu->cluster_id, ms->smp.clusters - 1); + return; + } + + if ((cpu->socket_id < 0) || (cpu->socket_id >= ms->smp.sockets)) { + error_setg(errp, "Invalid socket-id %u specified, correct range 0:%u", + cpu->socket_id, ms->smp.sockets - 1); + return; + } + + cs->cpu_index = virt_get_cpu_id_from_cpu_topo(ms, dev); + + cpu_slot = virt_find_cpu_slot(cs); + if (cpu_slot->cpu) { + error_setg(errp, "cpu(id%d=%d:%d:%d:%d) with arch-id %" PRIu64 " exist", + cs->cpu_index, cpu->socket_id, cpu->cluster_id, cpu->core_id, + cpu->thread_id, cpu_slot->arch_id); + return; + } + virt_cpu_set_properties(OBJECT(cs), cpu_slot, errp); +} + +static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + /* insert the cold/hot-plugged vcpu in the slot */ + cpu_slot = virt_find_cpu_slot(cs); + cpu_slot->cpu = CPU(dev); +} + static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2919,6 +3001,8 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM)) { virt_memory_pre_plug(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_pre_plug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_pre_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { @@ -2975,6 +3059,8 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev, virt_memory_plug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_plug(hotplug_dev, dev, errp); } if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { @@ -3059,7 +3145,8 @@ static HotplugHandler *virt_machine_get_hotplug_handler(MachineState *machine, if (device_is_dynamic_sysbus(mc, dev) || object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) || object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI) || - object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) { + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI) || + object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { return HOTPLUG_HANDLER(machine); } return NULL; @@ -3197,6 +3284,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->get_default_cpu_node_id = virt_get_default_cpu_node_id; mc->kvm_type = virt_kvm_type; mc->hvf_get_physical_address_range = virt_hvf_get_physical_address_range; + mc->has_hotpluggable_cpus = true; assert(!mc->get_hotplug_handler); mc->get_hotplug_handler = virt_machine_get_hotplug_handler; hc->pre_plug = virt_machine_device_pre_plug_cb; diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 5300e8d2bc..446c574c0d 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -246,4 +246,30 @@ static inline int virt_get_thread_id(const MachineState *ms, int cpu_index) return ms->possible_cpus->cpus[cpu_index].props.thread_id; } +static inline CPUArchId *virt_find_cpu_slot(CPUState *cs) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + CPUArchId *cpu_slot; + + assert(cs->cpu_index >= 0 && cs->cpu_index < ms->possible_cpus->len); + + cpu_slot = &ms->possible_cpus->cpus[cs->cpu_index]; + + /* + * The slot ID refers to the index where a vCPU with a specific architecture + * ID (e.g., MPIDR or affinity) is plugged in. The slot ID is more closely + * related to the machine configuration, while the architecture ID is tied + * directly to the vCPU itself. Currently, the code assumes that the slot ID + * and architecture ID are the same, which can make the concept of a slot + * somewhat vague. However, it makes more sense to associate the + * hot-(un)plugging of vCPUs with a slot as a metaphor. This could represent + * the smallest granularity for vCPU hot-(un)plugging. That said, we cannot + * rule out the possibility of extending this concept to die or socket + * level hot-(un)plugging in the future, should the ARM specification allow + * for it. + */ + + return cpu_slot; +} + #endif /* QEMU_ARM_VIRT_H */ diff --git a/target/arm/cpu.c b/target/arm/cpu.c index bda95366d1..3de0cb346b 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -2680,6 +2680,12 @@ static const TCGCPUOps arm_tcg_ops = { }; #endif /* CONFIG_TCG */ +static int64_t arm_cpu_get_arch_id(CPUState *cs) +{ + ARMCPU *cpu = ARM_CPU(cs); + return cpu->mp_affinity; +} + static void arm_cpu_class_init(ObjectClass *oc, void *data) { ARMCPUClass *acc = ARM_CPU_CLASS(oc); @@ -2699,6 +2705,7 @@ static void arm_cpu_class_init(ObjectClass *oc, void *data) cc->has_work = arm_cpu_has_work; cc->mmu_index = arm_cpu_mmu_index; cc->dump_state = arm_cpu_dump_state; + cc->get_arch_id = arm_cpu_get_arch_id; cc->set_pc = arm_cpu_set_pc; cc->get_pc = arm_cpu_get_pc; cc->gdb_read_register = arm_cpu_gdb_read_register; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index 458d1cee01..0e217f827e 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -803,7 +803,9 @@ static const gchar *aarch64_gdb_arch_name(CPUState *cs) static void aarch64_cpu_class_init(ObjectClass *oc, void *data) { CPUClass *cc = CPU_CLASS(oc); + DeviceClass *dc = DEVICE_CLASS(oc); + dc->user_creatable = true; cc->gdb_read_register = aarch64_cpu_gdb_read_register; cc->gdb_write_register = aarch64_cpu_gdb_write_register; cc->gdb_core_xml_file = "aarch64-core.xml"; From patchwork Tue Oct 15 09:59:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836023 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EC8E4CFC281 for ; Tue, 15 Oct 2024 10:03:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eOW-0001FW-95; Tue, 15 Oct 2024 06:03:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eO9-0000kW-66; Tue, 15 Oct 2024 06:02:50 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eO6-0000wm-WB; Tue, 15 Oct 2024 06:02:48 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV3k5NW4z6FGmc; Tue, 15 Oct 2024 18:01:06 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id E3F2D1404F9; Tue, 15 Oct 2024 18:02:43 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:02:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 05/30] arm/virt, kvm: Pre-create KVM vCPUs for all unplugged QOM vCPUs @machine init Date: Tue, 15 Oct 2024 10:59:47 +0100 Message-ID: <20241015100012.254223-6-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ARM CPU architecture does not allows CPU to be plugged after system has initialized. This is an constraint. Hence, Kernel must know all the CPUs being booted during its initialization. This applies to the Guest Kernel as well and therefore, number of KVM vCPUs need to be fixed at the VM initialization time. Also, The GIC must know all the CPUs it is connected to during its initialization, and this cannot change afterward. This must also be ensured during the initialization of the VGIC in KVM. This is necessary because: 1. The association between GICR and MPIDR must be fixed at VM initialization time. This is represented by the register `GICR_TYPER(mp_affinity, proc_num)` 2. Memory regions associated with GICR, etc., cannot be changed (added, deleted, or modified) after the VM has been initialized. This is not an ARM architectural constraint but rather invites a difficult and messy change in VGIC data structures. This patch adds support to pre-create all possible vCPUs within the KVM host as part of the virtual machine initialization. These vCPUs can later be unparked and attached to the hotplugged QOM vCPU. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] --- hw/arm/virt.c | 74 ++++++++++++++++++++++++++++++++++++++----- include/hw/core/cpu.h | 1 + target/arm/cpu64.c | 9 ++++++ target/arm/kvm.c | 41 +++++++++++++++++++++++- target/arm/kvm_arm.h | 10 ++++++ 5 files changed, 126 insertions(+), 9 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 48102d5a4c..858a19bc8b 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2366,17 +2366,12 @@ static void machvirt_init(MachineState *machine) assert(possible_cpus->len == max_cpus); for (n = 0; n < possible_cpus->len; n++) { + CPUArchId *cpu_slot; Object *cpuobj; CPUState *cs; - if (n >= smp_cpus) { - break; - } - cpuobj = object_new(possible_cpus->cpus[n].type); - cs = CPU(cpuobj); - cs->cpu_index = n; aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL); object_property_set_int(cpuobj, "socket-id", @@ -2388,8 +2383,61 @@ static void machvirt_init(MachineState *machine) object_property_set_int(cpuobj, "thread-id", virt_get_thread_id(machine, n), NULL); - qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); - object_unref(cpuobj); + if (n < smp_cpus) { + qdev_realize(DEVICE(cpuobj), NULL, &error_fatal); + object_unref(cpuobj); + } else { + /* + * Handling vCPUs that are yet to be hot-plugged requires the + * unrealized `ARMCPU` object for the following purposes: + * + * 1. To create the corresponding host KVM vCPU. + * 2. During the GICv3 realization phase, the `GICR_TYPER` value is + * derived using the fetched MPIDR/mp-affinity. It's worth + * considering modifying the GICv3 realization code to directly + * fetch the `arch-id`/mp-affinity from the possible vCPUs. + * 3. Additionally, the `ARMCPU` object must be retained until + * `virt_cpu_post_init`, as there may be late per-vCPU + * initializations. + * + * Once these tasks are completed, the initialized `ARMCPU` object + * can be safely released as those are not required and will be + * recreated when they are {hot,cold}-plugged later. + */ + cs->cpu_index = n; + cpu_slot = virt_find_cpu_slot(cs); + + /* + * We will pre-create the KVM vCPUs corresponding to the currently + * unplugged but possible QOM vCPUs and park them until they are + * actually hot-plugged. The ARM architecture does not allow new + * CPUs to be plugged after the system has been initialized, and + * this constraint is also reflected in KVM. + */ + if (kvm_enabled()) { + kvm_arm_create_host_vcpu(ARM_CPU(cs)); + /* + * Override the default architecture ID with the one fetched + * from KVM. After initialization, we will destroy the CPUState + * for disabled vCPUs; however, the CPU slot and its association + * with the architecture ID (and consequently the vCPU ID) will + * remain fixed for the entire lifetime of QEMU. + */ + cpu_slot->arch_id = arm_cpu_mp_affinity(ARM_CPU(cs)); + } + + /* + * GICv3 realization will need `mp-affinity` to derive `gicr_typer` + */ + object_property_set_int(cpuobj, "mp-affinity", cpu_slot->arch_id, + NULL); + + /* + * Add the unplugged vCPU to the vCPU slot temporarily. It will be + * released later in the virt_post_init_cpu() function + */ + cpu_slot->cpu = cs; + } } /* Now we've created the CPUs we can see if they have the hypvirt timer */ @@ -2992,6 +3040,16 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, /* insert the cold/hot-plugged vcpu in the slot */ cpu_slot = virt_find_cpu_slot(cs); cpu_slot->cpu = CPU(dev); + + if (kvm_enabled()) { + /* + * Override the default architecture ID with the one fetched from KVM + * Currently, KVM derives the architecture ID from the vCPU ID specified + * by QEMU. In the future, we might implement a change where the entire + * architecture ID can be configured directly by QEMU. + */ + cpu_slot->arch_id = arm_cpu_mp_affinity(ARM_CPU(cs)); + } } static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 299e96c45b..4e0cb325a0 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -534,6 +534,7 @@ struct CPUState { uint64_t dirty_pages; int kvm_vcpu_stats_fd; bool vcpu_dirty; + VMChangeStateEntry *vmcse; /* Use by accel-block: CPU is executing an ioctl() */ QemuLockCnt in_ioctl_lock; diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index 0e217f827e..d2f4624d61 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -791,6 +791,14 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp) } } +static void aarch64_cpu_initfn(Object *obj) +{ + CPUState *cs = CPU(obj); + + /* TODO: re-check if this is necessary still */ + cs->thread_id = 0; +} + static void aarch64_cpu_finalizefn(Object *obj) { } @@ -850,6 +858,7 @@ void aarch64_cpu_register(const ARMCPUInfo *info) static const TypeInfo aarch64_cpu_type_info = { .name = TYPE_AARCH64_CPU, .parent = TYPE_ARM_CPU, + .instance_init = aarch64_cpu_initfn, .instance_finalize = aarch64_cpu_finalizefn, .abstract = true, .class_init = aarch64_cpu_class_init, diff --git a/target/arm/kvm.c b/target/arm/kvm.c index f1f1b5b375..e82cb2aa8b 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1003,6 +1003,38 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) write_list_to_cpustate(cpu); } +void kvm_arm_create_host_vcpu(ARMCPU *cpu) +{ + CPUState *cs = CPU(cpu); + unsigned long vcpu_id = cs->cpu_index; + int ret; + + ret = kvm_create_vcpu(cs); + if (ret < 0) { + error_report("Failed to create host vcpu %ld", vcpu_id); + abort(); + } + + /* + * Initialize the vCPU in the host. This will reset the sys regs + * for this vCPU and related registers like MPIDR_EL1 etc. also + * gets programmed during this call to host. These are referred + * later while setting device attributes of the GICR during GICv3 + * reset + */ + ret = kvm_arch_init_vcpu(cs); + if (ret < 0) { + error_report("Failed to initialize host vcpu %ld", vcpu_id); + abort(); + } + + /* + * park the created vCPU. shall be used during kvm_get_vcpu() when + * threads are created during realization of ARM vCPUs. + */ + kvm_park_vcpu(cs); +} + /* * Update KVM's MP_STATE based on what QEMU thinks it is */ @@ -1874,7 +1906,14 @@ int kvm_arch_init_vcpu(CPUState *cs) return -EINVAL; } - qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, cpu); + /* + * Install VM change handler only when vCPU thread has been spawned + * i.e. vCPU is being realized + */ + if (cs->thread_id) { + cs->vmcse = qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, + cpu); + } /* Determine init features for this CPU */ memset(cpu->kvm_init_features, 0, sizeof(cpu->kvm_init_features)); diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index cfaa0d9bc7..93d12eeb74 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -96,6 +96,16 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu); */ void kvm_arm_reset_vcpu(ARMCPU *cpu); +/** + * kvm_arm_create_host_vcpu: + * @cpu: ARMCPU + * + * Called to pre-create possible KVM vCPU within the host during the + * `virt_machine` initialization phase. This pre-created vCPU will be parked and + * will be reused when ARM QOM vCPU is actually hotplugged. + */ +void kvm_arm_create_host_vcpu(ARMCPU *cpu); + #ifdef CONFIG_KVM /** * kvm_arm_create_scratch_host_vcpu: From patchwork Tue Oct 15 09:59:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C5B3CFC280 for ; Tue, 15 Oct 2024 10:03:25 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eOY-0001TW-Dm; Tue, 15 Oct 2024 06:03:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eOR-00013N-Mm; Tue, 15 Oct 2024 06:03:08 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eOP-0000yr-TH; Tue, 15 Oct 2024 06:03:07 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV5K4Yw7z6K98P; Tue, 15 Oct 2024 18:02:29 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 148CC1404F5; Tue, 15 Oct 2024 18:03:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:02:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 06/30] arm/virt, gicv3: Changes to pre-size GIC with possible vCPUs @machine init Date: Tue, 15 Oct 2024 10:59:48 +0100 Message-ID: <20241015100012.254223-7-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The GIC must be pre-sized with the possible vCPUs during initialization. This is essential because: 1. Memory regions and resources associated with GICC/GICR cannot be modified (i.e., added, deleted, or changed) once the VM has been initialized. 2. Additionally, the `GIC_TYPER` must be initialized with the `mp_affinity` and CPU interface number associations, which cannot be altered after initialization. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 15 ++++++++------- include/hw/arm/virt.h | 2 +- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 858a19bc8b..b286afc62c 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -769,6 +769,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) const char *gictype; int i; unsigned int smp_cpus = ms->smp.cpus; + unsigned int max_cpus = ms->smp.max_cpus; uint32_t nb_redist_regions = 0; int revision; @@ -793,7 +794,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) } vms->gic = qdev_new(gictype); qdev_prop_set_uint32(vms->gic, "revision", revision); - qdev_prop_set_uint32(vms->gic, "num-cpu", smp_cpus); + qdev_prop_set_uint32(vms->gic, "num-cpu", max_cpus); /* Note that the num-irq property counts both internal and external * interrupts; there are always 32 of the former (mandated by GIC spec). */ @@ -805,7 +806,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) if (vms->gic_version != VIRT_GIC_VERSION_2) { QList *redist_region_count; uint32_t redist0_capacity = virt_redist_capacity(vms, VIRT_GIC_REDIST); - uint32_t redist0_count = MIN(smp_cpus, redist0_capacity); + uint32_t redist0_count = MIN(max_cpus, redist0_capacity); nb_redist_regions = virt_gicv3_redist_region_count(vms); @@ -816,7 +817,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) virt_redist_capacity(vms, VIRT_HIGH_GIC_REDIST2); qlist_append_int(redist_region_count, - MIN(smp_cpus - redist0_count, redist1_capacity)); + MIN(max_cpus - redist0_count, redist1_capacity)); } qdev_prop_set_array(vms->gic, "redist-region-count", redist_region_count); @@ -889,7 +890,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) } else if (vms->virt) { qemu_irq irq = qdev_get_gpio_in(vms->gic, intidbase + ARCH_GIC_MAINT_IRQ); - sysbus_connect_irq(gicbusdev, i + 4 * smp_cpus, irq); + sysbus_connect_irq(gicbusdev, i + 4 * max_cpus, irq); } qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, @@ -897,11 +898,11 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) + VIRTUAL_PMU_IRQ)); sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); - sysbus_connect_irq(gicbusdev, i + smp_cpus, + sysbus_connect_irq(gicbusdev, i + max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); - sysbus_connect_irq(gicbusdev, i + 2 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 2 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); - sysbus_connect_irq(gicbusdev, i + 3 * smp_cpus, + sysbus_connect_irq(gicbusdev, i + 3 * max_cpus, qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); if (vms->gic_version != VIRT_GIC_VERSION_2) { diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 446c574c0d..362422413c 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -214,7 +214,7 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) assert(vms->gic_version != VIRT_GIC_VERSION_2); - return (MACHINE(vms)->smp.cpus > redist0_capacity && + return (MACHINE(vms)->smp.max_cpus > redist0_capacity && vms->highmem_redists) ? 2 : 1; } From patchwork Tue Oct 15 09:59:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19E3BCFC27F for ; Tue, 15 Oct 2024 10:03:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eOp-0003Dn-7b; Tue, 15 Oct 2024 06:03:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eOn-00033I-DD; Tue, 15 Oct 2024 06:03:29 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eOk-00013D-FX; Tue, 15 Oct 2024 06:03:29 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV1C6n42z6DB43; Tue, 15 Oct 2024 17:58:55 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 2F0F3140B63; Tue, 15 Oct 2024 18:03:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:03:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 07/30] arm/virt, gicv3: Introduce GICv3 CPU Interface *accessibility* flag and checks Date: Tue, 15 Oct 2024 10:59:49 +0100 Message-ID: <20241015100012.254223-8-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Introduce a `gicc_accessible` flag to indicate whether it is safe to access the GICv3 CPU interface. This flag will determine the availability of the GICv3 CPU interface based on whether the associated QOM vCPUs are enabled or disabled. Additionally, implement checks throughout the GICv3 codebase to ensure that any updates or accesses to GICC registers (e.g., `ICC_*_EL1`) occur only when the `gicc_accessible` flag is set. This ensures that operations such as KVM VGIC GICC register fetches or modifications are performed only for GICv3 CPU interfaces that are valid and associated with active QOM vCPUs. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- cpu-common.c | 11 +++++++++ hw/arm/virt.c | 1 + hw/intc/arm_gicv3_common.c | 14 ++++++++--- hw/intc/arm_gicv3_cpuif.c | 8 +++++++ hw/intc/arm_gicv3_kvm.c | 23 ++++++++++++++++++- include/hw/core/cpu.h | 12 ++++++++++ include/hw/intc/arm_gicv3_common.h | 37 ++++++++++++++++++++++++++++++ 7 files changed, 102 insertions(+), 4 deletions(-) diff --git a/cpu-common.c b/cpu-common.c index 6b262233a3..b96eb4ca55 100644 --- a/cpu-common.c +++ b/cpu-common.c @@ -24,6 +24,7 @@ #include "sysemu/cpus.h" #include "qemu/lockable.h" #include "trace/trace-root.h" +#include "hw/boards.h" QemuMutex qemu_cpu_list_lock; static QemuCond exclusive_cond; @@ -108,6 +109,16 @@ void cpu_list_remove(CPUState *cpu) cpu_list_generation_id++; } +CPUState *qemu_get_possible_cpu(int index) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + const CPUArchIdList *possible_cpus = ms->possible_cpus; + + assert((index >= 0) && (index < possible_cpus->len)); + + return CPU(possible_cpus->cpus[index].cpu); +} + CPUState *qemu_get_cpu(int index) { CPUState *cpu; diff --git a/hw/arm/virt.c b/hw/arm/virt.c index b286afc62c..00fd65b4e1 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -795,6 +795,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) vms->gic = qdev_new(gictype); qdev_prop_set_uint32(vms->gic, "revision", revision); qdev_prop_set_uint32(vms->gic, "num-cpu", max_cpus); + qdev_prop_set_uint32(vms->gic, "num-smp-cpu", smp_cpus); /* Note that the num-irq property counts both internal and external * interrupts; there are always 32 of the former (mandated by GIC spec). */ diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index bd50a1b079..218ced1d9f 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -436,13 +436,20 @@ static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) s->cpu = g_new0(GICv3CPUState, s->num_cpu); for (i = 0; i < s->num_cpu; i++) { - CPUState *cpu = qemu_get_cpu(i); + CPUState *cpu = qemu_get_possible_cpu(i); + bool gicc_accessible = (i < s->num_smp_cpu); uint64_t cpu_affid; - s->cpu[i].cpu = cpu; + /* + * Accordingly, set the QOM `GICv3CPUState` as either accessible or + * inaccessible based on the `CPUState` of the associated QOM vCPU. + */ + gicv3_set_cpustate(&s->cpu[i], cpu, gicc_accessible); + s->cpu[i].gic = s; /* Store GICv3CPUState in CPUARMState gicv3state pointer */ - gicv3_set_gicv3state(cpu, &s->cpu[i]); + if (gicc_accessible) + gicv3_set_gicv3state(cpu, &s->cpu[i]); /* Pre-construct the GICR_TYPER: * For our implementation: @@ -607,6 +614,7 @@ static void arm_gic_common_linux_init(ARMLinuxBootIf *obj, static Property arm_gicv3_common_properties[] = { DEFINE_PROP_UINT32("num-cpu", GICv3State, num_cpu, 1), + DEFINE_PROP_UINT32("num-smp-cpu", GICv3State, num_smp_cpu, 1), DEFINE_PROP_UINT32("num-irq", GICv3State, num_irq, 32), DEFINE_PROP_UINT32("revision", GICv3State, revision, 3), DEFINE_PROP_BOOL("has-lpi", GICv3State, lpi_enable, 0), diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index bdb13b00e9..151f957d42 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -1052,6 +1052,10 @@ void gicv3_cpuif_update(GICv3CPUState *cs) ARMCPU *cpu = ARM_CPU(cs->cpu); CPUARMState *env = &cpu->env; + if (!gicv3_cpu_accessible(cs)) { + return; + } + g_assert(bql_locked()); trace_gicv3_cpuif_update(gicv3_redist_affid(cs), cs->hppi.irq, @@ -2036,6 +2040,10 @@ static void icc_generate_sgi(CPUARMState *env, GICv3CPUState *cs, for (i = 0; i < s->num_cpu; i++) { GICv3CPUState *ocs = &s->cpu[i]; + if (!gicv3_cpu_accessible(ocs)) { + continue; + } + if (irm) { /* IRM == 1 : route to all CPUs except self */ if (cs == ocs) { diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index 9ea6b8e218..7e741ace50 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -24,6 +24,7 @@ #include "hw/intc/arm_gicv3_common.h" #include "qemu/error-report.h" #include "qemu/module.h" +#include "sysemu/cpus.h" #include "sysemu/kvm.h" #include "sysemu/runstate.h" #include "kvm_arm.h" @@ -458,6 +459,16 @@ static void kvm_arm_gicv3_put(GICv3State *s) GICv3CPUState *c = &s->cpu[ncpu]; int num_pri_bits; + /* + * We must ensure that we do not attempt to access or update KVM GICC + * registers if their corresponding QOM `GICv3CPUState` is marked as + * 'inaccessible', either because their corresponding QOM vCPU objects + * do not exist or are disabled due to hot-unplug action. + */ + if (!gicv3_cpu_accessible(c)) { + continue; + } + kvm_gicc_access(s, ICC_SRE_EL1, ncpu, &c->icc_sre_el1, true); kvm_gicc_access(s, ICC_CTLR_EL1, ncpu, &c->icc_ctlr_el1[GICV3_NS], true); @@ -616,6 +627,14 @@ static void kvm_arm_gicv3_get(GICv3State *s) GICv3CPUState *c = &s->cpu[ncpu]; int num_pri_bits; + /* + * don't attempt to access KVM VGIC for the disabled vCPUs where + * GICv3CPUState is inaccessible. + */ + if (!gicv3_cpu_accessible(c)) { + continue; + } + kvm_gicc_access(s, ICC_SRE_EL1, ncpu, &c->icc_sre_el1, false); kvm_gicc_access(s, ICC_CTLR_EL1, ncpu, &c->icc_ctlr_el1[GICV3_NS], false); @@ -815,7 +834,9 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) for (i = 0; i < s->num_cpu; i++) { ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + if (gicv3_cpu_accessible(&s->cpu[i])) { + define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + } } /* Try to create the device via the device control API */ diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 4e0cb325a0..0be1984698 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -950,6 +950,18 @@ static inline bool cpu_in_exclusive_context(const CPUState *cpu) */ CPUState *qemu_get_cpu(int index); +/** + * qemu_get_possible_cpu: + * @index: The CPUState@cpu_index value of the CPU to obtain. + * Input index MUST be in range [0, Max Possible CPUs) + * + * If CPUState object exists,then it gets a CPU matching + * @index in the possible CPU array. + * + * Returns: The possible CPU or %NULL if CPU does not exist. + */ +CPUState *qemu_get_possible_cpu(int index); + /** * cpu_exists: * @id: Guest-exposed CPU ID to lookup. diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h index cd09bee3bc..4383a4ce08 100644 --- a/include/hw/intc/arm_gicv3_common.h +++ b/include/hw/intc/arm_gicv3_common.h @@ -190,6 +190,7 @@ struct GICv3CPUState { uint64_t icc_apr[3][4]; uint64_t icc_igrpen[3]; uint64_t icc_ctlr_el3; + bool gicc_accessible; /* Virtualization control interface */ uint64_t ich_apr[3][4]; /* ich_apr[GICV3_G1][x] never used */ @@ -255,6 +256,7 @@ struct GICv3State { uint32_t nb_redist_regions; /* number of redist regions */ uint32_t num_cpu; + uint32_t num_smp_cpu; uint32_t num_irq; uint32_t revision; bool lpi_enable; @@ -353,4 +355,39 @@ void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, */ const char *gicv3_class_name(void); +/** + * gicv3_cpu_accessible + * + * The `GICv3CPUState` can become inaccessible if the associated `CPUState` is + * either unavailable or in a disabled state. This state is independent of the + * KVM VGIC and is not compliant with ARM CPU architecture (i.e. there is no + * way we can explicitly enable/disable ARM GIC CPU interface). This change + * is specific to QOM only. + * + * Returns: True if accessible otherwise False + */ +static inline bool gicv3_cpu_accessible(GICv3CPUState *gicc) +{ + assert(gicc); + return gicc->gicc_accessible; +} + +/** + * gicv3_set_cpustate + * + * Sets `GICv3CPUState` and the associated `CPUState` as accessible and + * available for use + */ +static inline void gicv3_set_cpustate(GICv3CPUState *s, + CPUState *cpu, + bool gicc_accessible) +{ + if (gicc_accessible) { + s->cpu = cpu; + s->gicc_accessible = true; + } else { + s->cpu = NULL; + s->gicc_accessible = false; + } +} #endif From patchwork Tue Oct 15 09:59:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 012BDCFC282 for ; Tue, 15 Oct 2024 10:03:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0ePA-00058Y-H3; Tue, 15 Oct 2024 06:03:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eP6-0004wT-Ud; Tue, 15 Oct 2024 06:03:48 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eP4-00019S-8y; Tue, 15 Oct 2024 06:03:48 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV4v0m9Cz6FH29; Tue, 15 Oct 2024 18:02:07 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 4F2791404F9; Tue, 15 Oct 2024 18:03:44 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:03:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 08/30] hw/intc/arm-gicv3*: Changes required to (re)init the GICv3 vCPU Interface Date: Tue, 15 Oct 2024 10:59:50 +0100 Message-ID: <20241015100012.254223-9-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The GICv3 CPU interface must be (re)initialized when a vCPU is either cold- or hot-plugged. System registers need to be defined and registered with the associated vCPU. For cold-plugged vCPUs, this occurs during the GICv3 realization phase, while for hot-plugged vCPUs, it happens during the GICv3 update notification. The latter will be addressed in subsequent patches. This process must be implemented for both emulation/TCG and KVM cases. This change adds the necessary support and refactors the existing code to maximize reuse for both cold and hotplug vCPU initialization. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/intc/arm_gicv3.c | 1 + hw/intc/arm_gicv3_cpuif.c | 245 ++++++++++++++--------------- hw/intc/arm_gicv3_cpuif_common.c | 13 ++ hw/intc/arm_gicv3_kvm.c | 14 +- hw/intc/gicv3_internal.h | 1 + include/hw/intc/arm_gicv3_common.h | 1 + 6 files changed, 143 insertions(+), 132 deletions(-) diff --git a/hw/intc/arm_gicv3.c b/hw/intc/arm_gicv3.c index 58e18fff54..2a30625916 100644 --- a/hw/intc/arm_gicv3.c +++ b/hw/intc/arm_gicv3.c @@ -459,6 +459,7 @@ static void arm_gicv3_class_init(ObjectClass *klass, void *data) ARMGICv3Class *agc = ARM_GICV3_CLASS(klass); agcc->post_load = arm_gicv3_post_load; + agcc->init_cpu_reginfo = gicv3_init_cpu_reginfo; device_class_set_parent_realize(dc, arm_gic_realize, &agc->parent_realize); } diff --git a/hw/intc/arm_gicv3_cpuif.c b/hw/intc/arm_gicv3_cpuif.c index 151f957d42..453d1296ea 100644 --- a/hw/intc/arm_gicv3_cpuif.c +++ b/hw/intc/arm_gicv3_cpuif.c @@ -3033,143 +3033,138 @@ static void gicv3_cpuif_el_change_hook(ARMCPU *cpu, void *opaque) gicv3_cpuif_virt_irq_fiq_update(cs); } -void gicv3_init_cpuif(GICv3State *s) +void gicv3_init_cpu_reginfo(CPUState *cs) { - /* Called from the GICv3 realize function; register our system - * registers with the CPU - */ - int i; + ARMCPU *cpu = ARM_CPU(cs); + GICv3CPUState *gcs = icc_cs_from_env(&cpu->env); - for (i = 0; i < s->num_cpu; i++) { - ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - GICv3CPUState *cs = &s->cpu[i]; + /* + * If the CPU doesn't define a GICv3 configuration, probably because + * in real hardware it doesn't have one, then we use default values + * matching the one used by most Arm CPUs. This applies to: + * cpu->gic_num_lrs + * cpu->gic_vpribits + * cpu->gic_vprebits + * cpu->gic_pribits + */ - /* - * If the CPU doesn't define a GICv3 configuration, probably because - * in real hardware it doesn't have one, then we use default values - * matching the one used by most Arm CPUs. This applies to: - * cpu->gic_num_lrs - * cpu->gic_vpribits - * cpu->gic_vprebits - * cpu->gic_pribits - */ + /* + * Note that we can't just use the GICv3CPUState as an opaque pointer + * in define_arm_cp_regs_with_opaque(), because when we're called back + * it might be with code translated by CPU 0 but run by CPU 1, in + * which case we'd get the wrong value. + * So instead we define the regs with no ri->opaque info, and + * get back to the GICv3CPUState from the CPUARMState. + * + * These CP regs callbacks can be called from either TCG or HVF code. + */ + define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); - /* Note that we can't just use the GICv3CPUState as an opaque pointer - * in define_arm_cp_regs_with_opaque(), because when we're called back - * it might be with code translated by CPU 0 but run by CPU 1, in - * which case we'd get the wrong value. - * So instead we define the regs with no ri->opaque info, and - * get back to the GICv3CPUState from the CPUARMState. - * - * These CP regs callbacks can be called from either TCG or HVF code. - */ - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); + /* + * If the CPU implements FEAT_NMI and FEAT_GICv3 it must also + * implement FEAT_GICv3_NMI, which is the CPU interface part + * of NMI support. This is distinct from whether the GIC proper + * (redistributors and distributor) have NMI support. In QEMU + * that is a property of the GIC device in s->nmi_support; + * cs->nmi_support indicates the CPU interface's support. + */ + if (cpu_isar_feature(aa64_nmi, cpu)) { + gcs->nmi_support = true; + define_arm_cp_regs(cpu, gicv3_cpuif_gicv3_nmi_reginfo); + } - /* - * If the CPU implements FEAT_NMI and FEAT_GICv3 it must also - * implement FEAT_GICv3_NMI, which is the CPU interface part - * of NMI support. This is distinct from whether the GIC proper - * (redistributors and distributor) have NMI support. In QEMU - * that is a property of the GIC device in s->nmi_support; - * cs->nmi_support indicates the CPU interface's support. - */ - if (cpu_isar_feature(aa64_nmi, cpu)) { - cs->nmi_support = true; - define_arm_cp_regs(cpu, gicv3_cpuif_gicv3_nmi_reginfo); - } + /* + * The CPU implementation specifies the number of supported + * bits of physical priority. For backwards compatibility + * of migration, we have a compat property that forces use + * of 8 priority bits regardless of what the CPU really has. + */ + if (gcs->gic->force_8bit_prio) { + gcs->pribits = 8; + } else { + gcs->pribits = cpu->gic_pribits ?: 5; + } - /* - * The CPU implementation specifies the number of supported - * bits of physical priority. For backwards compatibility - * of migration, we have a compat property that forces use - * of 8 priority bits regardless of what the CPU really has. - */ - if (s->force_8bit_prio) { - cs->pribits = 8; - } else { - cs->pribits = cpu->gic_pribits ?: 5; - } + /* + * The GICv3 has separate ID register fields for virtual priority + * and preemption bit values, but only a single ID register field + * for the physical priority bits. The preemption bit count is + * always the same as the priority bit count, except that 8 bits + * of priority means 7 preemption bits. We precalculate the + * preemption bits because it simplifies the code and makes the + * parallels between the virtual and physical bits of the GIC + * a bit clearer. + */ + gcs->prebits = gcs->pribits; + if (gcs->prebits == 8) { + gcs->prebits--; + } + /* + * Check that CPU code defining pribits didn't violate + * architectural constraints our implementation relies on. + */ + g_assert(gcs->pribits >= 4 && gcs->pribits <= 8); - /* - * The GICv3 has separate ID register fields for virtual priority - * and preemption bit values, but only a single ID register field - * for the physical priority bits. The preemption bit count is - * always the same as the priority bit count, except that 8 bits - * of priority means 7 preemption bits. We precalculate the - * preemption bits because it simplifies the code and makes the - * parallels between the virtual and physical bits of the GIC - * a bit clearer. - */ - cs->prebits = cs->pribits; - if (cs->prebits == 8) { - cs->prebits--; - } - /* - * Check that CPU code defining pribits didn't violate - * architectural constraints our implementation relies on. - */ - g_assert(cs->pribits >= 4 && cs->pribits <= 8); + /* + * gicv3_cpuif_reginfo[] defines ICC_AP*R0_EL1; add definitions + * for ICC_AP*R{1,2,3}_EL1 if the prebits value requires them. + */ + if (gcs->prebits >= 6) { + define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr1_reginfo); + } + if (gcs->prebits == 7) { + define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr23_reginfo); + } - /* - * gicv3_cpuif_reginfo[] defines ICC_AP*R0_EL1; add definitions - * for ICC_AP*R{1,2,3}_EL1 if the prebits value requires them. - */ - if (cs->prebits >= 6) { - define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr1_reginfo); - } - if (cs->prebits == 7) { - define_arm_cp_regs(cpu, gicv3_cpuif_icc_apxr23_reginfo); - } + if (arm_feature(&cpu->env, ARM_FEATURE_EL2)) { + int j; - if (arm_feature(&cpu->env, ARM_FEATURE_EL2)) { - int j; + gcs->num_list_regs = cpu->gic_num_lrs ?: 4; + gcs->vpribits = cpu->gic_vpribits ?: 5; + gcs->vprebits = cpu->gic_vprebits ?: 5; - cs->num_list_regs = cpu->gic_num_lrs ?: 4; - cs->vpribits = cpu->gic_vpribits ?: 5; - cs->vprebits = cpu->gic_vprebits ?: 5; - /* Check against architectural constraints: getting these - * wrong would be a bug in the CPU code defining these, - * and the implementation relies on them holding. - */ - g_assert(cs->vprebits <= cs->vpribits); - g_assert(cs->vprebits >= 5 && cs->vprebits <= 7); - g_assert(cs->vpribits >= 5 && cs->vpribits <= 8); + /* Check against architectural constraints: getting these + * wrong would be a bug in the CPU code defining these, + * and the implementation relies on them holding. + */ + g_assert(gcs->vprebits <= gcs->vpribits); + g_assert(gcs->vprebits >= 5 && gcs->vprebits <= 7); + g_assert(gcs->vpribits >= 5 && gcs->vpribits <= 8); - define_arm_cp_regs(cpu, gicv3_cpuif_hcr_reginfo); + define_arm_cp_regs(cpu, gicv3_cpuif_hcr_reginfo); - for (j = 0; j < cs->num_list_regs; j++) { - /* Note that the AArch64 LRs are 64-bit; the AArch32 LRs - * are split into two cp15 regs, LR (the low part, with the - * same encoding as the AArch64 LR) and LRC (the high part). - */ - ARMCPRegInfo lr_regset[] = { - { .name = "ICH_LRn_EL2", .state = ARM_CP_STATE_BOTH, - .opc0 = 3, .opc1 = 4, .crn = 12, - .crm = 12 + (j >> 3), .opc2 = j & 7, - .type = ARM_CP_IO | ARM_CP_NO_RAW, - .nv2_redirect_offset = 0x400 + 8 * j, - .access = PL2_RW, - .readfn = ich_lr_read, - .writefn = ich_lr_write, - }, - { .name = "ICH_LRCn_EL2", .state = ARM_CP_STATE_AA32, - .cp = 15, .opc1 = 4, .crn = 12, - .crm = 14 + (j >> 3), .opc2 = j & 7, - .type = ARM_CP_IO | ARM_CP_NO_RAW, - .access = PL2_RW, - .readfn = ich_lr_read, - .writefn = ich_lr_write, - }, - }; - define_arm_cp_regs(cpu, lr_regset); - } - if (cs->vprebits >= 6) { - define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr1_reginfo); - } - if (cs->vprebits == 7) { - define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo); - } + for (j = 0; j < gcs->num_list_regs; j++) { + /* Note that the AArch64 LRs are 64-bit; the AArch32 LRs + * are split into two cp15 regs, LR (the low part, with the + * same encoding as the AArch64 LR) and LRC (the high part). + */ + ARMCPRegInfo lr_regset[] = { + { .name = "ICH_LRn_EL2", .state = ARM_CP_STATE_BOTH, + .opc0 = 3, .opc1 = 4, .crn = 12, + .crm = 12 + (j >> 3), .opc2 = j & 7, + .type = ARM_CP_IO | ARM_CP_NO_RAW, + .nv2_redirect_offset = 0x400 + 8 * j, + .access = PL2_RW, + .readfn = ich_lr_read, + .writefn = ich_lr_write, + }, + { .name = "ICH_LRCn_EL2", .state = ARM_CP_STATE_AA32, + .cp = 15, .opc1 = 4, .crn = 12, + .crm = 14 + (j >> 3), .opc2 = j & 7, + .type = ARM_CP_IO | ARM_CP_NO_RAW, + .access = PL2_RW, + .readfn = ich_lr_read, + .writefn = ich_lr_write, + }, + }; + define_arm_cp_regs(cpu, lr_regset); + } + if (gcs->vprebits >= 6) { + define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr1_reginfo); + } + if (gcs->vprebits == 7) { + define_arm_cp_regs(cpu, gicv3_cpuif_ich_apxr23_reginfo); } if (tcg_enabled() || qtest_enabled()) { /* @@ -3177,7 +3172,7 @@ void gicv3_init_cpuif(GICv3State *s) * state only changes on EL changes involving EL2 or EL3, so for * the non-TCG case this is OK, as EL2 and EL3 can't exist. */ - arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, cs); + arm_register_el_change_hook(cpu, gicv3_cpuif_el_change_hook, gcs); } else { assert(!arm_feature(&cpu->env, ARM_FEATURE_EL2)); assert(!arm_feature(&cpu->env, ARM_FEATURE_EL3)); diff --git a/hw/intc/arm_gicv3_cpuif_common.c b/hw/intc/arm_gicv3_cpuif_common.c index ff1239f65d..ef9eef3e01 100644 --- a/hw/intc/arm_gicv3_cpuif_common.c +++ b/hw/intc/arm_gicv3_cpuif_common.c @@ -20,3 +20,16 @@ void gicv3_set_gicv3state(CPUState *cpu, GICv3CPUState *s) env->gicv3state = (void *)s; }; + +void gicv3_init_cpuif(GICv3State *s) +{ + ARMGICv3CommonClass *agcc = ARM_GICV3_COMMON_GET_CLASS(s); + int i; + + /* define and register `system registers` with the vCPU */ + for (i = 0; i < s->num_cpu; i++) { + if (gicv3_cpu_accessible(&s->cpu[i])) { + agcc->init_cpu_reginfo(s->cpu[i].cpu); + } + } +} diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index 7e741ace50..3e1e97d830 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -796,6 +796,10 @@ static void vm_change_state_handler(void *opaque, bool running, } } +static void kvm_gicv3_init_cpu_reginfo(CPUState *cs) +{ + define_arm_cp_regs(ARM_CPU(cs), gicv3_cpuif_reginfo); +} static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) { @@ -831,13 +835,8 @@ static void kvm_arm_gicv3_realize(DeviceState *dev, Error **errp) gicv3_init_irqs_and_mmio(s, kvm_arm_gicv3_set_irq, NULL); - for (i = 0; i < s->num_cpu; i++) { - ARMCPU *cpu = ARM_CPU(qemu_get_cpu(i)); - - if (gicv3_cpu_accessible(&s->cpu[i])) { - define_arm_cp_regs(cpu, gicv3_cpuif_reginfo); - } - } + /* initialize vCPU interface */ + gicv3_init_cpuif(s); /* Try to create the device via the device control API */ s->dev_fd = kvm_create_device(kvm_state, KVM_DEV_TYPE_ARM_VGIC_V3, false); @@ -923,6 +922,7 @@ static void kvm_arm_gicv3_class_init(ObjectClass *klass, void *data) agcc->pre_save = kvm_arm_gicv3_get; agcc->post_load = kvm_arm_gicv3_put; + agcc->init_cpu_reginfo = kvm_gicv3_init_cpu_reginfo; device_class_set_parent_realize(dc, kvm_arm_gicv3_realize, &kgc->parent_realize); resettable_class_set_parent_phases(rc, NULL, kvm_arm_gicv3_reset_hold, NULL, diff --git a/hw/intc/gicv3_internal.h b/hw/intc/gicv3_internal.h index bc9f518fe8..cc8edc499b 100644 --- a/hw/intc/gicv3_internal.h +++ b/hw/intc/gicv3_internal.h @@ -722,6 +722,7 @@ void gicv3_redist_vinvall(GICv3CPUState *cs, uint64_t vptaddr); void gicv3_redist_send_sgi(GICv3CPUState *cs, int grp, int irq, bool ns); void gicv3_init_cpuif(GICv3State *s); +void gicv3_init_cpu_reginfo(CPUState *cs); /** * gicv3_cpuif_update: diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h index 4383a4ce08..3f585162f3 100644 --- a/include/hw/intc/arm_gicv3_common.h +++ b/include/hw/intc/arm_gicv3_common.h @@ -340,6 +340,7 @@ struct ARMGICv3CommonClass { void (*pre_save)(GICv3State *s); void (*post_load)(GICv3State *s); + void (*init_cpu_reginfo)(CPUState *cs); }; void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, From patchwork Tue Oct 15 09:59:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836028 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2CC6CFC283 for ; Tue, 15 Oct 2024 10:04:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0ePR-0005w2-Tl; Tue, 15 Oct 2024 06:04:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0ePQ-0005tB-2D; Tue, 15 Oct 2024 06:04:08 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0ePO-0001Do-ER; Tue, 15 Oct 2024 06:04:07 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV6V1JRsz6K98V; Tue, 15 Oct 2024 18:03:30 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 509E4140A34; Tue, 15 Oct 2024 18:04:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:03:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 09/30] arm/acpi: Enable ACPI support for vCPU hotplug Date: Tue, 15 Oct 2024 10:59:51 +0100 Message-ID: <20241015100012.254223-10-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org ACPI is required to interface QEMU with the guest. Roughly falls into below cases, 1. Convey the possible vCPUs config at the machine init time to the guest using various DSDT tables like MADT etc. 2. Convey vCPU hotplug events to guest(using GED) 3. Assist in evaluation of various ACPI methods(like _EVT, _STA, _OST, _EJ0, _MAT etc.) 4. Provides ACPI CPU hotplug state and 12 Byte memory mapped vCPU hotplug control register interface to the OSPM/guest corresponding to each possible vcpu. The register interface consists of various R/W fields and their handling operations. These are called when ever register fields or memory regions are accessed (i.e. read or written) by OSPM when ever it evaluates various ACPI methods. Note: lot of this framework code is inherited from the changes already done for x86 but still some minor changes are required to make it compatible with ARM64.) Enable the ACPI support switch for vCPU hotplug feature. Actual ACPI changes required will follow in subsequent patches. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/hw/arm/Kconfig b/hw/arm/Kconfig index eac5070514..77cdbc5c27 100644 --- a/hw/arm/Kconfig +++ b/hw/arm/Kconfig @@ -33,6 +33,7 @@ config ARM_VIRT select ACPI_HW_REDUCED select ACPI_APEI select ACPI_VIOT + select ACPI_CPU_HOTPLUG select VIRTIO_MEM_SUPPORTED select ACPI_CXL select ACPI_HMAT From patchwork Tue Oct 15 09:59:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1DD8CFC280 for ; Tue, 15 Oct 2024 10:04:51 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0ePr-0006dc-Fb; Tue, 15 Oct 2024 06:04:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0ePn-0006Mu-CZ; Tue, 15 Oct 2024 06:04:33 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0ePl-0001Gq-IS; Tue, 15 Oct 2024 06:04:30 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV5g0mr5z6FGpc; Tue, 15 Oct 2024 18:02:47 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 4FE03140AA7; Tue, 15 Oct 2024 18:04:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:04:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 10/30] arm/virt: Enhance GED framework to handle vCPU hotplug events Date: Tue, 15 Oct 2024 10:59:52 +0100 Message-ID: <20241015100012.254223-11-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During GED device creation at Virt Machine initialization, add a new vCPU hotplug event to the existing set of supported GED device events. Additionally, initialize the memory map for the vCPU hotplug *control device*, which will provide an interface to exchange ACPI events between QEMU/VMM and the Guest Kernel. Signed-off-by: Salil Mehta Reviewed-by: Gavin Shan --- hw/arm/virt.c | 5 ++++- include/hw/arm/virt.h | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 00fd65b4e1..b9df428049 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -81,6 +81,7 @@ #include "hw/mem/pc-dimm.h" #include "hw/mem/nvdimm.h" #include "hw/acpi/generic_event_device.h" +#include "hw/acpi/cpu_hotplug.h" #include "hw/virtio/virtio-md-pci.h" #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" @@ -181,6 +182,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_NVDIMM_ACPI] = { 0x09090000, NVDIMM_ACPI_IO_LEN}, [VIRT_PVTIME] = { 0x090a0000, 0x00010000 }, [VIRT_SECURE_GPIO] = { 0x090b0000, 0x00001000 }, + [VIRT_CPUHP_ACPI] = { 0x090c0000, ACPI_CPU_HOTPLUG_REG_LEN}, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 }, @@ -678,7 +680,7 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) DeviceState *dev; MachineState *ms = MACHINE(vms); int irq = vms->irqmap[VIRT_ACPI_GED]; - uint32_t event = ACPI_GED_PWR_DOWN_EVT; + uint32_t event = ACPI_GED_PWR_DOWN_EVT | ACPI_GED_CPU_HOTPLUG_EVT; if (ms->ram_slots) { event |= ACPI_GED_MEM_HOTPLUG_EVT; @@ -694,6 +696,7 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, vms->memmap[VIRT_ACPI_GED].base); sysbus_mmio_map(SYS_BUS_DEVICE(dev), 1, vms->memmap[VIRT_PCDIMM_ACPI].base); + sysbus_mmio_map(SYS_BUS_DEVICE(dev), 3, vms->memmap[VIRT_CPUHP_ACPI].base); sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0, qdev_get_gpio_in(vms->gic, irq)); return dev; diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 362422413c..b5bfb75f71 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -78,6 +78,7 @@ enum { VIRT_PCDIMM_ACPI, VIRT_ACPI_GED, VIRT_NVDIMM_ACPI, + VIRT_CPUHP_ACPI, VIRT_PVTIME, VIRT_LOWMEMMAP_LAST, }; From patchwork Tue Oct 15 09:59:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D5E62CFC283 for ; Tue, 15 Oct 2024 10:05:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eQA-00077S-Ro; Tue, 15 Oct 2024 06:04:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eQ7-0006zN-J4; Tue, 15 Oct 2024 06:04:52 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eQ4-0001Is-IU; Tue, 15 Oct 2024 06:04:51 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV630ltTz6FH25; Tue, 15 Oct 2024 18:03:07 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 4FCC61400D9; Tue, 15 Oct 2024 18:04:44 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:04:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 11/30] arm/virt: Init PMU at host for all possible vCPUs Date: Tue, 15 Oct 2024 10:59:53 +0100 Message-ID: <20241015100012.254223-12-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The PMU for all possible vCPUs must be initialized during VM initialization. Refactor the existing code to accommodate possible vCPUs. This assumes that all processors being used are identical. It is an architectural constraint of ARM CPUs that all vCPUs MUST have identical feature sets, at least until the ARM specification is updated to allow otherwise. Past discussion for reference: Link: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00131.html Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 9 +++++---- include/hw/arm/virt.h | 1 + include/hw/core/cpu.h | 5 +++++ 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index b9df428049..6ac2d8826e 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2035,12 +2035,13 @@ static void finalize_gic_version(VirtMachineState *vms) */ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) { + CPUArchIdList *possible_cpus = vms->parent.possible_cpus; int max_cpus = MACHINE(vms)->smp.max_cpus; - bool aarch64, pmu, steal_time; + bool aarch64, steal_time; CPUState *cpu; aarch64 = object_property_get_bool(OBJECT(first_cpu), "aarch64", NULL); - pmu = object_property_get_bool(OBJECT(first_cpu), "pmu", NULL); + vms->pmu = object_property_get_bool(OBJECT(first_cpu), "pmu", NULL); steal_time = object_property_get_bool(OBJECT(first_cpu), "kvm-steal-time", NULL); @@ -2067,8 +2068,8 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) memory_region_add_subregion(sysmem, pvtime_reg_base, pvtime); } - CPU_FOREACH(cpu) { - if (pmu) { + CPU_FOREACH_POSSIBLE(cpu, possible_cpus) { + if (vms->pmu) { assert(arm_feature(&ARM_CPU(cpu)->env, ARM_FEATURE_PMU)); if (kvm_irqchip_in_kernel()) { kvm_arm_pmu_set_irq(ARM_CPU(cpu), VIRTUAL_PMU_IRQ); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index b5bfb75f71..98ce68eae1 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -161,6 +161,7 @@ struct VirtMachineState { bool mte; bool dtb_randomness; bool second_ns_uart_present; + bool pmu; OnOffAuto acpi; VirtGICType gic_version; VirtIOMMUType iommu; diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 0be1984698..4a74c383ab 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -618,6 +618,11 @@ extern CPUTailQ cpus_queue; #define CPU_FOREACH_SAFE(cpu, next_cpu) \ QTAILQ_FOREACH_SAFE_RCU(cpu, &cpus_queue, node, next_cpu) +#define CPU_FOREACH_POSSIBLE(cpu, poslist) \ + for (int iter = 0; \ + iter < (poslist)->len && ((cpu) = (poslist)->cpus[iter].cpu, 1); \ + iter++) + extern __thread CPUState *current_cpu; /** From patchwork Tue Oct 15 09:59:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5719DCFC280 for ; Tue, 15 Oct 2024 10:05:33 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eQX-0007y1-OV; Tue, 15 Oct 2024 06:05:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eQP-0007nD-Al; Tue, 15 Oct 2024 06:05:09 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eQN-0001Uo-Im; Tue, 15 Oct 2024 06:05:09 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV380b50z6DB4H; Tue, 15 Oct 2024 18:00:36 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 66461140519; Tue, 15 Oct 2024 18:05:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:04:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 12/30] arm/virt: Release objects for *disabled* possible vCPUs after init Date: Tue, 15 Oct 2024 10:59:54 +0100 Message-ID: <20241015100012.254223-13-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During `machvirt_init()`, QOM ARMCPU objects are pre-created along with the corresponding KVM vCPUs in the host for all possible vCPUs. This is necessary due to the architectural constraint that KVM restricts the deferred creation of KVM vCPUs and VGIC initialization/sizing after VM initialization. Hence, VGIC is pre-sized with possible vCPUs. After the initialization of the machine is complete, the disabled possible KVM vCPUs are parked in the per-virt-machine list "kvm_parked_vcpus," and we release the QOM ARMCPU objects for the disabled vCPUs. These will be re-created when the vCPU is hotplugged again. The QOM ARMCPU object is then re-attached to the corresponding parked KVM vCPU. Alternatively, we could have chosen not to release the QOM CPU objects and kept reusing them. This approach might require some modifications to the `qdevice_add()` interface to retrieve the old ARMCPU object instead of creating a new one for the hotplug request. Each of these approaches has its own pros and cons. This prototype uses the first approach (suggestions are welcome!). Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 6ac2d8826e..7cbc212130 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2037,6 +2037,7 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) { CPUArchIdList *possible_cpus = vms->parent.possible_cpus; int max_cpus = MACHINE(vms)->smp.max_cpus; + int smp_cpus = MACHINE(vms)->smp.cpus; bool aarch64, steal_time; CPUState *cpu; @@ -2095,6 +2096,30 @@ static void virt_cpu_post_init(VirtMachineState *vms, MemoryRegion *sysmem) } } } + + if (kvm_enabled() || tcg_enabled()) { + int i = 0; + CPU_FOREACH_POSSIBLE(cpu, possible_cpus) { + /* + * Release the disabled ARMCPU objects that were previously used + * during initialization for pre-sizing host KVM. + * + * We simulate the presence of these non-existent vCPUs to the guest + * via ACPI by setting `_STA.PRES = 1` (indicating they are present) + * and mark them as disabled vCPUs by setting `_STA.ENA = 0`, + * ensuring they cannot be used. These vCPUs can be added back to + * the guest later through hotplug operations when ARMCPU objects + * are recreated using the '-device_add' QMP command. + */ + if (i >= smp_cpus) { + CPUArchId *cpu_slot; + cpu_slot = virt_find_cpu_slot(cpu); + cpu_slot->cpu = NULL; + object_unref(OBJECT(cpu)); + } + i++; + } + } } static void virt_cpu_set_properties(Object *cpuobj, const CPUArchId *cpu_slot, From patchwork Tue Oct 15 09:59:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1BCCDCFC283 for ; Tue, 15 Oct 2024 10:05:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eQl-0008TU-OT; Tue, 15 Oct 2024 06:05:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eQi-0008E0-EP; Tue, 15 Oct 2024 06:05:28 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eQg-0001Wa-DY; Tue, 15 Oct 2024 06:05:28 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV6q1D5xz6FGpc; Tue, 15 Oct 2024 18:03:47 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 5FA4A140A34; Tue, 15 Oct 2024 18:05:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:05:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 13/30] arm/virt/acpi: Update ACPI DSDT Tbl to include CPUs AML with hotplug support Date: Tue, 15 Oct 2024 10:59:55 +0100 Message-ID: <20241015100012.254223-14-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_FILL_THIS_FORM_SHORT=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Support for Virtual CPU Hotplug requires a sequence of ACPI handshakes between QEMU and the guest kernel when a vCPU is plugged or unplugged. Most of the AML code to support these handshakes already exists. This AML needs to be built during VM initialization for the ARM architecture as well, if GED support exists. TODO (Peculiar Problem with AML): Encountering the following ACPI namespace error when the GED AML is placed after the CPU AML. Error excerpt: [ 75.795774] ACPI BIOS Error (bug): Could not resolve symbol [\_SB.GED.CSCN], [ 75.797237] ACPI Error: Aborting method \_SB.GED._EVT due to previous error [ 75.798530] acpi-ged ACPI0013:00: IRQ method execution failed Preliminary analysis: The error is definitely not related to a `forward declaration` issue but rather to associating the `CSCN` (CPU Scan event handler) method with the `GED` scope before the `\\_SB.GED` scope is created. Therefore, it appears that the GED AML should be initialized prior to the CPU AML. As a result, I had to move the GED AML before the CPU AML to resolve the issue. Everything about the two AML sections seems identical, except for their location. Any insights? ============================== Summary comparison of DSDT.dsl ============================== [1] Both working and not working DSDT.dsl Files are placed at below path: https://drive.google.com/drive/folders/1bbEvS18CtBn3vYFnGIVdgcSD_hggyODV?usp=drive_link [2] Configuration: -smp cpu 4, maxcpus 6 DSDT.dsl (Not Working) DSDT.dsl (Working) --------------------- ------------------ DefinitionBlock ("", "DSDT", 2, "BOCHS ", "BXPC ", 0x00000001) DefinitionBlock ("", "DSDT", 2, "BOCHS ", "BXPC ", 0x00000001) { { Scope (\_SB) Scope (\_SB) { { Scope (_SB) Device (\_SB.GED) { { Device (\_SB.PRES) Name (_HID, "ACPI0013" { Name (_UID, "GED") [...] Name (_CRS, ResourceTemplate () Device (\_SB.CPUS) [...] { Method (_EVT, 1, Serialized) Name (_HID, "ACPI0010") { Name (_CID, EisaId ("PNP0A05")) Local0 = ESEL /* \_SB_.GED_.ESEL */ Method (CTFY, 2, NotSerialized) If (((Local0 & 0x02) == 0x02)) { { [...] Notify (PWRB, 0x80) Method (CSTA, 1, Serialized) } { [...] If (((Local0 & 0x08) == 0x08)) Method (CEJ0, 1, Serialized) { { \_SB.GED.CSCN () [...] } Method (CSCN, 0, Serialized) } { } [...] Method (COST, 4, Serialized) Scope (_SB) { { [...] Device (\_SB.PRES) Device (C000) { { [...] [...] Device (\_SB.CPUS) Device (C001) { { Name (_HID, "ACPI0010") [...] Name (_CID, EisaId ("PNP0A05")) Device (C002) Method (CTFY, 2, NotSerialized) { { [...] [...] Device (C003) Method (CSTA, 1, Serialized) { { [...] [...] Device (C004) Method (CEJ0, 1, Serialized) { { [...] [...] Device (C005) Method (CSCN, 0, Serialized) { { } [...] } Method (COST, 4, Serialized) { Method (\_SB.GED.CSCN, 0, NotSerialized) [...] { Device (C000) \_SB.CPUS.CSCN () { } [...] Device (C001) Device (COM0) { { [...] [...] Device (C002) { Device (\_SB.GED) [...] { Device (C003) Name (_HID, "ACPI0013") { Name (_UID, "GED") [...] Name (_CRS, ResourceTemplate () Device (C004) { { [...] [...] OperationRegion (EREG, SystemMemory, 0x09080000, 0x04) Device (C005) Field (EREG, DWordAcc, NoLock, WriteAsZeros) { { } [...] } Method (_EVT, 1, Serialized) Method (\_SB.GED.CSCN, 0, NotSerialized) { { Local0 = ESEL \_SB.CPUS.CSCN () If (((Local0 & 0x02) == 0x02)) } { Notify (PWRB, 0x80) Device (COM0) } { [...] If (((Local0 & 0x08) == 0x08)) } { } \_SB.GED.CSCN () } } } Device (PWRB) { [...] } } Signed-off-by: Salil Mehta --- hw/arm/virt-acpi-build.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index f76fb117ad..32238170ab 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -805,6 +805,7 @@ static void build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) { VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms); + MachineClass *mc = MACHINE_GET_CLASS(vms); Aml *scope, *dsdt; MachineState *ms = MACHINE(vms); const MemMapEntry *memmap = vms->memmap; @@ -821,7 +822,28 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) * the RTC ACPI device at all when using UEFI. */ scope = aml_scope("\\_SB"); - acpi_dsdt_add_cpus(scope, vms); + if (vms->acpi_dev) { + build_ged_aml(scope, "\\_SB."GED_DEVICE, + HOTPLUG_HANDLER(vms->acpi_dev), + irqmap[VIRT_ACPI_GED] + ARM_SPI_BASE, AML_SYSTEM_MEMORY, + memmap[VIRT_ACPI_GED].base); + } else { + acpi_dsdt_add_gpio(scope, &memmap[VIRT_GPIO], + (irqmap[VIRT_GPIO] + ARM_SPI_BASE)); + } + + /* if GED is enabled then cpus AML shall be added as part build_cpus_aml */ + if (vms->acpi_dev && mc->has_hotpluggable_cpus) { + CPUHotplugFeatures opts = { + .acpi_1_compatible = false, + .has_legacy_cphp = false + }; + + build_cpus_aml(scope, ms, opts, NULL, memmap[VIRT_CPUHP_ACPI].base, + "\\_SB", AML_GED_EVT_CPU_SCAN_METHOD, AML_SYSTEM_MEMORY); + } else { + acpi_dsdt_add_cpus(scope, vms); + } acpi_dsdt_add_uart(scope, &memmap[VIRT_UART0], (irqmap[VIRT_UART0] + ARM_SPI_BASE), 0); if (vms->second_ns_uart_present) { @@ -836,15 +858,6 @@ build_dsdt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) (irqmap[VIRT_MMIO] + ARM_SPI_BASE), 0, NUM_VIRTIO_TRANSPORTS); acpi_dsdt_add_pci(scope, memmap, irqmap[VIRT_PCIE] + ARM_SPI_BASE, vms); - if (vms->acpi_dev) { - build_ged_aml(scope, "\\_SB."GED_DEVICE, - HOTPLUG_HANDLER(vms->acpi_dev), - irqmap[VIRT_ACPI_GED] + ARM_SPI_BASE, AML_SYSTEM_MEMORY, - memmap[VIRT_ACPI_GED].base); - } else { - acpi_dsdt_add_gpio(scope, &memmap[VIRT_GPIO], - (irqmap[VIRT_GPIO] + ARM_SPI_BASE)); - } if (vms->acpi_dev) { uint32_t event = object_property_get_uint(OBJECT(vms->acpi_dev), From patchwork Tue Oct 15 09:59:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6F43ECFC280 for ; Tue, 15 Oct 2024 10:06:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eR3-0000et-SM; Tue, 15 Oct 2024 06:05:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eR1-0000ct-Q8; Tue, 15 Oct 2024 06:05:47 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eR0-0001Yy-9F; Tue, 15 Oct 2024 06:05:47 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV8Q1jZPz6K98P; Tue, 15 Oct 2024 18:05:10 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 5F435140AA7; Tue, 15 Oct 2024 18:05:44 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:05:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 14/30] hw/acpi: Make _MAT method optional Date: Tue, 15 Oct 2024 10:59:56 +0100 Message-ID: <20241015100012.254223-15-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jean-Philippe Brucker The GICC interface on arm64 vCPUs is statically defined in the MADT, and doesn't require a _MAT entry. Although the GICC is indicated as present by the MADT entry, it can only be used from vCPU sysregs, which aren't accessible until hot-add. Signed-off-by: Jean-Philippe Brucker Reviewed-by: Gavin Shan Signed-off-by: Salil Mehta --- hw/acpi/cpu.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c index d34c1e601e..2cbeedffe8 100644 --- a/hw/acpi/cpu.c +++ b/hw/acpi/cpu.c @@ -731,10 +731,13 @@ void build_cpus_aml(Aml *table, MachineState *machine, CPUHotplugFeatures opts, aml_append(dev, method); /* build _MAT object */ - build_madt_cpu(i, arch_ids, madt_buf, true); /* set enabled flag */ - aml_append(dev, aml_name_decl("_MAT", + if (build_madt_cpu) { + build_madt_cpu(i, arch_ids, madt_buf, + true); /* set enabled flag */ + aml_append(dev, aml_name_decl("_MAT", aml_buffer(madt_buf->len, (uint8_t *)madt_buf->data))); - g_array_free(madt_buf, true); + g_array_free(madt_buf, true); + } if (CPU(arch_ids->cpus[i].cpu) != first_cpu) { method = aml_method("_EJ0", 1, AML_NOTSERIALIZED); From patchwork Tue Oct 15 09:59:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 738C4CFC280 for ; Tue, 15 Oct 2024 10:06:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eRR-0001TI-HB; Tue, 15 Oct 2024 06:06:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eRP-0001AP-1i; Tue, 15 Oct 2024 06:06:11 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eRL-0001bv-1Z; Tue, 15 Oct 2024 06:06:09 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV7b2GYpz6FH29; Tue, 15 Oct 2024 18:04:27 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 83382140451; Tue, 15 Oct 2024 18:06:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:05:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 15/30] hw/arm/acpi: MADT Tbl change to size the guest with possible vCPUs Date: Tue, 15 Oct 2024 10:59:57 +0100 Message-ID: <20241015100012.254223-16-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org When QEMU builds the MADT table, modifications are needed to include information about possible vCPUs that are exposed as ACPI-disabled (i.e., `_STA.Enabled=0`). This new information will help the guest kernel pre-size its resources during boot time. Pre-sizing based on possible vCPUs will facilitate the future hot-plugging of the currently disabled vCPUs. Additionally, this change addresses updates to the ACPI MADT GIC CPU interface flags, as introduced in the UEFI ACPI 6.5 specification. These updates enable deferred virtual CPU onlining in the guest kernel. Link: https://uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html#gic-cpu-interface-gicc-structure Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt-acpi-build.c | 36 ++++++++++++++++++++++++++++++------ include/hw/acpi/cpu.h | 19 +++++++++++++++++++ 2 files changed, 49 insertions(+), 6 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 32238170ab..ff104b3496 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -663,6 +663,29 @@ static void build_append_gicr(GArray *table_data, uint64_t base, uint32_t size) build_append_int_noprefix(table_data, size, 4); /* Discovery Range Length */ } +static uint32_t virt_acpi_get_gicc_flags(CPUState *cpu) +{ + MachineClass *mc = MACHINE_GET_CLASS(qdev_get_machine()); + + /* can only exist in 'enabled' state */ + if (!mc->has_hotpluggable_cpus) { + return 1; + } + + /* + * ARM GIC CPU Interface can be 'online-capable' or 'enabled' at boot + * We MUST set 'online-capable' bit for all hotpluggable CPUs except the + * first/boot CPU. Cold-booted CPUs without 'Id' can also be unplugged. + * Though as-of-now this is only used as a debugging feature. + * + * UEFI ACPI Specification 6.5 + * Section: 5.2.12.14. GIC CPU Interface (GICC) Structure + * Table: 5.37 GICC CPU Interface Flags + * Link: https://uefi.org/specs/ACPI/6.5 + */ + return cpu && !cpu->cpu_index ? 1 : (1 << 3); +} + static void build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) { @@ -689,12 +712,13 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, vms->gic_version, 1); build_append_int_noprefix(table_data, 0, 3); /* Reserved */ - for (i = 0; i < MACHINE(vms)->smp.cpus; i++) { - ARMCPU *armcpu = ARM_CPU(qemu_get_cpu(i)); + for (i = 0; i < MACHINE(vms)->smp.max_cpus; i++) { + CPUState *cpu = qemu_get_possible_cpu(i); uint64_t physical_base_address = 0, gich = 0, gicv = 0; uint32_t vgic_interrupt = vms->virt ? ARCH_GIC_MAINT_IRQ : 0; - uint32_t pmu_interrupt = arm_feature(&armcpu->env, ARM_FEATURE_PMU) ? - VIRTUAL_PMU_IRQ : 0; + uint32_t pmu_interrupt = vms->pmu ? VIRTUAL_PMU_IRQ : 0; + uint32_t flags = virt_acpi_get_gicc_flags(cpu); + uint64_t mpidr = acpi_get_cpu_archid(i); if (vms->gic_version == VIRT_GIC_VERSION_2) { physical_base_address = memmap[VIRT_GIC_CPU].base; @@ -709,7 +733,7 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, i, 4); /* GIC ID */ build_append_int_noprefix(table_data, i, 4); /* ACPI Processor UID */ /* Flags */ - build_append_int_noprefix(table_data, 1, 4); /* Enabled */ + build_append_int_noprefix(table_data, flags, 4); /* Parking Protocol Version */ build_append_int_noprefix(table_data, 0, 4); /* Performance Interrupt GSIV */ @@ -723,7 +747,7 @@ build_madt(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, vgic_interrupt, 4); build_append_int_noprefix(table_data, 0, 8); /* GICR Base Address*/ /* MPIDR */ - build_append_int_noprefix(table_data, arm_cpu_mp_affinity(armcpu), 8); + build_append_int_noprefix(table_data, mpidr, 8); /* Processor Power Efficiency Class */ build_append_int_noprefix(table_data, 0, 1); /* Reserved */ diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h index bd3f9973c9..c688bb2141 100644 --- a/include/hw/acpi/cpu.h +++ b/include/hw/acpi/cpu.h @@ -96,4 +96,23 @@ static inline bool acpi_persistent_cpu(CPUState *cpu) */ return cpu && cpu->acpi_persistent; } + +/** + * acpi_get_cpu_archid: + * @cpu_index: possible vCPU for which arch-id needs to be retreived + * + * Fetches the vCPU arch-id of the possible vCPU. This should be same + * same as the one configured at KVM Host. + * + * Returns: arch-id of the possible vCPU + */ +static inline uint64_t acpi_get_cpu_archid(int cpu_index) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + const CPUArchIdList *possible_cpus = ms->possible_cpus; + + assert((cpu_index >= 0) && (cpu_index < possible_cpus->len)); + + return possible_cpus->cpus[cpu_index].arch_id; +} #endif From patchwork Tue Oct 15 09:59:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C043CFC283 for ; Tue, 15 Oct 2024 10:06:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eRj-0002YB-79; Tue, 15 Oct 2024 06:06:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eRg-0002HS-Gd; Tue, 15 Oct 2024 06:06:28 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eRe-0001fj-MD; Tue, 15 Oct 2024 06:06:28 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV9B3Nnqz6K977; Tue, 15 Oct 2024 18:05:50 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 994881404F5; Tue, 15 Oct 2024 18:06:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:06:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 16/30] target/arm: Force ARM vCPU *present* status ACPI *persistent* Date: Tue, 15 Oct 2024 10:59:58 +0100 Message-ID: <20241015100012.254223-17-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The ARM CPU architecture does not permit changes to CPU presence after the kernel has booted. This is an immutable requirement from ARM and represents a strict architectural constraint [1][2]. The ACPI update [3] reinforces this by specifying that the `_STA.Present` bit in the ACPI specification cannot be modified once the system has booted. Consequently, the firmware, ACPI, and QEMU must provide the guest kernel with a persistent view of the vCPUs, even when they are not present in the QOM (i.e., when they are unplugged or have yet to be plugged into the QOM after the kernel has booted). References: [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don’t Support CPU Hotplug (like ARM64) a. Kernel Link: https://kvm-forum.qemu.org/2023/KVM-forum-cpu-hotplug_7OJ1YyJ.pdf b. Qemu Link: https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64) Link: https://kvmforum2020.sched.com/event/eE4m [3] Check comment 5 in the bugzilla entry Link: https://bugzilla.tianocore.org/show_bug.cgi?id=4481#c5 Signed-off-by: Salil Mehta --- target/arm/cpu64.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index d2f4624d61..c2af6a28f5 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -797,6 +797,13 @@ static void aarch64_cpu_initfn(Object *obj) /* TODO: re-check if this is necessary still */ cs->thread_id = 0; + /* + * To provide the guest with a persistent view of vCPU presence, ACPI may + * need to simulate the presence of vCPUs even when they are not present in + * the QOM or are in a disabled state. This flag is utilized during the + * initialization of ACPI hotplug state and during vCPU hot-unplug events. + */ + cs->acpi_persistent = true; } static void aarch64_cpu_finalizefn(Object *obj) From patchwork Tue Oct 15 09:59:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6EF3CCFC283 for ; Tue, 15 Oct 2024 10:06:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eS4-00031f-9l; Tue, 15 Oct 2024 06:06:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eS1-0002wY-LN; Tue, 15 Oct 2024 06:06:50 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eRz-0001iP-0M; Tue, 15 Oct 2024 06:06:49 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV8M2TzBz6FGg5; Tue, 15 Oct 2024 18:05:07 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 8C3411400D9; Tue, 15 Oct 2024 18:06:44 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:06:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 17/30] arm/virt: Add/update basic hot-(un)plug framework Date: Tue, 15 Oct 2024 10:59:59 +0100 Message-ID: <20241015100012.254223-18-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add CPU hot-unplug hooks and update hotplug hooks with additional sanity checks for use in hotplug paths. Note: The functional contents of the hooks (currently left with TODO comments) will be gradually filled in subsequent patches in an incremental approach to patch and logic building, which would roughly include the following: 1. (Un)wiring of interrupts between vCPU<->GIC. 2. Sending events to the guest for hot-(un)plug so that the guest can take appropriate actions. 3. Notifying the GIC about the hot-(un)plug action so that the vCPU can be (un)stitched to the GIC CPU interface. 4. Updating the guest with next boot information for this vCPU in the firmware. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 7cbc212130..d4bcaedb8f 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -86,6 +86,7 @@ #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" #include "qemu/guest-random.h" +#include "qapi/qmp/qdict.h" static GlobalProperty arm_virt_compat[] = { { TYPE_VIRTIO_IOMMU_PCI, "aw-bits", "48" }, @@ -3014,11 +3015,23 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); MachineState *ms = MACHINE(hotplug_dev); + MachineClass *mc = MACHINE_GET_CLASS(ms); ARMCPU *cpu = ARM_CPU(dev); CPUState *cs = CPU(dev); CPUArchId *cpu_slot; + if (dev->hotplugged && !vms->acpi_dev) { + error_setg(errp, "GED acpi device does not exists"); + return; + } + + if (dev->hotplugged && !mc->has_hotpluggable_cpus) { + error_setg(errp, "CPU hotplug not supported on this machine"); + return; + } + /* sanity check the cpu */ if (!object_dynamic_cast(OBJECT(cpu), ms->cpu_type)) { error_setg(errp, "Invalid CPU type, expected cpu type: '%s'", @@ -3060,11 +3073,30 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, return; } virt_cpu_set_properties(OBJECT(cs), cpu_slot, errp); + + /* + * Fix the GIC for the newly plugged vCPU. The QOM CPU object for this new + * vCPU needs to be updated in the corresponding QOM `GICv3CPUState` object. + * Additionally, the IRQs for this new CPU object must be re-wired. This + * update is confined to the QOM layer and does not affect KVM, as KVM was + * already pre-sized with possible CPUs during VM initialization. This + * serves as a workaround to the constraints posed by the ARM architecture + * in supporting CPU hotplug, for which no formal specification exists. + * + * This GIC IRQ patch-up is necessary for both cold- and hot-plugged vCPUs. + * Cold-initialized vCPUs have their GIC state initialized earlier during + * `machvirt_init()`. + */ + if (vms->acpi_dev) { + /* TODO: update GIC about this hotplug change here */ + /* TODO: wire the GIC<->CPU irqs */ + } } static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); CPUState *cs = CPU(dev); CPUArchId *cpu_slot; @@ -3081,6 +3113,74 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, */ cpu_slot->arch_id = arm_cpu_mp_affinity(ARM_CPU(cs)); } + + if (vms->acpi_dev) { + /* TODO: register cpu for reset & update F/W info for the next boot */ + } + + /* + * Update the ACPI hotplug state for vCPUs being both hot- and cold-plugged. + * vCPUs can be cold-plugged using the `-device` option. For vCPUs that are + * hot-plugged, the guest is also notified. + */ + if (vms->acpi_dev) { + /* TODO: update acpi hotplug state. Send cpu hotplug event to guest */ + } +} + +static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) +{ + MachineClass *mc = MACHINE_GET_CLASS(qdev_get_machine()); + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUState *cs = CPU(dev); + + if (!vms->acpi_dev) { + error_setg(errp, "GED does not exists or device is not realized!"); + return; + } + + if (!mc->has_hotpluggable_cpus) { + error_setg(errp, "CPU hot(un)plug not supported on this machine"); + return; + } + + if (cs->cpu_index == first_cpu->cpu_index) { + error_setg(errp, "Boot CPU(id%d=%d:%d:%d:%d) hot-unplug not supported", + first_cpu->cpu_index, cpu->socket_id, cpu->cluster_id, + cpu->core_id, cpu->thread_id); + return; + } + + /* TODO: request cpu hotplug from guest */ +} + +static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + CPUState *cs = CPU(dev); + CPUArchId *cpu_slot; + + if (!vms->acpi_dev) { + error_setg(errp, "GED does not exists or device is not realized!"); + return; + } + + cpu_slot = virt_find_cpu_slot(cs); + + /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ + + /* TODO: unwire the gic-cpu irqs here */ + /* TODO: update the GIC about this hot unplug change */ + + /* TODO: unregister cpu for reset & update F/W info for the next boot */ + + qobject_unref(dev->opts); + dev->opts = NULL; + + cpu_slot->cpu = NULL; } static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, @@ -3207,6 +3307,8 @@ static void virt_machine_device_unplug_request_cb(HotplugHandler *hotplug_dev, } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_unplug_request(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_unplug_request(hotplug_dev, dev, errp); } else { error_setg(errp, "device unplug request for unsupported device" " type: %s", object_get_typename(OBJECT(dev))); @@ -3220,6 +3322,8 @@ static void virt_machine_device_unplug_cb(HotplugHandler *hotplug_dev, virt_dimm_unplug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MD_PCI)) { virtio_md_pci_unplug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { + virt_cpu_unplug(hotplug_dev, dev, errp); } else { error_setg(errp, "virt: device unplug for unsupported device" " type: %s", object_get_typename(OBJECT(dev))); From patchwork Tue Oct 15 10:00:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EAE7ACFC280 for ; Tue, 15 Oct 2024 10:07:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eSb-0003m5-UV; Tue, 15 Oct 2024 06:07:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eSL-0003c7-S9; Tue, 15 Oct 2024 06:07:11 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eSJ-0001nN-NY; Tue, 15 Oct 2024 06:07:09 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV9y30m5z6K5sy; Tue, 15 Oct 2024 18:06:30 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 8B527140451; Tue, 15 Oct 2024 18:07:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:06:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 18/30] arm/virt: Changes to (un)wire GICC<->vCPU IRQs during hot-(un)plug Date: Tue, 15 Oct 2024 11:00:00 +0100 Message-ID: <20241015100012.254223-19-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Refactors the existing GIC create code to extract common code to wire the vcpu<->gic interrupts. This function could be used with cold-plug case and also used when vCPU is hot-plugged. It also introduces a new function to unwire the vcpu<->gic interrupts for the vCPU hot-unplug cases. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [4/05/2024: Issue with total number of PPI available during create GIC] Suggested-by: Miguel Luis [5/05/2024: Fix the total number of PPIs available as per ARM BSA to avoid overflow] Signed-off-by: Salil Mehta --- hw/arm/virt.c | 154 ++++++++++++++++++++++++++++------------- hw/core/gpio.c | 2 +- include/hw/qdev-core.h | 2 + target/arm/cpu-qom.h | 18 +++-- 4 files changed, 118 insertions(+), 58 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index d4bcaedb8f..0a912919ec 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -765,6 +765,107 @@ static bool gicv3_nmi_present(VirtMachineState *vms) (vms->gic_version != VIRT_GIC_VERSION_2); } +/* + * Mapping from the output timer irq lines from the CPU to the GIC PPI inputs + * we use for the virt board. + */ +const int timer_irq[] = { + [GTIMER_PHYS] = ARCH_TIMER_NS_EL1_IRQ, + [GTIMER_VIRT] = ARCH_TIMER_VIRT_IRQ, + [GTIMER_HYP] = ARCH_TIMER_NS_EL2_IRQ, + [GTIMER_SEC] = ARCH_TIMER_S_EL1_IRQ, +}; + +static void unwire_gic_cpu_irqs(VirtMachineState *vms, CPUState *cs) +{ + MachineState *ms = MACHINE(vms); + unsigned int max_cpus = ms->smp.max_cpus; + DeviceState *cpudev = DEVICE(cs); + DeviceState *gicdev = vms->gic; + int cpu = CPU(cs)->cpu_index; + int type = vms->gic_version; + int irq, num_gpio_in; + + for (irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) { + qdev_disconnect_gpio_out_named(cpudev, NULL, irq); + } + + if (type != VIRT_GIC_VERSION_2) { + qdev_disconnect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", + 0); + } else if (vms->virt) { + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, + cpu + 4 * max_cpus); + } + + /* + * RFC: Question: This currently does not takes care of intimating the + * devices which might be sitting on system bus. Do we need a + * sysbus_disconnect_irq() which also does the job of notification beside + * disconnection? + */ + qdev_disconnect_gpio_out_named(cpudev, "pmu-interrupt", 0); + + /* Unwire GIC's IRQ/FIQ/VIRQ/VFIQ/NMI/VINMI interrupt outputs to CPU */ + num_gpio_in = (vms->gic_version != VIRT_GIC_VERSION_2) ? + NUM_GPIO_IN : NUM_GICV2_GPIO_IN; + for (irq = 0; irq < num_gpio_in; irq++) { + qdev_disconnect_gpio_out_named(gicdev, SYSBUS_DEVICE_GPIO_IRQ, + cpu + irq * max_cpus); + } +} + +static void wire_gic_cpu_irqs(VirtMachineState *vms, CPUState *cs) +{ + MachineState *ms = MACHINE(vms); + unsigned int max_cpus = ms->smp.max_cpus; + DeviceState *cpudev = DEVICE(cs); + DeviceState *gicdev = vms->gic; + int cpu = CPU(cs)->cpu_index; + int type = vms->gic_version; + SysBusDevice *gicbusdev; + int intidbase; + int irqn; + + intidbase = NUM_IRQS + cpu * GIC_INTERNAL; + + for (irqn = 0; irqn < ARRAY_SIZE(timer_irq); irqn++) { + qdev_connect_gpio_out(cpudev, irqn, + qdev_get_gpio_in(gicdev, + intidbase + timer_irq[irqn])); + } + + gicbusdev = SYS_BUS_DEVICE(gicdev); + if (type != VIRT_GIC_VERSION_2) { + qemu_irq irq = qdev_get_gpio_in(gicdev, + intidbase + ARCH_GIC_MAINT_IRQ); + qdev_connect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", + 0, irq); + } else if (vms->virt) { + qemu_irq irq = qdev_get_gpio_in(gicdev, + intidbase + ARCH_GIC_MAINT_IRQ); + sysbus_connect_irq(gicbusdev, cpu + 4 * max_cpus, irq); + } + + qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, + qdev_get_gpio_in(gicdev, + intidbase + VIRTUAL_PMU_IRQ)); + + sysbus_connect_irq(gicbusdev, cpu, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); + sysbus_connect_irq(gicbusdev, cpu + max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); + sysbus_connect_irq(gicbusdev, cpu + 2 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); + sysbus_connect_irq(gicbusdev, cpu + 3 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); + if (vms->gic_version != VIRT_GIC_VERSION_2) { + sysbus_connect_irq(gicbusdev, cpu + 4 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_NMI)); + sysbus_connect_irq(gicbusdev, cpu + 5 * max_cpus, + qdev_get_gpio_in(cpudev, ARM_CPU_VINMI)); + } +} + static void create_gic(VirtMachineState *vms, MemoryRegion *mem) { MachineState *ms = MACHINE(vms); @@ -868,54 +969,7 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) * CPU's inputs. */ for (i = 0; i < smp_cpus; i++) { - DeviceState *cpudev = DEVICE(qemu_get_cpu(i)); - int intidbase = NUM_IRQS + i * GIC_INTERNAL; - /* Mapping from the output timer irq lines from the CPU to the - * GIC PPI inputs we use for the virt board. - */ - const int timer_irq[] = { - [GTIMER_PHYS] = ARCH_TIMER_NS_EL1_IRQ, - [GTIMER_VIRT] = ARCH_TIMER_VIRT_IRQ, - [GTIMER_HYP] = ARCH_TIMER_NS_EL2_IRQ, - [GTIMER_SEC] = ARCH_TIMER_S_EL1_IRQ, - [GTIMER_HYPVIRT] = ARCH_TIMER_NS_EL2_VIRT_IRQ, - }; - - for (unsigned irq = 0; irq < ARRAY_SIZE(timer_irq); irq++) { - qdev_connect_gpio_out(cpudev, irq, - qdev_get_gpio_in(vms->gic, - intidbase + timer_irq[irq])); - } - - if (vms->gic_version != VIRT_GIC_VERSION_2) { - qemu_irq irq = qdev_get_gpio_in(vms->gic, - intidbase + ARCH_GIC_MAINT_IRQ); - qdev_connect_gpio_out_named(cpudev, "gicv3-maintenance-interrupt", - 0, irq); - } else if (vms->virt) { - qemu_irq irq = qdev_get_gpio_in(vms->gic, - intidbase + ARCH_GIC_MAINT_IRQ); - sysbus_connect_irq(gicbusdev, i + 4 * max_cpus, irq); - } - - qdev_connect_gpio_out_named(cpudev, "pmu-interrupt", 0, - qdev_get_gpio_in(vms->gic, intidbase - + VIRTUAL_PMU_IRQ)); - - sysbus_connect_irq(gicbusdev, i, qdev_get_gpio_in(cpudev, ARM_CPU_IRQ)); - sysbus_connect_irq(gicbusdev, i + max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_FIQ)); - sysbus_connect_irq(gicbusdev, i + 2 * max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VIRQ)); - sysbus_connect_irq(gicbusdev, i + 3 * max_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VFIQ)); - - if (vms->gic_version != VIRT_GIC_VERSION_2) { - sysbus_connect_irq(gicbusdev, i + 4 * smp_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_NMI)); - sysbus_connect_irq(gicbusdev, i + 5 * smp_cpus, - qdev_get_gpio_in(cpudev, ARM_CPU_VINMI)); - } + wire_gic_cpu_irqs(vms, qemu_get_cpu(i)); } fdt_add_gic_node(vms); @@ -3089,7 +3143,7 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, */ if (vms->acpi_dev) { /* TODO: update GIC about this hotplug change here */ - /* TODO: wire the GIC<->CPU irqs */ + wire_gic_cpu_irqs(vms, cs); } } @@ -3172,7 +3226,7 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ - /* TODO: unwire the gic-cpu irqs here */ + unwire_gic_cpu_irqs(vms, cs); /* TODO: update the GIC about this hot unplug change */ /* TODO: unregister cpu for reset & update F/W info for the next boot */ diff --git a/hw/core/gpio.c b/hw/core/gpio.c index 80d07a6ec9..abb164d5c0 100644 --- a/hw/core/gpio.c +++ b/hw/core/gpio.c @@ -143,7 +143,7 @@ qemu_irq qdev_get_gpio_out_connector(DeviceState *dev, const char *name, int n) /* disconnect a GPIO output, returning the disconnected input (if any) */ -static qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, +qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, const char *name, int n) { char *propname = g_strdup_printf("%s[%d]", diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h index aa97c34a4b..71f169718c 100644 --- a/include/hw/qdev-core.h +++ b/include/hw/qdev-core.h @@ -739,6 +739,8 @@ qemu_irq qdev_get_gpio_out_connector(DeviceState *dev, const char *name, int n); */ qemu_irq qdev_intercept_gpio_out(DeviceState *dev, qemu_irq icpt, const char *name, int n); +qemu_irq qdev_disconnect_gpio_out_named(DeviceState *dev, + const char *name, int n); BusState *qdev_get_child_bus(DeviceState *dev, const char *name); diff --git a/target/arm/cpu-qom.h b/target/arm/cpu-qom.h index b497667d61..e49fb096de 100644 --- a/target/arm/cpu-qom.h +++ b/target/arm/cpu-qom.h @@ -37,13 +37,17 @@ DECLARE_CLASS_CHECKERS(AArch64CPUClass, AARCH64_CPU, #define ARM_CPU_TYPE_NAME(name) (name ARM_CPU_TYPE_SUFFIX) /* Meanings of the ARMCPU object's seven inbound GPIO lines */ -#define ARM_CPU_IRQ 0 -#define ARM_CPU_FIQ 1 -#define ARM_CPU_VIRQ 2 -#define ARM_CPU_VFIQ 3 -#define ARM_CPU_NMI 4 -#define ARM_CPU_VINMI 5 -#define ARM_CPU_VFNMI 6 +enum { + ARM_CPU_IRQ = 0, + ARM_CPU_FIQ = 1, + ARM_CPU_VIRQ = 2, + ARM_CPU_VFIQ = 3, + NUM_GICV2_GPIO_IN = (ARM_CPU_VFIQ+1), + ARM_CPU_NMI = 4, + ARM_CPU_VINMI = 5, + /* ARM_CPU_VFNMI = 6, */ /* not used? */ + NUM_GPIO_IN = (ARM_CPU_VINMI+1), +}; /* For M profile, some registers are banked secure vs non-secure; * these are represented as a 2-element array where the first element From patchwork Tue Oct 15 10:00:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 66D02CFC283 for ; Tue, 15 Oct 2024 10:08:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eTO-0004LF-PO; Tue, 15 Oct 2024 06:08:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eT0-0004AV-Aw; Tue, 15 Oct 2024 06:07:52 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eSv-0002CP-Ez; Tue, 15 Oct 2024 06:07:47 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV5r1Mr9z6DB5T; Tue, 15 Oct 2024 18:02:56 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 853B21404F5; Tue, 15 Oct 2024 18:07:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:07:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 19/30] hw/arm, gicv3: Changes to notify GICv3 CPU state with vCPU hot-(un)plug event Date: Tue, 15 Oct 2024 11:00:01 +0100 Message-ID: <20241015100012.254223-20-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Virtual CPU hot-(un)plug events must be communicated to the GIC. Introduce a notification mechanism to ensure these events are properly relayed to the GIC, allowing it to update the accessibility of the GIC CPU interface and adjust the vCPU-to-GIC CPU interface association accordingly. This approach deviates from the standard ARM CPU architecture specification, where the CPU-to-GIC interface is typically fixed and the accessibility of the GIC CPU interface cannot be disabled. However, this workaround is necessary to address limitations imposed by the ARM CPU architecture [1][2]. For more details regarding these constraints and the workarounds, please refer to the slides below: References: [1] KVMForum 2023 Presentation: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don’t Support CPU Hotplug (like ARM64) Link: https://kvm-forum.qemu.org/2023/Challenges_Revisited_in_Supporting_Virt_CPU_Hotplug_-__ii0iNb3.pdf (Slides 13,17,18) [2] KVMForum 2020 Presentation: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64) Link: https://kvmforum2020.sched.com/event/eE4m Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 31 ++++++++++++++- hw/intc/arm_gicv3_common.c | 60 +++++++++++++++++++++++++++++- include/hw/arm/virt.h | 1 + include/hw/intc/arm_gicv3_common.h | 23 ++++++++++++ 4 files changed, 112 insertions(+), 3 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 0a912919ec..295012bce1 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -703,6 +703,16 @@ static inline DeviceState *create_acpi_ged(VirtMachineState *vms) return dev; } +static void virt_add_gic_cpuhp_notifier(VirtMachineState *vms) +{ + MachineClass *mc = MACHINE_GET_CLASS(vms); + + if (mc->has_hotpluggable_cpus) { + Notifier *cpuhp_notifier = gicv3_cpuhp_notifier(vms->gic); + notifier_list_add(&vms->cpuhp_notifiers, cpuhp_notifier); + } +} + static void create_its(VirtMachineState *vms) { const char *itsclass = its_class_name(); @@ -979,6 +989,9 @@ static void create_gic(VirtMachineState *vms, MemoryRegion *mem) } else if (vms->gic_version == VIRT_GIC_VERSION_2) { create_v2m(vms); } + + /* add GIC CPU hot(un)plug update notifier */ + virt_add_gic_cpuhp_notifier(vms); } static void create_uart(const VirtMachineState *vms, int uart, @@ -2450,6 +2463,8 @@ static void machvirt_init(MachineState *machine) create_fdt(vms); + notifier_list_init(&vms->cpuhp_notifiers); + assert(possible_cpus->len == max_cpus); for (n = 0; n < possible_cpus->len; n++) { CPUArchId *cpu_slot; @@ -3066,6 +3081,18 @@ static void virt_memory_plug(HotplugHandler *hotplug_dev, dev, &error_abort); } +static void virt_update_gic(VirtMachineState *vms, CPUState *cs, bool plugging) +{ + GICv3CPUHotplugInfo gic_info = { + .gic = vms->gic, + .cpu = cs, + .cpu_plugging = plugging + }; + + /* notify gic to stitch GICC to this new cpu */ + notifier_list_notify(&vms->cpuhp_notifiers, &gic_info); +} + static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -3142,7 +3169,7 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, * `machvirt_init()`. */ if (vms->acpi_dev) { - /* TODO: update GIC about this hotplug change here */ + virt_update_gic(vms, cs, true); wire_gic_cpu_irqs(vms, cs); } } @@ -3227,7 +3254,7 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ unwire_gic_cpu_irqs(vms, cs); - /* TODO: update the GIC about this hot unplug change */ + virt_update_gic(vms, cs, false); /* TODO: unregister cpu for reset & update F/W info for the next boot */ diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index 218ced1d9f..c21fff4903 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -33,7 +33,6 @@ #include "hw/arm/linux-boot-if.h" #include "sysemu/kvm.h" - static void gicv3_gicd_no_migration_shift_bug_post_load(GICv3State *cs) { if (cs->gicd_no_migration_shift_bug) { @@ -366,6 +365,62 @@ void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, } } +static int arm_gicv3_get_proc_num(GICv3State *s, CPUState *cpu) +{ + uint64_t mp_affinity; + uint64_t gicr_typer; + uint64_t cpu_affid; + int i; + + mp_affinity = object_property_get_uint(OBJECT(cpu), "mp-affinity", NULL); + /* match the cpu mp-affinity to get the gic cpuif number */ + for (i = 0; i < s->num_cpu; i++) { + gicr_typer = s->cpu[i].gicr_typer; + cpu_affid = (gicr_typer >> 32) & 0xFFFFFF; + if (cpu_affid == mp_affinity) { + return i; + } + } + + return -1; +} + +static void arm_gicv3_cpu_update_notifier(Notifier *notifier, void * data) +{ + GICv3CPUHotplugInfo *gic_info = (GICv3CPUHotplugInfo *)data; + CPUState *cpu = gic_info->cpu; + ARMGICv3CommonClass *agcc; + int gic_cpuif_num; + GICv3State *s; + + s = ARM_GICV3_COMMON(gic_info->gic); + agcc = ARM_GICV3_COMMON_GET_CLASS(s); + + /* this shall get us mapped GICv3 CPU interface corresponding to MPIDR */ + gic_cpuif_num = arm_gicv3_get_proc_num(s, cpu); + if (gic_cpuif_num < 0) { + error_report("Failed to associate cpu %d with any GIC cpuif", + cpu->cpu_index); + abort(); + } + + /* Update the GICv3 CPU interface accessibiltiy accordingly */ + gicv3_set_cpustate(&s->cpu[gic_cpuif_num], cpu, gic_info->cpu_plugging); + + if (!gic_info->cpu_plugging) { + return; + } + + /* re-stitch the GICv3 CPU interface to this new vCPU */ + gicv3_set_gicv3state(cpu, &s->cpu[gic_cpuif_num]); + + /* + * define and register the GICv3 CPU interface `system registers` for + * this new vCPU being hotplugged + */ + agcc->init_cpu_reginfo(cpu); +} + static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) { GICv3State *s = ARM_GICV3_COMMON(dev); @@ -492,6 +547,8 @@ static void arm_gicv3_common_realize(DeviceState *dev, Error **errp) s->cpu[cpuidx - 1].gicr_typer |= GICR_TYPER_LAST; } + s->cpu_update_notifier.notify = arm_gicv3_cpu_update_notifier; + s->itslist = g_ptr_array_new(); } @@ -499,6 +556,7 @@ static void arm_gicv3_finalize(Object *obj) { GICv3State *s = ARM_GICV3_COMMON(obj); + notifier_remove(&s->cpu_update_notifier); g_free(s->redist_region_count); } diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 98ce68eae1..0202f0252c 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -186,6 +186,7 @@ struct VirtMachineState { char *oem_id; char *oem_table_id; bool ns_el2_virt_timer_irq; + NotifierList cpuhp_notifiers; }; #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM) diff --git a/include/hw/intc/arm_gicv3_common.h b/include/hw/intc/arm_gicv3_common.h index 3f585162f3..c02a810da9 100644 --- a/include/hw/intc/arm_gicv3_common.h +++ b/include/hw/intc/arm_gicv3_common.h @@ -295,6 +295,7 @@ struct GICv3State { GICv3CPUState *gicd_irouter_target[GICV3_MAXIRQ]; uint32_t gicd_nsacr[DIV_ROUND_UP(GICV3_MAXIRQ, 16)]; + Notifier cpu_update_notifier; GICv3CPUState *cpu; /* List of all ITSes connected to this GIC */ GPtrArray *itslist; @@ -345,6 +346,28 @@ struct ARMGICv3CommonClass { void gicv3_init_irqs_and_mmio(GICv3State *s, qemu_irq_handler handler, const MemoryRegionOps *ops); +/** + * Structure used by GICv3 CPU hotplug notifier + */ +typedef struct GICv3CPUHotplugInfo { + DeviceState *gic; /* GICv3State */ + CPUState *cpu; + bool cpu_plugging; /* CPU being plugged or unplugged */ +} GICv3CPUHotplugInfo; + +/** + * gicv3_cpuhp_notifier + * + * Returns CPU hotplug notifier which could be used to update GIC about any + * CPU hot(un)plug events. + * + * Returns: Notifier initialized with CPU Hot(un)plug update function + */ +static inline Notifier *gicv3_cpuhp_notifier(DeviceState *dev) +{ + GICv3State *s = ARM_GICV3_COMMON(dev); + return &s->cpu_update_notifier; +} /** * gicv3_class_name From patchwork Tue Oct 15 10:00:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 70946CFC284 for ; Tue, 15 Oct 2024 10:08:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eTY-0004cz-Pi; Tue, 15 Oct 2024 06:08:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTV-0004U2-Ib; Tue, 15 Oct 2024 06:08:21 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTP-0002E3-7I; Tue, 15 Oct 2024 06:08:21 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV9W2214z6FHFT; Tue, 15 Oct 2024 18:06:07 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 7A224140AA7; Tue, 15 Oct 2024 18:07:44 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:07:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 20/30] hw/arm: Changes required for reset and to support next boot Date: Tue, 15 Oct 2024 11:00:02 +0100 Message-ID: <20241015100012.254223-21-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Updates the firmware config with the next boot cpus information and also registers the reset callback to be called when guest reboots to reset the cpu. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/boot.c | 2 +- hw/arm/virt.c | 18 +++++++++++++++--- include/hw/arm/boot.h | 2 ++ include/hw/arm/virt.h | 1 + 4 files changed, 19 insertions(+), 4 deletions(-) diff --git a/hw/arm/boot.c b/hw/arm/boot.c index 5301d8d318..8bf8d003eb 100644 --- a/hw/arm/boot.c +++ b/hw/arm/boot.c @@ -682,7 +682,7 @@ fail: return -1; } -static void do_cpu_reset(void *opaque) +void do_cpu_reset(void *opaque) { ARMCPU *cpu = opaque; CPUState *cs = CPU(cpu); diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 295012bce1..b1eba8de8c 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -45,6 +45,8 @@ #include "sysemu/device_tree.h" #include "sysemu/numa.h" #include "sysemu/runstate.h" +#include "sysemu/reset.h" +#include "sysemu/sysemu.h" #include "sysemu/tpm.h" #include "sysemu/tcg.h" #include "sysemu/kvm.h" @@ -1429,7 +1431,7 @@ static FWCfgState *create_fw_cfg(const VirtMachineState *vms, AddressSpace *as) char *nodename; fw_cfg = fw_cfg_init_mem_wide(base + 8, base, 8, base + 16, as); - fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, (uint16_t)ms->smp.cpus); + fw_cfg_add_i16(fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); nodename = g_strdup_printf("/fw-cfg@%" PRIx64, base); qemu_fdt_add_subnode(ms->fdt, nodename); @@ -3196,7 +3198,13 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } if (vms->acpi_dev) { - /* TODO: register cpu for reset & update F/W info for the next boot */ + qemu_register_reset(do_cpu_reset, ARM_CPU(cs)); + } + + /* update the firmware information for the next boot. */ + vms->boot_cpus++; + if (vms->fw_cfg) { + fw_cfg_modify_i16(vms->fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); } /* @@ -3256,7 +3264,11 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, unwire_gic_cpu_irqs(vms, cs); virt_update_gic(vms, cs, false); - /* TODO: unregister cpu for reset & update F/W info for the next boot */ + qemu_unregister_reset(do_cpu_reset, ARM_CPU(cs)); + vms->boot_cpus--; + if (vms->fw_cfg) { + fw_cfg_modify_i16(vms->fw_cfg, FW_CFG_NB_CPUS, vms->boot_cpus); + } qobject_unref(dev->opts); dev->opts = NULL; diff --git a/include/hw/arm/boot.h b/include/hw/arm/boot.h index 80c492d742..f81326a1dc 100644 --- a/include/hw/arm/boot.h +++ b/include/hw/arm/boot.h @@ -178,6 +178,8 @@ AddressSpace *arm_boot_address_space(ARMCPU *cpu, int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo, hwaddr addr_limit, AddressSpace *as, MachineState *ms); +void do_cpu_reset(void *opaque); + /* Write a secure board setup routine with a dummy handler for SMCs */ void arm_write_secure_board_setup_dummy_smc(ARMCPU *cpu, const struct arm_boot_info *info, diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 0202f0252c..073d18281e 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -172,6 +172,7 @@ struct VirtMachineState { MemMapEntry *memmap; char *pciehb_nodename; const int *irqmap; + uint16_t boot_cpus; int fdt_size; uint32_t clock_phandle; uint32_t gic_phandle; From patchwork Tue Oct 15 10:00:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AC738CFC284 for ; Tue, 15 Oct 2024 10:09:15 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eTZ-0004ge-Lo; Tue, 15 Oct 2024 06:08:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTX-0004X2-2p; Tue, 15 Oct 2024 06:08:23 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTU-0002EX-Sf; Tue, 15 Oct 2024 06:08:22 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSVC62pjvz6K977; Tue, 15 Oct 2024 18:07:30 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 8593B1400D9; Tue, 15 Oct 2024 18:08:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:07:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 21/30] arm/virt: Update the guest(via GED) about vCPU hot-(un)plug events Date: Tue, 15 Oct 2024 11:00:03 +0100 Message-ID: <20241015100012.254223-22-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During any vCPU hot-(un)plug operation, the running guest VM must be notified about the addition of a new vCPU or the removal of an existing vCPU. This notification is handled via an ACPI GED event, which is eventually demultiplexed into a vCPU hotplug event, and then further into a specific hot-(un)plug event for the *targeted* vCPU. Introduce the required ACPI calls into the existing hot-(un)plug hooks, allowing ACPI GED events to be triggered from QEMU to the guest VM. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta --- hw/arm/virt.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index b1eba8de8c..15d9c2d8ac 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -3181,6 +3181,7 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, { VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); CPUState *cs = CPU(dev); + Error *local_err = NULL; CPUArchId *cpu_slot; /* insert the cold/hot-plugged vcpu in the slot */ @@ -3213,8 +3214,18 @@ static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, * hot-plugged, the guest is also notified. */ if (vms->acpi_dev) { - /* TODO: update acpi hotplug state. Send cpu hotplug event to guest */ + HotplugHandlerClass *hhc; + /* update acpi hotplug state and send cpu hotplug event to guest */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->plug(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) { + goto fail; + } } + + return; +fail: + error_propagate(errp, local_err); } static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, @@ -3223,7 +3234,9 @@ static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, MachineClass *mc = MACHINE_GET_CLASS(qdev_get_machine()); VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); ARMCPU *cpu = ARM_CPU(dev); + HotplugHandlerClass *hhc; CPUState *cs = CPU(dev); + Error *local_err = NULL; if (!vms->acpi_dev) { error_setg(errp, "GED does not exists or device is not realized!"); @@ -3242,14 +3255,25 @@ static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, return; } - /* TODO: request cpu hotplug from guest */ + /* request cpu hotplug from guest */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->unplug_request(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) { + goto fail; + } + + return; +fail: + error_propagate(errp, local_err); } static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { VirtMachineState *vms = VIRT_MACHINE(hotplug_dev); + HotplugHandlerClass *hhc; CPUState *cs = CPU(dev); + Error *local_err = NULL; CPUArchId *cpu_slot; if (!vms->acpi_dev) { @@ -3259,7 +3283,12 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, cpu_slot = virt_find_cpu_slot(cs); - /* TODO: update the acpi cpu hotplug state for cpu hot-unplug */ + /* update the acpi cpu hotplug state for cpu hot-unplug */ + hhc = HOTPLUG_HANDLER_GET_CLASS(vms->acpi_dev); + hhc->unplug(HOTPLUG_HANDLER(vms->acpi_dev), dev, &local_err); + if (local_err) { + goto fail; + } unwire_gic_cpu_irqs(vms, cs); virt_update_gic(vms, cs, false); @@ -3274,6 +3303,10 @@ static void virt_cpu_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, dev->opts = NULL; cpu_slot->cpu = NULL; + + return; +fail: + error_propagate(errp, local_err); } static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, From patchwork Tue Oct 15 10:00:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4B250CFC283 for ; Tue, 15 Oct 2024 10:09:09 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eTg-0005CU-DJ; Tue, 15 Oct 2024 06:08:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTc-0004ww-A2; Tue, 15 Oct 2024 06:08:28 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTa-0002H4-KK; Tue, 15 Oct 2024 06:08:28 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSVBH21j1z6F97V; Tue, 15 Oct 2024 18:06:47 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 7CB55140519; Tue, 15 Oct 2024 18:08:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:08:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 22/30] target/arm/cpu: Check if hotplugged ARM vCPU's FEAT match existing Date: Tue, 15 Oct 2024 11:00:04 +0100 Message-ID: <20241015100012.254223-23-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The ARM extensions configuration *must* match the existing vCPUs already initialized in KVM at VM initialization. ARM does not allow any per-vCPU features to be changed once the system has fully initialized. This is an immutable constraint of the ARM CPU architecture. Signed-off-by: Salil Mehta --- target/arm/cpu.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/target/arm/cpu.c b/target/arm/cpu.c index 3de0cb346b..14fcabc2c9 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -1912,6 +1912,49 @@ static void arm_cpu_finalizefn(Object *obj) #endif } +static void arm_cpu_check_features_change(ARMCPU *cpu, Error **errp) +{ +#if defined(TARGET_AARCH64) && !defined(CONFIG_USER_ONLY) + MachineClass *mc = MACHINE_GET_CLASS(qdev_get_machine()); + ARMCPU *firstcpu = ARM_CPU(first_cpu); + DeviceState *dev = DEVICE(cpu); + + if (!arm_feature(&cpu->env, ARM_FEATURE_AARCH64)) { + return; + } + + /* For now, features of hotplugged CPU MUST match earlier booted CPUs */ + if (!dev->hotplugged || !mc->has_hotpluggable_cpus) { + return; + } + + if (cpu_isar_feature(aa64_sve, cpu) && + (cpu->sve_max_vq != firstcpu->sve_max_vq || + cpu->sve_vq.map != firstcpu->sve_vq.map)) { + error_setg(errp, + "CPU %d: 'SVE' feature didn't match with existing CPUs", + CPU(cpu)->cpu_index); + return; + } + + if (cpu_isar_feature(aa64_sme, cpu) && + (cpu->sme_vq.map != firstcpu->sme_vq.map)) { + error_setg(errp, + "CPU %d: 'SME' feature didn't match with exisitng CPUs", + CPU(cpu)->cpu_index); + return; + } + + if (cpu_isar_feature(aa64_pauth, cpu) && + (cpu->prop_pauth != firstcpu->prop_pauth)) { + error_setg(errp, + "CPU %d: 'PAuth' feature didn't match with exisitng CPUs", + CPU(cpu)->cpu_index); + return; + } +#endif +} + void arm_cpu_finalize_features(ARMCPU *cpu, Error **errp) { Error *local_err = NULL; @@ -1961,6 +2004,13 @@ void arm_cpu_finalize_features(ARMCPU *cpu, Error **errp) return; } } + + /* + * As of now, we do not support heterogeneous computing, hence, features of + * all cpus should match. Hotplugged vCPUs are not allowed to have + * different features than the existing cold-plugged vCPUs + */ + arm_cpu_check_features_change(cpu, &local_err); } static void arm_cpu_realizefn(DeviceState *dev, Error **errp) From patchwork Tue Oct 15 10:00:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0EA3FCFC284 for ; Tue, 15 Oct 2024 10:09:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eU8-00062q-S3; Tue, 15 Oct 2024 06:09:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTz-0005j0-8Q; Tue, 15 Oct 2024 06:08:51 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eTw-0002PO-Fw; Tue, 15 Oct 2024 06:08:50 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV7N1nZWz6DB4H; Tue, 15 Oct 2024 18:04:16 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 91FF5140519; Tue, 15 Oct 2024 18:08:44 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:08:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 23/30] tcg: Update tcg_register_thread() leg to handle region alloc for hotplugged vCPU Date: Tue, 15 Oct 2024 11:00:05 +0100 Message-ID: <20241015100012.254223-24-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The TCG code cache consists of multiple regions shared among vCPUs in multi-threaded TCG mode. For cold-plugged vCPUs, these regions are sized and allocated during initialization in the `tcg_register_thread()` function when the vCPUs are realized. Later, these regions must be reallocated for hot-plugged vCPUs as well. If region allocation fails for hot-plugged vCPUs—due to the code cache being under stress—the TCG code cache must be flushed to create space for the newly hot-plugged vCPU. The only safe way to perform `tb_flush()` is to execute it synchronously within the `cpu_exec()` loop. Reported-by: Miguel Luis Signed-off-by: Miguel Luis Signed-off-by: Salil Mehta --- accel/tcg/tcg-accel-ops-mttcg.c | 2 +- accel/tcg/tcg-accel-ops-rr.c | 2 +- include/tcg/startup.h | 6 ++++++ include/tcg/tcg.h | 1 + tcg/region.c | 14 ++++++++++++++ tcg/tcg.c | 13 ++++++++++++- 6 files changed, 35 insertions(+), 3 deletions(-) diff --git a/accel/tcg/tcg-accel-ops-mttcg.c b/accel/tcg/tcg-accel-ops-mttcg.c index 49814ec4af..ab2f79d2c7 100644 --- a/accel/tcg/tcg-accel-ops-mttcg.c +++ b/accel/tcg/tcg-accel-ops-mttcg.c @@ -74,7 +74,7 @@ static void *mttcg_cpu_thread_fn(void *arg) force_rcu.notifier.notify = mttcg_force_rcu; force_rcu.cpu = cpu; rcu_add_force_rcu_notifier(&force_rcu.notifier); - tcg_register_thread(); + tcg_register_thread(cpu); bql_lock(); qemu_thread_get_self(cpu->thread); diff --git a/accel/tcg/tcg-accel-ops-rr.c b/accel/tcg/tcg-accel-ops-rr.c index 8ebadf8e9e..953231837c 100644 --- a/accel/tcg/tcg-accel-ops-rr.c +++ b/accel/tcg/tcg-accel-ops-rr.c @@ -186,7 +186,7 @@ static void *rr_cpu_thread_fn(void *arg) rcu_register_thread(); force_rcu.notify = rr_force_rcu; rcu_add_force_rcu_notifier(&force_rcu); - tcg_register_thread(); + tcg_register_thread(cpu); bql_lock(); qemu_thread_get_self(cpu->thread); diff --git a/include/tcg/startup.h b/include/tcg/startup.h index f71305765c..a565071516 100644 --- a/include/tcg/startup.h +++ b/include/tcg/startup.h @@ -25,6 +25,8 @@ #ifndef TCG_STARTUP_H #define TCG_STARTUP_H +#include "hw/core/cpu.h" + /** * tcg_init: Initialize the TCG runtime * @tb_size: translation buffer size @@ -43,7 +45,11 @@ void tcg_init(size_t tb_size, int splitwx, unsigned max_cpus); * accelerator's init_machine() method) must register with this * function before initiating translation. */ +#ifdef CONFIG_USER_ONLY void tcg_register_thread(void); +#else +void tcg_register_thread(CPUState *cpu); +#endif /** * tcg_prologue_init(): Generate the code for the TCG prologue diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h index 21d5884741..e3328cc600 100644 --- a/include/tcg/tcg.h +++ b/include/tcg/tcg.h @@ -518,6 +518,7 @@ struct TCGContext { /* Track which vCPU triggers events */ CPUState *cpu; /* *_trans */ + bool tbflush_pend; /* TB flush pending due to vCPU hotplug */ /* These structures are private to tcg-target.c.inc. */ #ifdef TCG_TARGET_NEED_LDST_LABELS diff --git a/tcg/region.c b/tcg/region.c index 478ec051c4..9007bfd71e 100644 --- a/tcg/region.c +++ b/tcg/region.c @@ -393,6 +393,20 @@ bool tcg_region_alloc(TCGContext *s) static void tcg_region_initial_alloc__locked(TCGContext *s) { bool err = tcg_region_alloc__locked(s); + + /* + * Hotplugged vCPUs may initially fail to find even a single available + * region. This could be due to the TB cache being under stress from the + * existing vCPUs. To mitigate this, the TB cache should be flushed. + * Therefore, the region allocation failure should be ignored, and a flag + * set to mark `tb_flush()` as pending. The flush will be performed later, + * synchronously in the context of `cpu_exec_loop()`/`tb_gen_code()`. + */ + if (err && s->cpu && DEVICE(s->cpu)->hotplugged) { + s->tbflush_pend = true; + return; + } + g_assert(!err); } diff --git a/tcg/tcg.c b/tcg/tcg.c index 34e3056380..5e9c6b2b4b 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -783,12 +783,13 @@ void tcg_register_thread(void) tcg_ctx = &tcg_init_ctx; } #else -void tcg_register_thread(void) +void tcg_register_thread(CPUState *cpu) { TCGContext *s = g_malloc(sizeof(*s)); unsigned int i, n; *s = tcg_init_ctx; + s->cpu = cpu; /* Relink mem_base. */ for (i = 0, n = tcg_init_ctx.nb_globals; i < n; ++i) { @@ -1388,6 +1389,16 @@ TranslationBlock *tcg_tb_alloc(TCGContext *s) TranslationBlock *tb; void *next; + /* + * The hotplugged vCPU's TCG context might not have any regions allocated. + * If this condition is detected, we should flush the TB cache to ensure + * that regions can be allocated for the newly hotplugged vCPU's TCGContext. + */ + if (s->tbflush_pend) { + s->tbflush_pend = false; + return NULL; + } + retry: tb = (void *)ROUND_UP((uintptr_t)s->code_gen_ptr, align); next = (void *)ROUND_UP((uintptr_t)(tb + 1), align); From patchwork Tue Oct 15 10:00:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF802CFC283 for ; Tue, 15 Oct 2024 10:10:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eUx-00070n-7q; Tue, 15 Oct 2024 06:09:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eUH-0006gQ-94; Tue, 15 Oct 2024 06:09:09 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eUE-0002TN-US; Tue, 15 Oct 2024 06:09:09 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSVC32sjSz6FGVm; Tue, 15 Oct 2024 18:07:27 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 99D2F140A71; Tue, 15 Oct 2024 18:09:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:08:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 24/30] target/arm: Add support to *unrealize* ARMCPU during vCPU Hot-unplug Date: Tue, 15 Oct 2024 11:00:06 +0100 Message-ID: <20241015100012.254223-25-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org vCPU Hot-unplug will result in QOM CPU object unrealization which will do away with all the vCPU thread creations, allocations, registrations that happened as part of the realization process. This change introduces the ARM CPU unrealize function taking care of exactly that. Note, initialized KVM vCPUs are not destroyed in host KVM rather their Qemu context is parked at the QEMU KVM layer. Co-developed-by: Keqian Zhu Signed-off-by: Keqian Zhu Signed-off-by: Salil Mehta Reported-by: Vishnu Pajjuri [VP: Identified CPU stall issue & suggested probable fix] Signed-off-by: Salil Mehta --- system/physmem.c | 8 ++- target/arm/cpu.c | 117 ++++++++++++++++++++++++++++++++++++++++- target/arm/cpu.h | 14 +++++ target/arm/gdbstub.c | 6 +++ target/arm/helper.c | 25 +++++++++ target/arm/internals.h | 3 ++ target/arm/kvm.c | 5 ++ 7 files changed, 176 insertions(+), 2 deletions(-) diff --git a/system/physmem.c b/system/physmem.c index dc1db3a384..17e09b406f 100644 --- a/system/physmem.c +++ b/system/physmem.c @@ -2613,8 +2613,14 @@ static void tcg_commit(MemoryListener *listener) * * That said, the listener is also called during realize, before * all of the tcg machinery for run-on is initialized: thus halt_cond. + * Similarly, the listener can also be triggered during the *unrealize* + * operation. In such a case, we should avoid using `run_on_cpu` because the + * TCG vCPU thread might already be terminated. As a result, the CPU work + * will never get processed, and `tcg_commit_cpu` will not be called. This + * means that operations like `tlb_flush()` might not be executed, + * potentially leading to inconsistencies. */ - if (cpu->halt_cond) { + if (cpu->halt_cond && !cpu->unplug) { async_run_on_cpu(cpu, tcg_commit_cpu, RUN_ON_CPU_HOST_PTR(cpuas)); } else { tcg_commit_cpu(cpu, RUN_ON_CPU_HOST_PTR(cpuas)); diff --git a/target/arm/cpu.c b/target/arm/cpu.c index 14fcabc2c9..19d2f89f5f 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -157,6 +157,16 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, QLIST_INSERT_HEAD(&cpu->pre_el_change_hooks, entry, node); } +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->pre_el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque) { @@ -168,6 +178,16 @@ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, QLIST_INSERT_HEAD(&cpu->el_change_hooks, entry, node); } +void arm_unregister_el_change_hooks(ARMCPU *cpu) +{ + ARMELChangeHook *entry, *next; + + QLIST_FOREACH_SAFE(entry, &cpu->el_change_hooks, node, next) { + QLIST_REMOVE(entry, node); + g_free(entry); + } +} + static void cp_reg_reset(gpointer key, gpointer value, gpointer opaque) { /* Reset a single ARMCPRegInfo register */ @@ -2642,6 +2662,98 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp) acc->parent_realize(dev, errp); } +#ifndef CONFIG_USER_ONLY +static void arm_cpu_unrealizefn(DeviceState *dev) +{ + ARMCPUClass *acc = ARM_CPU_GET_CLASS(dev); + ARMCPU *cpu = ARM_CPU(dev); + CPUARMState *env = &cpu->env; + CPUState *cs = CPU(dev); + bool has_secure; + + /* rock 'n' un-roll, whatever happened in the arm_cpu_realizefn cleanly */ + destroy_cpreg_list(cpu); + arm_cpu_unregister_gdb_regs(cpu); + unregister_cp_regs_for_features(cpu); + + if (cpu->sau_sregion && arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->sau.rbar); + g_free(env->sau.rlar); + } + + if (arm_feature(env, ARM_FEATURE_PMSA) && + arm_feature(env, ARM_FEATURE_V7) && + cpu->pmsav7_dregion) { + if (arm_feature(env, ARM_FEATURE_V8)) { + g_free(env->pmsav8.rbar[M_REG_NS]); + g_free(env->pmsav8.rlar[M_REG_NS]); + if (arm_feature(env, ARM_FEATURE_M_SECURITY)) { + g_free(env->pmsav8.rbar[M_REG_S]); + g_free(env->pmsav8.rlar[M_REG_S]); + } + } else { + g_free(env->pmsav7.drbar); + g_free(env->pmsav7.drsr); + g_free(env->pmsav7.dracr); + } + if (cpu->pmsav8r_hdregion) { + g_free(env->pmsav8.hprbar); + g_free(env->pmsav8.hprlar); + } + } + + if (arm_feature(env, ARM_FEATURE_PMU)) { + if (!kvm_enabled()) { + arm_unregister_pre_el_change_hooks(cpu); + arm_unregister_el_change_hooks(cpu); + } + + if (cpu->pmu_timer) { + timer_del(cpu->pmu_timer); + } + } + + cpu_remove_sync(CPU(dev)); + + /* + * We are intentionally destroying the CPU address space after the vCPU + * threads have been joined. This ensures that for TCG, any pending TLB + * flushes associated with the CPU are completed. The destruction of the + * address space also removes associated listeners, and joining threads + * after the address space no longer exists can lead to race conditions with + * already queued work for this CPU, which may result in a segmentation + * fault (SEGV) in `tcg_commit_cpu()`. + * + * Alternatively, Peter Maydell has suggested moving the CPU address space + * destruction to `cpu_common_unrealize()`, which would be called in the + * context of `parent_unrealize()`. This would also address the race + * condition in TCG. + * + * RFC: Question: Any additional thoughts or feedback on this approach would + * be appreciated? + */ + has_secure = cpu->has_el3 || arm_feature(env, ARM_FEATURE_M_SECURITY); + cpu_address_space_destroy(cs, ARMASIdx_NS); + if (cpu->tag_memory != NULL) { + cpu_address_space_destroy(cs, ARMASIdx_TagNS); + if (has_secure) { + cpu_address_space_destroy(cs, ARMASIdx_TagS); + } + } + if (has_secure) { + cpu_address_space_destroy(cs, ARMASIdx_S); + } + + acc->parent_unrealize(dev); + + timer_del(cpu->gt_timer[GTIMER_PHYS]); + timer_del(cpu->gt_timer[GTIMER_VIRT]); + timer_del(cpu->gt_timer[GTIMER_HYP]); + timer_del(cpu->gt_timer[GTIMER_SEC]); + timer_del(cpu->gt_timer[GTIMER_HYPVIRT]); +} +#endif + static ObjectClass *arm_cpu_class_by_name(const char *cpu_model) { ObjectClass *oc; @@ -2745,7 +2857,10 @@ static void arm_cpu_class_init(ObjectClass *oc, void *data) device_class_set_parent_realize(dc, arm_cpu_realizefn, &acc->parent_realize); - +#ifndef CONFIG_USER_ONLY + device_class_set_parent_unrealize(dc, arm_cpu_unrealizefn, + &acc->parent_unrealize); +#endif device_class_set_props(dc, arm_cpu_properties); resettable_class_set_parent_phases(rc, NULL, arm_cpu_reset_hold, NULL, diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 1277a0ddfc..07bd7d6542 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1128,6 +1128,7 @@ struct ARMCPUClass { const ARMCPUInfo *info; DeviceRealize parent_realize; + DeviceUnrealize parent_unrealize; ResettablePhases parent_phases; }; @@ -3293,6 +3294,13 @@ static inline AddressSpace *arm_addressspace(CPUState *cs, MemTxAttrs attrs) */ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_pre_el_change_hook: + * unregister all pre EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_pre_el_change_hooks(ARMCPU *cpu); + /** * arm_register_el_change_hook: * Register a hook function which will be called immediately after this @@ -3305,6 +3313,12 @@ void arm_register_pre_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, */ void arm_register_el_change_hook(ARMCPU *cpu, ARMELChangeHookFn *hook, void *opaque); +/** + * arm_unregister_el_change_hook: + * unregister all EL change hook functions. Generally called during + * unrealize'ing leg + */ +void arm_unregister_el_change_hooks(ARMCPU *cpu); /** * arm_rebuild_hflags: diff --git a/target/arm/gdbstub.c b/target/arm/gdbstub.c index 554b8736bb..58067e30a5 100644 --- a/target/arm/gdbstub.c +++ b/target/arm/gdbstub.c @@ -595,3 +595,9 @@ void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu) } #endif /* CONFIG_TCG */ } + +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu) +{ + CPUState *cs = CPU(cpu); + gdb_unregister_coprocessor_all(cs); +} diff --git a/target/arm/helper.c b/target/arm/helper.c index 3f77b40734..f5ea3ad4e6 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -264,6 +264,19 @@ void init_cpreg_list(ARMCPU *cpu) g_list_free(keys); } +void destroy_cpreg_list(ARMCPU *cpu) +{ + assert(cpu->cpreg_indexes); + assert(cpu->cpreg_values); + assert(cpu->cpreg_vmstate_indexes); + assert(cpu->cpreg_vmstate_values); + + g_free(cpu->cpreg_indexes); + g_free(cpu->cpreg_values); + g_free(cpu->cpreg_vmstate_indexes); + g_free(cpu->cpreg_vmstate_values); +} + static bool arm_pan_enabled(CPUARMState *env) { if (is_a64(env)) { @@ -9985,6 +9998,18 @@ void register_cp_regs_for_features(ARMCPU *cpu) #endif } +void unregister_cp_regs_for_features(ARMCPU *cpu) +{ + CPUARMState *env = &cpu->env; + if (arm_feature(env, ARM_FEATURE_M)) { + /* M profile has no coprocessor registers */ + return; + } + + /* empty it all. unregister all the coprocessor registers */ + g_hash_table_remove_all(cpu->cp_regs); +} + /* * Private utility function for define_one_arm_cp_reg_with_opaque(): * add a single reginfo struct to the hash table. diff --git a/target/arm/internals.h b/target/arm/internals.h index 1e5da81ce9..be2f14c6b3 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -367,9 +367,12 @@ void arm_cpu_register(const ARMCPUInfo *info); void aarch64_cpu_register(const ARMCPUInfo *info); void register_cp_regs_for_features(ARMCPU *cpu); +void unregister_cp_regs_for_features(ARMCPU *cpu); void init_cpreg_list(ARMCPU *cpu); +void destroy_cpreg_list(ARMCPU *cpu); void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu); +void arm_cpu_unregister_gdb_regs(ARMCPU *cpu); void arm_translate_init(void); void arm_cpu_register_gdb_commands(ARMCPU *cpu); diff --git a/target/arm/kvm.c b/target/arm/kvm.c index e82cb2aa8b..f61fd70db0 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1983,6 +1983,11 @@ int kvm_arch_init_vcpu(CPUState *cs) int kvm_arch_destroy_vcpu(CPUState *cs) { + /* vCPUs which are yet to be realized will not have handler */ + if (cs->thread_id) { + qemu_del_vm_change_state_handler(cs->vmcse); + } + return 0; } From patchwork Tue Oct 15 10:00:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35577CFC283 for ; Tue, 15 Oct 2024 10:10:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eVR-0007T6-LE; Tue, 15 Oct 2024 06:10:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eUb-0006zm-Ai; Tue, 15 Oct 2024 06:09:36 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eUZ-0002WC-It; Tue, 15 Oct 2024 06:09:29 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSVDf3Bchz6K98R; Tue, 15 Oct 2024 18:08:50 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 93A68140451; Tue, 15 Oct 2024 18:09:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:09:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 25/30] tcg/mttcg: Introduce MTTCG thread unregistration leg Date: Tue, 15 Oct 2024 11:00:07 +0100 Message-ID: <20241015100012.254223-26-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Miguel Luis Introduce the TCG thread unregistration leg which shall be called in context to TCG/vCPU unrealize. Reported-by: Salil Mehta Signed-off-by: Miguel Luis Signed-off-by: Salil Mehta --- accel/tcg/tcg-accel-ops-mttcg.c | 1 + include/tcg/startup.h | 7 +++++++ tcg/tcg.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/accel/tcg/tcg-accel-ops-mttcg.c b/accel/tcg/tcg-accel-ops-mttcg.c index ab2f79d2c7..4b26164ffd 100644 --- a/accel/tcg/tcg-accel-ops-mttcg.c +++ b/accel/tcg/tcg-accel-ops-mttcg.c @@ -122,6 +122,7 @@ static void *mttcg_cpu_thread_fn(void *arg) bql_unlock(); rcu_remove_force_rcu_notifier(&force_rcu.notifier); rcu_unregister_thread(); + tcg_unregister_thread(); return NULL; } diff --git a/include/tcg/startup.h b/include/tcg/startup.h index a565071516..c035d03f7e 100644 --- a/include/tcg/startup.h +++ b/include/tcg/startup.h @@ -51,6 +51,13 @@ void tcg_register_thread(void); void tcg_register_thread(CPUState *cpu); #endif +/** + * tcg_unregister_thread: Unregister this thread with the TCG runtime + * + * This leg shall be called whenever TCG vCPU is hot-unplugged + */ +void tcg_unregister_thread(void); + /** * tcg_prologue_init(): Generate the code for the TCG prologue * diff --git a/tcg/tcg.c b/tcg/tcg.c index 5e9c6b2b4b..3bdebdb332 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -811,6 +811,39 @@ void tcg_register_thread(CPUState *cpu) tcg_ctx = s; } + +static void tcg_free_plugin_context(TCGContext *s) +{ +#ifdef CONFIG_PLUGIN + unsigned i; + + if (s->plugin_tb) { + for (i = 0; i < s->plugin_tb->insns->len; i++) { + g_free(g_ptr_array_index(s->plugin_tb->insns, i)); + } + g_ptr_array_free(s->plugin_tb->insns, TRUE); + + if (!s->plugin_tb->insns) { + g_free(s->plugin_tb); + } + } +#endif +} + +void tcg_unregister_thread(void) +{ + TCGContext *s = tcg_ctx; + unsigned int n; + + /* unclaim an entry in tcg_ctxs */ + n = qatomic_fetch_dec(&tcg_cur_ctxs); + g_assert(n > 1); + qatomic_store_release(&tcg_ctxs[n - 1], 0); + + tcg_free_plugin_context(s); + + g_free(s); +} #endif /* !CONFIG_USER_ONLY */ /* pool based memory allocation */ From patchwork Tue Oct 15 10:00:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5B307CFC285 for ; Tue, 15 Oct 2024 10:11:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eW4-00084j-C2; Tue, 15 Oct 2024 06:11:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eUu-00076Z-Iq; Tue, 15 Oct 2024 06:09:55 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eUs-0002YL-Mr; Tue, 15 Oct 2024 06:09:48 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV8X2KtSz6DB4D; Tue, 15 Oct 2024 18:05:16 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id A3C5C1404F5; Tue, 15 Oct 2024 18:09:44 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:09:24 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 26/30] hw/intc/arm_gicv3_common: Add GICv3CPUState 'accessible' flag migration handling Date: Tue, 15 Oct 2024 11:00:08 +0100 Message-ID: <20241015100012.254223-27-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The QOM `GICv3CPUState` (and consequently the corresponding KVM VGIC `ICC_*_EL1` registers) can be either 'accessible' or 'inaccessible', depending on the state of the associated QOM vCPUs. This `gicc_accessible` state should be saved during migration at the source and restored at the destination. Ideally, the number of possible and enabled QOM vCPUs should match between the source and destination. Ensuring this consistency is the responsibility of the administrator. However, if the destination QEMU has more enabled vCPUs than the source, we can either fail the migration or override the destination QEMU’s vCPU configuration to match the source. We have adopted the latter approach as a mitigation for the mismatch. Nonetheless, the administrator should still ensure that the number of possible QOM vCPUs is consistent at both ends. Signed-off-by: Salil Mehta --- hw/intc/arm_gicv3_common.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/hw/intc/arm_gicv3_common.c b/hw/intc/arm_gicv3_common.c index c21fff4903..53045ad6bc 100644 --- a/hw/intc/arm_gicv3_common.c +++ b/hw/intc/arm_gicv3_common.c @@ -126,6 +126,29 @@ static int vmstate_gicv3_cpu_pre_load(void *opaque) return 0; } +static int vmstate_gicv3_cpu_post_load(void *opaque, int version_id) +{ + GICv3CPUState *cs = opaque; + + /* + * If the destination QEMU has more *enabled* vCPUs than the source, we can + * either *fail* the migration or override the destination QEMU’s vCPU + * configuration to match the source. Since it is safe to override the + * `CPUState` of the extra *enabled* vCPUs at the destination, we have + * adopted the latter approach as a mitigation for the mismatch. + * RFC: Question: any suggestions on this are welcome? + */ + if (cs->cpu && !gicv3_cpu_accessible((cs))) { + warn_report("Found CPU %d enabled, for incoming *disabled* GICC State", + cs->cpu->cpu_index); + warn_report("*Disabling* CPU %d, to match the incoming migrated state", + cs->cpu->cpu_index); + qdev_unrealize(DEVICE(cs->cpu)); + } + + return 0; +} + static bool icc_sre_el1_reg_needed(void *opaque) { GICv3CPUState *cs = opaque; @@ -186,6 +209,7 @@ static const VMStateDescription vmstate_gicv3_cpu = { .version_id = 1, .minimum_version_id = 1, .pre_load = vmstate_gicv3_cpu_pre_load, + .post_load = vmstate_gicv3_cpu_post_load, .fields = (const VMStateField[]) { VMSTATE_UINT32(level, GICv3CPUState), VMSTATE_UINT32(gicr_ctlr, GICv3CPUState), @@ -207,6 +231,7 @@ static const VMStateDescription vmstate_gicv3_cpu = { VMSTATE_UINT64_2DARRAY(icc_apr, GICv3CPUState, 3, 4), VMSTATE_UINT64_ARRAY(icc_igrpen, GICv3CPUState, 3), VMSTATE_UINT64(icc_ctlr_el3, GICv3CPUState), + VMSTATE_BOOL(gicc_accessible, GICv3CPUState), VMSTATE_END_OF_LIST() }, .subsections = (const VMStateDescription * const []) { From patchwork Tue Oct 15 10:00:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92FFBCFC283 for ; Tue, 15 Oct 2024 10:11:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eVz-00083a-LE; Tue, 15 Oct 2024 06:11:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eVG-0007KG-0n; Tue, 15 Oct 2024 06:10:11 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eVC-0002ld-Ta; Tue, 15 Oct 2024 06:10:09 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSV8w1nFvz6DB51; Tue, 15 Oct 2024 18:05:36 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 913991400D9; Tue, 15 Oct 2024 18:10:04 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:09:44 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 27/30] target/arm/kvm, tcg: Handle SMCCC hypercall exits in VMM during PSCI_CPU_{ON, OFF} Date: Tue, 15 Oct 2024 11:00:09 +0100 Message-ID: <20241015100012.254223-28-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Author Salil Mehta To support vCPU hotplug, we must trap any `HVC`/`SMC` `PSCI_CPU_{ON,OFF}` hypercalls from the host KVM to QEMU for policy checks. This ensures the following when a vCPU is brought online: 1. The vCPU is actually plugged in (i.e., present). 2. The vCPU is not disabled. Implement the registration and handling of `HVC`/`SMC` hypercall exits within the VMM, ensuring that proper policy checks and control flow are enforced during the vCPU onlining and offlining processes. Co-developed-by: Jean-Philippe Brucker Signed-off-by: Jean-Philippe Brucker Signed-off-by: Salil Mehta --- target/arm/arm-powerctl.c | 20 +++++--- target/arm/helper.c | 2 +- target/arm/internals.h | 11 ----- target/arm/kvm.c | 93 +++++++++++++++++++++++++++++++++++++ target/arm/kvm_arm.h | 14 ++++++ target/arm/meson.build | 1 + target/arm/{tcg => }/psci.c | 8 ++++ target/arm/tcg/meson.build | 4 -- 8 files changed, 131 insertions(+), 22 deletions(-) rename target/arm/{tcg => }/psci.c (97%) diff --git a/target/arm/arm-powerctl.c b/target/arm/arm-powerctl.c index 2b2055c6ac..51e7436d37 100644 --- a/target/arm/arm-powerctl.c +++ b/target/arm/arm-powerctl.c @@ -17,6 +17,7 @@ #include "qemu/main-loop.h" #include "sysemu/tcg.h" #include "target/arm/multiprocessing.h" +#include "hw/boards.h" #ifndef DEBUG_ARM_POWERCTL #define DEBUG_ARM_POWERCTL 0 @@ -31,14 +32,14 @@ CPUState *arm_get_cpu_by_id(uint64_t id) { + MachineState *ms = MACHINE(qdev_get_machine()); CPUState *cpu; DPRINTF("cpu %" PRId64 "\n", id); - CPU_FOREACH(cpu) { - ARMCPU *armcpu = ARM_CPU(cpu); - - if (arm_cpu_mp_affinity(armcpu) == id) { + /* with vCPU hotplug support, we must now check for all possible vCPUs */ + CPU_FOREACH_POSSIBLE(cpu, ms->possible_cpus) { + if (cpu && (arm_cpu_mp_affinity(ARM_CPU(cpu)) == id)) { return cpu; } } @@ -119,9 +120,16 @@ int arm_set_cpu_on(uint64_t cpuid, uint64_t entry, uint64_t context_id, /* Retrieve the cpu we are powering up */ target_cpu_state = arm_get_cpu_by_id(cpuid); + if (!target_cpu_state) { - /* The cpu was not found */ - return QEMU_ARM_POWERCTL_INVALID_PARAM; + /* + * The cpu is not plugged in or disabled. We should return appropriate + * value as introduced in DEN0022E PSCI 1.2 issue E + */ + qemu_log_mask(LOG_GUEST_ERROR, + "[ARM]%s: Denying attempt to online removed/disabled " + "CPU%" PRId64"\n", __func__, cpuid); + return QEMU_ARM_POWERCTL_IS_OFF; } target_cpu = ARM_CPU(target_cpu_state); diff --git a/target/arm/helper.c b/target/arm/helper.c index f5ea3ad4e6..2291fc8428 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -11840,7 +11840,7 @@ void arm_cpu_do_interrupt(CPUState *cs) env->exception.syndrome); } - if (tcg_enabled() && arm_is_psci_call(cpu, cs->exception_index)) { + if (arm_is_psci_call(cpu, cs->exception_index)) { arm_handle_psci_call(cpu); qemu_log_mask(CPU_LOG_INT, "...handled as PSCI call\n"); return; diff --git a/target/arm/internals.h b/target/arm/internals.h index be2f14c6b3..755c0a7b46 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -520,21 +520,10 @@ vaddr arm_adjust_watchpoint_address(CPUState *cs, vaddr addr, int len); /* Callback function for when a watchpoint or breakpoint triggers. */ void arm_debug_excp_handler(CPUState *cs); -#if defined(CONFIG_USER_ONLY) || !defined(CONFIG_TCG) -static inline bool arm_is_psci_call(ARMCPU *cpu, int excp_type) -{ - return false; -} -static inline void arm_handle_psci_call(ARMCPU *cpu) -{ - g_assert_not_reached(); -} -#else /* Return true if the r0/x0 value indicates that this SMC/HVC is a PSCI call. */ bool arm_is_psci_call(ARMCPU *cpu, int excp_type); /* Actually handle a PSCI call */ void arm_handle_psci_call(ARMCPU *cpu); -#endif /** * arm_clear_exclusive: clear the exclusive monitor diff --git a/target/arm/kvm.c b/target/arm/kvm.c index f61fd70db0..61ac7f6093 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -544,9 +544,51 @@ int kvm_arch_get_default_type(MachineState *ms) return fixed_ipa ? 0 : size; } +static bool kvm_arm_set_vm_attr(struct kvm_device_attr *attr, const char *name) +{ + int err; + + err = kvm_vm_ioctl(kvm_state, KVM_HAS_DEVICE_ATTR, attr); + if (err != 0) { + error_report("%s: KVM_HAS_DEVICE_ATTR: %s", name, strerror(-err)); + return false; + } + + err = kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, attr); + if (err != 0) { + error_report("%s: KVM_SET_DEVICE_ATTR: %s", name, strerror(-err)); + return false; + } + + return true; +} + +int kvm_arm_set_smccc_filter(uint64_t func, uint8_t faction) +{ + struct kvm_smccc_filter filter = { + .base = func, + .nr_functions = 1, + .action = faction, + }; + struct kvm_device_attr attr = { + .group = KVM_ARM_VM_SMCCC_CTRL, + .attr = KVM_ARM_VM_SMCCC_FILTER, + .flags = 0, + .addr = (uintptr_t)&filter, + }; + + if (!kvm_arm_set_vm_attr(&attr, "SMCCC Filter")) { + error_report("failed to set SMCCC filter in KVM Host"); + return -1; + } + + return 0; +} + int kvm_arch_init(MachineState *ms, KVMState *s) { int ret = 0; + /* For ARM interrupt delivery is always asynchronous, * whether we are using an in-kernel VGIC or not. */ @@ -609,6 +651,22 @@ int kvm_arch_init(MachineState *ms, KVMState *s) hw_breakpoints = g_array_sized_new(true, true, sizeof(HWBreakpoint), max_hw_bps); + /* + * To be able to handle PSCI CPU ON calls in QEMU, we need to install SMCCC + * filter in the Host KVM. This is required to support features like + * virtual CPU Hotplug on ARM platforms. + */ + if (kvm_arm_set_smccc_filter(PSCI_0_2_FN64_CPU_ON, + KVM_SMCCC_FILTER_FWD_TO_USER)) { + error_report("CPU On PSCI-to-user-space fwd filter install failed"); + abort(); + } + if (kvm_arm_set_smccc_filter(PSCI_0_2_FN_CPU_OFF, + KVM_SMCCC_FILTER_FWD_TO_USER)) { + error_report("CPU Off PSCI-to-user-space fwd filter install failed"); + abort(); + } + return ret; } @@ -1452,6 +1510,38 @@ static bool kvm_arm_handle_debug(ARMCPU *cpu, return false; } +static int kvm_arm_handle_hypercall(CPUState *cs, struct kvm_run *run) +{ + ARMCPU *cpu = ARM_CPU(cs); + CPUARMState *env = &cpu->env; + + kvm_cpu_synchronize_state(cs); + + /* + * hard coding immediate to 0 as we dont expect non-zero value as of now + * This might change in future versions. Hence, KVM_GET_ONE_REG could be + * used in such cases but it must be enhanced then only synchronize will + * also fetch ESR_EL2 value. + */ + if (run->hypercall.flags == KVM_HYPERCALL_EXIT_SMC) { + cs->exception_index = EXCP_SMC; + env->exception.syndrome = syn_aa64_smc(0); + } else { + cs->exception_index = EXCP_HVC; + env->exception.syndrome = syn_aa64_hvc(0); + } + env->exception.target_el = 1; + bql_lock(); + arm_cpu_do_interrupt(cs); + bql_unlock(); + + /* + * For PSCI, exit the kvm_run loop and process the work. Especially + * important if this was a CPU_OFF command and we can't return to the guest. + */ + return EXCP_INTERRUPT; +} + int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run) { ARMCPU *cpu = ARM_CPU(cs); @@ -1468,6 +1558,9 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run) ret = kvm_arm_handle_dabt_nisv(cpu, run->arm_nisv.esr_iss, run->arm_nisv.fault_ipa); break; + case KVM_EXIT_HYPERCALL: + ret = kvm_arm_handle_hypercall(cs, run); + break; default: qemu_log_mask(LOG_UNIMP, "%s: un-handled exit reason %d\n", __func__, run->exit_reason); diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h index 93d12eeb74..e59e1e2511 100644 --- a/target/arm/kvm_arm.h +++ b/target/arm/kvm_arm.h @@ -224,6 +224,15 @@ void kvm_arm_pvtime_init(ARMCPU *cpu, uint64_t ipa); int kvm_arm_set_irq(int cpu, int irqtype, int irq, int level); +/** + * kvm_arm_set_smccc_filter + * @func: funcion + * @faction: SMCCC filter action(handle, deny, fwd-to-user) to be deployed + * + * Sets the ARMs SMC-CC filter in KVM Host for selective hypercall exits + */ +int kvm_arm_set_smccc_filter(uint64_t func, uint8_t faction); + #else /* @@ -293,6 +302,11 @@ static inline uint32_t kvm_arm_sve_get_vls(ARMCPU *cpu) g_assert_not_reached(); } +static inline int kvm_arm_set_smccc_filter(uint64_t func, uint8_t faction) +{ + g_assert_not_reached(); +} + #endif #endif diff --git a/target/arm/meson.build b/target/arm/meson.build index 2e10464dbb..3e9f704f35 100644 --- a/target/arm/meson.build +++ b/target/arm/meson.build @@ -23,6 +23,7 @@ arm_system_ss.add(files( 'arm-qmp-cmds.c', 'cortex-regs.c', 'machine.c', + 'psci.c', 'ptw.c', )) diff --git a/target/arm/tcg/psci.c b/target/arm/psci.c similarity index 97% rename from target/arm/tcg/psci.c rename to target/arm/psci.c index 51d2ca3d30..b3fcb85079 100644 --- a/target/arm/tcg/psci.c +++ b/target/arm/psci.c @@ -21,7 +21,9 @@ #include "exec/helper-proto.h" #include "kvm-consts.h" #include "qemu/main-loop.h" +#include "qemu/error-report.h" #include "sysemu/runstate.h" +#include "sysemu/tcg.h" #include "internals.h" #include "arm-powerctl.h" #include "target/arm/multiprocessing.h" @@ -158,6 +160,11 @@ void arm_handle_psci_call(ARMCPU *cpu) case QEMU_PSCI_0_1_FN_CPU_SUSPEND: case QEMU_PSCI_0_2_FN_CPU_SUSPEND: case QEMU_PSCI_0_2_FN64_CPU_SUSPEND: + if (!tcg_enabled()) { + warn_report("CPU suspend not supported in non-tcg mode"); + break; + } +#ifdef CONFIG_TCG /* Affinity levels are not supported in QEMU */ if (param[1] & 0xfffe0000) { ret = QEMU_PSCI_RET_INVALID_PARAMS; @@ -170,6 +177,7 @@ void arm_handle_psci_call(ARMCPU *cpu) env->regs[0] = 0; } helper_wfi(env, 4); +#endif break; case QEMU_PSCI_1_0_FN_PSCI_FEATURES: switch (param[1]) { diff --git a/target/arm/tcg/meson.build b/target/arm/tcg/meson.build index 508932a249..5b43c84c40 100644 --- a/target/arm/tcg/meson.build +++ b/target/arm/tcg/meson.build @@ -54,9 +54,5 @@ arm_ss.add(when: 'TARGET_AARCH64', if_true: files( 'sve_helper.c', )) -arm_system_ss.add(files( - 'psci.c', -)) - arm_system_ss.add(when: 'CONFIG_ARM_V7M', if_true: files('cpu-v7m.c')) arm_user_ss.add(when: 'TARGET_AARCH64', if_false: files('cpu-v7m.c')) From patchwork Tue Oct 15 10:00:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 82252CFC285 for ; Tue, 15 Oct 2024 10:11:13 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eWD-0000ZF-6I; Tue, 15 Oct 2024 06:11:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eVZ-0007kQ-5i; Tue, 15 Oct 2024 06:10:31 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0eVW-0002oM-J9; Tue, 15 Oct 2024 06:10:28 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSVFp38SNz6K987; Tue, 15 Oct 2024 18:09:50 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 92B7F140B63; Tue, 15 Oct 2024 18:10:24 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:10:04 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 28/30] target/arm/kvm: Write vCPU's state back to KVM on cold-reset Date: Tue, 15 Oct 2024 11:00:10 +0100 Message-ID: <20241015100012.254223-29-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jean-Philippe Brucker Previously, all `PSCI_CPU_{ON, OFF}` calls were handled directly by KVM. However, with the introduction of vCPU hotplug, these hypervisor calls are now trapped to QEMU for policy checks. This shift can lead to inconsistent vCPU states between KVM and QEMU, particularly when the vCPU has been recently plugged in and is transitioning from the unparked state in QOM. Therefore, it is crucial to synchronize the vCPU state with KVM, especially in the context of a cold reset of the QOM vCPU. To ensure this synchronization, mark the QOM vCPU as "dirty" to trigger a call to `kvm_arch_put_registers()`. This guarantees that KVM’s `MP_STATE` is updated accordingly, forcing synchronization of the `mp_state` between QEMU and KVM. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Salil Mehta --- target/arm/kvm.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 61ac7f6093..4a10e63877 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1038,6 +1038,7 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu) void kvm_arm_reset_vcpu(ARMCPU *cpu) { int ret; + CPUState *cs = CPU(cpu); /* Re-init VCPU so that all registers are set to * their respective reset values. @@ -1059,6 +1060,12 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu) * for the same reason we do so in kvm_arch_get_registers(). */ write_list_to_cpustate(cpu); + + /* + * Ensure we call kvm_arch_put_registers(). The vCPU isn't marked dirty if + * it was parked in KVM and is now booting from a PSCI CPU_ON call. + */ + cs->vcpu_dirty = true; } void kvm_arm_create_host_vcpu(ARMCPU *cpu) From patchwork Tue Oct 15 10:16:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 984DDCFC283 for ; Tue, 15 Oct 2024 10:18:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0eci-0003lR-Ht; Tue, 15 Oct 2024 06:17:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0ecf-0003ki-St; Tue, 15 Oct 2024 06:17:49 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0ece-0003VN-2G; Tue, 15 Oct 2024 06:17:49 -0400 Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSVQB2VjXz6K985; Tue, 15 Oct 2024 18:17:06 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id 7FB101400D9; Tue, 15 Oct 2024 18:17:40 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:17:21 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 29/30] hw/intc/arm_gicv3_kvm: Pause all vCPU to ensure locking in KVM of resetting vCPU Date: Tue, 15 Oct 2024 11:16:56 +0100 Message-ID: <20241015101656.254330-1-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org vCPU reset can result in device access to VGIC CPU system registers using the `IOCTL KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS` interface. When accessing these registers in the KVM host, it is necessary to acquire a lock on all vCPUs during the `vgic_v3_attr_regs_access()` operation. This operation may fail if KVM is unable to acquire the necessary locks on all vCPUs. Therefore, to ensure proper locking of the vCPU being reset and prevent failures, we need to *pause all vCPUs* during this operation to facilitate successful locking within the host. Signed-off-by: Salil Mehta --- hw/intc/arm_gicv3_kvm.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/hw/intc/arm_gicv3_kvm.c b/hw/intc/arm_gicv3_kvm.c index 3e1e97d830..bcdbf83897 100644 --- a/hw/intc/arm_gicv3_kvm.c +++ b/hw/intc/arm_gicv3_kvm.c @@ -714,10 +714,19 @@ static void arm_gicv3_icc_reset(CPUARMState *env, const ARMCPRegInfo *ri) return; } + /* + * This shall be called even when vcpu is being hotplugged or onlined and + * other vcpus might be running. Host kernel KVM code to handle device + * access of IOCTLs KVM_{GET|SET}_DEVICE_ATTR might fail due to inability to + * grab vcpu locks for all the vcpus. Hence, we need to pause all vcpus to + * facilitate locking within host. + */ + pause_all_vcpus(); /* Initialize to actual HW supported configuration */ kvm_device_access(s->dev_fd, KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS, KVM_VGIC_ATTR(ICC_CTLR_EL1, c->gicr_typer), &c->icc_ctlr_el1[GICV3_NS], false, &error_abort); + resume_all_vcpus(); c->icc_ctlr_el1[GICV3_S] = c->icc_ctlr_el1[GICV3_NS]; } From patchwork Tue Oct 15 10:18:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Salil Mehta X-Patchwork-Id: 13836050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 367C7CFC283 for ; Tue, 15 Oct 2024 10:18:59 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t0ede-0004K2-3m; Tue, 15 Oct 2024 06:18:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0edV-0004EL-9R; Tue, 15 Oct 2024 06:18:42 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t0edT-0003l0-AG; Tue, 15 Oct 2024 06:18:41 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4XSVQ356K0z6FGyW; Tue, 15 Oct 2024 18:16:59 +0800 (CST) Received: from frapeml500007.china.huawei.com (unknown [7.182.85.172]) by mail.maildlp.com (Postfix) with ESMTPS id E8EA91404F5; Tue, 15 Oct 2024 18:18:36 +0800 (CST) Received: from 00293818-MRGF.huawei.com (10.48.146.149) by frapeml500007.china.huawei.com (7.182.85.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Tue, 15 Oct 2024 12:18:17 +0200 To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V5 30/30] hw/arm/virt: Expose cold-booted vCPUs as MADT GICC *Enabled* Date: Tue, 15 Oct 2024 11:18:00 +0100 Message-ID: <20241015101800.254351-1-salil.mehta@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015100012.254223-1-salil.mehta@huawei.com> References: <20241015100012.254223-1-salil.mehta@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.48.146.149] X-ClientProxiedBy: dggems701-chm.china.huawei.com (10.3.19.178) To frapeml500007.china.huawei.com (7.182.85.172) Received-SPF: pass client-ip=185.176.79.56; envelope-from=salil.mehta@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Salil Mehta X-Patchwork-Original-From: Salil Mehta via From: Salil Mehta Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Hotpluggable vCPUs must be exposed as "online-capable" according to the new UEFI specification [1][2]. However, marking cold-booted vCPUs as "online-capable" during boot may cause them to go undetected by legacy operating systems, potentially leading to compatibility issues. Hence, both 'online-capable' bit and 'Enabled' bit in GIC CPU Interface flags should not be mutually exclusive as they are now. Since implementing this specification change may take time, it is necessary to temporarily *disable* support for *unplugging* cold-booted vCPUs to maintain compatibility with legacy OS environments. As an alternative and temporary mitigation, we could introduce a property that controls whether cold-booted vCPUs are marked as unpluggable. Community feedback on this approach would be appreciated. References: [1] Original UEFI/ACPI proposed Change Bugzilla – TianoCore Link: https://bugzilla.tianocore.org/show_bug.cgi?id=3706 [2] Advanced Configuration and Power Interface (ACPI) Specification, Release 6.5, Aug 29, 2022 Section: 5.2.12.14 GIC CPU Interface (GICC) Structure / Table 5.37: GICC CPU Interface Flags Link: https://uefi.org/sites/default/files/resources/ACPI_Spec_6_5_Aug29.pdf (Pages 138, 140) Signed-off-by: Salil Mehta --- hw/arm/virt.c | 16 ++++++++++++++++ include/hw/core/cpu.h | 2 ++ 2 files changed, 18 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 15d9c2d8ac..d1172151be 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -3174,6 +3174,10 @@ static void virt_cpu_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, virt_update_gic(vms, cs, true); wire_gic_cpu_irqs(vms, cs); } + + if (!dev->hotplugged) { + cs->cold_booted = true; + } } static void virt_cpu_plug(HotplugHandler *hotplug_dev, DeviceState *dev, @@ -3248,6 +3252,18 @@ static void virt_cpu_unplug_request(HotplugHandler *hotplug_dev, return; } + /* + * UEFI ACPI standard change is required to make both 'enabled' and the + * 'online-capable' bit co-exist instead of being mutually exclusive. + * check virt_acpi_get_gicc_flags() for more details. + * + * Disable the unplugging of cold-booted vCPUs as a temporary mitigation. + */ + if (cs->cold_booted) { + error_setg(errp, "Hot-unplug of cold-booted CPU not supported!"); + return; + } + if (cs->cpu_index == first_cpu->cpu_index) { error_setg(errp, "Boot CPU(id%d=%d:%d:%d:%d) hot-unplug not supported", first_cpu->cpu_index, cpu->socket_id, cpu->cluster_id, diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 4a74c383ab..7b7ca2c32c 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -571,6 +571,8 @@ struct CPUState { uint32_t halted; int32_t exception_index; + bool cold_booted; + AccelCPUState *accel; /* Used to keep track of an outstanding cpu throttle thread for migration