From patchwork Tue Jun 19 01:41:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Roth X-Patchwork-Id: 10472829 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 32B006032A for ; Tue, 19 Jun 2018 01:53:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 141CF28831 for ; Tue, 19 Jun 2018 01:53:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 07AB028B3E; Tue, 19 Jun 2018 01:53:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 504FF28831 for ; Tue, 19 Jun 2018 01:53:06 +0000 (UTC) Received: from localhost ([::1]:38407 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fV5pR-0007JM-EL for patchwork-qemu-devel@patchwork.kernel.org; Mon, 18 Jun 2018 21:53:05 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44419) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fV5gY-0008SJ-SV for qemu-devel@nongnu.org; Mon, 18 Jun 2018 21:43:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fV5gU-0007bg-An for qemu-devel@nongnu.org; Mon, 18 Jun 2018 21:43:54 -0400 Received: from mail-ot0-x242.google.com ([2607:f8b0:4003:c0f::242]:36216) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fV5gU-0007ar-3k; Mon, 18 Jun 2018 21:43:50 -0400 Received: by mail-ot0-x242.google.com with SMTP id c15-v6so20774662otl.3; Mon, 18 Jun 2018 18:43:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=lskHDsTshqm8kYT9BCkBdOyQU+5alT3xAZIZr1jbusk=; b=KcunkjlNljBT1UIhYdWLykUYvUghatIclLIHll1d4DBP0GT3PisjnbJY1WMqa8TMZR hvIllE0nY5bZE+2jp04TgR6/xQ7k8TD+UAUpcvfzVk13RB4pAFS91OxQNjtN37cYZY6j gwaylvtYSLxzFy5Q/Xoi32PD2iy12pOU3euZsFIbnfh5uo+0jJVFfoMlPn07wZVl0H3+ f8IU8IK4HLNm2SUujrjiFAfvCec8fmiZLVgGoShDZdyXQkk8AQJsYB0LnGCitMJld1Py LYU8QlINLboapNTzGA1phNnt7pNE39Scjq3QBUl1ExwR5GW/umNq/pbfIZxNc0qFm8Sv RZ5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=lskHDsTshqm8kYT9BCkBdOyQU+5alT3xAZIZr1jbusk=; b=thmeYiMXBGulFEbfg+tVgaq5PEzRPlCUKIFDxSHCSxjcLQvrM/E1jGsUfqblLN3LJE 4ihLfRyRGVDdJAFuy+0Lu3uydXTCjLxlKqZJtUkdo4pcQMvH+myvK/lDuUhh+6DNL/I2 wEIpNueU/mIGv9GLANXa2VUA1nq/uL7ryojRQM9cwGKjAVv5jqE4Ce5VadD+o4lUPI87 rI0QOYPcAMfKz3lvvcoUWaXJqhX7gHw7fk6919aE8QX4YVareKAB1ZvjHpvfL+7cVXvW mdwufNbsovGMbNQZK3pKiAGBRLaO3es0PUaX8tZuCs7pxgwnlAVvru68df0cumJ13Zrg 6BUA== X-Gm-Message-State: APt69E1gaNEIakZO/BPjvUItPwYuF7B49ODKwz8PvwU1+YmSHroRzios hYdL0v32L7em4fvgoIKmZTmVi0Mi X-Google-Smtp-Source: ADUXVKL286qVFcHOOQFlsQYJiQPUifBZBZVV/F6qSp3kGo3R3ry3pj4wmfXMXaXcJiabnqDlTSNPGA== X-Received: by 2002:a9d:55d9:: with SMTP id z25-v6mr8695642oti.131.1529372628780; Mon, 18 Jun 2018 18:43:48 -0700 (PDT) Received: from localhost (76-251-165-188.lightspeed.austtx.sbcglobal.net. [76.251.165.188]) by smtp.gmail.com with ESMTPSA id t4-v6sm16635209otd.40.2018.06.18.18.43.47 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 18 Jun 2018 18:43:47 -0700 (PDT) From: Michael Roth To: qemu-devel@nongnu.org Date: Mon, 18 Jun 2018 20:41:35 -0500 Message-Id: <20180619014319.28272-10-mdroth@linux.vnet.ibm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180619014319.28272-1-mdroth@linux.vnet.ibm.com> References: <20180619014319.28272-1-mdroth@linux.vnet.ibm.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4003:c0f::242 Subject: [Qemu-devel] [PATCH 009/113] spapr: use spapr->vsmt to compute VCPU ids X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: David Gibson , qemu-stable@nongnu.org, Greg Kurz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Greg Kurz Since the introduction of VSMT in 2.11, the spacing of VCPU ids between cores is controllable through a machine property instead of being only dictated by the SMT mode of the host: cpu->vcpu_id = (cc->core_id * spapr->vsmt / smp_threads) + i Until recently, the machine code would try to change the SMT mode of the host to be equal to VSMT or exit. This allowed the rest of the code to assume that kvmppc_smt_threads() == spapr->vsmt is always true. Recent commit "8904e5a75005 spapr: Adjust default VSMT value for better migration compatibility" relaxed the rule. If the VSMT mode cannot be set in KVM for some reasons, but the requested CPU topology is compatible with the current SMT mode, then we let the guest run with kvmppc_smt_threads() != spapr->vsmt. This breaks quite a few places in the code, in particular when calculating DRC indexes. This is what happens on a POWER host with subcores-per-core=2 (ie, supports up to SMT4) when passing the following topology: -smp threads=4,maxcpus=16 \ -device host-spapr-cpu-core,core-id=4,id=core1 \ -device host-spapr-cpu-core,core-id=8,id=core2 qemu-system-ppc64: warning: Failed to set KVM's VSMT mode to 8 (errno -22) This is expected since KVM is limited to SMT4, but the guest is started anyway because this topology can run on SMT4 even with a VSMT8 spacing. But when we look at the DT, things get nastier: cpus { ... ibm,drc-indexes = <0x4 0x10000000 0x10000004 0x10000008 0x1000000c>; This means that we have the following association: CPU core device | DRC | VCPU id -----------------+------------+--------- boot core | 0x10000000 | 0 core1 | 0x10000004 | 4 core2 | 0x10000008 | 8 core3 | 0x1000000c | 12 But since the spacing of VCPU ids is 8, the DRC for core1 points to a VCPU that doesn't exist, the DRC for core2 points to the first VCPU of core1 and and so on... ... PowerPC,POWER8@0 { ... ibm,my-drc-index = <0x10000000>; ... }; PowerPC,POWER8@8 { ... ibm,my-drc-index = <0x10000008>; ... }; PowerPC,POWER8@10 { ... No ibm,my-drc-index property for this core since 0x10000010 doesn't exist in ibm,drc-indexes above. ... }; }; ... interrupt-controller { ... ibm,interrupt-server-ranges = <0x0 0x10>; With a spacing of 8, the highest VCPU id for the given topology should be: 16 * 8 / 4 = 32 and not 16 ... linux,phandle = <0x7e7323b8>; interrupt-controller; }; And CPU hot-plug/unplug is broken: (qemu) device_del core1 pseries-hotplug-cpu: Cannot find CPU (drc index 10000004) to remove (qemu) device_del core2 cpu 4 (hwid 8) Ready to die... cpu 5 (hwid 9) Ready to die... cpu 6 (hwid 10) Ready to die... cpu 7 (hwid 11) Ready to die... These are the VCPU ids of core1 actually (qemu) device_add host-spapr-cpu-core,core-id=12,id=core3 (qemu) device_del core3 pseries-hotplug-cpu: Cannot find CPU (drc index 1000000c) to remove This patches all the code in hw/ppc/spapr.c to assume the VSMT spacing when manipulating VCPU ids. Fixes: 8904e5a75005 Signed-off-by: Greg Kurz Signed-off-by: David Gibson (cherry picked from commit 72194664c8a16b67865eb95054f984dd169cfa86) Signed-off-by: Greg Kurz Signed-off-by: Greg Kurz --- hw/ppc/spapr.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 6f78c1cf7a..094e75f97d 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -161,9 +161,9 @@ static void pre_2_10_vmstate_unregister_dummy_icp(int i) (void *)(uintptr_t) i); } -static inline int xics_max_server_number(void) +static int xics_max_server_number(sPAPRMachineState *spapr) { - return DIV_ROUND_UP(max_cpus * kvmppc_smt_threads(), smp_threads); + return DIV_ROUND_UP(max_cpus * spapr->vsmt, smp_threads); } static void xics_system_init(MachineState *machine, int nr_irqs, Error **errp) @@ -195,7 +195,7 @@ static void xics_system_init(MachineState *machine, int nr_irqs, Error **errp) if (smc->pre_2_10_has_unused_icps) { int i; - for (i = 0; i < xics_max_server_number(); i++) { + for (i = 0; i < xics_max_server_number(spapr); i++) { /* Dummy entries get deregistered when real ICPState objects * are registered during CPU core hotplug. */ @@ -338,7 +338,6 @@ static int spapr_fixup_cpu_dt(void *fdt, sPAPRMachineState *spapr) int ret = 0, offset, cpus_offset; CPUState *cs; char cpu_model[32]; - int smt = kvmppc_smt_threads(); uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)}; CPU_FOREACH(cs) { @@ -347,7 +346,7 @@ static int spapr_fixup_cpu_dt(void *fdt, sPAPRMachineState *spapr) int index = spapr_vcpu_id(cpu); int compat_smt = MIN(smp_threads, ppc_compat_max_threads(cpu)); - if ((index % smt) != 0) { + if (index % spapr->vsmt != 0) { continue; } @@ -614,7 +613,6 @@ static void spapr_populate_cpus_dt_node(void *fdt, sPAPRMachineState *spapr) CPUState *cs; int cpus_offset; char *nodename; - int smt = kvmppc_smt_threads(); cpus_offset = fdt_add_subnode(fdt, 0, "cpus"); _FDT(cpus_offset); @@ -632,7 +630,7 @@ static void spapr_populate_cpus_dt_node(void *fdt, sPAPRMachineState *spapr) DeviceClass *dc = DEVICE_GET_CLASS(cs); int offset; - if ((index % smt) != 0) { + if (index % spapr->vsmt != 0) { continue; } @@ -1105,7 +1103,7 @@ static void *spapr_build_fdt(sPAPRMachineState *spapr, _FDT(fdt_setprop_cell(fdt, 0, "#size-cells", 2)); /* /interrupt controller */ - spapr_dt_xics(xics_max_server_number(), fdt, PHANDLE_XICP); + spapr_dt_xics(xics_max_server_number(spapr), fdt, PHANDLE_XICP); ret = spapr_populate_memory(spapr, fdt); if (ret < 0) { @@ -2198,7 +2196,6 @@ static void spapr_init_cpus(sPAPRMachineState *spapr) MachineState *machine = MACHINE(spapr); MachineClass *mc = MACHINE_GET_CLASS(machine); const char *type = spapr_get_cpu_core_type(machine->cpu_type); - int smt = kvmppc_smt_threads(); const CPUArchIdList *possible_cpus; int boot_cores_nr = smp_cpus / smp_threads; int i; @@ -2233,7 +2230,7 @@ static void spapr_init_cpus(sPAPRMachineState *spapr) if (mc->has_hotpluggable_cpus) { spapr_dr_connector_new(OBJECT(spapr), TYPE_SPAPR_DRC_CPU, - (core_id / smp_threads) * smt); + (core_id / smp_threads) * spapr->vsmt); } if (i < boot_cores_nr) { @@ -3262,10 +3259,10 @@ static void spapr_core_unplug_request(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { + sPAPRMachineState *spapr = SPAPR_MACHINE(OBJECT(hotplug_dev)); int index; sPAPRDRConnector *drc; CPUCore *cc = CPU_CORE(dev); - int smt = kvmppc_smt_threads(); if (!spapr_find_cpu_slot(MACHINE(hotplug_dev), cc->core_id, &index)) { error_setg(errp, "Unable to find CPU core with core-id: %d", @@ -3277,7 +3274,7 @@ void spapr_core_unplug_request(HotplugHandler *hotplug_dev, DeviceState *dev, return; } - drc = spapr_drc_by_id(TYPE_SPAPR_DRC_CPU, index * smt); + drc = spapr_drc_by_id(TYPE_SPAPR_DRC_CPU, index * spapr->vsmt); g_assert(drc); spapr_drc_detach(drc); @@ -3296,7 +3293,6 @@ static void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, CPUState *cs = CPU(core->threads); sPAPRDRConnector *drc; Error *local_err = NULL; - int smt = kvmppc_smt_threads(); CPUArchId *core_slot; int index; bool hotplugged = spapr_drc_hotplugged(dev); @@ -3307,7 +3303,7 @@ static void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, cc->core_id); return; } - drc = spapr_drc_by_id(TYPE_SPAPR_DRC_CPU, index * smt); + drc = spapr_drc_by_id(TYPE_SPAPR_DRC_CPU, index * spapr->vsmt); g_assert(drc || !mc->has_hotpluggable_cpus);