From patchwork Fri Jun 17 06:36:35 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 9182685 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A3ED96075D for ; Fri, 17 Jun 2016 06:49:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8FFC021327 for ; Fri, 17 Jun 2016 06:49:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 846BA2839C; Fri, 17 Jun 2016 06:49:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 978D521327 for ; Fri, 17 Jun 2016 06:49:26 +0000 (UTC) Received: from localhost ([::1]:54268 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bDnan-0003Hd-ML for patchwork-qemu-devel@patchwork.kernel.org; Fri, 17 Jun 2016 02:49:25 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50370) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bDnN2-0005b2-O0 for qemu-devel@nongnu.org; Fri, 17 Jun 2016 02:35:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bDnMz-0008GH-5S for qemu-devel@nongnu.org; Fri, 17 Jun 2016 02:35:11 -0400 Received: from ozlabs.org ([2401:3900:2:1::2]:50890) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bDnMy-0008AQ-GA; Fri, 17 Jun 2016 02:35:09 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 3rW9Tv1vXTz9t2S; Fri, 17 Jun 2016 16:34:58 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1466145299; bh=9rRrrKbi1sTjoMrt68IcDnOiUE5d6ails+E+Ilr5+pQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=faUr4aW6wzPXU1s5f4vLWt1ogyVPTZJi7Xxq4+obnO1QVcWc6UUBofRs8CDXpL8W/ WqbN0IPV9c75WvVFXaSsVrhto5ep4FHKM9Kg6RY3F4t8Iv3jkPAre16tM/vUfFuwYL 3t/u91mU+2dIjUx2OHX3Lv30yITec4rCRmDXF49M= From: David Gibson To: peter.maydell@linaro.org Date: Fri, 17 Jun 2016 16:36:35 +1000 Message-Id: <1466145399-32209-15-git-send-email-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1466145399-32209-1-git-send-email-david@gibson.dropbear.id.au> References: <1466145399-32209-1-git-send-email-david@gibson.dropbear.id.au> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2401:3900:2:1::2 Subject: [Qemu-devel] [PULL 14/18] spapr: CPU hotplug support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, aik@ozlabs.ru, mdroth@linux.vnet.ibm.com, agraf@suse.de, qemu-ppc@nongnu.org, bharata@linux.vnet.ibm.com, imammedo@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Bharata B Rao Set up device tree entries for the hotplugged CPU core and use the exising RTAS event logging infrastructure to send CPU hotplug notification to the guest. Signed-off-by: Bharata B Rao Reviewed-by: David Gibson Signed-off-by: David Gibson --- hw/ppc/spapr.c | 78 +++++++++++++++++++++++++++++++++------- hw/ppc/spapr_cpu_core.c | 80 +++++++++++++++++++++++++++++++++++++++++ hw/ppc/spapr_events.c | 3 ++ hw/ppc/spapr_rtas.c | 24 +++++++++++++ include/hw/ppc/spapr.h | 2 ++ include/hw/ppc/spapr_cpu_core.h | 2 ++ 6 files changed, 176 insertions(+), 13 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 52e89af..c444a86 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -605,6 +605,16 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset, size_t page_sizes_prop_size; uint32_t vcpus_per_socket = smp_threads * smp_cores; uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)}; + sPAPRDRConnector *drc; + sPAPRDRConnectorClass *drck; + int drc_index; + + drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index); + if (drc) { + drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + drc_index = drck->get_index(drc); + _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index))); + } /* Note: we keep CI large pages off for now because a 64K capable guest * provisioned with large pages might otherwise try to map a qemu @@ -1005,6 +1015,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr, _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB)); } + if (smc->dr_cpu_enabled) { + int offset = fdt_path_offset(fdt, "/cpus"); + ret = spapr_drc_populate_dt(fdt, offset, NULL, + SPAPR_DR_CONNECTOR_TYPE_CPU); + if (ret < 0) { + error_report("Couldn't set up CPU DR device tree properties"); + exit(1); + } + } + _FDT((fdt_pack(fdt))); if (fdt_totalsize(fdt) > FDT_MAX_SIZE) { @@ -1775,21 +1795,30 @@ static void ppc_spapr_init(MachineState *machine) char *type = spapr_get_cpu_core_type(machine->cpu_model); spapr->cores = g_new0(Object *, spapr_max_cores); - for (i = 0; i < spapr_cores; i++) { + for (i = 0; i < spapr_max_cores; i++) { int core_dt_id = i * smt; - Object *core; - - if (!object_class_by_name(type)) { - error_report("Unable to find sPAPR CPU Core definition"); - exit(1); + sPAPRDRConnector *drc = + spapr_dr_connector_new(OBJECT(spapr), + SPAPR_DR_CONNECTOR_TYPE_CPU, core_dt_id); + + qemu_register_reset(spapr_drc_reset, drc); + + if (i < spapr_cores) { + char *type = spapr_get_cpu_core_type(machine->cpu_model); + Object *core; + + if (!object_class_by_name(type)) { + error_report("Unable to find sPAPR CPU Core definition"); + exit(1); + } + + core = object_new(type); + object_property_set_int(core, smp_threads, "nr-threads", + &error_fatal); + object_property_set_int(core, core_dt_id, CPU_CORE_PROP_CORE_ID, + &error_fatal); + object_property_set_bool(core, true, "realized", &error_fatal); } - - core = object_new(type); - object_property_set_int(core, smp_threads, "nr-threads", - &error_fatal); - object_property_set_int(core, core_dt_id, CPU_CORE_PROP_CORE_ID, - &error_fatal); - object_property_set_bool(core, true, "realized", &error_fatal); } g_free(type); } else { @@ -2211,6 +2240,27 @@ out: error_propagate(errp, local_err); } +void *spapr_populate_hotplug_cpu_dt(CPUState *cs, int *fdt_offset, + sPAPRMachineState *spapr) +{ + PowerPCCPU *cpu = POWERPC_CPU(cs); + DeviceClass *dc = DEVICE_GET_CLASS(cs); + int id = ppc_get_vcpu_dt_id(cpu); + void *fdt; + int offset, fdt_size; + char *nodename; + + fdt = create_device_tree(&fdt_size); + nodename = g_strdup_printf("%s@%x", dc->fw_name, id); + offset = fdt_add_subnode(fdt, 0, nodename); + + spapr_populate_cpu_dt(cs, fdt, offset, spapr); + g_free(nodename); + + *fdt_offset = offset; + return fdt; +} + static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2255,6 +2305,8 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, } spapr_memory_plug(hotplug_dev, dev, node, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) { + spapr_core_plug(hotplug_dev, dev, errp); } } diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c index d747c26..d5fa4e6 100644 --- a/hw/ppc/spapr_cpu_core.c +++ b/hw/ppc/spapr_cpu_core.c @@ -41,6 +41,8 @@ static void spapr_cpu_reset(void *opaque) void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp) { CPUPPCState *env = &cpu->env; + CPUState *cs = CPU(cpu); + int i; /* Set time-base frequency to 512 MHz */ cpu_ppc_tb_init(env, SPAPR_TIMEBASE_FREQ); @@ -58,9 +60,18 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp) } } + /* Set NUMA node for the added CPUs */ + for (i = 0; i < nb_numa_nodes; i++) { + if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) { + cs->numa_node = i; + break; + } + } + xics_cpu_setup(spapr->icp, cpu); qemu_register_reset(spapr_cpu_reset, cpu); + spapr_cpu_reset(cpu); } /* @@ -77,10 +88,74 @@ char *spapr_get_cpu_core_type(const char *model) return core_type; } +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(OBJECT(hotplug_dev)); + sPAPRMachineState *spapr = SPAPR_MACHINE(OBJECT(hotplug_dev)); + sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev)); + CPUCore *cc = CPU_CORE(dev); + CPUState *cs = CPU(core->threads); + sPAPRDRConnector *drc; + sPAPRDRConnectorClass *drck; + Error *local_err = NULL; + void *fdt = NULL; + int fdt_offset = 0; + int index; + int smt = kvmppc_smt_threads(); + + drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, cc->core_id); + index = cc->core_id / smt; + spapr->cores[index] = OBJECT(dev); + + if (!smc->dr_cpu_enabled) { + /* + * This is a cold plugged CPU core but the machine doesn't support + * DR. So skip the hotplug path ensuring that the core is brought + * up online with out an associated DR connector. + */ + return; + } + + g_assert(drc); + + /* + * Setup CPU DT entries only for hotplugged CPUs. For boot time or + * coldplugged CPUs DT entries are setup in spapr_finalize_fdt(). + */ + if (dev->hotplugged) { + fdt = spapr_populate_hotplug_cpu_dt(cs, &fdt_offset, spapr); + } + + drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err); + if (local_err) { + g_free(fdt); + spapr->cores[index] = NULL; + error_propagate(errp, local_err); + return; + } + + if (dev->hotplugged) { + /* + * Send hotplug notification interrupt to the guest only in case + * of hotplugged CPUs. + */ + spapr_hotplug_req_add_by_index(drc); + } else { + /* + * Set the right DRC states for cold plugged CPU. + */ + drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE); + drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED); + } +} + void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { MachineState *machine = MACHINE(OBJECT(hotplug_dev)); + sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(OBJECT(hotplug_dev)); sPAPRMachineState *spapr = SPAPR_MACHINE(OBJECT(hotplug_dev)); int spapr_max_cores = max_cpus / smp_threads; int index; @@ -95,6 +170,11 @@ void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, goto out; } + if (!smc->dr_cpu_enabled && dev->hotplugged) { + error_setg(&local_err, "CPU hotplug not supported for this machine"); + goto out; + } + if (cc->nr_threads != smp_threads) { error_setg(&local_err, "threads must be %d", smp_threads); goto out; diff --git a/hw/ppc/spapr_events.c b/hw/ppc/spapr_events.c index 049fb1b..af80992 100644 --- a/hw/ppc/spapr_events.c +++ b/hw/ppc/spapr_events.c @@ -449,6 +449,9 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action, case SPAPR_DR_CONNECTOR_TYPE_LMB: hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_MEMORY; break; + case SPAPR_DR_CONNECTOR_TYPE_CPU: + hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_CPU; + break; default: /* we shouldn't be signaling hotplug events for resources * that don't support them diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c index 43e2c68..dc058e5 100644 --- a/hw/ppc/spapr_rtas.c +++ b/hw/ppc/spapr_rtas.c @@ -36,6 +36,7 @@ #include "hw/ppc/spapr.h" #include "hw/ppc/spapr_vio.h" +#include "hw/ppc/ppc.h" #include "qapi-event.h" #include "hw/boards.h" @@ -164,6 +165,27 @@ static void rtas_query_cpu_stopped_state(PowerPCCPU *cpu_, rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); } +/* + * Set the timebase offset of the CPU to that of first CPU. + * This helps hotplugged CPU to have the correct timebase offset. + */ +static void spapr_cpu_update_tb_offset(PowerPCCPU *cpu) +{ + PowerPCCPU *fcpu = POWERPC_CPU(first_cpu); + + cpu->env.tb_env->tb_offset = fcpu->env.tb_env->tb_offset; +} + +static void spapr_cpu_set_endianness(PowerPCCPU *cpu) +{ + PowerPCCPU *fcpu = POWERPC_CPU(first_cpu); + PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(fcpu); + + if (!pcc->interrupts_big_endian(fcpu)) { + cpu->env.spr[SPR_LPCR] |= LPCR_ILE; + } +} + static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr, uint32_t token, uint32_t nargs, target_ulong args, @@ -200,6 +222,8 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr, env->nip = start; env->gpr[3] = r3; cs->halted = 0; + spapr_cpu_set_endianness(cpu); + spapr_cpu_update_tb_offset(cpu); qemu_cpu_kick(cs); diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 2a892bd..e1f8274 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -587,6 +587,8 @@ void spapr_hotplug_req_add_by_count(sPAPRDRConnectorType drc_type, void spapr_hotplug_req_remove_by_count(sPAPRDRConnectorType drc_type, uint32_t count); void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp); +void *spapr_populate_hotplug_cpu_dt(CPUState *cs, int *fdt_offset, + sPAPRMachineState *spapr); /* rtas-configure-connector state */ struct sPAPRConfigureConnectorState { diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h index 401381b..7cb0515 100644 --- a/include/hw/ppc/spapr_cpu_core.h +++ b/include/hw/ppc/spapr_cpu_core.h @@ -29,4 +29,6 @@ typedef struct sPAPRCPUCore { void spapr_core_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp); char *spapr_get_cpu_core_type(const char *model); +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp); #endif