From patchwork Fri Mar 11 04:54:36 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bharata B Rao X-Patchwork-Id: 8562151 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 42AF3C0553 for ; Fri, 11 Mar 2016 04:57:34 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 402FF20122 for ; Fri, 11 Mar 2016 04:57:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 15AA4200F0 for ; Fri, 11 Mar 2016 04:57:32 +0000 (UTC) Received: from localhost ([::1]:52720 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeF8l-0000SB-Ib for patchwork-qemu-devel@patchwork.kernel.org; Thu, 10 Mar 2016 23:57:31 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37890) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeF6b-00054E-Sw for qemu-devel@nongnu.org; Thu, 10 Mar 2016 23:55:19 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aeF6Z-0004Ck-Gu for qemu-devel@nongnu.org; Thu, 10 Mar 2016 23:55:17 -0500 Received: from e28smtp01.in.ibm.com ([125.16.236.1]:47766) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aeF6Y-0004Bv-Rj for qemu-devel@nongnu.org; Thu, 10 Mar 2016 23:55:15 -0500 Received: from localhost by e28smtp01.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Mar 2016 10:25:11 +0530 Received: from d28relay05.in.ibm.com (9.184.220.62) by e28smtp01.in.ibm.com (192.168.1.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 11 Mar 2016 10:25:08 +0530 X-IBM-Helo: d28relay05.in.ibm.com X-IBM-MailFrom: bharata@linux.vnet.ibm.com X-IBM-RcptTo: qemu-ppc@nongnu.org;qemu-devel@nongnu.org Received: from d28av05.in.ibm.com (d28av05.in.ibm.com [9.184.220.67]) by d28relay05.in.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u2B4spYJ15401342; Fri, 11 Mar 2016 10:24:52 +0530 Received: from d28av05.in.ibm.com (localhost [127.0.0.1]) by d28av05.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u2B4t538022091; Fri, 11 Mar 2016 10:25:07 +0530 Received: from bharata.in.ibm.com ([9.77.192.47]) by d28av05.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u2B4se3U021136; Fri, 11 Mar 2016 10:25:03 +0530 From: Bharata B Rao To: qemu-devel@nongnu.org Date: Fri, 11 Mar 2016 10:24:36 +0530 Message-Id: <1457672078-17307-8-git-send-email-bharata@linux.vnet.ibm.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1457672078-17307-1-git-send-email-bharata@linux.vnet.ibm.com> References: <1457672078-17307-1-git-send-email-bharata@linux.vnet.ibm.com> X-TM-AS-MML: disable x-cbid: 16031104-4790-0000-0000-00000DDBB789 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 125.16.236.1 Cc: mjrosato@linux.vnet.ibm.com, thuth@redhat.com, pkrempa@redhat.com, ehabkost@redhat.com, aik@ozlabs.ru, Bharata B Rao , armbru@redhat.com, agraf@suse.de, borntraeger@de.ibm.com, qemu-ppc@nongnu.org, pbonzini@redhat.com, imammedo@redhat.com, mdroth@linux.vnet.ibm.com, afaerber@suse.de, david@gibson.dropbear.id.au Subject: [Qemu-devel] [RFC PATCH v2 7/9] spapr: CPU hotplug support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Set up device tree entries for the hotplugged CPU core and use the exising RTAS event logging infrastructure to send CPU hotplug notification to the guest. Signed-off-by: Bharata B Rao --- hw/ppc/spapr.c | 64 ++++++++++++++++++++++++++++++++++++++ hw/ppc/spapr_cpu_core.c | 69 +++++++++++++++++++++++++++++++++++++++++ hw/ppc/spapr_events.c | 3 ++ hw/ppc/spapr_rtas.c | 24 ++++++++++++++ include/hw/ppc/spapr.h | 2 ++ include/hw/ppc/spapr_cpu_core.h | 2 ++ 6 files changed, 164 insertions(+) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index cffe8c8..822c87d 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset, size_t page_sizes_prop_size; uint32_t vcpus_per_socket = smp_threads * smp_cores; uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)}; + sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine()); + sPAPRDRConnector *drc; + sPAPRDRConnectorClass *drck; + int drc_index; + + if (smc->dr_cpu_enabled) { + drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index); + g_assert(drc); + drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + drc_index = drck->get_index(drc); + _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index))); + } /* Note: we keep CI large pages off for now because a 64K capable guest * provisioned with large pages might otherwise try to map a qemu @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr, _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB)); } + if (smc->dr_cpu_enabled) { + int offset = fdt_path_offset(fdt, "/cpus"); + ret = spapr_drc_populate_dt(fdt, offset, NULL, + SPAPR_DR_CONNECTOR_TYPE_CPU); + if (ret < 0) { + error_report("Couldn't set up CPU DR device tree properties"); + exit(1); + } + } + _FDT((fdt_pack(fdt))); if (fdt_totalsize(fdt) > FDT_MAX_SIZE) { @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device, void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp) { CPUPPCState *env = &cpu->env; + CPUState *cs = CPU(cpu); + int i; /* Set time-base frequency to 512 MHz */ cpu_ppc_tb_init(env, TIMEBASE_FREQ); @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp) } } + /* Set NUMA node for the added CPUs */ + for (i = 0; i < nb_numa_nodes; i++) { + if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) { + cs->numa_node = i; + break; + } + } + xics_cpu_setup(spapr->icp, cpu); qemu_register_reset(spapr_cpu_reset, cpu); @@ -1824,6 +1856,11 @@ static void ppc_spapr_init(MachineState *machine) for (i = 0; i < spapr_max_cores; i++) { int core_dt_id = i * smt; + sPAPRDRConnector *drc = + spapr_dr_connector_new(OBJECT(spapr), + SPAPR_DR_CONNECTOR_TYPE_CPU, core_dt_id); + + qemu_register_reset(spapr_drc_reset, drc); if (i < spapr_cores) { Object *core = object_new(TYPE_SPAPR_CPU_CORE); @@ -2246,6 +2283,27 @@ out: error_propagate(errp, local_err); } +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs, + int *fdt_offset, sPAPRMachineState *spapr) +{ + PowerPCCPU *cpu = POWERPC_CPU(cs); + DeviceClass *dc = DEVICE_GET_CLASS(cs); + int id = ppc_get_vcpu_dt_id(cpu); + void *fdt; + int offset, fdt_size; + char *nodename; + + fdt = create_device_tree(&fdt_size); + nodename = g_strdup_printf("%s@%x", dc->fw_name, id); + offset = fdt_add_subnode(fdt, 0, nodename); + + spapr_populate_cpu_dt(cs, fdt, offset, spapr); + g_free(nodename); + + *fdt_offset = offset; + return fdt; +} + static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2286,6 +2344,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, } spapr_memory_plug(hotplug_dev, dev, node, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) { + if (!smc->dr_cpu_enabled && dev->hotplugged) { + error_setg(errp, "CPU hotplug not supported for this machine"); + return; + } + spapr_core_plug(hotplug_dev, dev, errp); } } diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c index 8c6d71d..db8de32 100644 --- a/hw/ppc/spapr_cpu_core.c +++ b/hw/ppc/spapr_cpu_core.c @@ -15,6 +15,75 @@ #include #include "target-ppc/kvm_ppc.h" +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine()); + sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine()); + sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev)); + CPUState *cs = CPU(&core->threads[0]); + sPAPRDRConnector *drc; + sPAPRDRConnectorClass *drck; + Error *local_err = NULL; + void *fdt = NULL; + int fdt_offset = 0; + int core_id, core_dt_id; + int smt = kvmppc_smt_threads(); + + /* TODO: Should we cache core_id in sPAPRCPUCore ? */ + core_dt_id = object_property_get_int(OBJECT(dev), "core", &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, core_dt_id); + core_id = core_dt_id / smt; + spapr->cores[core_id] = OBJECT(dev); + + if (!smc->dr_cpu_enabled) { + /* + * This is a cold plugged CPU core but the machine doesn't support + * DR. So skip the hotplug path ensuring that the core is brought + * up online with out an associated DR connector. + */ + return; + } + + g_assert(drc); + + /* + * Setup CPU DT entries only for hotplugged CPUs. For boot time or + * coldplugged CPUs DT entries are setup in spapr_finalize_fdt(). + */ + if (dev->hotplugged) { + fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, spapr); + dev->hotplugged = true; + } + + drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err); + if (local_err) { + g_free(fdt); + spapr->cores[core_id] = NULL; + error_propagate(errp, local_err); + return; + } + + if (dev->hotplugged) { + /* + * Send hotplug notification interrupt to the guest only in case + * of hotplugged CPUs. + */ + spapr_hotplug_req_add_by_index(drc); + } else { + /* + * Set the right DRC states for cold plugged CPU. + */ + drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE); + drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED); + } +} + static void spapr_cpu_core_create_threads(sPAPRCPUCore *core, int threads, Error **errp) { diff --git a/hw/ppc/spapr_events.c b/hw/ppc/spapr_events.c index 39f4682..10340e1 100644 --- a/hw/ppc/spapr_events.c +++ b/hw/ppc/spapr_events.c @@ -437,6 +437,9 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action, case SPAPR_DR_CONNECTOR_TYPE_LMB: hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_MEMORY; break; + case SPAPR_DR_CONNECTOR_TYPE_CPU: + hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_CPU; + break; default: /* we shouldn't be signaling hotplug events for resources * that don't support them diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c index b7c5ebd..cc0369e 100644 --- a/hw/ppc/spapr_rtas.c +++ b/hw/ppc/spapr_rtas.c @@ -34,6 +34,7 @@ #include "hw/ppc/spapr.h" #include "hw/ppc/spapr_vio.h" +#include "hw/ppc/ppc.h" #include "qapi-event.h" #include "hw/boards.h" @@ -161,6 +162,27 @@ static void rtas_query_cpu_stopped_state(PowerPCCPU *cpu_, rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); } +/* + * Set the timebase offset of the CPU to that of first CPU. + * This helps hotplugged CPU to have the correct timebase offset. + */ +static void spapr_cpu_update_tb_offset(PowerPCCPU *cpu) +{ + PowerPCCPU *fcpu = POWERPC_CPU(first_cpu); + + cpu->env.tb_env->tb_offset = fcpu->env.tb_env->tb_offset; +} + +static void spapr_cpu_set_endianness(PowerPCCPU *cpu) +{ + PowerPCCPU *fcpu = POWERPC_CPU(first_cpu); + PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(fcpu); + + if (!pcc->interrupts_big_endian(fcpu)) { + cpu->env.spr[SPR_LPCR] |= LPCR_ILE; + } +} + static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr, uint32_t token, uint32_t nargs, target_ulong args, @@ -197,6 +219,8 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr, env->nip = start; env->gpr[3] = r3; cs->halted = 0; + spapr_cpu_set_endianness(cpu); + spapr_cpu_update_tb_offset(cpu); qemu_cpu_kick(cs); diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index c099c3c..8957072 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -589,6 +589,8 @@ void spapr_hotplug_req_remove_by_count(sPAPRDRConnectorType drc_type, uint32_t count); void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp); void spapr_cpu_reset(void *opaque); +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs, + int *fdt_offset, sPAPRMachineState *spapr); /* rtas-configure-connector state */ struct sPAPRConfigureConnectorState { diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h index 48fb76a..980d8ae 100644 --- a/include/hw/ppc/spapr_cpu_core.h +++ b/include/hw/ppc/spapr_cpu_core.h @@ -25,4 +25,6 @@ typedef struct sPAPRCPUCore { PowerPCCPU *threads; } sPAPRCPUCore; +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp); #endif