From patchwork Fri Mar 4 06:54:19 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bharata B Rao X-Patchwork-Id: 8499481 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 44E18C0553 for ; Fri, 4 Mar 2016 06:55:25 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 37FA120218 for ; Fri, 4 Mar 2016 06:55:24 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F0246201FE for ; Fri, 4 Mar 2016 06:55:22 +0000 (UTC) Received: from localhost ([::1]:39097 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abjdy-0002GQ-Dw for patchwork-qemu-devel@patchwork.kernel.org; Fri, 04 Mar 2016 01:55:22 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57355) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abjdV-0001pF-9s for qemu-devel@nongnu.org; Fri, 04 Mar 2016 01:54:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1abjdT-0005Vt-G3 for qemu-devel@nongnu.org; Fri, 04 Mar 2016 01:54:53 -0500 Received: from e28smtp03.in.ibm.com ([125.16.236.3]:39377) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abjdS-0005VO-Qe for qemu-devel@nongnu.org; Fri, 04 Mar 2016 01:54:51 -0500 Received: from localhost by e28smtp03.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 4 Mar 2016 12:24:48 +0530 Received: from d28relay03.in.ibm.com (9.184.220.60) by e28smtp03.in.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 4 Mar 2016 12:24:47 +0530 X-IBM-Helo: d28relay03.in.ibm.com X-IBM-MailFrom: bharata@linux.vnet.ibm.com X-IBM-RcptTo: qemu-ppc@nongnu.org;qemu-devel@nongnu.org Received: from d28av03.in.ibm.com (d28av03.in.ibm.com [9.184.220.65]) by d28relay03.in.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u246slui4653368; Fri, 4 Mar 2016 12:24:47 +0530 Received: from d28av03.in.ibm.com (localhost [127.0.0.1]) by d28av03.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u246sivG030296; Fri, 4 Mar 2016 12:24:46 +0530 Received: from bharata.in.ibm.com ([9.124.212.244]) by d28av03.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u246sQDY029500; Fri, 4 Mar 2016 12:24:42 +0530 From: Bharata B Rao To: qemu-devel@nongnu.org Date: Fri, 4 Mar 2016 12:24:19 +0530 Message-Id: <1457074461-14285-9-git-send-email-bharata@linux.vnet.ibm.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1457074461-14285-1-git-send-email-bharata@linux.vnet.ibm.com> References: <1457074461-14285-1-git-send-email-bharata@linux.vnet.ibm.com> X-TM-AS-MML: disable x-cbid: 16030406-0009-0000-0000-00000AE3B3F2 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 125.16.236.3 Cc: mjrosato@linux.vnet.ibm.com, thuth@redhat.com, pkrempa@redhat.com, ehabkost@redhat.com, aik@ozlabs.ru, Bharata B Rao , armbru@redhat.com, agraf@suse.de, borntraeger@de.ibm.com, qemu-ppc@nongnu.org, pbonzini@redhat.com, imammedo@redhat.com, mdroth@linux.vnet.ibm.com, afaerber@suse.de, david@gibson.dropbear.id.au Subject: [Qemu-devel] [RFC PATCH v1 08/10] spapr: CPU hotplug support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Set up device tree entries for the hotplugged CPU core and use the exising EPOW event infrastructure to send CPU hotplug notification to the guest. Signed-off-by: Bharata B Rao --- hw/ppc/spapr.c | 73 ++++++++++++++++++++++++++++++++++++++++- hw/ppc/spapr_cpu_core.c | 60 +++++++++++++++++++++++++++++++++ hw/ppc/spapr_events.c | 3 ++ hw/ppc/spapr_rtas.c | 24 ++++++++++++++ include/hw/ppc/spapr.h | 4 +++ include/hw/ppc/spapr_cpu_core.h | 2 ++ 6 files changed, 165 insertions(+), 1 deletion(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 5acb612..6c4ac50 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -603,6 +603,18 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, int offset, size_t page_sizes_prop_size; uint32_t vcpus_per_socket = smp_threads * smp_cores; uint32_t pft_size_prop[] = {0, cpu_to_be32(spapr->htab_shift)}; + sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine()); + sPAPRDRConnector *drc; + sPAPRDRConnectorClass *drck; + int drc_index; + + if (smc->dr_cpu_enabled) { + drc = spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, index); + g_assert(drc); + drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + drc_index = drck->get_index(drc); + _FDT((fdt_setprop_cell(fdt, offset, "ibm,my-drc-index", drc_index))); + } /* Note: we keep CI large pages off for now because a 64K capable guest * provisioned with large pages might otherwise try to map a qemu @@ -987,6 +999,16 @@ static void spapr_finalize_fdt(sPAPRMachineState *spapr, _FDT(spapr_drc_populate_dt(fdt, 0, NULL, SPAPR_DR_CONNECTOR_TYPE_LMB)); } + if (smc->dr_cpu_enabled) { + int offset = fdt_path_offset(fdt, "/cpus"); + ret = spapr_drc_populate_dt(fdt, offset, NULL, + SPAPR_DR_CONNECTOR_TYPE_CPU); + if (ret < 0) { + error_report("Couldn't set up CPU DR device tree properties"); + exit(1); + } + } + _FDT((fdt_pack(fdt))); if (fdt_totalsize(fdt) > FDT_MAX_SIZE) { @@ -1181,7 +1203,7 @@ static void ppc_spapr_reset(void) } -static void spapr_cpu_reset(void *opaque) +void spapr_cpu_reset(void *opaque) { sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine()); PowerPCCPU *cpu = opaque; @@ -1622,6 +1644,8 @@ static void spapr_boot_set(void *opaque, const char *boot_device, void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp) { CPUPPCState *env = &cpu->env; + CPUState *cs = CPU(cpu); + int i; /* Set time-base frequency to 512 MHz */ cpu_ppc_tb_init(env, TIMEBASE_FREQ); @@ -1646,6 +1670,14 @@ void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp) } } + /* Set NUMA node for the added CPUs */ + for (i = 0; i < nb_numa_nodes; i++) { + if (test_bit(cs->cpu_index, numa_info[i].node_cpu)) { + cs->numa_node = i; + break; + } + } + xics_cpu_setup(spapr->icp, cpu); qemu_register_reset(spapr_cpu_reset, cpu); } @@ -1768,6 +1800,7 @@ static void ppc_spapr_init(MachineState *machine) char *filename; int spapr_cores = smp_cpus / smp_threads; int spapr_max_cores = max_cpus / smp_threads; + int smt = kvmppc_smt_threads(); if (smp_cpus % smp_threads) { error_report("smp_cpus (%u) must be multiple of threads (%u)", @@ -1834,6 +1867,15 @@ static void ppc_spapr_init(MachineState *machine) spapr_validate_node_memory(machine, &error_fatal); } + if (smc->dr_cpu_enabled) { + for (i = 0; i < spapr_max_cores; i++) { + sPAPRDRConnector *drc = + spapr_dr_connector_new(OBJECT(spapr), + SPAPR_DR_CONNECTOR_TYPE_CPU, i * smt); + qemu_register_reset(spapr_drc_reset, drc); + } + } + /* init CPUs */ if (machine->cpu_model == NULL) { machine->cpu_model = kvm_enabled() ? "host" : "POWER7"; @@ -2267,6 +2309,27 @@ out: error_propagate(errp, local_err); } +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs, + int *fdt_offset, sPAPRMachineState *spapr) +{ + PowerPCCPU *cpu = POWERPC_CPU(cs); + DeviceClass *dc = DEVICE_GET_CLASS(cs); + int id = ppc_get_vcpu_dt_id(cpu); + void *fdt; + int offset, fdt_size; + char *nodename; + + fdt = create_device_tree(&fdt_size); + nodename = g_strdup_printf("%s@%x", dc->fw_name, id); + offset = fdt_add_subnode(fdt, 0, nodename); + + spapr_populate_cpu_dt(cs, fdt, offset, spapr); + g_free(nodename); + + *fdt_offset = offset; + return fdt; +} + static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -2307,6 +2370,12 @@ static void spapr_machine_device_plug(HotplugHandler *hotplug_dev, } spapr_memory_plug(hotplug_dev, dev, node, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_SPAPR_CPU_CORE)) { + if (!smc->dr_cpu_enabled && dev->hotplugged) { + error_setg(errp, "CPU hotplug not supported for this machine"); + return; + } + spapr_core_plug(hotplug_dev, dev, errp); } } @@ -2366,6 +2435,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data) mc->cpu_index_to_socket_id = spapr_cpu_index_to_socket_id; smc->dr_lmb_enabled = true; + smc->dr_cpu_enabled = true; fwc->get_dev_path = spapr_get_fw_dev_path; nc->nmi_monitor_handler = spapr_nmi; } @@ -2445,6 +2515,7 @@ static void spapr_machine_2_5_class_options(MachineClass *mc) spapr_machine_2_6_class_options(mc); smc->use_ohci_by_default = true; + smc->dr_cpu_enabled = false; SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_5); } diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c index 9ddf3ce..4c233d7 100644 --- a/hw/ppc/spapr_cpu_core.c +++ b/hw/ppc/spapr_cpu_core.c @@ -14,6 +14,65 @@ #include "qapi/visitor.h" #include +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp) +{ + sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(qdev_get_machine()); + sPAPRMachineState *ms = SPAPR_MACHINE(qdev_get_machine()); + sPAPRCPUCore *core = SPAPR_CPU_CORE(OBJECT(dev)); + PowerPCCPU *cpu = &core->threads[0]; + CPUState *cs = CPU(cpu); + int id = ppc_get_vcpu_dt_id(cpu); + sPAPRDRConnector *drc = + spapr_dr_connector_by_id(SPAPR_DR_CONNECTOR_TYPE_CPU, id); + sPAPRDRConnectorClass *drck; + Error *local_err = NULL; + void *fdt = NULL; + int fdt_offset = 0; + + if (!smc->dr_cpu_enabled) { + /* + * This is a cold plugged CPU core but the machine doesn't support + * DR. So skip the hotplug path ensuring that the core is brought + * up online with out an associated DR connector. + */ + return; + } + + g_assert(drc); + + /* + * Setup CPU DT entries only for hotplugged CPUs. For boot time or + * coldplugged CPUs DT entries are setup in spapr_finalize_fdt(). + */ + if (dev->hotplugged) { + fdt = spapr_populate_hotplug_cpu_dt(dev, cs, &fdt_offset, ms); + dev->hotplugged = true; + } + + drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc); + drck->attach(drc, dev, fdt, fdt_offset, !dev->hotplugged, &local_err); + if (local_err) { + g_free(fdt); + error_propagate(errp, local_err); + return; + } + + if (dev->hotplugged) { + /* + * Send hotplug notification interrupt to the guest only in case + * of hotplugged CPUs. + */ + spapr_hotplug_req_add_by_index(drc); + } else { + /* + * Set the right DRC states for cold plugged CPU. + */ + drck->set_allocation_state(drc, SPAPR_DR_ALLOCATION_STATE_USABLE); + drck->set_isolation_state(drc, SPAPR_DR_ISOLATION_STATE_UNISOLATED); + } +} + static int spapr_cpu_core_realize_child(Object *child, void *opaque) { Error **errp = opaque; @@ -30,6 +89,7 @@ static int spapr_cpu_core_realize_child(Object *child, void *opaque) if (*errp) { return 1; } + spapr_cpu_reset(cpu); return 0; } diff --git a/hw/ppc/spapr_events.c b/hw/ppc/spapr_events.c index 39f4682..10340e1 100644 --- a/hw/ppc/spapr_events.c +++ b/hw/ppc/spapr_events.c @@ -437,6 +437,9 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action, case SPAPR_DR_CONNECTOR_TYPE_LMB: hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_MEMORY; break; + case SPAPR_DR_CONNECTOR_TYPE_CPU: + hp->hotplug_type = RTAS_LOG_V6_HP_TYPE_CPU; + break; default: /* we shouldn't be signaling hotplug events for resources * that don't support them diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c index b7c5ebd..cc0369e 100644 --- a/hw/ppc/spapr_rtas.c +++ b/hw/ppc/spapr_rtas.c @@ -34,6 +34,7 @@ #include "hw/ppc/spapr.h" #include "hw/ppc/spapr_vio.h" +#include "hw/ppc/ppc.h" #include "qapi-event.h" #include "hw/boards.h" @@ -161,6 +162,27 @@ static void rtas_query_cpu_stopped_state(PowerPCCPU *cpu_, rtas_st(rets, 0, RTAS_OUT_PARAM_ERROR); } +/* + * Set the timebase offset of the CPU to that of first CPU. + * This helps hotplugged CPU to have the correct timebase offset. + */ +static void spapr_cpu_update_tb_offset(PowerPCCPU *cpu) +{ + PowerPCCPU *fcpu = POWERPC_CPU(first_cpu); + + cpu->env.tb_env->tb_offset = fcpu->env.tb_env->tb_offset; +} + +static void spapr_cpu_set_endianness(PowerPCCPU *cpu) +{ + PowerPCCPU *fcpu = POWERPC_CPU(first_cpu); + PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(fcpu); + + if (!pcc->interrupts_big_endian(fcpu)) { + cpu->env.spr[SPR_LPCR] |= LPCR_ILE; + } +} + static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr, uint32_t token, uint32_t nargs, target_ulong args, @@ -197,6 +219,8 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, sPAPRMachineState *spapr, env->nip = start; env->gpr[3] = r3; cs->halted = 0; + spapr_cpu_set_endianness(cpu); + spapr_cpu_update_tb_offset(cpu); qemu_cpu_kick(cs); diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 06e6cec..d1a0af8 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -36,6 +36,7 @@ struct sPAPRMachineClass { /*< public >*/ bool dr_lmb_enabled; /* enable dynamic-reconfig/hotplug of LMBs */ + bool dr_cpu_enabled; /* enable dynamic-reconfig/hotplug of CPUs */ bool use_ohci_by_default; /* use USB-OHCI instead of XHCI */ }; @@ -589,6 +590,9 @@ void spapr_hotplug_req_add_by_count(sPAPRDRConnectorType drc_type, void spapr_hotplug_req_remove_by_count(sPAPRDRConnectorType drc_type, uint32_t count); void spapr_cpu_init(sPAPRMachineState *spapr, PowerPCCPU *cpu, Error **errp); +void spapr_cpu_reset(void *opaque); +void *spapr_populate_hotplug_cpu_dt(DeviceState *dev, CPUState *cs, + int *fdt_offset, sPAPRMachineState *spapr); /* rtas-configure-connector state */ struct sPAPRConfigureConnectorState { diff --git a/include/hw/ppc/spapr_cpu_core.h b/include/hw/ppc/spapr_cpu_core.h index ddaff6f..9879398 100644 --- a/include/hw/ppc/spapr_cpu_core.h +++ b/include/hw/ppc/spapr_cpu_core.h @@ -26,4 +26,6 @@ typedef struct sPAPRCPUCore { PowerPCCPU *threads; } sPAPRCPUCore; +void spapr_core_plug(HotplugHandler *hotplug_dev, DeviceState *dev, + Error **errp); #endif