From patchwork Wed Jun 16 01:19:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12323705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90C0FC48BDF for ; Wed, 16 Jun 2021 01:29:15 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 13CF761159 for ; Wed, 16 Jun 2021 01:29:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13CF761159 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58242 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKMk-0007nq-7H for qemu-devel@archiver.kernel.org; Tue, 15 Jun 2021 21:29:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltKDr-0003FN-Sn; Tue, 15 Jun 2021 21:20:03 -0400 Received: from mail-qk1-x72e.google.com ([2607:f8b0:4864:20::72e]:34624) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ltKDq-0004Ev-1l; Tue, 15 Jun 2021 21:20:03 -0400 Received: by mail-qk1-x72e.google.com with SMTP id k11so994753qkk.1; Tue, 15 Jun 2021 18:20:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AqJ/CANn+mLhqYcCkmI2qej991jLbuz2Di1qLnOGcV8=; b=YZtgJWh99NgYWFpDKsnPgduJMkYDAIof+Z6BzcWYf5ABL1LvjwFyLbPKZTHyJtsIM4 OX5CZhvFGwiy0L8Mvbpc0btwAk8qT9jEhDqPQYcdb1rxeaJUWY7pqQRA1Gx+BOOnBqDv D07n+cj/R49TlEsEoYALB52wJjCPOU6ZpxyQElvBTBX1vTZHzoLhOtl0Pga/jINzFVaZ M7AJ7csAzxPm93bh8BcyfCNYXA/sMJDoPRpxY33RvIHV2njeSFb/CgEcdtxL2lWeiLVe s5NVPiA2iUj1yujDov89ku5tLDcHTmw60pG+w2jDSZiLaFo1icAydQoY0PQHPlM9L6qd awtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AqJ/CANn+mLhqYcCkmI2qej991jLbuz2Di1qLnOGcV8=; b=gyYaTIDC/XPdUCn1VaF2KjhS/+2ZHDN1MbTnRNo9Pot16SiHDSSZ9J27curfpp4p3U TxQlHoO0X0Rg3NnpFExXT+c6zJWpkjOAYFWQ1XD8KDe4nQCLPSUwRgwOQ332rBOuuv7U 6bG35Zko25ko2Ckjdh/9A9fOxM6IXLoara4oK/jFQsHlc8+wjO8idbAv1acwoEhBCjAa uRO3vXBbegbI4zpzK0WqbafxxL4dLwhEeghHxFTDh1hsQS5F7ObtgKRkifnSHWX/OfU2 w51Na6CvUAFRMfcJcwJa3GK299gIrJG71q+VAeGMvTKwi8jEuJboZYTDxMxy7h85sch0 cjmA== X-Gm-Message-State: AOAM530cxTXd0IUhV22UmCu5itCI/+9mM2wB0dBuj1SHmOpdw2b13vxX TCXJXpqlxuWIJHjSHS8wQofV2Rlx4uE= X-Google-Smtp-Source: ABdhPJzMTSkCr8et+f/QjsKhXhavnF1mCTf2JtvG9KPYzqroVzpP3w8sApX6sh0z9FPLsUkj94nesQ== X-Received: by 2002:a05:620a:1129:: with SMTP id p9mr2632100qkk.163.1623806399735; Tue, 15 Jun 2021 18:19:59 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7c6:4d9a:784f:a7de:c020:5f1e]) by smtp.gmail.com with ESMTPSA id f19sm627994qkg.70.2021.06.15.18.19.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 18:19:59 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 1/7] spapr_numa.c: split FORM1 code into helpers Date: Tue, 15 Jun 2021 22:19:38 -0300 Message-Id: <20210616011944.2996399-2-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210616011944.2996399-1-danielhb413@gmail.com> References: <20210616011944.2996399-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::72e; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x72e.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.ibm.com, Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The upcoming FORM2 NUMA affinity will support asymmetric NUMA topologies and doesn't need be concerned with all the legacy support for older pseries FORM1 guests. We're also not going to calculate associativity domains based on numa distance (via spapr_numa_define_associativity_domains) since the distances will be written directly into new DT properties. Let's split FORM1 code into its own functions to allow for easier insertion of FORM2 logic later on. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr_numa.c | 61 +++++++++++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 22 deletions(-) diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 779f18b994..04a86f9b5b 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -155,6 +155,32 @@ static void spapr_numa_define_associativity_domains(SpaprMachineState *spapr) } +/* + * Set NUMA machine state data based on FORM1 affinity semantics. + */ +static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, + MachineState *machine) +{ + bool using_legacy_numa = spapr_machine_using_legacy_numa(spapr); + + /* + * Legacy NUMA guests (pseries-5.1 and older, or guests with only + * 1 NUMA node) will not benefit from anything we're going to do + * after this point. + */ + if (using_legacy_numa) { + return; + } + + if (!spapr_numa_is_symmetrical(machine)) { + error_report("Asymmetrical NUMA topologies aren't supported " + "in the pSeries machine"); + exit(EXIT_FAILURE); + } + + spapr_numa_define_associativity_domains(spapr); +} + void spapr_numa_associativity_init(SpaprMachineState *spapr, MachineState *machine) { @@ -210,22 +236,7 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr, spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i); } - /* - * Legacy NUMA guests (pseries-5.1 and older, or guests with only - * 1 NUMA node) will not benefit from anything we're going to do - * after this point. - */ - if (using_legacy_numa) { - return; - } - - if (!spapr_numa_is_symmetrical(machine)) { - error_report("Asymmetrical NUMA topologies aren't supported " - "in the pSeries machine"); - exit(EXIT_FAILURE); - } - - spapr_numa_define_associativity_domains(spapr); + spapr_numa_FORM1_affinity_init(spapr, machine); } void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, @@ -302,12 +313,8 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *fdt, return ret; } -/* - * Helper that writes ibm,associativity-reference-points and - * max-associativity-domains in the RTAS pointed by @rtas - * in the DT @fdt. - */ -void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) +static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, + void *fdt, int rtas) { MachineState *ms = MACHINE(spapr); SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr); @@ -365,6 +372,16 @@ void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) maxdomains, sizeof(maxdomains))); } +/* + * Helper that writes ibm,associativity-reference-points and + * max-associativity-domains in the RTAS pointed by @rtas + * in the DT @fdt. + */ +void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) +{ + spapr_numa_FORM1_write_rtas_dt(spapr, fdt, rtas); +} + static target_ulong h_home_node_associativity(PowerPCCPU *cpu, SpaprMachineState *spapr, target_ulong opcode, From patchwork Wed Jun 16 01:19:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12323707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06F9FC48BDF for ; Wed, 16 Jun 2021 01:31:06 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 44C8361356 for ; Wed, 16 Jun 2021 01:31:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 44C8361356 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:35984 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKOW-0003e5-A3 for qemu-devel@archiver.kernel.org; Tue, 15 Jun 2021 21:31:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53294) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltKDu-0003Mj-4j; Tue, 15 Jun 2021 21:20:06 -0400 Received: from mail-qt1-x82b.google.com ([2607:f8b0:4864:20::82b]:42518) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ltKDs-0004GE-1a; Tue, 15 Jun 2021 21:20:05 -0400 Received: by mail-qt1-x82b.google.com with SMTP id v6so603854qta.9; Tue, 15 Jun 2021 18:20:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3xyanZGLhmsSqK7jaC1yFAjCkmQEu0N/Cuc+eCkqoiA=; b=CdvjTcl6kA01zuZfab+UXuf408zXsmh0dVBb+i3p5ndWhzJDI8e1C1qQk1gqk47h74 XI5BWTHweARe3BacyMAsuW55YrCj2egAISScvDtDjvi7lFL2AyU4z7ZXzuH3TYdZp9AA 7fMGg6YpqfWT5vsEeTpQ+1kJ+wO+leBEtdVnF/WzPgYJiiALoiLsgPofHYmaBeq++H6q vkmu8wOEBgHMzOQrpjtOXWLNOfEWOTRG/XVyYc/ppIkIRTxOzmxSR5FjYu2Qc/AxZUJL wGpB0PgqjPird0/+8uoLsxPCRvC6rZDki6jDvkq+RlLH8IxRJAMuKCWPoqW7DkEvkzfL Bkhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3xyanZGLhmsSqK7jaC1yFAjCkmQEu0N/Cuc+eCkqoiA=; b=LIuJr2S7seYFZTz0GEqKIfyU+NCAf62QxQsXXdde8rC8Nrm4EXLR9gwLfWwK03qFx1 bE/W8FeF6/zGX0n+cHTNqBTq9c87SzJH+pjx3ZCmnCzbcbhYnnXM2sZh/jI8OQYBS2dN fslCHG+hIjXVTPg3dVTaE8JfCKfW/Ff6vpfV0FjErUVnxq3guVA4OU2cw68ooXAAm43h eyAc0XwgnAN8W6+YfaHEHxkH882UR5ImoD0yRIlayNQfYX+EOkv6avFBtbZXGJ3TPaPf mHj0CjXhjmFLTJ0NXkMS3jX5XcRioOjdjhgH9uyLlz1BnxDNFxaPXa0eOhoAfQ5VKJ+e XJ0Q== X-Gm-Message-State: AOAM531lOW6y8WtizFSoAkD7eJ2a42ffU5oTlT+Qqk8rDXOVYdEXsxV0 F+mzR9oBK3OvVuD6RDtya+DDlnyN1Ok= X-Google-Smtp-Source: ABdhPJzv14fnlyLwADBZf2sjRljNUNwQmXR+/2b/9BroOyDKT7Ez41NS+H9He6GqMbBdNwfTJh6dNA== X-Received: by 2002:ac8:5c48:: with SMTP id j8mr2580918qtj.154.1623806402291; Tue, 15 Jun 2021 18:20:02 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7c6:4d9a:784f:a7de:c020:5f1e]) by smtp.gmail.com with ESMTPSA id f19sm627994qkg.70.2021.06.15.18.20.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 18:20:02 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 2/7] spapr: move NUMA data init to post-CAS Date: Tue, 15 Jun 2021 22:19:39 -0300 Message-Id: <20210616011944.2996399-3-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210616011944.2996399-1-danielhb413@gmail.com> References: <20210616011944.2996399-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::82b; envelope-from=danielhb413@gmail.com; helo=mail-qt1-x82b.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.ibm.com, Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The pSeries machine will support a new NUMA affinity form, FORM2. This new FORM will be negotiated via ibm,architecture-vec5 during CAS. All artifacts and assumptions that are currently on use for FORM1 affinity will be deprecated in a guest that chooses to use FORM2. This means that we're going to wait until CAS to determine whether we're going to use FORM1 and FORM2. This patch does that by moving all NUMA data init functions to post-CAS time. spapr_numa_associativity_init() is moved from spapr_machine_init() to do_client_architecture_support(). Straightforward change since the initialization of spapr->numa_assoc_array is transparent to the guest. spapr_numa_write_rtas_dt() is more complex. The function is called from spapr_dt_rtas(), which in turned is called by spapr_build_fdt(). spapr_build_fdt() is called in two places: spapr_machine_reset() and do_client_architecture_support(). This means that we're writing RTAS nodes with NUMA artifacts without going through CAS first, and then writing it again post CAS. This is not an issue because, at this moment, we always write the same FORM1 NUMA affinity properties in the DT. With the upcoming FORM2 support, we're now reliant on guest choice to decide what to write. The proposed solution is to change spapr_numa_write_rtas_dt() to not write the DT until we're post-CAS. This is a benign guest visible change (a well behaved guest wouldn't bother to read NUMA properties before CAs), but to be on the safe side, let's wrap it with a machine class flag to skip this logic unless we're running with the latest machine type. This also means that FORM2 support will not be available for older machine types. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 6 +++--- hw/ppc/spapr_hcall.c | 4 ++++ hw/ppc/spapr_numa.c | 11 +++++++++++ include/hw/ppc/spapr.h | 1 + 4 files changed, 19 insertions(+), 3 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 4dd90b75cc..8c8029e190 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -2790,9 +2790,6 @@ static void spapr_machine_init(MachineState *machine) spapr->gpu_numa_id = spapr_numa_initial_nvgpu_numa_id(machine); - /* Init numa_assoc_array */ - spapr_numa_associativity_init(spapr, machine); - if ((!kvm_enabled() || kvmppc_has_cap_mmu_radix()) && ppc_type_check_compat(machine->cpu_type, CPU_POWERPC_LOGICAL_3_00, 0, spapr->max_compat_pvr)) { @@ -4645,8 +4642,11 @@ DEFINE_SPAPR_MACHINE(6_1, "6.1", true); */ static void spapr_machine_6_0_class_options(MachineClass *mc) { + SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc); + spapr_machine_6_1_class_options(mc); compat_props_add(mc->compat_props, hw_compat_6_0, hw_compat_6_0_len); + smc->pre_6_1_numa_affinity = true; } DEFINE_SPAPR_MACHINE(6_0, "6.0", false); diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index f25014afda..6d6ffcc92b 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -11,6 +11,7 @@ #include "helper_regs.h" #include "hw/ppc/spapr.h" #include "hw/ppc/spapr_cpu_core.h" +#include "hw/ppc/spapr_numa.h" #include "mmu-hash64.h" #include "cpu-models.h" #include "trace.h" @@ -1197,6 +1198,9 @@ target_ulong do_client_architecture_support(PowerPCCPU *cpu, spapr->cas_pre_isa3_guest = !spapr_ovec_test(ov1_guest, OV1_PPC_3_00); spapr_ovec_cleanup(ov1_guest); + /* Init numa_assoc_array */ + spapr_numa_associativity_init(spapr, MACHINE(spapr)); + /* * Ensure the guest asks for an interrupt mode we support; * otherwise terminate the boot. diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 04a86f9b5b..b0bd056546 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -379,6 +379,17 @@ static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, */ void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) { + SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr); + + /* + * pre-6.1 machine types were writing RTAS information before + * CAS. Check if that's case before changing their existing + * behavior. + */ + if (spapr_ovec_empty(spapr->ov5_cas) && !smc->pre_6_1_numa_affinity) { + return; + } + spapr_numa_FORM1_write_rtas_dt(spapr, fdt, rtas); } diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index f05219f75e..aca6832d1c 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -142,6 +142,7 @@ struct SpaprMachineClass { hwaddr rma_limit; /* clamp the RMA to this size */ bool pre_5_1_assoc_refpoints; bool pre_5_2_numa_associativity; + bool pre_6_1_numa_affinity; bool (*phb_placement)(SpaprMachineState *spapr, uint32_t index, uint64_t *buid, hwaddr *pio, From patchwork Wed Jun 16 01:19:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12323711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 020A9C48BE5 for ; Wed, 16 Jun 2021 01:31:49 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 533DE6115C for ; Wed, 16 Jun 2021 01:31:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 533DE6115C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37846 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKPD-0004wh-EH for qemu-devel@archiver.kernel.org; Tue, 15 Jun 2021 21:31:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53318) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltKDw-0003VH-T5; Tue, 15 Jun 2021 21:20:08 -0400 Received: from mail-qv1-xf36.google.com ([2607:f8b0:4864:20::f36]:34334) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ltKDu-0004Hz-8B; Tue, 15 Jun 2021 21:20:08 -0400 Received: by mail-qv1-xf36.google.com with SMTP id 5so769996qvf.1; Tue, 15 Jun 2021 18:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=p2HUBwJU498l6zjmIzeCkAS9jKLDEKJ6AtdgcGE3JCM=; b=N9ku13QWQ3ER9kOO346ynkSJI981s3KuPDQed1p6ZjGgx5DslDOzFyU7NRUY9VAtKs R0O4ocwfvv6I9X77CqXwzTxvSzXb7KnDvu042OTG1K9H4y9mL6df49dtAgtumb+GiV/Q RzKF/HRFgLHViTLekgbWempbwTqFahA/qJKJuxOzLeuSTPrBejRezfomfNnQr5OaRcAf PuIGitf547hDrrLdivvBPs6gMq0kU5bgcSAlz/2ljubrzme0GCit0X6DMhPPw4m3yVmd NU/W77jYpJ4C+T6nR7o6mh8pCB8v2CuUD7+dDwOZooLVc3KnjCqcP6NqSLDn8yXe+Nlt ELVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=p2HUBwJU498l6zjmIzeCkAS9jKLDEKJ6AtdgcGE3JCM=; b=IERg/n34rIkN6uCqgHPBTlkaR3GB3WghcaIoBqlnQpRVhcYbcvpWs5lDDPlDm5SHNG KcMcYAl2n90ZPZui3TsNnUqk5o7TJZV4Rd9jrXMEB49OzJHyWOzKcXax+MxcYApGlXyo gp8LC+sNYTkvahgDaHULEcsGV7eklQLbrd7wkOMSyN72DXb3Vbuf3D2EyjR/yzZkV4XU Pn1GRmRrC1cABFRLnV53Jqy8ChI/8GVWuvdJL0HOlhjxNeTZm2A6TtVfQ1IBa/Sj6UNN ocW2xmcEmMxeIx1+nGTtTJJQppWhPp2MujsPp9V0XDaLeTXx2L5ZzlOVHKNF7tMNZhAE MtaA== X-Gm-Message-State: AOAM530sG5OH54p8cdPDIZZ5ewS9vXlxqjfffRR5zGjKpLNvhe3sBYF8 9WPOS9EMrM5DkqO76APpFmEAZ5sZ3B0= X-Google-Smtp-Source: ABdhPJyRCSre5+1Yb8hVAnZ792UmjbwPgIU3eK3D8ih6ZxqrTP4rjCuE3GvoCAs7Ypx3hBhD1AcxJA== X-Received: by 2002:a0c:b38c:: with SMTP id t12mr7970847qve.44.1623806404597; Tue, 15 Jun 2021 18:20:04 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7c6:4d9a:784f:a7de:c020:5f1e]) by smtp.gmail.com with ESMTPSA id f19sm627994qkg.70.2021.06.15.18.20.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 18:20:04 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 3/7] spapr_numa.c: base FORM2 NUMA affinity support Date: Tue, 15 Jun 2021 22:19:40 -0300 Message-Id: <20210616011944.2996399-4-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210616011944.2996399-1-danielhb413@gmail.com> References: <20210616011944.2996399-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::f36; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf36.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.ibm.com, Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The main feature of FORM2 affinity support is the separation of NUMA distances from ibm,associativity information. This allows for a more flexible and straightforward NUMA distance assignment without relying on complex associations between several levels of NUMA via ibm,associativity matches. Another feature is its extensibility. This base support contains the facilities for NUMA distance assignment, but in the future more facilities will be added for latency, performance, bandwidth and so on. This patch implements the base FORM2 affinity support as follows: - the use of FORM2 associativity is indicated by using bit 2 of byte 5 of ibm,architecture-vec-5. A FORM2 aware guest can choose to use FORM1 or FORM2 affinity. Setting both forms will default to FORM2. We rely on CAS to decide what to write in the DT, so all writing is postponed until client architecture support is done. This also mean that we're not advertising FORM2 for pseries-6.0 and older machine versions to prevent guest visible changes; - ibm,associativity-reference-points has a new semantic. Instead of being used to calculate distances via NUMA levels, the concept of subdomain was introduced. The primary domain is the first element of the array, secondary domain is the second element and so on. At this moment, the only usage of this semantic is with NVDIMMs. This will be explained further in the next patches; - two new RTAS DT artifacts are introduced: ibm,numa-lookup-index-table and ibm,numa-distance-table. The index table is used to list all the NUMA logical domains of the platform, in ascending order, and allows for spartial NUMA configurations (although QEMU ATM doesn't support that). ibm,numa-distance-table is an array that contains all the distances from the first NUMA node to all other nodes, then the second NUMA node distances to all other nodes and so on; - spapr_post_load changes: since we're adding a new NUMA affinity that isn't compatible with the existing one, migration must be handled accordingly because we can't be certain of whether the guest went through CAS in the source. The solution chosen is to initiate the NUMA associativity data in spapr_post_load() unconditionally. The worst case would be to write the DT twice if the guest is in pre-CAS stage. Otherwise, we're making sure that a FORM1 guest will have the spapr->numa_assoc_array initialized with the proper information based on user distance, something that we're not doing with FORM2. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 21 ++++++ hw/ppc/spapr_numa.c | 140 +++++++++++++++++++++++++++++++++++- include/hw/ppc/spapr_ovec.h | 1 + 3 files changed, 161 insertions(+), 1 deletion(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 8c8029e190..3f655d5084 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -1770,6 +1770,22 @@ static int spapr_post_load(void *opaque, int version_id) return err; } + /* + * NUMA data init is made in CAS time. There is no reliable + * way of telling whether the guest already went through CAS + * in the source due to how spapr_ov5_cas_needed works: a + * FORM1 guest can be migrated with ov5_cas empty regardless + * of going through CAS first. + * + * One solution is to always call numa_associativity_init. The + * downside is that a guest migrated before CAS will run + * numa_associativity_init again when going through it, but + * at least we're making sure spapr->numa_assoc_array will be + * initialized and hotplug operations won't fail in both before + * and after CAS migration cases. + */ + spapr_numa_associativity_init(spapr, MACHINE(spapr)); + return err; } @@ -2734,6 +2750,11 @@ static void spapr_machine_init(MachineState *machine) spapr_ovec_set(spapr->ov5, OV5_FORM1_AFFINITY); + /* Do not advertise FORM2 support for pseries-6.0 and older */ + if (!smc->pre_6_1_numa_affinity) { + spapr_ovec_set(spapr->ov5, OV5_FORM2_AFFINITY); + } + /* advertise support for dedicated HP event source to guests */ if (spapr->use_hotplug_event_source) { spapr_ovec_set(spapr->ov5, OV5_HP_EVT); diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index b0bd056546..4298a4ecdb 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -202,6 +202,16 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr, spapr->numa_assoc_array[i][0] = cpu_to_be32(MAX_DISTANCE_REF_POINTS); spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i); + /* + * For FORM2 affinity the initialization above is enough. No + * need to fill non-zero NUMA nodes with node_id because + * there is no associativity domain match to calculate + * NUMA distances in FORM2. + */ + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + continue; + } + /* * Fill all associativity domains of non-zero NUMA nodes with * node_id. This is required because the default value (0) is @@ -236,7 +246,16 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr, spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i); } - spapr_numa_FORM1_affinity_init(spapr, machine); + /* + * We test for !FORM2 instead of testing for FORM1 because, + * as per spapr_ov5_cas_needed, setting FORM1 is not enough + * to get ov5_cas migrated, but setting FORM2 is. Since we're + * dealing with either FORM1 or FORM2, test for the option + * that is guaranteed to be set after a migration. + */ + if (!spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + spapr_numa_FORM1_affinity_init(spapr, machine); + } } void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, @@ -313,6 +332,120 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *fdt, return ret; } +static void spapr_numa_FORM2_write_rtas_tables(SpaprMachineState *spapr, + void *fdt, int rtas) +{ + MachineState *ms = MACHINE(spapr); + NodeInfo *numa_info = ms->numa_state->nodes; + int nb_numa_nodes = ms->numa_state->num_nodes; + int distance_table_entries = nb_numa_nodes * nb_numa_nodes; + g_autofree uint32_t *lookup_index_table = NULL; + g_autofree uint32_t *distance_table = NULL; + int src, dst, i, distance_table_size; + uint8_t *node_distances; + + /* + * ibm,numa-lookup-index-table: array with length and a + * list of NUMA ids present in the guest. + */ + lookup_index_table = g_new0(uint32_t, nb_numa_nodes + 1); + lookup_index_table[0] = cpu_to_be32(nb_numa_nodes); + + for (i = 0; i < nb_numa_nodes; i++) { + lookup_index_table[i + 1] = cpu_to_be32(i); + } + + _FDT(fdt_setprop(fdt, rtas, "ibm,numa-lookup-index-table", + lookup_index_table, + (nb_numa_nodes + 1) * sizeof(uint32_t))); + + /* + * ibm,numa-distance-table: contains all node distances. First + * element is the size of the table as uint32, followed up + * by all the uint8 distances from the first NUMA node, then all + * distances from the second NUMA node and so on. + * + * ibm,numa-lookup-index-table is used by guest to navigate this + * array because NUMA ids can be sparse (node 0 is the first, + * node 8 is the second ...). + */ + distance_table = g_new0(uint32_t, distance_table_entries + 1); + distance_table[0] = cpu_to_be32(distance_table_entries); + + node_distances = (uint8_t *)&distance_table[1]; + i = 0; + + for (src = 0; src < nb_numa_nodes; src++) { + for (dst = 0; dst < nb_numa_nodes; dst++) { + node_distances[i++] = numa_info[src].distance[dst]; + } + } + + distance_table_size = distance_table_entries * sizeof(uint8_t) + + sizeof(uint32_t); + _FDT(fdt_setprop(fdt, rtas, "ibm,numa-distance-table", + distance_table, distance_table_size)); +} + +/* + * This helper could be compressed in a single function with + * FORM1 logic since we're setting the same DT values, with the + * difference being a call to spapr_numa_FORM2_write_rtas_tables() + * in the end. The separation was made to avoid clogging FORM1 code + * which already has to deal with compat modes from previous + * QEMU machine types. + */ +static void spapr_numa_FORM2_write_rtas_dt(SpaprMachineState *spapr, + void *fdt, int rtas) +{ + MachineState *ms = MACHINE(spapr); + uint32_t number_nvgpus_nodes = spapr->gpu_numa_id - + spapr_numa_initial_nvgpu_numa_id(ms); + + /* + * From the kernel docs: "With Form2 the primary domainID and + * secondary domainID are used to identify the NUMA nodes + * the kernel should use when using persistent memory devices." + * + * Persistent memory devices, such as spapr-nvdimm, will use the + * primary domainID to indicate the NUMA node number the guest + * OS should use when using the device as regular memory. The + * secondary domainID indicates the numa node number that should + * be used when the device is used as persistent memory. + * + * FORM2 specification allows for further subdomains such as + * tertiary and quaternary. To avoid having to update + * ibm,associativity-reference-points every time a new subdomain + * level is going to be used by the kernel, let's set all available + * subdomains QEMU is willing to support without increasing + * MAX_DISTANCE_REF_POINTS. + */ + uint32_t refpoints[] = { + cpu_to_be32(0x4), + cpu_to_be32(0x3), + cpu_to_be32(0x2), + cpu_to_be32(0x1), + }; + + uint32_t nr_refpoints = ARRAY_SIZE(refpoints); + uint32_t maxdomain = ms->numa_state->num_nodes + number_nvgpus_nodes; + uint32_t maxdomains[] = { + cpu_to_be32(4), + cpu_to_be32(maxdomain), + cpu_to_be32(maxdomain), + cpu_to_be32(maxdomain), + cpu_to_be32(maxdomain) + }; + + _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", + refpoints, nr_refpoints * sizeof(refpoints[0]))); + + _FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", + maxdomains, sizeof(maxdomains))); + + spapr_numa_FORM2_write_rtas_tables(spapr, fdt, rtas); +} + static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) { @@ -390,6 +523,11 @@ void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) return; } + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + spapr_numa_FORM2_write_rtas_dt(spapr, fdt, rtas); + return; + } + spapr_numa_FORM1_write_rtas_dt(spapr, fdt, rtas); } diff --git a/include/hw/ppc/spapr_ovec.h b/include/hw/ppc/spapr_ovec.h index 48b716a060..c3e8b98e7e 100644 --- a/include/hw/ppc/spapr_ovec.h +++ b/include/hw/ppc/spapr_ovec.h @@ -49,6 +49,7 @@ typedef struct SpaprOptionVector SpaprOptionVector; /* option vector 5 */ #define OV5_DRCONF_MEMORY OV_BIT(2, 2) #define OV5_FORM1_AFFINITY OV_BIT(5, 0) +#define OV5_FORM2_AFFINITY OV_BIT(5, 2) #define OV5_HP_EVT OV_BIT(6, 5) #define OV5_HPT_RESIZE OV_BIT(6, 7) #define OV5_DRMEM_V2 OV_BIT(22, 0) From patchwork Wed Jun 16 01:19:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12323713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0292FC48BDF for ; Wed, 16 Jun 2021 01:33:32 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8F6DC6115C for ; Wed, 16 Jun 2021 01:33:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F6DC6115C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42242 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKQs-0007yv-Kz for qemu-devel@archiver.kernel.org; Tue, 15 Jun 2021 21:33:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53342) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltKDy-0003am-UU; Tue, 15 Jun 2021 21:20:10 -0400 Received: from mail-qv1-xf2e.google.com ([2607:f8b0:4864:20::f2e]:43773) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ltKDw-0004JA-Ua; Tue, 15 Jun 2021 21:20:10 -0400 Received: by mail-qv1-xf2e.google.com with SMTP id e18so740021qvm.10; Tue, 15 Jun 2021 18:20:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E2AxgRPQ+y0q+DrPNTDfWMYzC9qLv0yalxAo5OxvugA=; b=UZibSLqsuK86RoXygrsu1SFy4KuzXufxkdpSjcm+voyGi+ImDpUbYTBwTDMIeCF9pV NFmAv4feM1S2KUWKIJ4pZVBvRmQ2Nxfdd3fJcvlJB2gjbxNGqXwhM1b7t5VveaV/qiDo w/Y5snHMYp0DKGwGQhTEZ8RLZTQy5eXxP8tLCLilnihDWSmzr9nNcexUzCmBzjYUUSAg SAsJbtjneIacsv8Dn1xX1LhOnRTXfdQP/nvTxyC4tN1wlHEzbFBu2UWbglLgshQajLmg HLkv2LiIOtjiAehK48Sg1SpGkr6let+KeCFs0LR7q+aezF5jOS+qtPowJ2m6LQYjHSlc Ft+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E2AxgRPQ+y0q+DrPNTDfWMYzC9qLv0yalxAo5OxvugA=; b=Mg4SaIyDqNh6gHPmS9M9FvS4MPPif9j1CpMTHeLeo2SqeHRC/j7Wt6lCULFAp8Knne qt6hHSYZc1+zEHBTsflDOaFB7yZmcBY41nEIFZqU8TIAKFhUmlBLxJERN69qxDmVRZIN cddOV2XmKfltZ9fR/iCf6sPSqBAyN+ZuOpGt2IU/EH9w0SBZXhBrvjD+uiuv1p0K/A7b MkAECcb+tIAw9saRtl8zlpQfMPhQEqKmCvujFaI01dfFYofLkmSRn5zwoS/Nj2B520n7 abFYjj8SY/ulZ7bRJSReezdWHDJOM0A6BpG1cTbO85lLGbXq1nNJlhqJM3LBa/amQi7/ mnUg== X-Gm-Message-State: AOAM532UyvbQ1NeZWhmLKQUSo3TdWc7LCQyuF3d/GPPwWMGLV2VFRONq mOjIbgtTWnfo/IMs+TK4XooHC/hp31E= X-Google-Smtp-Source: ABdhPJySKQzLtmGonH0wy9gtaQQtn4yRfYMi6iDheZDSGEkutTwXO+Yp2/iY2+RQCtfib+yxBc1wVQ== X-Received: by 2002:a05:6214:258c:: with SMTP id fq12mr8394499qvb.30.1623806407309; Tue, 15 Jun 2021 18:20:07 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7c6:4d9a:784f:a7de:c020:5f1e]) by smtp.gmail.com with ESMTPSA id f19sm627994qkg.70.2021.06.15.18.20.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 18:20:07 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 4/7] spapr: simplify spapr_numa_associativity_init params Date: Tue, 15 Jun 2021 22:19:41 -0300 Message-Id: <20210616011944.2996399-5-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210616011944.2996399-1-danielhb413@gmail.com> References: <20210616011944.2996399-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::f2e; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf2e.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.ibm.com, Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When spapr_numa_associativity_init() was introduced it was being called from spapr_machine_init(), where we have pointers to a SpaprMachineState and a MachineState. Having both being passed as params spared us from calling a macro to get the MachineState. Previous patches moved the function away from spapr_machine_init(), and the new locations doesn't have a pointer to MachineState ready. This resulted in MACHINE(spapr) macro calls as the second parameter in both callers. Simplify the function by folding the 'MACHINE(spapr)' macro into the function body, which can now receive only a pointer to SpaprMachineState. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 2 +- hw/ppc/spapr_hcall.c | 2 +- hw/ppc/spapr_numa.c | 4 ++-- include/hw/ppc/spapr_numa.h | 3 +-- 4 files changed, 5 insertions(+), 6 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 3f655d5084..b6b822a065 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -1784,7 +1784,7 @@ static int spapr_post_load(void *opaque, int version_id) * initialized and hotplug operations won't fail in both before * and after CAS migration cases. */ - spapr_numa_associativity_init(spapr, MACHINE(spapr)); + spapr_numa_associativity_init(spapr); return err; } diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index 6d6ffcc92b..73aca0aed6 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -1199,7 +1199,7 @@ target_ulong do_client_architecture_support(PowerPCCPU *cpu, spapr_ovec_cleanup(ov1_guest); /* Init numa_assoc_array */ - spapr_numa_associativity_init(spapr, MACHINE(spapr)); + spapr_numa_associativity_init(spapr); /* * Ensure the guest asks for an interrupt mode we support; diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 4298a4ecdb..8678ff4272 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -181,9 +181,9 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, spapr_numa_define_associativity_domains(spapr); } -void spapr_numa_associativity_init(SpaprMachineState *spapr, - MachineState *machine) +void spapr_numa_associativity_init(SpaprMachineState *spapr) { + MachineState *machine = MACHINE(spapr); SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr); int nb_numa_nodes = machine->numa_state->num_nodes; int i, j, max_nodes_with_gpus; diff --git a/include/hw/ppc/spapr_numa.h b/include/hw/ppc/spapr_numa.h index 6f9f02d3de..adaec8e163 100644 --- a/include/hw/ppc/spapr_numa.h +++ b/include/hw/ppc/spapr_numa.h @@ -22,8 +22,7 @@ * function. spapr_machine_init() is the only caller for it, and * it has both pointers resolved already. */ -void spapr_numa_associativity_init(SpaprMachineState *spapr, - MachineState *machine); +void spapr_numa_associativity_init(SpaprMachineState *spapr); void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas); void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, int offset, int nodeid); From patchwork Wed Jun 16 01:19:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12323723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00931C48BDF for ; Wed, 16 Jun 2021 01:37:52 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 95C03611BE for ; Wed, 16 Jun 2021 01:37:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 95C03611BE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58000 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKV4-00027E-OE for qemu-devel@archiver.kernel.org; Tue, 15 Jun 2021 21:37:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53370) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltKE3-0003kT-Ax; Tue, 15 Jun 2021 21:20:15 -0400 Received: from mail-qk1-x72a.google.com ([2607:f8b0:4864:20::72a]:34622) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ltKE0-0004LF-07; Tue, 15 Jun 2021 21:20:15 -0400 Received: by mail-qk1-x72a.google.com with SMTP id k11so995155qkk.1; Tue, 15 Jun 2021 18:20:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MKj6WbNGYNDETat7wPp4iyJPvKx6ZccNLrkYbFNYwsw=; b=TZTYmCO6KofkU1d5z1x4EJdQT0GRkIgAfllsn322IPw3Ix1ff1u949noA5P59lBv8C ArJ0w9if2ES4oYTEQfKDNj6uK1pfFiiHjzaGkR6aI50A0Vh3P+3CgQox7T/8rC3U/Ord GGtNHx3G1ucOEjrFmhCQnVOy+Rj1Z3++DVOOLmygBZAcX8zghh30Rf5jmtsVVG0OjVv7 1iMQ6Mmck0bE4RFMnmo3VPlKgBpxTSD9gzOH12M3811arXMedW6Yrxh/CJX5nNfWHcvf H8MfaUv7pd1fWg372MJ6kiZW2e/sj34Je2MKj7Y0CjPSAfeWHwd+ri3avhB5VYQXiA// wYvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MKj6WbNGYNDETat7wPp4iyJPvKx6ZccNLrkYbFNYwsw=; b=uM3WvzHZO4tJDJaS/AutrdfwKT2Sy975K7JotC5dkHgM5Z7Oe0lTH5RXURItQZ+F7w XB9FVvDEbU3cuU4W/vLmZsxtvYkhe9+abv0SdkcivX9GBS+a4lYpKbMeNwjIWP05Qf9J e0g86E1bQS57sGf5Sg5TmHwOjTj0l5hW0xt1D1gSUYj0KksuDDO4W6jAR3OEQf7sZKDL 3YUxIxuwIaKJDlv5LXDCtxba6mA9khFfqnlqIwSIouisAhlHZLd07qrdzrRi9lazdJuZ NVhd+RXHTOQS0dIRxkCLptYqJOZVyTLtrm5d4FAmgXTgF7IolyGbznP6htd82Pzr/O2+ iiAA== X-Gm-Message-State: AOAM533vimuOsETHFX0CGTdBk/X7O6kVEBoz1HBzsfINh607cT/j4Dhw ReOJ2RIdSyps4UQdnSX1BHydKJvGEr4= X-Google-Smtp-Source: ABdhPJwt/QEgyubEOVz+k+dW752PvDMG3wyULMoQSETgBUCsKF0UrmONj815vEjqa9fKj1n0Qjwgng== X-Received: by 2002:ae9:ed96:: with SMTP id c144mr2537017qkg.401.1623806410221; Tue, 15 Jun 2021 18:20:10 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7c6:4d9a:784f:a7de:c020:5f1e]) by smtp.gmail.com with ESMTPSA id f19sm627994qkg.70.2021.06.15.18.20.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 18:20:09 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 5/7] nvdimm: add PPC64 'device-node' property Date: Tue, 15 Jun 2021 22:19:42 -0300 Message-Id: <20210616011944.2996399-6-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210616011944.2996399-1-danielhb413@gmail.com> References: <20210616011944.2996399-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::72a; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x72a.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shivaprasad G Bhat , aneesh.kumar@linux.ibm.com, Daniel Henrique Barboza , groug@kaod.org, qemu-ppc@nongnu.org, Igor Mammedov , david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The spapr-nvdimm driver is able to operate in two ways. The first one is as a regular memory in which the NUMA node of the parent pc-dimm class is used. The second mode, as persistent memory, allows for a different NUMA node to be used based on the locality of the device. At this moment we don't have a way to express this second NUMA node for the persistent memory mode. This patch introduces a new 'device-node' property that will be used by the PPC64 spapr-nvdimm driver to set a second NUMA node for the nvdimm. CC: Shivaprasad G Bhat CC: Igor Mammedov Signed-off-by: Daniel Henrique Barboza --- hw/mem/nvdimm.c | 31 +++++++++++++++++++++++++++++++ include/hw/mem/nvdimm.h | 12 ++++++++++++ 2 files changed, 43 insertions(+) diff --git a/hw/mem/nvdimm.c b/hw/mem/nvdimm.c index 7397b67156..96298e3fda 100644 --- a/hw/mem/nvdimm.c +++ b/hw/mem/nvdimm.c @@ -96,15 +96,46 @@ static void nvdimm_set_uuid(Object *obj, Visitor *v, const char *name, g_free(value); } +static void nvdimm_get_device_node(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + NVDIMMDevice *nvdimm = NVDIMM(obj); + int64_t value = nvdimm->device_node; + + visit_type_int(v, name, &value, errp); +} + +static void nvdimm_set_device_node(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + NVDIMMDevice *nvdimm = NVDIMM(obj); + int64_t value; + + if (!visit_type_int(v, name, &value, errp)) { + return; + } + + nvdimm->device_node = value; +} static void nvdimm_init(Object *obj) { + NVDIMMDevice *nvdimm = NVDIMM(obj); + object_property_add(obj, NVDIMM_LABEL_SIZE_PROP, "int", nvdimm_get_label_size, nvdimm_set_label_size, NULL, NULL); object_property_add(obj, NVDIMM_UUID_PROP, "QemuUUID", nvdimm_get_uuid, nvdimm_set_uuid, NULL, NULL); + + nvdimm->device_node = -1; + object_property_add(obj, NVDIMM_DEVICE_NODE, "int", + nvdimm_get_device_node, + nvdimm_set_device_node, + NULL, NULL); } static void nvdimm_finalize(Object *obj) diff --git a/include/hw/mem/nvdimm.h b/include/hw/mem/nvdimm.h index bcf62f825c..e3298ef841 100644 --- a/include/hw/mem/nvdimm.h +++ b/include/hw/mem/nvdimm.h @@ -49,6 +49,7 @@ OBJECT_DECLARE_TYPE(NVDIMMDevice, NVDIMMClass, NVDIMM) #define NVDIMM_LABEL_SIZE_PROP "label-size" +#define NVDIMM_DEVICE_NODE "device-node" #define NVDIMM_UUID_PROP "uuid" #define NVDIMM_UNARMED_PROP "unarmed" @@ -89,6 +90,17 @@ struct NVDIMMDevice { * The PPC64 - spapr requires each nvdimm device have a uuid. */ QemuUUID uuid; + + /* + * The spapr-nvdimm (PPC64 NVDIMM) driver supports two modes of + * operation: regular memory and persistent memory. When using the + * device as regular memory, the NUMA nodeid that comes from + * PC_DIMM_NODEPROP is to be used. When used as persistent memory, + * the guest should consider the 'device-node' instead since it + * indicates the locality of the device to an established NUMA + * node, which is more relevant to this type of usage. + */ + int device_node; }; struct NVDIMMClass { From patchwork Wed Jun 16 01:19:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12323717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E239C48BDF for ; Wed, 16 Jun 2021 01:34:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9DFDA611BE for ; Wed, 16 Jun 2021 01:34:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9DFDA611BE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:46364 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKRt-0002O6-RO for qemu-devel@archiver.kernel.org; Tue, 15 Jun 2021 21:34:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53392) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltKE6-0003qO-RT; Tue, 15 Jun 2021 21:20:20 -0400 Received: from mail-qk1-x72b.google.com ([2607:f8b0:4864:20::72b]:45040) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ltKE3-0004Mu-Bo; Tue, 15 Jun 2021 21:20:18 -0400 Received: by mail-qk1-x72b.google.com with SMTP id c18so926759qkc.11; Tue, 15 Jun 2021 18:20:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qYzg2P4ERYOO3xCYvasy/0+KMqAX65ioY5WE3TNXiZw=; b=bkf9UolgTt5k5449PunXE6z0TgcC+6eHvby9qKPZp1ZgQX5+Fv8m3O5EHZpCVmr8EU MEUvXeZIEwO+Xx/I1tU++1eCXHLJbFnFSdDLspocWtqhPnMyVoqhbNSQbLHu1+Llup9H WRCOeYNhAEXW3abPPPZGPftPbGoO0Le7kp79nTJ4yRAS7pBaTJWHXU/7kMLT8+2LfGTZ WLesw9YhfIl+HAMaxtiCxDDYAlyqYIdaKzWtkXQ0ggUhF3ZIa7uBea0maFV1NqCg1xnK TdvYOfI39igqVVwWF2SGeZl4Zb1sBVdgABEDNXuPuImIH9tDEHAMbY98nQ0+WdIzi4No 45wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qYzg2P4ERYOO3xCYvasy/0+KMqAX65ioY5WE3TNXiZw=; b=nIeNbxIiwOCY+R2cpZr2KAmiU5X5wLmI7FNhIOW10jZElafz0EWLTzRTHQvTsrn/mY ZazsZXMLG6TwLDXBIs3F0TAG6qUaLiELQkEwsThjK5gAzJr4DLrOxpGho+iFPQXLjZGu laeW/118sbtT1z72MzzpTJftC6Lffp1ZZKZy+UuogcPvjEYolfwMWWogetjfEsI8ddOp HO04VMQDy/qIyXIX2rtFjT6DMfPX/1EKYVR7EbaaBPVEHvudXsL4hxHoAMdO4aQEf0Yo hc2WoSMICv0+49td2uoWDml6ZltMYfa3HMOzopH+9zTE8ecIcrK6SSt5gFlIlLP74I5H brHg== X-Gm-Message-State: AOAM533xPIhK5aLrdTjuVr9n0uzVStzYo7/XQcj9vBvUcF+j3znij+h5 +AlHMA3xFNwqQ6RqHEwCYNCBq99wsc8= X-Google-Smtp-Source: ABdhPJyj8eGBx2iT8JUgXlJTWeiDoHVvUXc5QohEFrOV8lAQz+07J57rSTIxtHiz/YqAaSy0JJzaLQ== X-Received: by 2002:a05:620a:414a:: with SMTP id k10mr2551446qko.37.1623806412736; Tue, 15 Jun 2021 18:20:12 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7c6:4d9a:784f:a7de:c020:5f1e]) by smtp.gmail.com with ESMTPSA id f19sm627994qkg.70.2021.06.15.18.20.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 18:20:12 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 6/7] spapr_numa, spapr_nvdimm: write secondary NUMA domain for nvdimms Date: Tue, 15 Jun 2021 22:19:43 -0300 Message-Id: <20210616011944.2996399-7-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210616011944.2996399-1-danielhb413@gmail.com> References: <20210616011944.2996399-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::72b; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x72b.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shivaprasad G Bhat , aneesh.kumar@linux.ibm.com, Daniel Henrique Barboza , groug@kaod.org, qemu-ppc@nongnu.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Using the new 'device-node' property, write it in the nvdimm DT to set a secondary domain for the persistent memory operation mode. If 'device-node' isn't set, secondary domain is equal to the primary domain. Note that this is only available in FORM2 affinity. FORM1 affinity NVDIMMs aren't affected by this change. CC: Shivaprasad G Bhat Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr_numa.c | 20 ++++++++++++++++++++ hw/ppc/spapr_nvdimm.c | 5 +++-- include/hw/ppc/spapr_numa.h | 3 +++ 3 files changed, 26 insertions(+), 2 deletions(-) diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 8678ff4272..e7d455d304 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -266,6 +266,26 @@ void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, sizeof(spapr->numa_assoc_array[nodeid])))); } +void spapr_numa_write_nvdimm_assoc_dt(SpaprMachineState *spapr, void *fdt, + int offset, int nodeid, + int device_node) +{ + uint32_t *nvdimm_assoc_array = spapr->numa_assoc_array[nodeid]; + + /* + * 'device-node' is the secondary domain for NVDIMMs when + * using FORM2. The secondary domain for FORM2 in QEMU + * is 0x3. + */ + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + nvdimm_assoc_array[0x3] = cpu_to_be32(device_node); + } + + _FDT((fdt_setprop(fdt, offset, "ibm,associativity", + nvdimm_assoc_array, + sizeof(spapr->numa_assoc_array[nodeid])))); +} + static uint32_t *spapr_numa_get_vcpu_assoc(SpaprMachineState *spapr, PowerPCCPU *cpu) { diff --git a/hw/ppc/spapr_nvdimm.c b/hw/ppc/spapr_nvdimm.c index 91de1052f2..7cc4e9a28f 100644 --- a/hw/ppc/spapr_nvdimm.c +++ b/hw/ppc/spapr_nvdimm.c @@ -92,7 +92,6 @@ bool spapr_nvdimm_validate(HotplugHandler *hotplug_dev, NVDIMMDevice *nvdimm, return true; } - void spapr_add_nvdimm(DeviceState *dev, uint64_t slot) { SpaprDrc *drc; @@ -126,6 +125,7 @@ static int spapr_dt_nvdimm(SpaprMachineState *spapr, void *fdt, uint64_t lsize = nvdimm->label_size; uint64_t size = object_property_get_int(OBJECT(nvdimm), PC_DIMM_SIZE_PROP, NULL); + int device_node = nvdimm->device_node != -1 ? nvdimm->device_node : node; drc = spapr_drc_by_id(TYPE_SPAPR_DRC_PMEM, slot); g_assert(drc); @@ -142,7 +142,8 @@ static int spapr_dt_nvdimm(SpaprMachineState *spapr, void *fdt, _FDT((fdt_setprop_string(fdt, child_offset, "compatible", "ibm,pmemory"))); _FDT((fdt_setprop_string(fdt, child_offset, "device_type", "ibm,pmemory"))); - spapr_numa_write_associativity_dt(spapr, fdt, child_offset, node); + spapr_numa_write_nvdimm_assoc_dt(spapr, fdt, child_offset, + node, device_node); buf = qemu_uuid_unparse_strdup(&nvdimm->uuid); _FDT((fdt_setprop_string(fdt, child_offset, "ibm,unit-guid", buf))); diff --git a/include/hw/ppc/spapr_numa.h b/include/hw/ppc/spapr_numa.h index adaec8e163..af25741e70 100644 --- a/include/hw/ppc/spapr_numa.h +++ b/include/hw/ppc/spapr_numa.h @@ -26,6 +26,9 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr); void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas); void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, int offset, int nodeid); +void spapr_numa_write_nvdimm_assoc_dt(SpaprMachineState *spapr, void *fdt, + int offset, int nodeid, + int device_node); int spapr_numa_fixup_cpu_dt(SpaprMachineState *spapr, void *fdt, int offset, PowerPCCPU *cpu); int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *fdt, From patchwork Wed Jun 16 01:19:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12323721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 410D8C48BE5 for ; Wed, 16 Jun 2021 01:35:33 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C2E67611BE for ; Wed, 16 Jun 2021 01:35:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C2E67611BE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:49746 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ltKSp-0004im-QZ for qemu-devel@archiver.kernel.org; Tue, 15 Jun 2021 21:35:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53404) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ltKE8-0003rC-KV; Tue, 15 Jun 2021 21:20:21 -0400 Received: from mail-qt1-x82e.google.com ([2607:f8b0:4864:20::82e]:46950) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ltKE5-0004PC-7o; Tue, 15 Jun 2021 21:20:20 -0400 Received: by mail-qt1-x82e.google.com with SMTP id a15so581987qtx.13; Tue, 15 Jun 2021 18:20:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Otc4Xxg9oKPK1ZYhzXJX913Wp5VnSfIZ1yBwKpJgpy0=; b=s3zK5HrrEYVeMkUvVhsNJOtZ1jS7aYXgb4HuwjQZDTU//6yRrBwlyVFtJizqgdmLif 2VCOMSXF03rTiDMYT2vAIX+D5l5H+OYyzpCR7VrRKMiZ09C7MHVrHRrC2wjuqNmueZBS NqrhZQj85gpZEDBTjWz31Wy5iuJFbNxtsS/MJbTR5cdZrskKUZNZd4nuwEiqEbpn2knH +pSlzcWxaRwY7V4iVX3kQyvheEy9cLvsvRX41RLK15Z+R8gP7nRM1FLz3lkLARVhJ3NF ajanLOSRPkaypW8Mdna6mXW4SsdHWP3S4MyHvO7srFsx+C7n2DHonzB/r9xiNHXqLIFj +2mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Otc4Xxg9oKPK1ZYhzXJX913Wp5VnSfIZ1yBwKpJgpy0=; b=q26I8QLJbCgK1xV2fBOvDwYwfnhWPWOMWkGkbjLPpe9iC9pZcm+5clRwkIfhYn2SDu fuYcl89gyCSSA9YCLlR5dNOZOt6mANwneKrWjCH5ISeil8klGlEOsq998JmvojUNMDv+ 5/oMRsv9MsnPd/0gpM0ZdS7l09AH5MNmkQusH46Pj7Oo97XgmohiZ/Nqe/6qnwI+q19w HzAa4MTFWckqM5Z4DVK8UasUN4LHvze7HJV+FVuoeOyJNCME/KPNu0Pt0ehuQ6rR+9Aa ZT+6Au4LgR1/sQUcKQu6gw4h5Fell9l0IQq07gOH4L+9xLD09iU20/grOdcMHlx2ekvP NJjw== X-Gm-Message-State: AOAM530TgrRfIRtag0OagoXYsX3Nq4C1JG0y67shMj8Gwg+eV8ZmekYj MKK5lXO28hZjuvi6si66RYG8ZxXOX4w= X-Google-Smtp-Source: ABdhPJxB2iPSmW907mRDfirlcC4YiSmsjr5/S87H6HVFnJYOdKBX0nyU2Qu1p3Pe0vlyGKQCHhzjCA== X-Received: by 2002:ac8:670f:: with SMTP id e15mr2504979qtp.291.1623806415741; Tue, 15 Jun 2021 18:20:15 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7c6:4d9a:784f:a7de:c020:5f1e]) by smtp.gmail.com with ESMTPSA id f19sm627994qkg.70.2021.06.15.18.20.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 18:20:15 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 7/7] spapr: move memory/cpu less check to spapr_numa_FORM1_affinity_init() Date: Tue, 15 Jun 2021 22:19:44 -0300 Message-Id: <20210616011944.2996399-8-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210616011944.2996399-1-danielhb413@gmail.com> References: <20210616011944.2996399-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::82e; envelope-from=danielhb413@gmail.com; helo=mail-qt1-x82e.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.ibm.com, Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" FORM2 NUMA affinity is prepared to deal with memory/cpu less NUMA nodes. This is used by the DAX KMEM driver to locate a PAPR SCM device that has a different latency than the original NUMA node from the regular memory. Move this verification to FORM1 affinity code. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 33 --------------------------------- hw/ppc/spapr_numa.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 33 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index b6b822a065..2216693e21 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -2776,39 +2776,6 @@ static void spapr_machine_init(MachineState *machine) /* init CPUs */ spapr_init_cpus(spapr); - /* - * check we don't have a memory-less/cpu-less NUMA node - * Firmware relies on the existing memory/cpu topology to provide the - * NUMA topology to the kernel. - * And the linux kernel needs to know the NUMA topology at start - * to be able to hotplug CPUs later. - */ - if (machine->numa_state->num_nodes) { - for (i = 0; i < machine->numa_state->num_nodes; ++i) { - /* check for memory-less node */ - if (machine->numa_state->nodes[i].node_mem == 0) { - CPUState *cs; - int found = 0; - /* check for cpu-less node */ - CPU_FOREACH(cs) { - PowerPCCPU *cpu = POWERPC_CPU(cs); - if (cpu->node_id == i) { - found = 1; - break; - } - } - /* memory-less and cpu-less node */ - if (!found) { - error_report( - "Memory-less/cpu-less nodes are not supported (node %d)", - i); - exit(1); - } - } - } - - } - spapr->gpu_numa_id = spapr_numa_initial_nvgpu_numa_id(machine); if ((!kvm_enabled() || kvmppc_has_cap_mmu_radix()) && diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index e7d455d304..6a2eafcae1 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -162,6 +162,39 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, MachineState *machine) { bool using_legacy_numa = spapr_machine_using_legacy_numa(spapr); + int i; + + /* + * check we don't have a memory-less/cpu-less NUMA node + * Firmware relies on the existing memory/cpu topology to provide the + * NUMA topology to the kernel. + * And the linux kernel needs to know the NUMA topology at start + * to be able to hotplug CPUs later. + */ + if (machine->numa_state->num_nodes) { + for (i = 0; i < machine->numa_state->num_nodes; ++i) { + /* check for memory-less node */ + if (machine->numa_state->nodes[i].node_mem == 0) { + CPUState *cs; + int found = 0; + /* check for cpu-less node */ + CPU_FOREACH(cs) { + PowerPCCPU *cpu = POWERPC_CPU(cs); + if (cpu->node_id == i) { + found = 1; + break; + } + } + /* memory-less and cpu-less node */ + if (!found) { + error_report( + "Memory-less/cpu-less nodes are not supported (node %d)", + i); + exit(1); + } + } + } + } /* * Legacy NUMA guests (pseries-5.1 and older, or guests with only