From patchwork Thu Sep 16 01:29:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12497969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A55C5C433EF for ; Thu, 16 Sep 2021 01:32:56 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2AE2161186 for ; Thu, 16 Sep 2021 01:32:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2AE2161186 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:59960 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQgGk-0001HB-Ql for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 21:32:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45208) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQgEK-0006ol-Ip; Wed, 15 Sep 2021 21:30:25 -0400 Received: from mail-qk1-x72d.google.com ([2607:f8b0:4864:20::72d]:37820) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQgEH-0004BI-2T; Wed, 15 Sep 2021 21:30:24 -0400 Received: by mail-qk1-x72d.google.com with SMTP id 73so680884qki.4; Wed, 15 Sep 2021 18:30:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uCwUlAEuHu9NPWjIuULqWyr2j0RMt56Ie8slVpbbpAE=; b=qmuVWrU+TMBQXERIC7Gf7G+6NBPIx6lkENmzyOXyTFWINyAEAWXJIG//wGZSmvHy+b f4F2t7HwuUyCXYtOStb4ECwlYWIrERTj9L1dItrarDpb6HSoJEOWihuBKcboh5Y0hdCG v6XGCgSVgzGwjtMtU4SnJzR8llfiblPoD8YMPopmTvvGTgzorr2HA3lJoxWciKJ7EU3a DLKyPgPTE3Z1UMEFpb8HjKvorgTTyKs9dsmeVzwwEGOIj8QpUtSZv59P8ZWpRwRTgdJi Q/AUjhnCF+7fAslV/RaoNgIPhZQN7nwIesE0sbQ8DjnQa4v6q/lTe81ZkpsRgpKYEZ/z j9rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uCwUlAEuHu9NPWjIuULqWyr2j0RMt56Ie8slVpbbpAE=; b=V2RTjY4+zl+i/vn4vkq0mx+avddXieugL3q2zwL+tvPdyXMJVFvQjjOL/CecaTtFee FrZLY+GU21DeSZHcFHavVaOCfT8kKL33KxgMt4nB9HaGP9qSQ5ZbUGe+mRIhwRu0Wldg R9IqbB77mu+NWNc3unLrVBudL/DZQydSrtdjZMGu2axiORxWW/kP7PgjUQbQEps2QJDH zI14ACnhhCC9d7j64c6TOerzNa6NBNdCqQn0xVdzWERgWV9UzMVg4qszCb4Pm5WKJdiw RkKoBQPquuJhDjIhH1N84g+50wm2C/LZ5ZLTyEMwbAfJ7PFcjLmp5MAqPYT5psR/AtBo MYTQ== X-Gm-Message-State: AOAM5306Aq+P26UcouxtnNO+J4L8TuWkERqTIEKnsKeQfWW/W4cI9AeH m7efh8BYuJ0RHjLKynDqmXt6gjAF3ZA= X-Google-Smtp-Source: ABdhPJxxeeOtyWMevjtayT+ghd5zHVhw+d/nAQTIB2BDBRM+6+m3KVP1ZKydY8J42y6IrGVvIwmnlg== X-Received: by 2002:a37:9cc8:: with SMTP id f191mr2894696qke.113.1631755818949; Wed, 15 Sep 2021 18:30:18 -0700 (PDT) Received: from rekt.COMFAST ([177.189.43.50]) by smtp.gmail.com with ESMTPSA id 67sm1369417qkl.1.2021.09.15.18.30.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 18:30:18 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v7 1/7] spapr_numa.c: split FORM1 code into helpers Date: Wed, 15 Sep 2021 22:29:58 -0300 Message-Id: <20210916013004.272059-2-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210916013004.272059-1-danielhb413@gmail.com> References: <20210916013004.272059-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::72d; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x72d.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The upcoming FORM2 NUMA affinity will support asymmetric NUMA topologies and doesn't need be concerned with all the legacy support for older pseries FORM1 guests. We're also not going to calculate associativity domains based on numa distance (via spapr_numa_define_associativity_domains) since the distances will be written directly into new DT properties. Let's split FORM1 code into its own functions to allow for easier insertion of FORM2 logic later on. Reviewed-by: Greg Kurz Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr_numa.c | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 779f18b994..786def7c73 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -92,7 +92,7 @@ static uint8_t spapr_numa_get_numa_level(uint8_t distance) return 0; } -static void spapr_numa_define_associativity_domains(SpaprMachineState *spapr) +static void spapr_numa_define_FORM1_domains(SpaprMachineState *spapr) { MachineState *ms = MACHINE(spapr); NodeInfo *numa_info = ms->numa_state->nodes; @@ -155,8 +155,11 @@ static void spapr_numa_define_associativity_domains(SpaprMachineState *spapr) } -void spapr_numa_associativity_init(SpaprMachineState *spapr, - MachineState *machine) +/* + * Set NUMA machine state data based on FORM1 affinity semantics. + */ +static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, + MachineState *machine) { SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr); int nb_numa_nodes = machine->numa_state->num_nodes; @@ -225,7 +228,13 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr, exit(EXIT_FAILURE); } - spapr_numa_define_associativity_domains(spapr); + spapr_numa_define_FORM1_domains(spapr); +} + +void spapr_numa_associativity_init(SpaprMachineState *spapr, + MachineState *machine) +{ + spapr_numa_FORM1_affinity_init(spapr, machine); } void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, @@ -302,12 +311,8 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *fdt, return ret; } -/* - * Helper that writes ibm,associativity-reference-points and - * max-associativity-domains in the RTAS pointed by @rtas - * in the DT @fdt. - */ -void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) +static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, + void *fdt, int rtas) { MachineState *ms = MACHINE(spapr); SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr); @@ -365,6 +370,16 @@ void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) maxdomains, sizeof(maxdomains))); } +/* + * Helper that writes ibm,associativity-reference-points and + * max-associativity-domains in the RTAS pointed by @rtas + * in the DT @fdt. + */ +void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) +{ + spapr_numa_FORM1_write_rtas_dt(spapr, fdt, rtas); +} + static target_ulong h_home_node_associativity(PowerPCCPU *cpu, SpaprMachineState *spapr, target_ulong opcode, From patchwork Thu Sep 16 01:29:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12497971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E6A3C433FE for ; Thu, 16 Sep 2021 01:32:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 966BE61165 for ; Thu, 16 Sep 2021 01:32:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 966BE61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:59964 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQgGm-0001HU-Cy for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 21:32:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45230) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQgEM-0006p8-6u; Wed, 15 Sep 2021 21:30:26 -0400 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]:33575) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQgEI-0004E5-2c; Wed, 15 Sep 2021 21:30:25 -0400 Received: by mail-qk1-x72f.google.com with SMTP id b64so5833295qkg.0; Wed, 15 Sep 2021 18:30:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9snP2SQ90EsaopUkwhiL3G+yqbxkthjiliUkz9xzmQM=; b=KocPYGQiQFi6KyhqwGdoPSLcrgGPTiCcJCxr8DZNpaIHBNl0OOPirySZ+QjCgcepBa naKZgM/RalfkkscnJ4gFGVYAotjlpKHufaRU2TCHmR457uV/nfYvn4UojaJpkTwoL1zb YzDySo7l4Pni2YDilgkz+hXbNyVSdUwGVIG6GfuRkNb7h+0m4FKAoHFiW1zQuOjegq0G gFbo+2ZpopnI/QiOJz6oEg9xerC7drsgRlPsZn7ybvLfOrXzidgHUEJpQv72OqA4qw1s oRQd+oWl0zJumJC9izcdfvcksjpPLaPpSluP0w6MwbY5xWGq3CABHKLx4s9LNG7ib3yF jTVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9snP2SQ90EsaopUkwhiL3G+yqbxkthjiliUkz9xzmQM=; b=zb1s4Yq/pzLE9h6oeyxgb0Lq4abmWFJPNdvH0GCVEWc9709wIkq2GIw1RUmoJZMIua HSo6SRkGEz1v2HTyg4irc0q/p+CJZv5XYTHKfUupA9VI5/eDEXO9l4N+tntvNQgooPYc eLEUDyMnRrZCLgxcm+emMJVJHdBoUwcTnaPobo8w7jF3TwdTs4a6co8lc1Kzy33ynsr5 c8NvS4PG2908fkllr4AbrDSzowfFnjMcA1fovB6FYKASy403resqIMHiNy28Tmv49yR/ DQ6uLq4fwAbeXwrpbF/Qo0kNEg5T3uoqLbliU5g5R93A3eOlb9HFzgNFPnGUERH360Uv q6xA== X-Gm-Message-State: AOAM533HZyV/f7spih3t7gpNykqzGMsE94ofj2RdLPto9u9guMMi4707 KoH73mFDvWQSI1bfbYk7JAtUXRI9wqk= X-Google-Smtp-Source: ABdhPJwImA6cMq2yk0vwdL2Csubxx0R9Ah89hSmrem7xavAzasmb/ztxXPmxLKERyUwmP63FepTuhw== X-Received: by 2002:a37:43cf:: with SMTP id q198mr2938101qka.470.1631755820792; Wed, 15 Sep 2021 18:30:20 -0700 (PDT) Received: from rekt.COMFAST ([177.189.43.50]) by smtp.gmail.com with ESMTPSA id 67sm1369417qkl.1.2021.09.15.18.30.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 18:30:20 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v7 2/7] spapr_numa.c: scrap 'legacy_numa' concept Date: Wed, 15 Sep 2021 22:29:59 -0300 Message-Id: <20210916013004.272059-3-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210916013004.272059-1-danielhb413@gmail.com> References: <20210916013004.272059-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::72f; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x72f.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When first introduced, 'legacy_numa' was a way to refer to guests that either wouldn't be affected by associativity domain calculations, namely the ones with only 1 NUMA node, and pre 5.2 guests that shouldn't be affected by it because it would be an userspace change. Calling these cases 'legacy_numa' was a convenient way to label these cases. We're about to introduce a new NUMA affinity, FORM2, and this concept of 'legacy_numa' is now a bit misleading because, although it is called 'legacy' it is in fact a FORM1 exclusive contraint. This patch removes spapr_machine_using_legacy_numa() and open code the conditions in each caller. While we're at it, move the chunk inside spapr_numa_FORM1_affinity_init() that sets all numa_assoc_array domains with 'node_id' to spapr_numa_define_FORM1_domains(). This chunk was being executed if !pre_5_2_numa_associativity and num_nodes => 1, the same conditions in which spapr_numa_define_FORM1_domains() is called shortly after. Reviewed-by: Greg Kurz Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr_numa.c | 47 +++++++++++++++++++-------------------------- 1 file changed, 20 insertions(+), 27 deletions(-) diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 786def7c73..bf520d42b2 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -19,15 +19,6 @@ /* Moved from hw/ppc/spapr_pci_nvlink2.c */ #define SPAPR_GPU_NUMA_ID (cpu_to_be32(1)) -static bool spapr_machine_using_legacy_numa(SpaprMachineState *spapr) -{ - MachineState *machine = MACHINE(spapr); - SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(machine); - - return smc->pre_5_2_numa_associativity || - machine->numa_state->num_nodes <= 1; -} - static bool spapr_numa_is_symmetrical(MachineState *ms) { int src, dst; @@ -97,7 +88,18 @@ static void spapr_numa_define_FORM1_domains(SpaprMachineState *spapr) MachineState *ms = MACHINE(spapr); NodeInfo *numa_info = ms->numa_state->nodes; int nb_numa_nodes = ms->numa_state->num_nodes; - int src, dst, i; + int src, dst, i, j; + + /* + * Fill all associativity domains of non-zero NUMA nodes with + * node_id. This is required because the default value (0) is + * considered a match with associativity domains of node 0. + */ + for (i = 1; i < nb_numa_nodes; i++) { + for (j = 1; j < MAX_DISTANCE_REF_POINTS; j++) { + spapr->numa_assoc_array[i][j] = cpu_to_be32(i); + } + } for (src = 0; src < nb_numa_nodes; src++) { for (dst = src; dst < nb_numa_nodes; dst++) { @@ -164,7 +166,6 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, SpaprMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr); int nb_numa_nodes = machine->numa_state->num_nodes; int i, j, max_nodes_with_gpus; - bool using_legacy_numa = spapr_machine_using_legacy_numa(spapr); /* * For all associativity arrays: first position is the size, @@ -178,17 +179,6 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, for (i = 0; i < nb_numa_nodes; i++) { spapr->numa_assoc_array[i][0] = cpu_to_be32(MAX_DISTANCE_REF_POINTS); spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i); - - /* - * Fill all associativity domains of non-zero NUMA nodes with - * node_id. This is required because the default value (0) is - * considered a match with associativity domains of node 0. - */ - if (!using_legacy_numa && i != 0) { - for (j = 1; j < MAX_DISTANCE_REF_POINTS; j++) { - spapr->numa_assoc_array[i][j] = cpu_to_be32(i); - } - } } /* @@ -214,11 +204,13 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, } /* - * Legacy NUMA guests (pseries-5.1 and older, or guests with only - * 1 NUMA node) will not benefit from anything we're going to do - * after this point. + * Guests pseries-5.1 and older uses zeroed associativity domains, + * i.e. no domain definition based on NUMA distance input. + * + * Same thing with guests that have only one NUMA node. */ - if (using_legacy_numa) { + if (smc->pre_5_2_numa_associativity || + machine->numa_state->num_nodes <= 1) { return; } @@ -334,7 +326,8 @@ static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, cpu_to_be32(maxdomain) }; - if (spapr_machine_using_legacy_numa(spapr)) { + if (smc->pre_5_2_numa_associativity || + ms->numa_state->num_nodes <= 1) { uint32_t legacy_refpoints[] = { cpu_to_be32(0x4), cpu_to_be32(0x4), From patchwork Thu Sep 16 01:30:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12497975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5E88C433F5 for ; Thu, 16 Sep 2021 01:35:30 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4D9DD61164 for ; Thu, 16 Sep 2021 01:35:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4D9DD61164 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:40220 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQgJF-0006sT-GH for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 21:35:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45252) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQgEN-0006pw-ML; Wed, 15 Sep 2021 21:30:27 -0400 Received: from mail-qv1-xf31.google.com ([2607:f8b0:4864:20::f31]:40595) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQgEK-0004Ff-Fe; Wed, 15 Sep 2021 21:30:27 -0400 Received: by mail-qv1-xf31.google.com with SMTP id 93so3189813qva.7; Wed, 15 Sep 2021 18:30:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GFf5m4deEIFrWeeUm0Gsm9NDplBK01kQmjI0YYnhmOY=; b=WPGH/TThd2EPfUc6Le5OI6Wf3O36hxxldMc0ku2j1Vkfiym43ZHgYcimwIvDKhDzWW g0ZJflI4unQQtqdQmlN8l6N0jiQ/UGqy6F05ZD2ZWifq05OglowJgSBg+wR3jwLBOqhq qZZy1Uk79eCQF6uXu1bFn+OGmGC88wLNctOJ3ANSXmRh1PyGUV0T95IQiHULlRUtvZqd /Al0y/KBNpqgZp6Q/9+vYJvYrNM0xvt9TLhbTXJNG1uZQGN1vmjWZg+vLgayfycmVrk6 Z96dlR5Xf5MODPFluJhIJ4wvasqTPU95gNWdZ8bBF/GfHY1O0/mp7ucnRiewt3CxDNUH 2CCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GFf5m4deEIFrWeeUm0Gsm9NDplBK01kQmjI0YYnhmOY=; b=EgbXpyaPpIIWXdjcYJ/0q6/5iPhWtO0EBmLo7VlIrzHrEep/6P7lNBii2zHBfD/63m oMVRerHYXzc/mUUI0ApM1nRnLlQDc63iazmhPia9h60nrap1GTbdOnIu3/r1V1x/huce zr8TGMqiGnwTtfBXGzhcssQKNbzC1DRg3AReeG3lhI3A8lkCmKzJDtav6BYMAYmmwasu CCMRCEDheFqGn+5IbltVFwUvc4JagOpxUxUpjRojuME44U9GiABjz1UINcwYdCGGNGR7 MQqce3pGnW5XcJLQ3GxpOIiHwvvtzf5ZOHteExReG6P8LmLBB/oTQIlBu/A3KYWdV2Dw +MAA== X-Gm-Message-State: AOAM532YMZBQEeJruFN5XMQgLJdoxbHF/KDuFwt7ct8KtwJ6wcvFfSd/ JTIDaO2dMEC9C82x70fCRq4i92HiHi0= X-Google-Smtp-Source: ABdhPJyxNIHF5oO4pT5KPpaukLfb8/tRsmIm2UOKQBptEzOmxf64KJuwvRUODZdHCqlyc+DpMT511w== X-Received: by 2002:a0c:dc09:: with SMTP id s9mr2828347qvk.61.1631755822775; Wed, 15 Sep 2021 18:30:22 -0700 (PDT) Received: from rekt.COMFAST ([177.189.43.50]) by smtp.gmail.com with ESMTPSA id 67sm1369417qkl.1.2021.09.15.18.30.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 18:30:22 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v7 3/7] spapr_numa.c: parametrize FORM1 macros Date: Wed, 15 Sep 2021 22:30:00 -0300 Message-Id: <20210916013004.272059-4-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210916013004.272059-1-danielhb413@gmail.com> References: <20210916013004.272059-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::f31; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf31.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The next preliminary step to introduce NUMA FORM2 affinity is to make the existing code independent of FORM1 macros and values, i.e. MAX_DISTANCE_REF_POINTS, NUMA_ASSOC_SIZE and VCPU_ASSOC_SIZE. This patch accomplishes that by doing the following: - move the NUMA related macros from spapr.h to spapr_numa.c where they are used. spapr.h gets instead a 'NUMA_NODES_MAX_NUM' macro that is used to refer to the maximum number of NUMA nodes, including GPU nodes, that the machine can support; - MAX_DISTANCE_REF_POINTS and NUMA_ASSOC_SIZE are renamed to FORM1_DIST_REF_POINTS and FORM1_NUMA_ASSOC_SIZE. These FORM1 specific macros are used in FORM1 init functions; - code that uses MAX_DISTANCE_REF_POINTS now retrieves the max_dist_ref_points value using get_max_dist_ref_points(). NUMA_ASSOC_SIZE is replaced by get_numa_assoc_size() and VCPU_ASSOC_SIZE is replaced by get_vcpu_assoc_size(). These functions are used by the generic device tree functions and h_home_node_associativity() and will allow them to switch between FORM1 and FORM2 without changing their core logic. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr_numa.c | 76 ++++++++++++++++++++++++++++++------------ include/hw/ppc/spapr.h | 28 ++++++++-------- 2 files changed, 69 insertions(+), 35 deletions(-) diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index bf520d42b2..1755c9d772 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -19,6 +19,35 @@ /* Moved from hw/ppc/spapr_pci_nvlink2.c */ #define SPAPR_GPU_NUMA_ID (cpu_to_be32(1)) +/* + * Retrieves max_dist_ref_points of the current NUMA affinity. + */ +static int get_max_dist_ref_points(SpaprMachineState *spapr) +{ + /* No FORM2 affinity implemented yet */ + return FORM1_DIST_REF_POINTS; +} + +/* + * Retrieves numa_assoc_size of the current NUMA affinity. + */ +static int get_numa_assoc_size(SpaprMachineState *spapr) +{ + /* No FORM2 affinity implemented yet */ + return FORM1_NUMA_ASSOC_SIZE; +} + +/* + * Retrieves vcpu_assoc_size of the current NUMA affinity. + * + * vcpu_assoc_size is the size of ibm,associativity array + * for CPUs, which has an extra element (vcpu_id) in the end. + */ +static int get_vcpu_assoc_size(SpaprMachineState *spapr) +{ + return get_numa_assoc_size(spapr) + 1; +} + static bool spapr_numa_is_symmetrical(MachineState *ms) { int src, dst; @@ -96,7 +125,7 @@ static void spapr_numa_define_FORM1_domains(SpaprMachineState *spapr) * considered a match with associativity domains of node 0. */ for (i = 1; i < nb_numa_nodes; i++) { - for (j = 1; j < MAX_DISTANCE_REF_POINTS; j++) { + for (j = 1; j < FORM1_DIST_REF_POINTS; j++) { spapr->numa_assoc_array[i][j] = cpu_to_be32(i); } } @@ -134,7 +163,7 @@ static void spapr_numa_define_FORM1_domains(SpaprMachineState *spapr) * * The Linux kernel will assume that the distance between src and * dst, in this case of no match, is 10 (local distance) doubled - * for each NUMA it didn't match. We have MAX_DISTANCE_REF_POINTS + * for each NUMA it didn't match. We have FORM1_DIST_REF_POINTS * levels (4), so this gives us 10*2*2*2*2 = 160. * * This logic can be seen in the Linux kernel source code, as of @@ -169,7 +198,7 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, /* * For all associativity arrays: first position is the size, - * position MAX_DISTANCE_REF_POINTS is always the numa_id, + * position FORM1_DIST_REF_POINTS is always the numa_id, * represented by the index 'i'. * * This will break on sparse NUMA setups, when/if QEMU starts @@ -177,8 +206,8 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, * 'i' will be a valid node_id set by the user. */ for (i = 0; i < nb_numa_nodes; i++) { - spapr->numa_assoc_array[i][0] = cpu_to_be32(MAX_DISTANCE_REF_POINTS); - spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i); + spapr->numa_assoc_array[i][0] = cpu_to_be32(FORM1_DIST_REF_POINTS); + spapr->numa_assoc_array[i][FORM1_DIST_REF_POINTS] = cpu_to_be32(i); } /* @@ -192,15 +221,15 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, max_nodes_with_gpus = nb_numa_nodes + NVGPU_MAX_NUM; for (i = nb_numa_nodes; i < max_nodes_with_gpus; i++) { - spapr->numa_assoc_array[i][0] = cpu_to_be32(MAX_DISTANCE_REF_POINTS); + spapr->numa_assoc_array[i][0] = cpu_to_be32(FORM1_DIST_REF_POINTS); - for (j = 1; j < MAX_DISTANCE_REF_POINTS; j++) { + for (j = 1; j < FORM1_DIST_REF_POINTS; j++) { uint32_t gpu_assoc = smc->pre_5_1_assoc_refpoints ? SPAPR_GPU_NUMA_ID : cpu_to_be32(i); spapr->numa_assoc_array[i][j] = gpu_assoc; } - spapr->numa_assoc_array[i][MAX_DISTANCE_REF_POINTS] = cpu_to_be32(i); + spapr->numa_assoc_array[i][FORM1_DIST_REF_POINTS] = cpu_to_be32(i); } /* @@ -234,13 +263,15 @@ void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, { _FDT((fdt_setprop(fdt, offset, "ibm,associativity", spapr->numa_assoc_array[nodeid], - sizeof(spapr->numa_assoc_array[nodeid])))); + get_numa_assoc_size(spapr) * sizeof(uint32_t)))); } static uint32_t *spapr_numa_get_vcpu_assoc(SpaprMachineState *spapr, PowerPCCPU *cpu) { - uint32_t *vcpu_assoc = g_new(uint32_t, VCPU_ASSOC_SIZE); + int max_distance_ref_points = get_max_dist_ref_points(spapr); + int vcpu_assoc_size = get_vcpu_assoc_size(spapr); + uint32_t *vcpu_assoc = g_new(uint32_t, vcpu_assoc_size); int index = spapr_get_vcpu_id(cpu); /* @@ -249,10 +280,10 @@ static uint32_t *spapr_numa_get_vcpu_assoc(SpaprMachineState *spapr, * 0, put cpu_id last, then copy the remaining associativity * domains. */ - vcpu_assoc[0] = cpu_to_be32(MAX_DISTANCE_REF_POINTS + 1); - vcpu_assoc[VCPU_ASSOC_SIZE - 1] = cpu_to_be32(index); + vcpu_assoc[0] = cpu_to_be32(max_distance_ref_points + 1); + vcpu_assoc[vcpu_assoc_size - 1] = cpu_to_be32(index); memcpy(vcpu_assoc + 1, spapr->numa_assoc_array[cpu->node_id] + 1, - (VCPU_ASSOC_SIZE - 2) * sizeof(uint32_t)); + (vcpu_assoc_size - 2) * sizeof(uint32_t)); return vcpu_assoc; } @@ -261,12 +292,13 @@ int spapr_numa_fixup_cpu_dt(SpaprMachineState *spapr, void *fdt, int offset, PowerPCCPU *cpu) { g_autofree uint32_t *vcpu_assoc = NULL; + int vcpu_assoc_size = get_vcpu_assoc_size(spapr); vcpu_assoc = spapr_numa_get_vcpu_assoc(spapr, cpu); /* Advertise NUMA via ibm,associativity */ return fdt_setprop(fdt, offset, "ibm,associativity", vcpu_assoc, - VCPU_ASSOC_SIZE * sizeof(uint32_t)); + vcpu_assoc_size * sizeof(uint32_t)); } @@ -274,17 +306,18 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *fdt, int offset) { MachineState *machine = MACHINE(spapr); + int max_distance_ref_points = get_max_dist_ref_points(spapr); int nb_numa_nodes = machine->numa_state->num_nodes; int nr_nodes = nb_numa_nodes ? nb_numa_nodes : 1; uint32_t *int_buf, *cur_index, buf_len; int ret, i; /* ibm,associativity-lookup-arrays */ - buf_len = (nr_nodes * MAX_DISTANCE_REF_POINTS + 2) * sizeof(uint32_t); + buf_len = (nr_nodes * max_distance_ref_points + 2) * sizeof(uint32_t); cur_index = int_buf = g_malloc0(buf_len); int_buf[0] = cpu_to_be32(nr_nodes); /* Number of entries per associativity list */ - int_buf[1] = cpu_to_be32(MAX_DISTANCE_REF_POINTS); + int_buf[1] = cpu_to_be32(max_distance_ref_points); cur_index += 2; for (i = 0; i < nr_nodes; i++) { /* @@ -293,8 +326,8 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *fdt, */ uint32_t *associativity = spapr->numa_assoc_array[i]; memcpy(cur_index, ++associativity, - sizeof(uint32_t) * MAX_DISTANCE_REF_POINTS); - cur_index += MAX_DISTANCE_REF_POINTS; + sizeof(uint32_t) * max_distance_ref_points); + cur_index += max_distance_ref_points; } ret = fdt_setprop(fdt, offset, "ibm,associativity-lookup-arrays", int_buf, (cur_index - int_buf) * sizeof(uint32_t)); @@ -383,6 +416,7 @@ static target_ulong h_home_node_associativity(PowerPCCPU *cpu, target_ulong procno = args[1]; PowerPCCPU *tcpu; int idx, assoc_idx; + int vcpu_assoc_size = get_vcpu_assoc_size(spapr); /* only support procno from H_REGISTER_VPA */ if (flags != 0x1) { @@ -401,7 +435,7 @@ static target_ulong h_home_node_associativity(PowerPCCPU *cpu, * 12 associativity domains for vcpus. Assert and bail if that's * not the case. */ - G_STATIC_ASSERT((VCPU_ASSOC_SIZE - 1) <= 12); + g_assert((vcpu_assoc_size - 1) <= 12); vcpu_assoc = spapr_numa_get_vcpu_assoc(spapr, tcpu); /* assoc_idx starts at 1 to skip associativity size */ @@ -422,9 +456,9 @@ static target_ulong h_home_node_associativity(PowerPCCPU *cpu, * macro. The ternary will fill the remaining registers with -1 * after we went through vcpu_assoc[]. */ - a = assoc_idx < VCPU_ASSOC_SIZE ? + a = assoc_idx < vcpu_assoc_size ? be32_to_cpu(vcpu_assoc[assoc_idx++]) : -1; - b = assoc_idx < VCPU_ASSOC_SIZE ? + b = assoc_idx < vcpu_assoc_size ? be32_to_cpu(vcpu_assoc[assoc_idx++]) : -1; args[idx] = ASSOCIATIVITY(a, b); diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 637652ad16..814e087e98 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -100,23 +100,23 @@ typedef enum { #define FDT_MAX_SIZE 0x200000 +/* Max number of GPUs per system */ +#define NVGPU_MAX_NUM 6 + +/* Max number of NUMA nodes */ +#define NUMA_NODES_MAX_NUM (MAX_NODES + NVGPU_MAX_NUM) + /* - * NUMA related macros. MAX_DISTANCE_REF_POINTS was taken - * from Linux kernel arch/powerpc/mm/numa.h. It represents the - * amount of associativity domains for non-CPU resources. + * NUMA FORM1 macros. FORM1_DIST_REF_POINTS was taken from + * MAX_DISTANCE_REF_POINTS in arch/powerpc/mm/numa.h from Linux + * kernel source. It represents the amount of associativity domains + * for non-CPU resources. * - * NUMA_ASSOC_SIZE is the base array size of an ibm,associativity + * FORM1_NUMA_ASSOC_SIZE is the base array size of an ibm,associativity * array for any non-CPU resource. - * - * VCPU_ASSOC_SIZE represents the size of ibm,associativity array - * for CPUs, which has an extra element (vcpu_id) in the end. */ -#define MAX_DISTANCE_REF_POINTS 4 -#define NUMA_ASSOC_SIZE (MAX_DISTANCE_REF_POINTS + 1) -#define VCPU_ASSOC_SIZE (NUMA_ASSOC_SIZE + 1) - -/* Max number of these GPUsper a physical box */ -#define NVGPU_MAX_NUM 6 +#define FORM1_DIST_REF_POINTS 4 +#define FORM1_NUMA_ASSOC_SIZE (FORM1_DIST_REF_POINTS + 1) typedef struct SpaprCapabilities SpaprCapabilities; struct SpaprCapabilities { @@ -249,7 +249,7 @@ struct SpaprMachineState { unsigned gpu_numa_id; SpaprTpmProxy *tpm_proxy; - uint32_t numa_assoc_array[MAX_NODES + NVGPU_MAX_NUM][NUMA_ASSOC_SIZE]; + uint32_t numa_assoc_array[NUMA_NODES_MAX_NUM][FORM1_NUMA_ASSOC_SIZE]; Error *fwnmi_migration_blocker; }; From patchwork Thu Sep 16 01:30:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12497981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33B32C433F5 for ; Thu, 16 Sep 2021 01:38:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AC33261185 for ; Thu, 16 Sep 2021 01:38:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AC33261185 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:46624 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQgM1-0002og-Of for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 21:38:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45256) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQgEO-0006qK-08; Wed, 15 Sep 2021 21:30:29 -0400 Received: from mail-qv1-xf35.google.com ([2607:f8b0:4864:20::f35]:37862) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQgEM-0004GY-6u; Wed, 15 Sep 2021 21:30:27 -0400 Received: by mail-qv1-xf35.google.com with SMTP id a12so3197396qvz.4; Wed, 15 Sep 2021 18:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E4cTJhDdTLXAMtGYBMI9JgRxBriEQsQoJGfBVqvbuaw=; b=cRlNv3QIF8Tmc3ulS7iqqCPvpg7eymLGsvBbj0pHYVbArogoSRRuMR4d9Ea+BE8EKJ y77EW1iZI5NMdYm24tOwqs06Qfp6Qe+lHedklxCGWtz36JQ4hWfLb0niVCBFZvhvd4gK L05sFNSyh0sTJskslmKgaQVQevk804fFKzkisKW3RE3N7L5gWpwLDpCsqvxybRkImgkN R4XWsX/VxE5xoGnCguOY1eUNojcAlLs7F7PuRj/xYLXHr3KbP2goYuJTyWlpw0heqNFp kyxIwg+7Et5HffmSYOKNy8PyXrwSyGJGNevZwM2C0Am0Ksd8d2i6Bo7naKx5wN6U98jA CiUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E4cTJhDdTLXAMtGYBMI9JgRxBriEQsQoJGfBVqvbuaw=; b=cKRz3eYTXr0xwCWf/tueHDlK3sVycibqN91wI3o/ACskCbSyZrOJV2osEeST7zBRbF tLt5BHxe75Kugw2Xy/7jbYf3DRza1hsjK6zCxgtwo543pIoEYnWb7xKBAbMGSQ5gI7+i N+C8eISsZyJeUYLBwZc77bjfkHbAovB1qdKdhJ/dCXy7ixu3VMEZp6BMBimsmHQXpoig 6hkAlXQuXUrU4tukiZVdZJJeJeu+PvZ6xPw3HSRYxFFAqdVpWG6j6vcvg0lfDo8Gf9sH sRQI3oYUESQzZWOqRyWb0ylFgw/3+JQK8Dnc3QSEB6b+lE7OHn8boVbzsUJaqww9tIRM c0kQ== X-Gm-Message-State: AOAM532M2vwp7TMIFDy/JiSHsDBHdaIGycZPX6bFETEE/OxqOm1RjYAD NQNQS7siOnBWiUUlKpXOgB/Uh55ilcI= X-Google-Smtp-Source: ABdhPJzn53WZ7DsBJs0LVWbEy89R75Y362T+JXjLuEzd+nU7VzBPcwW1mA52Hwx7D0t8YcQm+J1N5Q== X-Received: by 2002:ad4:4a93:: with SMTP id h19mr2893175qvx.41.1631755824950; Wed, 15 Sep 2021 18:30:24 -0700 (PDT) Received: from rekt.COMFAST ([177.189.43.50]) by smtp.gmail.com with ESMTPSA id 67sm1369417qkl.1.2021.09.15.18.30.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 18:30:24 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v7 4/7] spapr_numa.c: rename numa_assoc_array to FORM1_assoc_array Date: Wed, 15 Sep 2021 22:30:01 -0300 Message-Id: <20210916013004.272059-5-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210916013004.272059-1-danielhb413@gmail.com> References: <20210916013004.272059-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::f35; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf35.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Introducing a new NUMA affinity, FORM2, requires a new mechanism to switch between affinity modes after CAS. Also, we want FORM2 data structures and functions to be completely separated from the existing FORM1 code, allowing us to avoid adding new code that inherits the existing complexity of FORM1. The idea of switching values used by the write_dt() functions in spapr_numa.c was already introduced in the previous patch, and the same approach will be used when dealing with the FORM1 and FORM2 arrays. We can accomplish that by that by renaming the existing numa_assoc_array to FORM1_assoc_array, which now is used exclusively to handle FORM1 affinity data. A new helper get_associativity() is then introduced to be used by the write_dt() functions to retrieve the current ibm,associativity array of a given node, after considering affinity selection that might have been done during CAS. All code that was using numa_assoc_array now needs to retrieve the array by calling this function. This will allow for an easier plug of FORM2 data later on. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr_hcall.c | 1 + hw/ppc/spapr_numa.c | 38 +++++++++++++++++++++++++------------- include/hw/ppc/spapr.h | 2 +- 3 files changed, 27 insertions(+), 14 deletions(-) diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index 0e9a5b2e40..9056644890 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -17,6 +17,7 @@ #include "kvm_ppc.h" #include "hw/ppc/fdt.h" #include "hw/ppc/spapr_ovec.h" +#include "hw/ppc/spapr_numa.h" #include "mmu-book3s-v3.h" #include "hw/mem/memory-device.h" diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 1755c9d772..39f9a73429 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -48,6 +48,15 @@ static int get_vcpu_assoc_size(SpaprMachineState *spapr) return get_numa_assoc_size(spapr) + 1; } +/* + * Retrieves the ibm,associativity array of NUMA node 'node_id' + * for the current NUMA affinity. + */ +static uint32_t *get_associativity(SpaprMachineState *spapr, int node_id) +{ + return spapr->FORM1_assoc_array[node_id]; +} + static bool spapr_numa_is_symmetrical(MachineState *ms) { int src, dst; @@ -126,7 +135,7 @@ static void spapr_numa_define_FORM1_domains(SpaprMachineState *spapr) */ for (i = 1; i < nb_numa_nodes; i++) { for (j = 1; j < FORM1_DIST_REF_POINTS; j++) { - spapr->numa_assoc_array[i][j] = cpu_to_be32(i); + spapr->FORM1_assoc_array[i][j] = cpu_to_be32(i); } } @@ -178,8 +187,8 @@ static void spapr_numa_define_FORM1_domains(SpaprMachineState *spapr) * and going up to 0x1. */ for (i = n_level; i > 0; i--) { - assoc_src = spapr->numa_assoc_array[src][i]; - spapr->numa_assoc_array[dst][i] = assoc_src; + assoc_src = spapr->FORM1_assoc_array[src][i]; + spapr->FORM1_assoc_array[dst][i] = assoc_src; } } } @@ -206,8 +215,8 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, * 'i' will be a valid node_id set by the user. */ for (i = 0; i < nb_numa_nodes; i++) { - spapr->numa_assoc_array[i][0] = cpu_to_be32(FORM1_DIST_REF_POINTS); - spapr->numa_assoc_array[i][FORM1_DIST_REF_POINTS] = cpu_to_be32(i); + spapr->FORM1_assoc_array[i][0] = cpu_to_be32(FORM1_DIST_REF_POINTS); + spapr->FORM1_assoc_array[i][FORM1_DIST_REF_POINTS] = cpu_to_be32(i); } /* @@ -221,15 +230,15 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, max_nodes_with_gpus = nb_numa_nodes + NVGPU_MAX_NUM; for (i = nb_numa_nodes; i < max_nodes_with_gpus; i++) { - spapr->numa_assoc_array[i][0] = cpu_to_be32(FORM1_DIST_REF_POINTS); + spapr->FORM1_assoc_array[i][0] = cpu_to_be32(FORM1_DIST_REF_POINTS); for (j = 1; j < FORM1_DIST_REF_POINTS; j++) { uint32_t gpu_assoc = smc->pre_5_1_assoc_refpoints ? SPAPR_GPU_NUMA_ID : cpu_to_be32(i); - spapr->numa_assoc_array[i][j] = gpu_assoc; + spapr->FORM1_assoc_array[i][j] = gpu_assoc; } - spapr->numa_assoc_array[i][FORM1_DIST_REF_POINTS] = cpu_to_be32(i); + spapr->FORM1_assoc_array[i][FORM1_DIST_REF_POINTS] = cpu_to_be32(i); } /* @@ -261,8 +270,10 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr, void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, int offset, int nodeid) { + uint32_t *associativity = get_associativity(spapr, nodeid); + _FDT((fdt_setprop(fdt, offset, "ibm,associativity", - spapr->numa_assoc_array[nodeid], + associativity, get_numa_assoc_size(spapr) * sizeof(uint32_t)))); } @@ -272,6 +283,7 @@ static uint32_t *spapr_numa_get_vcpu_assoc(SpaprMachineState *spapr, int max_distance_ref_points = get_max_dist_ref_points(spapr); int vcpu_assoc_size = get_vcpu_assoc_size(spapr); uint32_t *vcpu_assoc = g_new(uint32_t, vcpu_assoc_size); + uint32_t *associativity = get_associativity(spapr, cpu->node_id); int index = spapr_get_vcpu_id(cpu); /* @@ -282,7 +294,7 @@ static uint32_t *spapr_numa_get_vcpu_assoc(SpaprMachineState *spapr, */ vcpu_assoc[0] = cpu_to_be32(max_distance_ref_points + 1); vcpu_assoc[vcpu_assoc_size - 1] = cpu_to_be32(index); - memcpy(vcpu_assoc + 1, spapr->numa_assoc_array[cpu->node_id] + 1, + memcpy(vcpu_assoc + 1, associativity + 1, (vcpu_assoc_size - 2) * sizeof(uint32_t)); return vcpu_assoc; @@ -321,10 +333,10 @@ int spapr_numa_write_assoc_lookup_arrays(SpaprMachineState *spapr, void *fdt, cur_index += 2; for (i = 0; i < nr_nodes; i++) { /* - * For the lookup-array we use the ibm,associativity array, - * from numa_assoc_array. without the first element (size). + * For the lookup-array we use the ibm,associativity array of the + * current NUMA affinity, without the first element (size). */ - uint32_t *associativity = spapr->numa_assoc_array[i]; + uint32_t *associativity = get_associativity(spapr, i); memcpy(cur_index, ++associativity, sizeof(uint32_t) * max_distance_ref_points); cur_index += max_distance_ref_points; diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 814e087e98..6b3dfc5dc2 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -249,7 +249,7 @@ struct SpaprMachineState { unsigned gpu_numa_id; SpaprTpmProxy *tpm_proxy; - uint32_t numa_assoc_array[NUMA_NODES_MAX_NUM][FORM1_NUMA_ASSOC_SIZE]; + uint32_t FORM1_assoc_array[NUMA_NODES_MAX_NUM][FORM1_NUMA_ASSOC_SIZE]; Error *fwnmi_migration_blocker; }; From patchwork Thu Sep 16 01:30:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12497977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53423C433EF for ; Thu, 16 Sep 2021 01:35:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E91C961164 for ; Thu, 16 Sep 2021 01:35:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E91C961164 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:40876 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQgJO-0007Ku-3l for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 21:35:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45282) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQgER-0006rW-8h; Wed, 15 Sep 2021 21:30:32 -0400 Received: from mail-qv1-xf31.google.com ([2607:f8b0:4864:20::f31]:35487) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQgEO-0004Hz-BP; Wed, 15 Sep 2021 21:30:31 -0400 Received: by mail-qv1-xf31.google.com with SMTP id g11so3212248qvd.2; Wed, 15 Sep 2021 18:30:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2C5b76jtImHGxpA+qB3vnX40UTYGfv0o/HjtPEWFh8k=; b=QlYxHVrgmtSjneWsJ9UE6j/sLXg3ttNzGX2urkemoJgEZ6QP8fDeXnAHG2bshO7+bX 6RBJcHwA496AQV/+r/D2J1wctOPWBSfQ7e3eDh60QjayfgVX7G2mTfVdgc0klf+ePMvc nBrMHEpsqfn0hGd//h31iW1HbfKcuntH4lYEG10wN0jTzuxM+POkzHsjgKTloE2P/tvF FfMtHB6N9iECrlXB7+2MKGVzykw/7UYAq8rKn1eZM34xrBwI65YCL1Ao2L56HbjqDJpl PUBnbgq0knpkkrBsu3UFRHQTBSYhCmLHCpdZlP4KStZv2P69DjtyxZpFDK9pI5O+HYb+ SY4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2C5b76jtImHGxpA+qB3vnX40UTYGfv0o/HjtPEWFh8k=; b=wOhF/2+crUczxKhX+2YQ2Vz7Ioa51E3BFTsbQ3J02dVmdbW7Z/3g+4LSh1MPGHtjcO 3yaumxjoNXspCHhokMrK6lHW6x6MZgmzoDRzCeu41WvXpSwjFZDVIuGZrnb7/uzemY6j hkZeDZzQmoYrgRV6JjYXppaJyKVBHOQoZK7EgS6LvECCqmy5doIlNA5rUfJHVlyFE3u3 GONIbu4hgTZlrmmXIzmR1be9P4zzTvVXe7RHHnIxmc04kKVdYXJQxTxSYXo4UMQ9nSbD kH5mMATUcOrUjHFsu4XFAzQDwk+YijC3PL0YQk0EozvXUOecBAP2qLTyrPKITvfH90Fm 2DtA== X-Gm-Message-State: AOAM530gbGsagbJ1Q1IJ7uNoJwlBt3tiYR0u76fVOYTasYU8zeVPj0hH XVIz1vSLiAj546Qq4CPI2jzwes525QM= X-Google-Smtp-Source: ABdhPJwdUco/KvrsOancQ2K/bVR/fY7EXAFRAqJI+SkcgNZi8po+z49kMQpgmuGhQUisy3D7j9+lfQ== X-Received: by 2002:a0c:d44d:: with SMTP id r13mr2877872qvh.50.1631755827064; Wed, 15 Sep 2021 18:30:27 -0700 (PDT) Received: from rekt.COMFAST ([177.189.43.50]) by smtp.gmail.com with ESMTPSA id 67sm1369417qkl.1.2021.09.15.18.30.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 18:30:26 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v7 5/7] spapr: move FORM1 verifications to post CAS Date: Wed, 15 Sep 2021 22:30:02 -0300 Message-Id: <20210916013004.272059-6-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210916013004.272059-1-danielhb413@gmail.com> References: <20210916013004.272059-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::f31; envelope-from=danielhb413@gmail.com; helo=mail-qv1-xf31.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" FORM2 NUMA affinity is prepared to deal with empty (memory/cpu less) NUMA nodes. This is used by the DAX KMEM driver to locate a PAPR SCM device that has a different latency than the original NUMA node from the regular memory. FORM2 is also able to deal with asymmetric NUMA distances gracefully, something that our FORM1 implementation doesn't do. Move these FORM1 verifications to a new function and wait until after CAS, when we're sure that we're sticking with FORM1, to enforce them. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 33 ----------------------- hw/ppc/spapr_hcall.c | 6 +++++ hw/ppc/spapr_numa.c | 53 ++++++++++++++++++++++++++++++++----- include/hw/ppc/spapr_numa.h | 1 + 4 files changed, 54 insertions(+), 39 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index d39fd4e644..ada85ee083 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -2773,39 +2773,6 @@ static void spapr_machine_init(MachineState *machine) /* init CPUs */ spapr_init_cpus(spapr); - /* - * check we don't have a memory-less/cpu-less NUMA node - * Firmware relies on the existing memory/cpu topology to provide the - * NUMA topology to the kernel. - * And the linux kernel needs to know the NUMA topology at start - * to be able to hotplug CPUs later. - */ - if (machine->numa_state->num_nodes) { - for (i = 0; i < machine->numa_state->num_nodes; ++i) { - /* check for memory-less node */ - if (machine->numa_state->nodes[i].node_mem == 0) { - CPUState *cs; - int found = 0; - /* check for cpu-less node */ - CPU_FOREACH(cs) { - PowerPCCPU *cpu = POWERPC_CPU(cs); - if (cpu->node_id == i) { - found = 1; - break; - } - } - /* memory-less and cpu-less node */ - if (!found) { - error_report( - "Memory-less/cpu-less nodes are not supported (node %d)", - i); - exit(1); - } - } - } - - } - spapr->gpu_numa_id = spapr_numa_initial_nvgpu_numa_id(machine); /* Init numa_assoc_array */ diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c index 9056644890..222c1b6bbd 100644 --- a/hw/ppc/spapr_hcall.c +++ b/hw/ppc/spapr_hcall.c @@ -1198,6 +1198,12 @@ target_ulong do_client_architecture_support(PowerPCCPU *cpu, spapr->cas_pre_isa3_guest = !spapr_ovec_test(ov1_guest, OV1_PPC_3_00); spapr_ovec_cleanup(ov1_guest); + /* + * Check for NUMA affinity conditions now that we know which NUMA + * affinity the guest will use. + */ + spapr_numa_associativity_check(spapr); + /* * Ensure the guest asks for an interrupt mode we support; * otherwise terminate the boot. diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 39f9a73429..3f7f612b03 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -195,6 +195,48 @@ static void spapr_numa_define_FORM1_domains(SpaprMachineState *spapr) } +static void spapr_numa_FORM1_affinity_check(MachineState *machine) +{ + int i; + + /* + * Check we don't have a memory-less/cpu-less NUMA node + * Firmware relies on the existing memory/cpu topology to provide the + * NUMA topology to the kernel. + * And the linux kernel needs to know the NUMA topology at start + * to be able to hotplug CPUs later. + */ + if (machine->numa_state->num_nodes) { + for (i = 0; i < machine->numa_state->num_nodes; ++i) { + /* check for memory-less node */ + if (machine->numa_state->nodes[i].node_mem == 0) { + CPUState *cs; + int found = 0; + /* check for cpu-less node */ + CPU_FOREACH(cs) { + PowerPCCPU *cpu = POWERPC_CPU(cs); + if (cpu->node_id == i) { + found = 1; + break; + } + } + /* memory-less and cpu-less node */ + if (!found) { + error_report( +"Memory-less/cpu-less nodes are not supported with FORM1 NUMA (node %d)", i); + exit(EXIT_FAILURE); + } + } + } + } + + if (!spapr_numa_is_symmetrical(machine)) { + error_report( +"Asymmetrical NUMA topologies aren't supported in the pSeries machine using FORM1 NUMA"); + exit(EXIT_FAILURE); + } +} + /* * Set NUMA machine state data based on FORM1 affinity semantics. */ @@ -252,12 +294,6 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, return; } - if (!spapr_numa_is_symmetrical(machine)) { - error_report("Asymmetrical NUMA topologies aren't supported " - "in the pSeries machine"); - exit(EXIT_FAILURE); - } - spapr_numa_define_FORM1_domains(spapr); } @@ -267,6 +303,11 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr, spapr_numa_FORM1_affinity_init(spapr, machine); } +void spapr_numa_associativity_check(SpaprMachineState *spapr) +{ + spapr_numa_FORM1_affinity_check(MACHINE(spapr)); +} + void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, int offset, int nodeid) { diff --git a/include/hw/ppc/spapr_numa.h b/include/hw/ppc/spapr_numa.h index 6f9f02d3de..7cb3367400 100644 --- a/include/hw/ppc/spapr_numa.h +++ b/include/hw/ppc/spapr_numa.h @@ -24,6 +24,7 @@ */ void spapr_numa_associativity_init(SpaprMachineState *spapr, MachineState *machine); +void spapr_numa_associativity_check(SpaprMachineState *spapr); void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas); void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt, int offset, int nodeid); From patchwork Thu Sep 16 01:30:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12497983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8061C433F5 for ; Thu, 16 Sep 2021 01:38:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 28BA061186 for ; Thu, 16 Sep 2021 01:38:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 28BA061186 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:47242 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQgMA-0003Dc-D8 for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 21:38:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45316) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQgEU-0006td-JD; Wed, 15 Sep 2021 21:30:36 -0400 Received: from mail-qk1-x730.google.com ([2607:f8b0:4864:20::730]:46995) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQgER-0004JW-9B; Wed, 15 Sep 2021 21:30:34 -0400 Received: by mail-qk1-x730.google.com with SMTP id m21so5692300qkm.13; Wed, 15 Sep 2021 18:30:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KhLH/nvSmOuKDMaYuKsd/xSXDgshC53klgdBSGwSd8I=; b=jI/edX6ZwMCgFWdnSgyfzedj892X5gsOBGawtTx9cgt/jj12iPsayd0p3fMtrv9Tgm rwTzTbwGPzgmVW8xv9x6OU+xxj9Z+DwCkznwLy8NFIcDFHlo3RsN7Rq/BUqleA9PcZww U/b3sOZ6qho+fqbU5pmHqbLiNaBCzKtVXYTdf4QIxQ9HmHS3abmF75nfLGHmfrDJ+Aph pTodnfbwWJ2Z6Q1uFp80+tOlwLsJtL/c1B4FlXmRITVlisjr6NSKwNk4lEo4V6al2uIW oGGsL8cFAVL1qpygYSm/PcmKDNmZ+8u2E24SrOddQBxzKmW31oH97ZqwjvaSn8lLz343 v/VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KhLH/nvSmOuKDMaYuKsd/xSXDgshC53klgdBSGwSd8I=; b=OG4G6+CS+XLmv3rGhZ8ybLN/D7A4ulvCUk5/+pvEKjz1garp7vNUhzQ+0ZFfciki3p GDDY33JI7CJNffh2OeZY2e3PitAStMXZ6tYqNcYuxhywggW2kUKPEEONYUWdW2PFsCx9 CfNKyY86SoZ5fi0OqKIE7auV8RVIwxVfW0ZNwRYg5leICM5mKaNbIhhsh6Vm3+2ChDuC VTT2Cowurh2aJg8XsaK3Z33E4vNezvIRRFAzRY/SpIvGi/Lq5pPSYIWg+3XUHLbiCgOS qABK3iuNrNwbhIS59ZQDILTgEb7OjVPH3DndEzJ7K73DpoYJZ8PLKczB/jjpPN+qyKqc SfeA== X-Gm-Message-State: AOAM530p0c5vz6j/gNwe4qbicSBi0YdMYhsP9jhOj0XTDoQrComeqEGc SjQH5YwC4iqzn88wMs41EjRIUY/VvFw= X-Google-Smtp-Source: ABdhPJyq3oXw2AmbIRRkeuFsofnkuDN8m3M6FUbTgtrudpSjHqEncSEapDbpwJIqNG7z97f+knvA0w== X-Received: by 2002:a05:620a:20db:: with SMTP id f27mr2870305qka.496.1631755828748; Wed, 15 Sep 2021 18:30:28 -0700 (PDT) Received: from rekt.COMFAST ([177.189.43.50]) by smtp.gmail.com with ESMTPSA id 67sm1369417qkl.1.2021.09.15.18.30.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 18:30:28 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v7 6/7] spapr_numa.c: FORM2 NUMA affinity support Date: Wed, 15 Sep 2021 22:30:03 -0300 Message-Id: <20210916013004.272059-7-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210916013004.272059-1-danielhb413@gmail.com> References: <20210916013004.272059-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::730; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x730.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The main feature of FORM2 affinity support is the separation of NUMA distances from ibm,associativity information. This allows for a more flexible and straightforward NUMA distance assignment without relying on complex associations between several levels of NUMA via ibm,associativity matches. Another feature is its extensibility. This base support contains the facilities for NUMA distance assignment, but in the future more facilities will be added for latency, performance, bandwidth and so on. This patch implements the base FORM2 affinity support as follows: - the use of FORM2 associativity is indicated by using bit 2 of byte 5 of ibm,architecture-vec-5. A FORM2 aware guest can choose to use FORM1 or FORM2 affinity. Setting both forms will default to FORM2. We're not advertising FORM2 for pseries-6.1 and older machine versions to prevent guest visible changes in those; - ibm,associativity-reference-points has a new semantic. Instead of being used to calculate distances via NUMA levels, it's now used to indicate the primary domain index in the ibm,associativity domain of each resource. In our case it's set to {0x4}, matching the position where we already place logical_domain_id; - two new RTAS DT artifacts are introduced: ibm,numa-lookup-index-table and ibm,numa-distance-table. The index table is used to list all the NUMA logical domains of the platform, in ascending order, and allows for spartial NUMA configurations (although QEMU ATM doesn't support that). ibm,numa-distance-table is an array that contains all the distances from the first NUMA node to all other nodes, then the second NUMA node distances to all other nodes and so on; - get_max_dist_ref_points(), get_numa_assoc_size() and get_associativity() now checks for OV5_FORM2_AFFINITY and returns FORM2 values if the guest selected FORM2 affinity during CAS. Signed-off-by: Daniel Henrique Barboza --- hw/ppc/spapr.c | 8 ++ hw/ppc/spapr_numa.c | 148 +++++++++++++++++++++++++++++++++++- include/hw/ppc/spapr.h | 9 +++ include/hw/ppc/spapr_ovec.h | 1 + 4 files changed, 164 insertions(+), 2 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index ada85ee083..babb662845 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -2752,6 +2752,11 @@ static void spapr_machine_init(MachineState *machine) spapr_ovec_set(spapr->ov5, OV5_FORM1_AFFINITY); + /* Do not advertise FORM2 NUMA support for pseries-6.1 and older */ + if (!smc->pre_6_2_numa_affinity) { + spapr_ovec_set(spapr->ov5, OV5_FORM2_AFFINITY); + } + /* advertise support for dedicated HP event source to guests */ if (spapr->use_hotplug_event_source) { spapr_ovec_set(spapr->ov5, OV5_HP_EVT); @@ -4667,8 +4672,11 @@ DEFINE_SPAPR_MACHINE(6_2, "6.2", true); */ static void spapr_machine_6_1_class_options(MachineClass *mc) { + SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc); + spapr_machine_6_2_class_options(mc); compat_props_add(mc->compat_props, hw_compat_6_1, hw_compat_6_1_len); + smc->pre_6_2_numa_affinity = true; } DEFINE_SPAPR_MACHINE(6_1, "6.1", false); diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 3f7f612b03..659513b405 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -24,7 +24,10 @@ */ static int get_max_dist_ref_points(SpaprMachineState *spapr) { - /* No FORM2 affinity implemented yet */ + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + return FORM2_DIST_REF_POINTS; + } + return FORM1_DIST_REF_POINTS; } @@ -33,7 +36,10 @@ static int get_max_dist_ref_points(SpaprMachineState *spapr) */ static int get_numa_assoc_size(SpaprMachineState *spapr) { - /* No FORM2 affinity implemented yet */ + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + return FORM2_NUMA_ASSOC_SIZE; + } + return FORM1_NUMA_ASSOC_SIZE; } @@ -54,6 +60,9 @@ static int get_vcpu_assoc_size(SpaprMachineState *spapr) */ static uint32_t *get_associativity(SpaprMachineState *spapr, int node_id) { + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + return spapr->FORM2_assoc_array[node_id]; + } return spapr->FORM1_assoc_array[node_id]; } @@ -297,14 +306,50 @@ static void spapr_numa_FORM1_affinity_init(SpaprMachineState *spapr, spapr_numa_define_FORM1_domains(spapr); } +/* + * Init NUMA FORM2 machine state data + */ +static void spapr_numa_FORM2_affinity_init(SpaprMachineState *spapr) +{ + int i; + + /* + * For all resources but CPUs, FORM2 associativity arrays will + * be a size 2 array with the following format: + * + * ibm,associativity = {1, numa_id} + * + * CPUs will write an additional 'vcpu_id' on top of the arrays + * being initialized here. 'numa_id' is represented by the + * index 'i' of the loop. + * + * Given that this initialization is also valid for GPU associativity + * arrays, handle everything in one single step by populating the + * arrays up to NUMA_NODES_MAX_NUM. + */ + for (i = 0; i < NUMA_NODES_MAX_NUM; i++) { + spapr->FORM2_assoc_array[i][0] = cpu_to_be32(1); + spapr->FORM2_assoc_array[i][1] = cpu_to_be32(i); + } +} + void spapr_numa_associativity_init(SpaprMachineState *spapr, MachineState *machine) { spapr_numa_FORM1_affinity_init(spapr, machine); + spapr_numa_FORM2_affinity_init(spapr); } void spapr_numa_associativity_check(SpaprMachineState *spapr) { + /* + * FORM2 does not have any restrictions we need to handle + * at CAS time, for now. + */ + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + return; + } + spapr_numa_FORM1_affinity_check(MACHINE(spapr)); } @@ -449,6 +494,100 @@ static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, maxdomains, sizeof(maxdomains))); } +static void spapr_numa_FORM2_write_rtas_tables(SpaprMachineState *spapr, + void *fdt, int rtas) +{ + MachineState *ms = MACHINE(spapr); + NodeInfo *numa_info = ms->numa_state->nodes; + int nb_numa_nodes = ms->numa_state->num_nodes; + int distance_table_entries = nb_numa_nodes * nb_numa_nodes; + g_autofree uint32_t *lookup_index_table = NULL; + g_autofree uint32_t *distance_table = NULL; + int src, dst, i, distance_table_size; + uint8_t *node_distances; + + /* + * ibm,numa-lookup-index-table: array with length and a + * list of NUMA ids present in the guest. + */ + lookup_index_table = g_new0(uint32_t, nb_numa_nodes + 1); + lookup_index_table[0] = cpu_to_be32(nb_numa_nodes); + + for (i = 0; i < nb_numa_nodes; i++) { + lookup_index_table[i + 1] = cpu_to_be32(i); + } + + _FDT(fdt_setprop(fdt, rtas, "ibm,numa-lookup-index-table", + lookup_index_table, + (nb_numa_nodes + 1) * sizeof(uint32_t))); + + /* + * ibm,numa-distance-table: contains all node distances. First + * element is the size of the table as uint32, followed up + * by all the uint8 distances from the first NUMA node, then all + * distances from the second NUMA node and so on. + * + * ibm,numa-lookup-index-table is used by guest to navigate this + * array because NUMA ids can be sparse (node 0 is the first, + * node 8 is the second ...). + */ + distance_table = g_new0(uint32_t, distance_table_entries + 1); + distance_table[0] = cpu_to_be32(distance_table_entries); + + node_distances = (uint8_t *)&distance_table[1]; + i = 0; + + for (src = 0; src < nb_numa_nodes; src++) { + for (dst = 0; dst < nb_numa_nodes; dst++) { + node_distances[i++] = numa_info[src].distance[dst]; + } + } + + distance_table_size = distance_table_entries * sizeof(uint8_t) + + sizeof(uint32_t); + _FDT(fdt_setprop(fdt, rtas, "ibm,numa-distance-table", + distance_table, distance_table_size)); +} + +/* + * This helper could be compressed in a single function with + * FORM1 logic since we're setting the same DT values, with the + * difference being a call to spapr_numa_FORM2_write_rtas_tables() + * in the end. The separation was made to avoid clogging FORM1 code + * which already has to deal with compat modes from previous + * QEMU machine types. + */ +static void spapr_numa_FORM2_write_rtas_dt(SpaprMachineState *spapr, + void *fdt, int rtas) +{ + MachineState *ms = MACHINE(spapr); + uint32_t number_nvgpus_nodes = spapr->gpu_numa_id - + spapr_numa_initial_nvgpu_numa_id(ms); + + /* + * In FORM2, ibm,associativity-reference-points will point to + * the element in the ibm,associativity array that contains the + * primary domain index (for FORM2, the first element). + * + * This value (in our case, the numa-id) is then used as an index + * to retrieve all other attributes of the node (distance, + * bandwidth, latency) via ibm,numa-lookup-index-table and other + * ibm,numa-*-table properties. + */ + uint32_t refpoints[] = { cpu_to_be32(1) }; + + uint32_t maxdomain = ms->numa_state->num_nodes + number_nvgpus_nodes; + uint32_t maxdomains[] = { cpu_to_be32(1), cpu_to_be32(maxdomain) }; + + _FDT(fdt_setprop(fdt, rtas, "ibm,associativity-reference-points", + refpoints, sizeof(refpoints))); + + _FDT(fdt_setprop(fdt, rtas, "ibm,max-associativity-domains", + maxdomains, sizeof(maxdomains))); + + spapr_numa_FORM2_write_rtas_tables(spapr, fdt, rtas); +} + /* * Helper that writes ibm,associativity-reference-points and * max-associativity-domains in the RTAS pointed by @rtas @@ -456,6 +595,11 @@ static void spapr_numa_FORM1_write_rtas_dt(SpaprMachineState *spapr, */ void spapr_numa_write_rtas_dt(SpaprMachineState *spapr, void *fdt, int rtas) { + if (spapr_ovec_test(spapr->ov5_cas, OV5_FORM2_AFFINITY)) { + spapr_numa_FORM2_write_rtas_dt(spapr, fdt, rtas); + return; + } + spapr_numa_FORM1_write_rtas_dt(spapr, fdt, rtas); } diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 6b3dfc5dc2..ee7504b976 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -118,6 +118,13 @@ typedef enum { #define FORM1_DIST_REF_POINTS 4 #define FORM1_NUMA_ASSOC_SIZE (FORM1_DIST_REF_POINTS + 1) +/* + * FORM2 NUMA affinity has a single associativity domain, giving + * us a assoc size of 2. + */ +#define FORM2_DIST_REF_POINTS 1 +#define FORM2_NUMA_ASSOC_SIZE (FORM2_DIST_REF_POINTS + 1) + typedef struct SpaprCapabilities SpaprCapabilities; struct SpaprCapabilities { uint8_t caps[SPAPR_CAP_NUM]; @@ -145,6 +152,7 @@ struct SpaprMachineClass { hwaddr rma_limit; /* clamp the RMA to this size */ bool pre_5_1_assoc_refpoints; bool pre_5_2_numa_associativity; + bool pre_6_2_numa_affinity; bool (*phb_placement)(SpaprMachineState *spapr, uint32_t index, uint64_t *buid, hwaddr *pio, @@ -250,6 +258,7 @@ struct SpaprMachineState { SpaprTpmProxy *tpm_proxy; uint32_t FORM1_assoc_array[NUMA_NODES_MAX_NUM][FORM1_NUMA_ASSOC_SIZE]; + uint32_t FORM2_assoc_array[NUMA_NODES_MAX_NUM][FORM2_NUMA_ASSOC_SIZE]; Error *fwnmi_migration_blocker; }; diff --git a/include/hw/ppc/spapr_ovec.h b/include/hw/ppc/spapr_ovec.h index 48b716a060..c3e8b98e7e 100644 --- a/include/hw/ppc/spapr_ovec.h +++ b/include/hw/ppc/spapr_ovec.h @@ -49,6 +49,7 @@ typedef struct SpaprOptionVector SpaprOptionVector; /* option vector 5 */ #define OV5_DRCONF_MEMORY OV_BIT(2, 2) #define OV5_FORM1_AFFINITY OV_BIT(5, 0) +#define OV5_FORM2_AFFINITY OV_BIT(5, 2) #define OV5_HP_EVT OV_BIT(6, 5) #define OV5_HPT_RESIZE OV_BIT(6, 7) #define OV5_DRMEM_V2 OV_BIT(22, 0) From patchwork Thu Sep 16 01:30:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Henrique Barboza X-Patchwork-Id: 12497979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B733DC433EF for ; Thu, 16 Sep 2021 01:35:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7D88E61165 for ; Thu, 16 Sep 2021 01:35:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7D88E61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:41830 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mQgJd-00081d-LB for qemu-devel@archiver.kernel.org; Wed, 15 Sep 2021 21:35:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45324) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mQgEV-0006tj-7x; Wed, 15 Sep 2021 21:30:36 -0400 Received: from mail-qk1-x733.google.com ([2607:f8b0:4864:20::733]:42573) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mQgES-0004KX-9t; Wed, 15 Sep 2021 21:30:35 -0400 Received: by mail-qk1-x733.google.com with SMTP id t4so5728909qkb.9; Wed, 15 Sep 2021 18:30:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=D/FfgWFSiS86WF5yRhDm3S5AMj1DHorE3wbzzE0S9U0=; b=iXtMIzFHctCQTUzGoYsNESyB9GEB2A9AEagf14BhF+hc1D2aktySD/guL7C6gJqbUC 9xoZPX/Nxxfgpd4BC1UlS3OvrceDreNb1GMsWlO+PmEzaU+hO2zk5pFYFX5CeGYwcsJa 9qPccnMZdDbIQ0pn1q5bxe7mmQd6f+aesZlEaD0tZnGjWepOttZn5C/Qsyb/+r/QCgAx vhVqznAaKZxI0pQcvYjj19DkVqaXKDxsxZBlwByKMcII1Nhm9ckDe272iuuashoaAEZv 5pz7AgcZsO0dWuZ3ckdlOE18qGDooN48pQ9aa64j2s5E/CHpR4jI4ygOqxZTyqXwv+VR a7Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D/FfgWFSiS86WF5yRhDm3S5AMj1DHorE3wbzzE0S9U0=; b=dSbIPIrMkHGLpCefyPBb3lV4C6UlPica4bqjZx2z/nfiKJZEj79tinhP49EFqcQIZb 1ErrxV12+ciKchhvqL4FbT2dYR8jTez4ZDuJPPVDRhfHeqBmYjoT+wZU9JCrDov0dkgA fMcW8oZZHQ4sFT94ifnLG7Sp0XYl/G1MiLe0TC2QwwqxmUSzhGb3rCPssDIw3vecGpZa IA5Km9Xf8haaw6DAW7vUmOqff6uCDQLRinyZwYQkHhEh3ukMc/ztKsK/XDS9Ail2eLcX V4gjnSUypGCbcSgfSUuYtKFImAKzVcyn4QKq29FuhSUwDTd9WzVtDBAHVB1hgxPDhh5k 3J+A== X-Gm-Message-State: AOAM532KNE4KVlCyRPCAyOV9JiWjDllVV6MPUkyk5Gtmyd5rklXQA4DN h70CQkDYUc51bS5VKGTEdqkaW827EwM= X-Google-Smtp-Source: ABdhPJyMWTJ4IpgTO8bHowDf1dhHFBpT5FZ3aky+z81cyQUGkPA0FcQICEsvgIi+uUTWjU/twqyqaQ== X-Received: by 2002:a05:620a:c53:: with SMTP id u19mr2900189qki.144.1631755830996; Wed, 15 Sep 2021 18:30:30 -0700 (PDT) Received: from rekt.COMFAST ([177.189.43.50]) by smtp.gmail.com with ESMTPSA id 67sm1369417qkl.1.2021.09.15.18.30.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Sep 2021 18:30:30 -0700 (PDT) From: Daniel Henrique Barboza To: qemu-devel@nongnu.org Subject: [PATCH v7 7/7] spapr_numa.c: handle auto NUMA node with no distance info Date: Wed, 15 Sep 2021 22:30:04 -0300 Message-Id: <20210916013004.272059-8-danielhb413@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210916013004.272059-1-danielhb413@gmail.com> References: <20210916013004.272059-1-danielhb413@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::733; envelope-from=danielhb413@gmail.com; helo=mail-qk1-x733.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Igor Mammedov , Daniel Henrique Barboza , qemu-ppc@nongnu.org, groug@kaod.org, david@gibson.dropbear.id.au Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" numa_complete_configuration() in hw/core/numa.c always adds a NUMA node for the pSeries machine if none was specified, but without node distance information for the single node created. This added node is also not accounted for in numa_state->num_nodes, which returns zero. NUMA FORM1 affinity code didn't rely on numa_state information to do its job, but FORM2 does. As is now, this is the result of a pSeries guest with NUMA FORM2 affinity when no NUMA nodes is specified: $ numactl -H available: 1 nodes (0) node 0 cpus: 0 node 0 size: 16222 MB node 0 free: 15681 MB No distance information available. This can be amended in spapr_numa_FORM2_write_rtas_tables(). We're always expecting at least one NUMA node, and we're going to enforce that the local distance (the distance to the node to itself) is always 10. This allows for the proper creation of the NUMA distance tables, fixing the output of 'numactl -H' in the guest: $ numactl -H available: 1 nodes (0) node 0 cpus: 0 node 0 size: 16222 MB node 0 free: 15685 MB node distances: node 0 0: 10 CC: Igor Mammedov Signed-off-by: Daniel Henrique Barboza --- Igor, CCing you as a FYI. I wasn't sure whether there is a reason for numa_complete_configuration() not adding distance info an update 'num_nodes' for the auto-generated NUMA node, I decided to handle this case in pseries side instead. hw/ppc/spapr_numa.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c index 659513b405..d8caf5f6bd 100644 --- a/hw/ppc/spapr_numa.c +++ b/hw/ppc/spapr_numa.c @@ -499,7 +499,7 @@ static void spapr_numa_FORM2_write_rtas_tables(SpaprMachineState *spapr, { MachineState *ms = MACHINE(spapr); NodeInfo *numa_info = ms->numa_state->nodes; - int nb_numa_nodes = ms->numa_state->num_nodes; + int nb_numa_nodes = ms->numa_state->num_nodes ?: 1; int distance_table_entries = nb_numa_nodes * nb_numa_nodes; g_autofree uint32_t *lookup_index_table = NULL; g_autofree uint32_t *distance_table = NULL; @@ -539,6 +539,17 @@ static void spapr_numa_FORM2_write_rtas_tables(SpaprMachineState *spapr, for (src = 0; src < nb_numa_nodes; src++) { for (dst = 0; dst < nb_numa_nodes; dst++) { + /* + * We need to be explicit with the local distance + * value to cover the case where the user didn't added any + * NUMA nodes, but QEMU adds the default NUMA node without + * adding the numa_info to retrieve the info from. + */ + if (src == dst) { + node_distances[i++] = 10; + continue; + } + node_distances[i++] = numa_info[src].distance[dst]; } }