From patchwork Fri Sep 15 02:45:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13386310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E456EDE989 for ; Fri, 15 Sep 2023 02:47:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgyqZ-0007Lz-IN; Thu, 14 Sep 2023 22:46:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqU-0007Kr-AD; Thu, 14 Sep 2023 22:46:14 -0400 Received: from mail-sn1nam02on2060a.outbound.protection.outlook.com ([2a01:111:f400:7ea9::60a] helo=NAM02-SN1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqR-0001Um-Be; Thu, 14 Sep 2023 22:46:14 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EEXFshB/v/QJwcxCvK0OsfnzzbUTtytF4763U28UMYo8zFAMz5sP1qK8anpIeDI1tScZdJ/JCaXXnoQxwm9ERCtbBHxJx+FCpPR/M3I8geO3XYaLsZtw7LVO0pHQ5KXIiYCVsAbKBB6FNhzlUBKCsGcpATIDunztwBWzoa/bGjSuLNfE6lSMZ4xmTE869ktLg2E8IRotdbu/ySLqRCbJrrNFgoXEDkFH02MZ5vB9IC0SUz3Qp01jLvM/y9c5KOW6bSHORqmk9bMDVv3fNwMWXj7KGDURnZsBfjFUgA7QF0DQl5y2SlL8pcTGyrf8cHVEnHA/FrKUj6kAv2s203pnug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2Wb+fDgWUHVWN492avgFz7D0P7TzZZRyrDY7sVUaidA=; b=nZVMzWEdXQCJu7oWNPxAJ3CwedVr2hUrQIkBCC2O7DTJSxf6XhksWvLD8247gpdQDf7xx5Me2qFhVoe24988jTxcHD/jYY1DKoNts9HW194CckRMmNw51syoOahyn8A8g79VyfNiBDlhYuZgtUimdSHUzZDdqiNabrhUKc0rLg70buA2boo0Z7BE3nXS3XpuC1Fk34CK2d6sHIIczy0sO3su76h7VPGWsDH7dhOiq1htisKbg4zeeL9TgxW5oaCQmE6XS84KAhX2X5T4xrOk5gUvzYn3rRNE4mYqvbS89HE/E7hz0aCOUkWbBVUZMDM27//+KHLbQWyt0Xf1XEwkjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2Wb+fDgWUHVWN492avgFz7D0P7TzZZRyrDY7sVUaidA=; b=a85JsVjcq6fP8wDH6LJqCBzO0IbK9V0mbw2DnfjY8dyuVvMZ2m2MnMEE6JyTL/ze7cAcfwGvX9sEsuqh9NDQutDr6/3lf3fLh0Ca2rhBV2Xc8AEVO7ht48mvKne6OT2lLoWXq8RVhVt/2tUxSDmtWdOtuhWKbe7h8wqXnZENqhcTCj3+HKlq2iosxJ4bIl12+CnPkPwWgF9KcT/o1loQGACzmEusAso1NUEY1PSxCi3NngFYUauo9vBjB8wsoolOntOmStTeNuWPCWrpp6ePhWJ/EgUcx2Gc+SyRn7Kq3LO77fAHqMDOBb3YKi8PW3IRa2cyhUZxT73Qvb7+Mzz9Zg== Received: from MW4P223CA0005.NAMP223.PROD.OUTLOOK.COM (2603:10b6:303:80::10) by IA0PR12MB7698.namprd12.prod.outlook.com (2603:10b6:208:432::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.34; Fri, 15 Sep 2023 02:46:04 +0000 Received: from CO1PEPF000044FC.namprd21.prod.outlook.com (2603:10b6:303:80:cafe::b8) by MW4P223CA0005.outlook.office365.com (2603:10b6:303:80::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.21 via Frontend Transport; Fri, 15 Sep 2023 02:46:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000044FC.mail.protection.outlook.com (10.167.241.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Fri, 15 Sep 2023 02:46:03 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:00 -0700 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:00 -0700 Received: from ankita-dt2.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Thu, 14 Sep 2023 19:46:00 -0700 From: To: , , , , , , CC: , , , , , , , Subject: [PATCH v1 1/4] vfio: new command line params for device memory NUMA nodes Date: Thu, 14 Sep 2023 19:45:56 -0700 Message-ID: <20230915024559.6565-2-ankita@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230915024559.6565-1-ankita@nvidia.com> References: <20230915024559.6565-1-ankita@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044FC:EE_|IA0PR12MB7698:EE_ X-MS-Office365-Filtering-Correlation-Id: f4d2a141-cac7-4e47-bef2-08dbb595e74b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 09zunvNbk9AF92eWojs46+9ZWNDpRCXuIOyuJixuvCqsLXNpgVN+mwlbp65TCcN8WzZlpMaK4HJTvp1T53I5iyT5BPlzWmouZaSnj63XpZJCBqi5dkqtNVLSlE9PDEKbATuJujb9rl5aRRFZktOupPNjpP4exaiAwbygN/71BG4uEKIJANIuhnpBQE97mpCkqYgHHX3oOfAi0cgCZjBWRf3AFbzuXt6nuZ43sx0HSA+h0dr8RIujdmty43KFyPeRwLS0r1pRyeinGpKJi5c/slVq11ZhvcEBIkxWX8SVOJPuFY0oCM2RJ5vo+BW+D1zwccCbet0LId1Nwf56oBv2TQdag/lO1l+BNZ1Ry6PNa2wb5Azu2dTdTfTB1uF79SptKukG+1CYJRM21mhfuSLwxsv7DOEQ9WuvasK2k4WRyUKCD6U2EfYqua+Yo53E9I/0aiiQWSe3sXo/yMRV1jeStFknjpSE1jeiM6DheBHXZ1iK6x2C5rdCZwmgaT+MnZylQGWLgElPbExioj0701htnlQGje8dGEZHIknu/hABGqxU3aBj+jHEbcxiHQVvxABjq/oWqbXLNhH8F4wbyTuQgBT9rOSdwXMlJ5eRhEiRhWUK8KKPBEJPaaWvNy+WaqPc/bL5XIB9fqbFsFfFPfx52MtHs6DhwgXx21fatIg6wViKAZ7cFZ2JwZFK+3OOuNzVQHsF3YcYBCmRUavhvtF9qPW19y3wTWpdOUMh7wwwrUDOk1Gk9SXpzC8K/j1OkZc6 X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(136003)(376002)(396003)(346002)(39860400002)(82310400011)(186009)(451199024)(1800799009)(40470700004)(46966006)(36840700001)(36860700001)(40460700003)(47076005)(41300700001)(36756003)(110136005)(7636003)(316002)(2876002)(82740400003)(8676002)(8936002)(40480700001)(2906002)(70206006)(70586007)(4326008)(356005)(54906003)(426003)(336012)(6666004)(5660300002)(83380400001)(7696005)(1076003)(86362001)(26005)(2616005)(478600001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 02:46:03.7375 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f4d2a141-cac7-4e47-bef2-08dbb595e74b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044FC.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7698 Received-SPF: softfail client-ip=2a01:111:f400:7ea9::60a; envelope-from=ankita@nvidia.com; helo=NAM02-SN1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Ankit Agrawal The CPU cache coherent device memory can be added as a set of NUMA nodes distinct from the system memory nodes. The Qemu currently do not provide a mechanism to support node creation for a vfio-pci device. Introduce new command line parameters to allow host admin provide the desired starting NUMA node id (pxm-ns) and the number of such nodes (pxm-nc) associated with the device. In this implementation, a numerically consecutive nodes from pxm-ns to pxm-ns + pxm-nc is created. Also validate the requested range of nodes to check for conflict with other nodes and to ensure that the id do not cross QEMU limit. Since the QEMU's SRAT and DST builder code needs the proximity domain (PXM) id range, expose PXM start and count as device object properties. The device driver module communicates support for such feature through sysfs. Check the presence of the feature to activate the code. E.g. the following argument adds 8 PXM nodes starting from id 0x10. -device vfio-pci-nohotplug,host=,pxm-ns=0x10,pxm-nc=8 Signed-off-by: Ankit Agrawal --- hw/vfio/pci.c | 144 ++++++++++++++++++++++++++++++++++++ hw/vfio/pci.h | 2 + include/hw/pci/pci_device.h | 3 + 3 files changed, 149 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a205c6b113..cc0c516161 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -42,6 +42,8 @@ #include "qapi/error.h" #include "migration/blocker.h" #include "migration/qemu-file.h" +#include "qapi/visitor.h" +#include "include/hw/boards.h" #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" @@ -2955,6 +2957,22 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) } } +static void vfio_pci_get_dev_mem_pxm_start(Object *obj, Visitor *v, + const char *name, + void *opaque, Error **errp) +{ + uint64_t pxm_start = (uintptr_t) opaque; + visit_type_uint64(v, name, &pxm_start, errp); +} + +static void vfio_pci_get_dev_mem_pxm_count(Object *obj, Visitor *v, + const char *name, + void *opaque, Error **errp) +{ + uint64_t pxm_count = (uintptr_t) opaque; + visit_type_uint64(v, name, &pxm_count, errp); +} + static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) { Error *err = NULL; @@ -2974,6 +2992,125 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } +static int validate_dev_numa(uint32_t dev_node_start, uint32_t num_nodes) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + unsigned int i; + + if (num_nodes >= MAX_NODES) { + return -EINVAL; + } + + for (i = 0; i < num_nodes; i++) { + if (ms->numa_state->nodes[dev_node_start + i].present) { + return -EBUSY; + } + } + + return 0; +} + +static int mark_dev_node_present(uint32_t dev_node_start, uint32_t num_nodes) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + unsigned int i; + + for (i = 0; i < num_nodes; i++) { + ms->numa_state->nodes[dev_node_start + i].present = true; + } + + return 0; +} + + +static bool vfio_pci_read_cohmem_support_sysfs(VFIODevice *vdev) +{ + gchar *contents = NULL; + gsize length; + char *path; + bool ret = false; + uint32_t supported; + + path = g_strdup_printf("%s/coherent_mem", vdev->sysfsdev); + if (g_file_get_contents(path, &contents, &length, NULL) && length > 0) { + if ((sscanf(contents, "%u", &supported) == 1) && supported) { + ret = true; + } + } + + if (length) { + g_free(contents); + } + g_free(path); + + return ret; +} + +static int vfio_pci_dev_mem_probe(VFIOPCIDevice *vPciDev, + Error **errp) +{ + Object *obj = NULL; + VFIODevice *vdev = &vPciDev->vbasedev; + MachineState *ms = MACHINE(qdev_get_machine()); + int ret = 0; + uint32_t dev_node_start = vPciDev->dev_node_start; + uint32_t dev_node_count = vPciDev->dev_nodes; + + if (!vdev->sysfsdev || !vfio_pci_read_cohmem_support_sysfs(vdev)) { + ret = -ENODEV; + goto done; + } + + if (vdev->type == VFIO_DEVICE_TYPE_PCI) { + obj = vfio_pci_get_object(vdev); + } + + /* Since this device creates new NUMA node, hotplug is not supported. */ + if (!obj || DEVICE_CLASS(object_get_class(obj))->hotpluggable) { + ret = -EINVAL; + goto done; + } + + /* + * This device has memory that is coherently accessible from the CPU. + * The memory can be represented seperate memory-only NUMA nodes. + */ + vPciDev->pdev.has_coherent_memory = true; + + /* + * The device can create several NUMA nodes with consecutive IDs + * from dev_node_start to dev_node_start + dev_node_count. + * Verify + * - whether any node ID is occupied in the desired range. + * - Node ID is not crossing MAX_NODE. + */ + ret = validate_dev_numa(dev_node_start, dev_node_count); + if (ret) { + goto done; + } + + /* Reserve the node by marking as present */ + mark_dev_node_present(dev_node_start, dev_node_count); + + /* + * To have multiple unique nodes in the VM, a series of PXM nodes are + * required to be added to VM's SRAT. Send the information about the + * starting node ID and the node count to the ACPI builder code. + */ + object_property_add(OBJECT(vPciDev), "dev_mem_pxm_start", "uint64", + vfio_pci_get_dev_mem_pxm_start, NULL, NULL, + (void *) (uintptr_t) dev_node_start); + + object_property_add(OBJECT(vPciDev), "dev_mem_pxm_count", "uint64", + vfio_pci_get_dev_mem_pxm_count, NULL, NULL, + (void *) (uintptr_t) dev_node_count); + + ms->numa_state->num_nodes += dev_node_count; + +done: + return ret; +} + static void vfio_realize(PCIDevice *pdev, Error **errp) { VFIOPCIDevice *vdev = VFIO_PCI(pdev); @@ -3291,6 +3428,11 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } } + ret = vfio_pci_dev_mem_probe(vdev, errp); + if (ret && ret != -ENODEV) { + error_report("Failed to setup device memory with error %d", ret); + } + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); vfio_setup_resetfn_quirk(vdev); @@ -3454,6 +3596,8 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_UINT32("x-pci-sub-device-id", VFIOPCIDevice, sub_device_id, PCI_ANY_ID), DEFINE_PROP_UINT32("x-igd-gms", VFIOPCIDevice, igd_gms, 0), + DEFINE_PROP_UINT32("pxm-ns", VFIOPCIDevice, dev_node_start, 0), + DEFINE_PROP_UINT32("pxm-nc", VFIOPCIDevice, dev_nodes, 0), DEFINE_PROP_UNSIGNED_NODEFAULT("x-nv-gpudirect-clique", VFIOPCIDevice, nv_gpudirect_clique, qdev_prop_nv_gpudirect_clique, uint8_t), diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index a2771b9ff3..eef5ddfd06 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -158,6 +158,8 @@ struct VFIOPCIDevice { uint32_t display_yres; int32_t bootindex; uint32_t igd_gms; + uint32_t dev_node_start; + uint32_t dev_nodes; OffAutoPCIBAR msix_relo; uint8_t pm_cap; uint8_t nv_gpudirect_clique; diff --git a/include/hw/pci/pci_device.h b/include/hw/pci/pci_device.h index d3dd0f64b2..aacd2279ae 100644 --- a/include/hw/pci/pci_device.h +++ b/include/hw/pci/pci_device.h @@ -157,6 +157,9 @@ struct PCIDevice { MSIVectorReleaseNotifier msix_vector_release_notifier; MSIVectorPollNotifier msix_vector_poll_notifier; + /* GPU coherent memory */ + bool has_coherent_memory; + /* ID of standby device in net_failover pair */ char *failover_pair_id; uint32_t acpi_index; From patchwork Fri Sep 15 02:45:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13386309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B942CEEAA42 for ; Fri, 15 Sep 2023 02:47:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgyqf-0007Od-6U; Thu, 14 Sep 2023 22:46:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqZ-0007MG-Pb; Thu, 14 Sep 2023 22:46:21 -0400 Received: from mail-dm6nam04on20623.outbound.protection.outlook.com ([2a01:111:f400:7e8b::623] helo=NAM04-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqX-0001Vn-7R; Thu, 14 Sep 2023 22:46:18 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZrHyFnqFnZMOLXUn+HZb+dzOUuZ9ikOU27PHMnh2MAV8b6pWYuhBi2cvRHXRsvWcMnK1qnCygyYtLW9WaaP5GjIB8Lj9dOy8wZ0ltV5yfy+l4WThkQJ/keRHJudtHRo+OavnLc5ESy0wFKbB/krO/ng0z26YEMgNSD4zKRv2rh4txDs0s136tY6HGBrckQuJ7gc1ZiaC1vV0mTzDNZG2PE+/H5FjKfsiEDHQHvJGzjOTFH+yLU6W9DEBFbqV7/3nYpjidnq7vwPBWZsCCI7zVBamV+oksfNndmI5dRzjZz4Nypb5HNIpwU9V7fBI3RdYp7eHxXXRgB+K/+/sIMLBtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QVzFIaGT2MT/s8xKxDAjSmCB7BCNbTEWJ4g0vSgjfio=; b=Ng4qVHTDi7jo+Op76tH2daPcHp2tgUOHTlsgMi7x4sOxL+zWOmBDoHMomeRfdwBzlLR6myxpHKmJ74NPhcpH95900c1VqBl+PCWLvz03Z2P59E/3D/iZzSZQXk+/CGHyLqW9EQGgu9RrhfMYmZdnxqoVDu+guY2wtxMwx+UBDYOTE6imsjB1/ZsxOKo21smPSkPhL4IIIGJWRVUZHSMwaF5xopNNgdA7m7wFBGt9P3VDHvbM95hDWFKvGPVhP2fDbpwOm+PfP3764B7zi9DMxBSVZ9VZ9gqny3h541Fy2WzJfKSQ60cYZ5TAYWzDVINl/uSTzQFcdX7/Zo+Xt4cPnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QVzFIaGT2MT/s8xKxDAjSmCB7BCNbTEWJ4g0vSgjfio=; b=oB9XzG/RSjP8DEomwggmez97fOpEe4lI921jTS2FkL2XddV6HRca+Ipt38eFFhgNxoaR9dX4SUsxzHUW4UOA/GhEXgousRKZCoFg+LKkFZUcwgTChbOYaUwWLaRP2MfTLKkq/vzD5lysfZPHbO1s2HugVK/EjhL2+bNMDnA5QjluAZTfyY3oJmeHn3uzXh693jjAqKAaEmV29N3ufoI3gFselwru3/7dKIPpjhqzNWu9vCeCnFmTvjo+16PpeFZP3xd8WtfY1uq/DRiWrK6yfxno4rW6pqE1YESu3yjlC2i6vffn20BxM17thEEJmmOyxpJIAe1WHVEA80s8nulzTA== Received: from BLAPR03CA0023.namprd03.prod.outlook.com (2603:10b6:208:32b::28) by SJ0PR12MB6943.namprd12.prod.outlook.com (2603:10b6:a03:44b::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.30; Fri, 15 Sep 2023 02:46:10 +0000 Received: from BL6PEPF0001AB4B.namprd04.prod.outlook.com (2603:10b6:208:32b:cafe::f) by BLAPR03CA0023.outlook.office365.com (2603:10b6:208:32b::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Fri, 15 Sep 2023 02:46:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF0001AB4B.mail.protection.outlook.com (10.167.242.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Fri, 15 Sep 2023 02:46:09 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:00 -0700 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:00 -0700 Received: from ankita-dt2.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Thu, 14 Sep 2023 19:46:00 -0700 From: To: , , , , , , CC: , , , , , , , Subject: [PATCH v1 2/4] vfio: assign default values to node params Date: Thu, 14 Sep 2023 19:45:57 -0700 Message-ID: <20230915024559.6565-3-ankita@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230915024559.6565-1-ankita@nvidia.com> References: <20230915024559.6565-1-ankita@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4B:EE_|SJ0PR12MB6943:EE_ X-MS-Office365-Filtering-Correlation-Id: c162bee1-2648-41ef-f2e3-08dbb595eaf4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IDY1D7Zkcceh+c7aCsoXJhPnfYzRcgN6k9nPcZBSLvZnevSI9PZmafx15bEjB2zBWeoDx/xBavQfVCEgW+JE8mWn5lLsJTpnr7utncr46wtDiXL1+5s/euhhRiJs0pE6r7roRY48ig5kMMnsmn9ZBzDQe81yIEOOMwQG7CGtxQcCEpccBLNnmhcGlyut3E0GKhttiHAqshl5NMC2rjzb1eZyuTlAiICgLWcfcgwIQuENyEuBxncDjOd4GMz+tbh4fOlSPrEitFE9CVFbe8aHbnTJavmnoIux+8sWMfVA14kXFprxZ5+XDB5HPb2M+PddG40APdGKmWtc9l50Ai3Q4hX5w+pkbWMyC186fjKrsQyuvk03sC9mYbyEUpLIL0nJFA3ur/fuV9gFJJXs6KIuii9Iv4Ts4Z5XOTse7SmgC2mya4WulEYoXlBzW4AOE/O6Jj93tRTU58dY5+4asRBbP8+CUfVdLNximD0/nxH3n5N2JBozE8UQ16fpTtvWuPhrDpnq+WPKj2mbSOu6pBZXuqEfqwL4ibhuQOsJPMVDYL9agkW7ZFwtS++58hAGvBAhnIWsE3lLqSqYN+/GgG0PdmHQNssnc7J0GLvr97xEKubh/5jtF0mrExrL0MNba1XmEiyEwvilEJmliKRtgktL0IxBJs3XevxSiKPWkh1qOqyf862/q3PLnycR0CeUJ0mPsSvcL9u3rClzr//uFPrzWoes4mkeX/xEw/5NIAf1RArZx0b33qinPnXpHWZK+eUa X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(39860400002)(396003)(136003)(346002)(376002)(451199024)(1800799009)(186009)(82310400011)(40470700004)(36840700001)(46966006)(70206006)(7636003)(356005)(2616005)(7696005)(6666004)(36756003)(82740400003)(86362001)(36860700001)(47076005)(40480700001)(110136005)(40460700003)(54906003)(26005)(336012)(2906002)(5660300002)(83380400001)(41300700001)(478600001)(70586007)(426003)(4326008)(8936002)(316002)(2876002)(8676002)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 02:46:09.7858 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c162bee1-2648-41ef-f2e3-08dbb595eaf4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6943 Received-SPF: softfail client-ip=2a01:111:f400:7e8b::623; envelope-from=ankita@nvidia.com; helo=NAM04-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Ankit Agrawal It may be desirable for some deployments to have QEMU automatically pick a range and create the NUMA nodes. So the admin need not care about passing any additional params. Another advantage is that the feature is not dependent on newer libvirt that support the new parameters pxm-ns and pxm-nc. Assign default values to pxm-ns (first available node) and pxm-nc (8). This makes the new params optional and the feature will work on older libvirt. Signed-off-by: Ankit Agrawal --- hw/vfio/pci.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index cc0c516161..0bba161172 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3053,8 +3053,10 @@ static int vfio_pci_dev_mem_probe(VFIOPCIDevice *vPciDev, VFIODevice *vdev = &vPciDev->vbasedev; MachineState *ms = MACHINE(qdev_get_machine()); int ret = 0; - uint32_t dev_node_start = vPciDev->dev_node_start; - uint32_t dev_node_count = vPciDev->dev_nodes; + uint32_t dev_node_start = vPciDev->dev_node_start ? + vPciDev->dev_node_start : + ms->numa_state->num_nodes; + uint32_t dev_node_count = vPciDev->dev_nodes ? vPciDev->dev_nodes : 8; if (!vdev->sysfsdev || !vfio_pci_read_cohmem_support_sysfs(vdev)) { ret = -ENODEV; From patchwork Fri Sep 15 02:45:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13386306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A64E3EEAA42 for ; Fri, 15 Sep 2023 02:46:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgyqX-0007LF-7g; Thu, 14 Sep 2023 22:46:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqT-0007Kk-Vx; Thu, 14 Sep 2023 22:46:14 -0400 Received: from mail-mw2nam10on20610.outbound.protection.outlook.com ([2a01:111:f400:7e89::610] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqR-0001Us-7K; Thu, 14 Sep 2023 22:46:13 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z51ydGkEYVRrlSe2Y//XRHIq2oMaAxu0r6Eq9m5qmz/yj15u2Kio0+zaiRSDZxMGn7NhBa6Pw0eitGmYw7I6ITm89szgSY1yyV1K0J6VMmQjsgqLSkZySAeR5fb+zdN0K2B3VSaBJWKaWdtCMO0x1KfIod5vP4wgfLlsqoN8YmCyQLrLaoJLUEd2ra1mNsPbH8cBbpGUFq/QvW3V+HwayxNZPU44x+LC2NcU5VJ+ka/8F0TdYGNTj1wZ9FkT1jURRO+jZw95EvCtZ1eeeu3DNChFx5DSwlYCdri2tw+obBGAwvWcEENS4MRJQ9EPb3l/t29XhicNaSCOKJOdbELtrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PVXjSxuojgA09dgSl7142csCmbKd+a8IJ9YZCxwmDqo=; b=kaoRnWqAtQphkS9/X9Iuuds8brLZXuC4D+XXDiL/6S56sE9RJNpns0p4JtQzkUsRFusEsRx5WN5a6VX9qCCAQkUYzTbVY+ZaULlbEPHVMm/vDm/NCVP6VDsocIMPNmdtvZ0LQ1UhjMERo0J++wLrpr/UlbA+LcIyFrYOhDcLuHuVHR0uiAUEJ1GNx5EIARLyoeZ03YlD6oHTUr8Tajbh/vpm5jVxtCXbAvhuIUkmq/fFQR4RgaMsbRFumpsXq7ec+lGIlY9NX9ztT2V/0cxiiD9HUrMxbJM1SlYOr2KPYrjOHSQ85dA6wS9nRqo1XFa3BLyGYwhKjdG++wfJx2468w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PVXjSxuojgA09dgSl7142csCmbKd+a8IJ9YZCxwmDqo=; b=OAKVJianuSQJex+XvMsrcPJSVPU19ixQhknoZYUJLPk9gKgO4hooKoW+T6LX4WC+eSpKQYxzGCFSBStUyhnsquKaTknpHuZDR+w7wNVY0kclpGET1tJ+aq3djLxq3T7XyOm85fu45A4FbJAsaT+E7WzGIB7qASDFgUS40xmAHwD94fiMxqGl7v19qSTT5+5E9wVoTSLN8gGpo4j22LFSroGKcygyMTEj4htUE9Cynp4mr6R4VsVeDKkwSMOBgpx9fOEHwyrhWr6gZHaunem0UtKABVbNWJZzNVBYjeJ03mhlmwMt8O+RkZIGRMxwmH9WV0FbIGsxZPhCHXLiuaYVRA== Received: from MW3PR05CA0003.namprd05.prod.outlook.com (2603:10b6:303:2b::8) by SA3PR12MB7830.namprd12.prod.outlook.com (2603:10b6:806:315::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.19; Fri, 15 Sep 2023 02:46:05 +0000 Received: from CO1PEPF000044FD.namprd21.prod.outlook.com (2603:10b6:303:2b:cafe::a7) by MW3PR05CA0003.outlook.office365.com (2603:10b6:303:2b::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.21 via Frontend Transport; Fri, 15 Sep 2023 02:46:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000044FD.mail.protection.outlook.com (10.167.241.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Fri, 15 Sep 2023 02:46:05 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:01 -0700 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:00 -0700 Received: from ankita-dt2.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Thu, 14 Sep 2023 19:46:00 -0700 From: To: , , , , , , CC: , , , , , , , Subject: [PATCH v1 3/4] hw/arm/virt-acpi-build: patch guest SRAT for NUMA nodes Date: Thu, 14 Sep 2023 19:45:58 -0700 Message-ID: <20230915024559.6565-4-ankita@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230915024559.6565-1-ankita@nvidia.com> References: <20230915024559.6565-1-ankita@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044FD:EE_|SA3PR12MB7830:EE_ X-MS-Office365-Filtering-Correlation-Id: 9236e475-72c7-43d6-79cd-08dbb595e823 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: arLCz1XQUTBh2+Ria8sspmyDZDOFBxrNT7Amn9m+dl+0y0bV9+fBENuLE+lgI5Zuy40w8vA+tPjNPeeYVwuYGxM0Ffc7QyVUqFqtepoDVjr0L+JWbktr9PJp0h6V0SitWfrxjEoPXvKSctdu0JAex5w82VxgBcI+5wWgadNqAY4JjU/sWX/Ufj/IbTYbpx0QR6TF8YSOQDShjoNZf4Y9BOKyCaWlpUtJDN+aBMQ9tp/+QiYmY4tZDXQ+enNaPoZdV075I6BiM4+pnk6V5LXFfuk+oLuM0oW/egmsrFEy6bJYLjImwuqecuVvZ44Xc/UryyBxWRi78CmYyoHOqjb+ziTIXti5DhquxuOzmbf3Morcjz2VACG9n7KnxbZ3QsEsbdmR8shu0PShqwaa5Z2I7yc0Iucrh1emahIUpV/atL4cJt+EPZpjxP/QT+luMeL9cY/g2ZAS242mrkGYdkbP5aaDNElAiYNn+TziSiQt4GTCTM86+mv0jHx7LslARSek6Dmyl8gHMVKB92tXsAhTROrMvoWuCPjdP1iB+VHc5GtPhdhLObP0qTKB7u5THt31TQpgrVMmkpDEVlljUuzP4qWn3RPeR53AQkdSiQP8m74oKXU4henTWheWndbhrQ/yt5/D+zGDEiXZFYgccU5Cmmd4XyU1iZJjTIWwDbffPG9N7okFeXDjDLVHEK9aFqY/qxr8ZFB5RIsUCkE6YKar5VDdH7huIu9ecabSpaZ8Noc= X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(346002)(376002)(39860400002)(396003)(136003)(451199024)(186009)(1800799009)(82310400011)(36840700001)(40470700004)(46966006)(2906002)(5660300002)(7636003)(36756003)(8936002)(82740400003)(86362001)(40460700003)(4326008)(356005)(8676002)(316002)(54906003)(47076005)(41300700001)(36860700001)(40480700001)(2616005)(1076003)(110136005)(26005)(70586007)(7696005)(70206006)(426003)(2876002)(478600001)(336012); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 02:46:05.1357 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9236e475-72c7-43d6-79cd-08dbb595e823 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044FD.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7830 Received-SPF: softfail client-ip=2a01:111:f400:7e89::610; envelope-from=ankita@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Ankit Agrawal During bootup, Linux kernel parse the ACPI SRAT to determine the PXM ids. This allows for the creation of NUMA nodes for each unique id. Insert a series of the unique PXM ids in the VM SRAT ACPI table. The range of nodes can be determined from the "dev_mem_pxm_start" and "dev_mem_pxm_count" object properties associated with the device. These nodes as made MEM_AFFINITY_HOTPLUGGABLE. This allows the kernel to create memory-less NUMA nodes on bootup to which a subrange (or entire range) of device memory can be added/removed. Signed-off-by: Ankit Agrawal --- hw/arm/virt-acpi-build.c | 54 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 6b674231c2..6d1e3b6b8a 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -46,6 +46,7 @@ #include "hw/acpi/hmat.h" #include "hw/pci/pcie_host.h" #include "hw/pci/pci.h" +#include "hw/vfio/pci.h" #include "hw/pci/pci_bus.h" #include "hw/pci-host/gpex.h" #include "hw/arm/virt.h" @@ -515,6 +516,57 @@ build_spcr(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) acpi_table_end(linker, &table); } +static int devmem_device_list(Object *obj, void *opaque) +{ + GSList **list = opaque; + + if (object_dynamic_cast(obj, TYPE_VFIO_PCI)) { + *list = g_slist_append(*list, DEVICE(obj)); + } + + object_child_foreach(obj, devmem_device_list, opaque); + return 0; +} + +static GSList *devmem_get_device_list(void) +{ + GSList *list = NULL; + + object_child_foreach(qdev_get_machine(), devmem_device_list, &list); + return list; +} + +static void build_srat_devmem(GArray *table_data) +{ + GSList *device_list, *list = devmem_get_device_list(); + + for (device_list = list; device_list; device_list = device_list->next) { + DeviceState *dev = device_list->data; + Object *obj = OBJECT(dev); + VFIOPCIDevice *pcidev + = ((VFIOPCIDevice *)object_dynamic_cast(OBJECT(obj), + TYPE_VFIO_PCI)); + + if (pcidev->pdev.has_coherent_memory) { + uint64_t start_node = object_property_get_uint(obj, + "dev_mem_pxm_start", &error_abort); + uint64_t node_count = object_property_get_uint(obj, + "dev_mem_pxm_count", &error_abort); + uint64_t node_index; + + /* + * Add the node_count PXM domains starting from start_node as + * hot pluggable. The VM kernel parse the PXM domains and + * creates NUMA nodes. + */ + for (node_index = 0; node_index < node_count; node_index++) + build_srat_memory(table_data, 0, 0, start_node + node_index, + MEM_AFFINITY_ENABLED | MEM_AFFINITY_HOTPLUGGABLE); + } + } + g_slist_free(list); +} + /* * ACPI spec, Revision 5.1 * 5.2.16 System Resource Affinity Table (SRAT) @@ -569,6 +621,8 @@ build_srat(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) MEM_AFFINITY_HOTPLUGGABLE | MEM_AFFINITY_ENABLED); } + build_srat_devmem(table_data); + acpi_table_end(linker, &table); } From patchwork Fri Sep 15 02:45:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13386307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3EF1EDE989 for ; Fri, 15 Sep 2023 02:47:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgyqf-0007Ot-RU; Thu, 14 Sep 2023 22:46:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqZ-0007MH-Px; Thu, 14 Sep 2023 22:46:21 -0400 Received: from mail-bn7nam10on20608.outbound.protection.outlook.com ([2a01:111:f400:7e8a::608] helo=NAM10-BN7-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgyqX-0001W6-1U; Thu, 14 Sep 2023 22:46:18 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jBSYbynCCb5/BgyettXuRf9mNBsT9Bq0CHqX8VTwFQKEvHw3BxVqpPUO5bLwDgpBrZq4k/+sZyRVO8YCswMokmbyKS4Sls9ahoimW4ubvobfbMEGGJbh7PFOIU8MH/+F5S1ZFw+WjQWrEpYPKKhkcqU9MehxduI79P/9QqYZJw+kHS4SNdtyv24lsH0Ck/czef7CUVxzT/NJYIKrWGRUTuMEKR0ntJPdwNJSUARgkkvyUAZFKIMxR4f/t68MHF2JWTCTE42F7KII3fZrSTi2WndN7KxzfIS5NsOKmD2TW26fDohiMeQIJ/xq+aY4MB0crVfKF8dPAV83Pbd1vGLfsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NTUmTnIUmHO4yCTxzTlZ1vqntUu0P0NkSfWyDbf1qLo=; b=QF6nkPYzjrzfwWtY+7qIlVJ0YAthMig0kjPdFO9AacvrAUJB/N8TXBz/W8P5Lu4B3LJ3JmvoGYVE+H96DsEzO8J0tTfvwcqRPbqPf5lP8IFOU8pkos779aIYBwdcBBFcBb9uMNR5vvl5GIeKHmfmvu73yriw2x0lm/Edh0z9QMcQHNYchmD8hrTAGGPswCSieDl47hA5V6V3yY3bmxVcf3ilGQUMuMu2XEgmRdOLwi26Os69NGcSl51D2E1FKHjGFzlP4N/uh2JJRnanRZYmIFpEeLSpfYtQblyiUXUoPD6HLkwYwTLOZHhRn3R9/F2YSiKCiYIURBhop75l0gSdCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NTUmTnIUmHO4yCTxzTlZ1vqntUu0P0NkSfWyDbf1qLo=; b=Cx6lR1eIqJ1kRrhh2QkJQTeYPpC1MD0TATScW0oXU+afC0OaAptJO5Iy5HeiUfCpxKpb6RVO6lvTyqexwKYU8l8hHOXogNWGXvpPlNglSjT0oero2b6iPr2GXTa9ckudLAqk9WLCRZnd7mIlBwY1/HMYbphLW6aX2Y0BpjRt79xdMRUuAhmqPSGvNyIC+TSMoN1++l+zMdBXBvJ/QMmznEdD4s8Wz3gr6UWj03yBwkDNgyaUzbNi5aBiJWxoKKT2iKxirXgqG6lXBURMsZLYimkSmNhmpleZoyOMfTX9U2AyXdNaQ/56bIK8oePw909OMWUqY4idozJiNe9HHpNm+A== Received: from BLAPR03CA0025.namprd03.prod.outlook.com (2603:10b6:208:32b::30) by PH8PR12MB6793.namprd12.prod.outlook.com (2603:10b6:510:1c4::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.30; Fri, 15 Sep 2023 02:46:12 +0000 Received: from BL6PEPF0001AB4B.namprd04.prod.outlook.com (2603:10b6:208:32b:cafe::3b) by BLAPR03CA0025.outlook.office365.com (2603:10b6:208:32b::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.21 via Frontend Transport; Fri, 15 Sep 2023 02:46:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF0001AB4B.mail.protection.outlook.com (10.167.242.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.20 via Frontend Transport; Fri, 15 Sep 2023 02:46:11 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:01 -0700 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 14 Sep 2023 19:46:01 -0700 Received: from ankita-dt2.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Thu, 14 Sep 2023 19:46:01 -0700 From: To: , , , , , , CC: , , , , , , , Subject: [PATCH v1 4/4] acpi/gpex: patch guest DSDT for dev mem information Date: Thu, 14 Sep 2023 19:45:59 -0700 Message-ID: <20230915024559.6565-5-ankita@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230915024559.6565-1-ankita@nvidia.com> References: <20230915024559.6565-1-ankita@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4B:EE_|PH8PR12MB6793:EE_ X-MS-Office365-Filtering-Correlation-Id: b5b2ac45-3420-486b-32a1-08dbb595ec39 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IJtR2mEBFNYf4dFXQzL35SOH9aLrZxxK5TwAmRb/TS5fG2McCF5FuKblikQab5qQZ6IJBYXM3FEDA68pnxtiqq8OD47KjYeZFwSH8zdjB6+cmA5AaxXrgjWg2R/UlIydyYf8W9j6bzBl6BhATNy6kl0f39yDrplkddXJ0X0MVVaf1myPD4fELWHLxN5stNs1pBQYYKl/+0m82knpgtPQbJcz/d/YlJqAJfqhf2DhnZVMUQEI5yRvDkvkrU+4Y+7PVFSx8vqVQWARU3ZnmzQKO9F++a+mc/hXqxbJ6XkT7yJ2qfKA4hYLKcX6xyKEeYqWcY2AuuwnjLLnLEL4zOgkLELowhkuskh4MjoAEnF5idW4b+5X3NqbvUNXBDrk5tRrSzHwPudlhFfDg8ZtQaAkSI9IRJm1NaZ2G9iRafpgHT3HIjTKkgZlQrA0l/9FBjYC0wwHVqFPX/sekWyeykD05rqRNh32G60mxiWuXEcxHbiPpfYB4A61n97nqNi7SpMjPzOU2tKylkCbYF7Qc3RxdY0nzKy9qRdwoHTyS084WA1bA4aCRn2OKuC2UHZwPE9D4EIEL0IBgyay3lpFZOuT+ZedeK/yfKwUe+6pM8xw1xAfHv4uL1xaYTNmTfEQtZaAMcvYeAbmAcTkF/2eU1KQBh0YP42VvlTOxZl7g/ZyC/sjseORvpfeUSoggHMo0WPtsb6eluKHR+UR2JEGL8ObDF7/ZyD71B1r3AVFs56nAFbmaBUT1bxFREkiDnqg91Vp X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(39860400002)(396003)(136003)(376002)(346002)(451199024)(82310400011)(1800799009)(186009)(46966006)(40470700004)(36840700001)(36756003)(86362001)(40480700001)(2616005)(40460700003)(478600001)(356005)(82740400003)(7636003)(36860700001)(2876002)(316002)(1076003)(2906002)(336012)(426003)(26005)(47076005)(5660300002)(41300700001)(7696005)(8676002)(70206006)(4326008)(110136005)(70586007)(54906003)(8936002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2023 02:46:11.9108 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b5b2ac45-3420-486b-32a1-08dbb595ec39 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6793 Received-SPF: softfail client-ip=2a01:111:f400:7e8a::608; envelope-from=ankita@nvidia.com; helo=NAM10-BN7-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Ankit Agrawal To add the memory in the guest as NUMA nodes, it needs the PXM node index and the total count of nodes associated with the memory. The range of proximity domains are communicated to the VM as part of the guest ACPI using the nvidia,gpu-mem-pxm-start and nvidia,gpu-mem-pxm-count DSD properties. These value respectively represent the staring proximity domain id and the count. Kernel modules can then fetch this information and determine the numa node id using pxm_to_node(). Signed-off-by: Ankit Agrawal --- hw/pci-host/gpex-acpi.c | 69 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/hw/pci-host/gpex-acpi.c b/hw/pci-host/gpex-acpi.c index 7c7316bc96..0548feace1 100644 --- a/hw/pci-host/gpex-acpi.c +++ b/hw/pci-host/gpex-acpi.c @@ -49,6 +49,72 @@ static void acpi_dsdt_add_pci_route_table(Aml *dev, uint32_t irq) } } +static void acpi_dsdt_add_cohmem_device(Aml *dev, int32_t devfn, + uint64_t dev_mem_pxm_start, + uint64_t dev_mem_pxm_count) +{ + Aml *memdev = aml_device("CMD%X", PCI_SLOT(devfn)); + Aml *pkg = aml_package(2); + Aml *pkg1 = aml_package(2); + Aml *pkg2 = aml_package(2); + Aml *dev_pkg = aml_package(2); + Aml *UUID; + + aml_append(memdev, aml_name_decl("_ADR", aml_int(PCI_SLOT(devfn) << 16))); + + aml_append(pkg1, aml_string("dev-mem-pxm-start")); + aml_append(pkg1, aml_int(dev_mem_pxm_start)); + + aml_append(pkg2, aml_string("dev-mem-pxm-count")); + aml_append(pkg2, aml_int(dev_mem_pxm_count)); + + aml_append(pkg, pkg1); + aml_append(pkg, pkg2); + + UUID = aml_touuid("DAFFD814-6EBA-4D8C-8A91-BC9BBF4AA301"); + aml_append(dev_pkg, UUID); + aml_append(dev_pkg, pkg); + + aml_append(memdev, aml_name_decl("_DSD", dev_pkg)); + aml_append(dev, memdev); +} + +static void find_mem_device(PCIBus *bus, PCIDevice *pdev, + void *opaque) +{ + Aml *dev = (Aml *)opaque; + + if (bus == NULL) { + return; + } + + if (pdev->has_coherent_memory) { + Object *po = OBJECT(pdev); + + if (po == NULL) { + return; + } + + uint64_t pxm_start + = object_property_get_uint(po, "dev_mem_pxm_start", NULL); + uint64_t pxm_count + = object_property_get_uint(po, "dev_mem_pxm_count", NULL); + + acpi_dsdt_add_cohmem_device(dev, pdev->devfn, pxm_start, pxm_count); + } +} + +static void acpi_dsdt_find_and_add_cohmem_device(PCIBus *bus, Aml *dev) +{ + if (bus == NULL) { + return; + } + + pci_for_each_device_reverse(bus, pci_bus_num(bus), + find_mem_device, dev); + +} + static void acpi_dsdt_add_pci_osc(Aml *dev) { Aml *method, *UUID, *ifctx, *ifctx1, *elsectx, *buf; @@ -207,7 +273,10 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg) acpi_dsdt_add_pci_route_table(dev, cfg->irq); + acpi_dsdt_find_and_add_cohmem_device(cfg->bus, dev); + method = aml_method("_CBA", 0, AML_NOTSERIALIZED); + aml_append(method, aml_return(aml_int(cfg->ecam.base))); aml_append(dev, method);