From patchwork Wed Jun 26 00:28:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C019CC2BBCA for ; Wed, 26 Jun 2024 02:37:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMI5N-0003pQ-0m; Tue, 25 Jun 2024 22:08:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMI51-0003nI-C6 for qemu-devel@nongnu.org; Tue, 25 Jun 2024 22:08:18 -0400 Received: from mail-bn1nam02on20616.outbound.protection.outlook.com ([2a01:111:f403:2407::616] helo=NAM02-BN1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMI4Q-0003fc-Dx for qemu-devel@nongnu.org; Tue, 25 Jun 2024 22:08:10 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h9mz20+7Zc+3nDMJPq8YOF2kAb37d2iuqXdrXBPOlJ34CuWWSgG2mGlmjrVurLq/GGvIJJdpjP4LVmboT3qRD8MSlX6KwUK1Auqctf+WNSXmmp+8CZghVJ1LD1oNY7ZdWM+iuh6FLOvE+s1cXSm6EpI/bLhotKEeZHo0WbQ/+MdYi3rEHsusG4NCmvRhIYz8Yu5YgE92YUvaSL/+UoaxCkpHUvWKJ5iN8f/WoXQ4XiSMahJWRpHK+kOFdn8Sv/uZ3Ncd4RMyjGXw57Qm0Jp3b6LyXoYV98EWf4DrkE5NWPvY1KtR5vJ2dhMgmARiFn9FqOn9OwFhlOu2BkHGokFw+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3JMy6Or3KW8qCfpz3EkC9MUSXl+ggzi3XZvt+LeIJ38=; b=i6ysUjPO3E7dJD17VW8Z7eyKsVO/4XqVjw9cHr2G6FLf0h4w6AiJ2hAAPbxLVyEp9Efvt0ueVvK7gX438eWtWiAHB+foHDdGPEu1vT+E2p747S9S1ex4FTPXnte+LRTnNB4WoX0TM2h1VhQ5i5uM5N0ySu778vixFOI28EX/O/mdos/V6mjGXpnHY72Qt/6/exWdg/NocVM2RKoqDp+YqC31cWgPn2ZVbT+3A4NJjCo/l528uFfagQdLXyPV35EpO7ppshbTwnfUDGbvob/Si1mGZl4VNWWhQxJJTwkSTkHdxDuYXbaDThLIsCkkQtJhqRNDM4jIEGh7UxdEDWsbzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3JMy6Or3KW8qCfpz3EkC9MUSXl+ggzi3XZvt+LeIJ38=; b=s3Z7khQMR8T9G2v1ft9JqpNMQtnTKgnWwhmlMQhHUrUiqyvXvkifPsmYMwsHC74WaXEg5VEgK7Quvz8YzMRX/Dr9hAf51dnDkXZDufV7AHDh8TZzrdtNzOfg8Igtbk8obhtjmn+IIntpbYPLKP4yJe+3+MN9fx7PbpsQhWaRJwwtiTB85uQYlx2vLLH2tiZXDZqRziNnraS6cRKZ+4q042AcNy0IRctdYmz4Lzhlo6I7+BxNDnWlqk0TFmXkjwE5+cc9LNxhvhgEWN6+YbShCaBPcmGYJ28zrgXO1KiBotldJNSFVa72iind5pwyaRZsz8KUlXqdL2J78ii+vVWBjA== Received: from BN0PR08CA0023.namprd08.prod.outlook.com (2603:10b6:408:142::17) by LV3PR12MB9216.namprd12.prod.outlook.com (2603:10b6:408:1a5::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.30; Wed, 26 Jun 2024 00:29:32 +0000 Received: from BN3PEPF0000B077.namprd04.prod.outlook.com (2603:10b6:408:142:cafe::7c) by BN0PR08CA0023.outlook.office365.com (2603:10b6:408:142::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7719.22 via Frontend Transport; Wed, 26 Jun 2024 00:29:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BN3PEPF0000B077.mail.protection.outlook.com (10.167.243.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:31 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:25 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:24 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:24 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 01/10] hw/arm/virt-acpi-build: Add IORT RMR regions to handle MSI nested binding Date: Tue, 25 Jun 2024 17:28:28 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B077:EE_|LV3PR12MB9216:EE_ X-MS-Office365-Filtering-Correlation-Id: aa58b68c-3716-4f24-dbbb-08dc95770c44 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: Z+UkL8PqjmL27JtixBriGNDW0AkCocb8MiF9RcGOwiLDdeamk+Im8/VB2RFbr2B7ucowSA0czC3cgWF/HIzSkTQAIrYsgWtp9ebMfhlxQcYwrs3iJkiIApa51Zi+eOAGDUign46CbDrBuX/qo8URSXXsAOUJ1HOh+Yv89DO7oIizWLTTmZ+JQrhxsMzE+4PJWBonEQF5F73bjv3z5gA+g46RBL9OWK2/ybEpjZV+YJ1Bm8gPAXsilZ5EA0Qi+w/53fc10SACPfD95+c6tz63T/Z4IpRtiYvDhA57FStQIKX4l409a+B/1BwQgkomp9VnCHqTWWyIdrgJkpdlkrhB4fi5ahIREty77vYEoWqBrsW+Sa0x7v8ntZI7OExqvufkbpTwbvQ/qrp39wDV9shv8f8U1+RS18T7Fpq4fnSSi04YXG96bije+IM9Kbr4QvtOYTfvKqPMko6CUio5mgnPE99ePBEWMvW8QGcXxPAaLfW9UlwNRwFxkX0C/rnl7aoSnENz/Fu7Ra5RBpu9Vj8PpxaJX9QQFLSm7DOyxc4KTW/zlnHJETaglXwsclkEbj4EBlMQ2doy7zrNzuQAWCLmhr6dkxFYsW6rozqBoKc9yJIAPnQj+XcUrB2MAfXaV5yH8zyfs036bYmNl44sVpcg0qVQPM/gUKWLtZ8FJe62MwTY6hpfm8HIRNW90WuG7z7jXqHFBOQpxO6ATVqLE9B9YRGeFsjOwlidQa/U0CJBEhq2lnBxfA6uR0P2fmYxVWFwMf5yy2V/3hvs8QQlqh8UyFcOLWmiSQZP4lf1gXb6+QdjUfwaMwCtNZWU5QGjevgqGocJz8AIUW6BBnUXa+hEZegmBGKJ8wXqMXzYaD1+8K47T8RJkpN9GsO1cG5h1tSsB7SVxN3lILUF1jYeRpypk7a32v2rE9EB9SA5WBd4VCXClE0uFa0gncudBgZ7K3vOeUic+faKdLNBakQYX8RKRnigvjaAlCj9K36+FsbN8jUXA1jXnfX3KJJybN9xKT9P5JNi0ebDxr8RlpUWb2ZoG+z32ii+CQppQItH3DFAAkZkkoSFFERhRSb5jQ/mhtdp/PVeFIoSjZH33hwwZ4fHqorc1rUAKhononsg9Eq6kUxhdHki2XWMtfSGIMRJgcFE5xl+nYwJZtORjIfOwD829IlrJ4tWYN8SYZO2ngoWw8kH3SuYSOVa/2/0HAD+bRnmOIiAuAO4Q29g0SCYHLSqCSccKgdvowfuIz+0fYVi3p3dTBUL8X9Mp0+/c8+KDTaeeoZAYNwbyUyvwdTUTLaKDkifjt/UZDpsZ1kmfMNCFcIMqkfn/WY8SRgHwmhgqoQ/yFor/TNMFM42USavParSBWtpNTncjoszYQj+8nnJ3Lll8pHq/QjOWvB49rm3MVEn7+eNPheEhYMDtoYf4PWZOw== X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:31.7222 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aa58b68c-3716-4f24-dbbb-08dc95770c44 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B077.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9216 Received-SPF: softfail client-ip=2a01:111:f403:2407::616; envelope-from=nicolinc@nvidia.com; helo=NAM02-BN1-obe.outbound.protection.outlook.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, KHOP_HELO_FCRDNS=0.4, T_SPF_TEMPERROR=0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Eric Auger To handle SMMUv3 nested stage support it is practical to expose the guest with reserved memory regions (RMRs) covering the IOVAs used by the host kernel to map physical MSI doorbells. Those IOVAs belong to [0x8000000, 0x8100000] matching MSI_IOVA_BASE and MSI_IOVA_LENGTH definitions in kernel arm-smmu-v3 driver. This is the window used to allocate IOVAs matching physical MSI doorbells. With those RMRs, the guest is forced to use a flat mapping for this range. Hence the assigned device is programmed with one IOVA from this range. Stage 1, owned by the guest has a flat mapping for this IOVA. Stage2, owned by the VMM then enforces a mapping from this IOVA to the physical MSI doorbell. The creation of those RMR nodes only is relevant if nested stage SMMU is in use, along with VFIO. As VFIO devices can be hotplugged, all RMRs need to be created in advance. Hence the patch introduces a new arm virt "nested-smmuv3" iommu type. ARM DEN 0049E.b IORT specification also mandates that when RMRs are present, the OS must preserve PCIe configuration performed by the boot FW. So along with the RMR IORT nodes, a _DSM function #5, as defined by PCI FIRMWARE SPECIFICATION EVISION 3.3, chapter 4.6.5 is added to PCIe host bridge and PCIe expander bridge objects. Signed-off-by: Eric Auger Suggested-by: Jean-Philippe Brucker Signed-off-by: Nicolin Chen --- v2 -> v3: - Comply with IORT E.d spec. RMR node rev = 3; IORT rev = 5 With this spec revision, the restriction on number of Stream IDs that can be associated with memory ranges in an RMR node was removed. So no need anymore to define 1 RMR node per SID! --- hw/arm/virt-acpi-build.c | 84 +++++++++++++++++++++++++++++++++------- hw/arm/virt.c | 7 +++- include/hw/arm/virt.h | 7 ++++ 3 files changed, 84 insertions(+), 14 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 94363a6d65..d5e72800f6 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -132,6 +132,14 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap, .bus = vms->bus, }; + /* + * Nested SMMU requires RMRs for MSI 1-1 mapping, which + * require _DSM for PreservingPCI Boot Configurations + */ + if (vms->iommu == VIRT_IOMMU_NESTED_SMMUV3) { + cfg.preserve_config = true; + } + if (vms->highmem_mmio) { cfg.mmio64 = memmap[VIRT_HIGH_PCIE_MMIO]; } @@ -216,16 +224,16 @@ static void acpi_dsdt_add_tpm(Aml *scope, VirtMachineState *vms) * * Note that @id_count gets internally subtracted by one, following the spec. */ -static void build_iort_id_mapping(GArray *table_data, uint32_t input_base, - uint32_t id_count, uint32_t out_ref) +static void +build_iort_id_mapping(GArray *table_data, uint32_t input_base, + uint32_t id_count, uint32_t out_ref, uint32_t flags) { build_append_int_noprefix(table_data, input_base, 4); /* Input base */ /* Number of IDs - The number of IDs in the range minus one */ build_append_int_noprefix(table_data, id_count - 1, 4); build_append_int_noprefix(table_data, input_base, 4); /* Output base */ build_append_int_noprefix(table_data, out_ref, 4); /* Output Reference */ - /* Flags */ - build_append_int_noprefix(table_data, 0 /* Single mapping (disabled) */, 4); + build_append_int_noprefix(table_data, flags, 4); /* Flags */ } struct AcpiIortIdMapping { @@ -267,6 +275,48 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b) return idmap_a->input_base - idmap_b->input_base; } +static void +build_iort_rmr_nodes(GArray *table_data, GArray *smmu_idmaps, int smmu_offset, uint32_t *id) { + AcpiIortIdMapping *range; + int i; + + for (i = 0; i < smmu_idmaps->len; i++) { + range = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i); + int bdf = range->input_base; + + /* Table 18 Reserved Memory Range Node */ + + build_append_int_noprefix(table_data, 6 /* RMR */, 1); /* Type */ + /* Length */ + build_append_int_noprefix(table_data, 28 + ID_MAPPING_ENTRY_SIZE + 20, 2); + build_append_int_noprefix(table_data, 3, 1); /* Revision */ + build_append_int_noprefix(table_data, *id, 4); /* Identifier */ + /* Number of ID mappings */ + build_append_int_noprefix(table_data, 1, 4); + /* Reference to ID Array */ + build_append_int_noprefix(table_data, 28, 4); + + /* RMR specific data */ + + /* Flags */ + build_append_int_noprefix(table_data, 0 /* Disallow remapping */, 4); + /* Number of Memory Range Descriptors */ + build_append_int_noprefix(table_data, 1 , 4); + /* Reference to Memory Range Descriptors */ + build_append_int_noprefix(table_data, 28 + ID_MAPPING_ENTRY_SIZE, 4); + build_iort_id_mapping(table_data, bdf, range->id_count, smmu_offset, 1); + + /* Table 19 Memory Range Descriptor */ + + /* Physical Range offset */ + build_append_int_noprefix(table_data, 0x8000000, 8); + /* Physical Range length */ + build_append_int_noprefix(table_data, 0x100000, 8); + build_append_int_noprefix(table_data, 0, 4); /* Reserved */ + *id += 1; + } +} + /* * Input Output Remapping Table (IORT) * Conforms to "IO Remapping Table System Software on ARM Platforms", @@ -282,17 +332,19 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) GArray *smmu_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping)); GArray *its_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping)); - AcpiTable table = { .sig = "IORT", .rev = 3, .oem_id = vms->oem_id, + AcpiTable table = { .sig = "IORT", .rev = 5, .oem_id = vms->oem_id, .oem_table_id = vms->oem_table_id }; /* Table 2 The IORT */ acpi_table_begin(&table, table_data); - if (vms->iommu == VIRT_IOMMU_SMMUV3) { + if (virt_has_smmuv3(vms)) { AcpiIortIdMapping next_range = {0}; object_child_foreach_recursive(object_get_root(), iort_host_bridges, smmu_idmaps); + nb_nodes = 3; /* RC, ITS, SMMUv3 */ + /* Sort the smmu idmap by input_base */ g_array_sort(smmu_idmaps, iort_idmap_compare); @@ -309,6 +361,9 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) } next_range.input_base = idmap->input_base + idmap->id_count; + if (vms->iommu == VIRT_IOMMU_NESTED_SMMUV3) { + nb_nodes++; + } } /* Append the last RC -> ITS ID mapping */ @@ -317,7 +372,6 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) g_array_append_val(its_idmaps, next_range); } - nb_nodes = 3; /* RC, ITS, SMMUv3 */ rc_mapping_count = smmu_idmaps->len + its_idmaps->len; } else { nb_nodes = 2; /* RC, ITS */ @@ -342,7 +396,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) /* GIC ITS Identifier Array */ build_append_int_noprefix(table_data, 0 /* MADT translation_id */, 4); - if (vms->iommu == VIRT_IOMMU_SMMUV3) { + if (virt_has_smmuv3(vms)) { int irq = vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE; smmu_offset = table_data->len - table.table_offset; @@ -372,7 +426,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, 0, 4); /* output IORT node is the ITS group node (the first node) */ - build_iort_id_mapping(table_data, 0, 0x10000, IORT_NODE_OFFSET); + build_iort_id_mapping(table_data, 0, 0x10000, IORT_NODE_OFFSET, 0); } /* Table 17 Root Complex Node */ @@ -405,7 +459,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) build_append_int_noprefix(table_data, 0, 3); /* Reserved */ /* Output Reference */ - if (vms->iommu == VIRT_IOMMU_SMMUV3) { + if (virt_has_smmuv3(vms)) { AcpiIortIdMapping *range; /* translated RIDs connect to SMMUv3 node: RC -> SMMUv3 -> ITS */ @@ -413,7 +467,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) range = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i); /* output IORT node is the smmuv3 node */ build_iort_id_mapping(table_data, range->input_base, - range->id_count, smmu_offset); + range->id_count, smmu_offset, 0); } /* bypassed RIDs connect to ITS group node directly: RC -> ITS */ @@ -421,11 +475,15 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) range = &g_array_index(its_idmaps, AcpiIortIdMapping, i); /* output IORT node is the ITS group node (the first node) */ build_iort_id_mapping(table_data, range->input_base, - range->id_count, IORT_NODE_OFFSET); + range->id_count, IORT_NODE_OFFSET, 0); } } else { /* output IORT node is the ITS group node (the first node) */ - build_iort_id_mapping(table_data, 0, 0x10000, IORT_NODE_OFFSET); + build_iort_id_mapping(table_data, 0, 0x10000, IORT_NODE_OFFSET, 0); + } + + if (vms->iommu == VIRT_IOMMU_NESTED_SMMUV3) { + build_iort_rmr_nodes(table_data, smmu_idmaps, smmu_offset, &id); } acpi_table_end(linker, &table); diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 3c93c0c0a6..78af2d2195 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1396,7 +1396,7 @@ static void create_smmu(const VirtMachineState *vms, DeviceState *dev; MachineState *ms = MACHINE(vms); - if (vms->iommu != VIRT_IOMMU_SMMUV3 || !vms->iommu_phandle) { + if (!virt_has_smmuv3(vms) || !vms->iommu_phandle) { return; } @@ -1578,6 +1578,7 @@ static void create_pcie(VirtMachineState *vms) switch (vms->iommu) { case VIRT_IOMMU_SMMUV3: + case VIRT_IOMMU_NESTED_SMMUV3: create_smmu(vms, vms->bus); qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map", 0x0, vms->iommu_phandle, 0x0, 0x10000); @@ -2653,6 +2654,8 @@ static char *virt_get_iommu(Object *obj, Error **errp) return g_strdup("none"); case VIRT_IOMMU_SMMUV3: return g_strdup("smmuv3"); + case VIRT_IOMMU_NESTED_SMMUV3: + return g_strdup("nested-smmuv3"); default: g_assert_not_reached(); } @@ -2664,6 +2667,8 @@ static void virt_set_iommu(Object *obj, const char *value, Error **errp) if (!strcmp(value, "smmuv3")) { vms->iommu = VIRT_IOMMU_SMMUV3; + } else if (!strcmp(value, "nested-smmuv3")) { + vms->iommu = VIRT_IOMMU_NESTED_SMMUV3; } else if (!strcmp(value, "none")) { vms->iommu = VIRT_IOMMU_NONE; } else { diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index bb486d36b1..7df0813e28 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -89,6 +89,7 @@ enum { typedef enum VirtIOMMUType { VIRT_IOMMU_NONE, VIRT_IOMMU_SMMUV3, + VIRT_IOMMU_NESTED_SMMUV3, VIRT_IOMMU_VIRTIO, } VirtIOMMUType; @@ -209,4 +210,10 @@ static inline int virt_gicv3_redist_region_count(VirtMachineState *vms) vms->highmem_redists) ? 2 : 1; } +static inline bool virt_has_smmuv3(const VirtMachineState *vms) +{ + return vms->iommu == VIRT_IOMMU_SMMUV3 || + vms->iommu == VIRT_IOMMU_NESTED_SMMUV3; +} + #endif /* QEMU_ARM_VIRT_H */ From patchwork Wed Jun 26 00:28:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712204 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4CB05C2BBCA for ; Wed, 26 Jun 2024 02:43:32 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMGbd-0003ik-8B; Tue, 25 Jun 2024 20:33:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGbR-0003ZU-Nc for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:33:39 -0400 Received: from mail-mw2nam10on20631.outbound.protection.outlook.com ([2a01:111:f403:2412::631] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGaD-00059n-IG for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:33:21 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QDU4wccu7At/2lnzehvUoVYpReiKnL02oaPQWXs/n4PohA7BLeGf04g4U9AePm95le0Lp+0JRAa4XtDBdOqt+pfjzHvuEmzL4EqeNurTO5dEsmA+whC4VfkyrSdnLFeJOQ/8ChSIN/nlyqN98ZYOGJndQ170M1CTTTGLpT5HFi0AKXUk4RzDd2Z2T436htmOz9DtoopPs5yAEaDYFM/KFU6VystRxiL1g/ro1mjp4K70N1faO9zBNVYLwZGQ49otPRMC7uHz3ZW81UYrHz8bSlAWsabxcHb334fgNFmz2Db6fK8rZayV5oTRKp5YAlIy8jF3HRGIc0Z0NJ2c46FMjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GQBTV5apUoNhqJeQQWMFVr+h9r6HSkwkDNEkif0DRXY=; b=cpEslryKheOKIGgZZefgLgoL0rfK5KUtn4+ynFMXfTGcE5sNni2DA67GoIdZOJBCBW47vYpHOv3fUErzwOLSiYhVHDKAZ//edjKC1lb+3dwPspqfv1DTow45/he/1fpYKUJ3bfTQX5F10CLca3dD+X6VPTTI9yHUwsLTdySr8Thc/jT4bILioe+Tr3SyqmMwspzoTiSThoquGvDwSUkCS12sdfk+q7ZjAshMLVp2tNBAVOc6dCmALeTcOeiaBUgfNlWrWAsIM7+fyH0v7Siczu2mDeSXUN/uC3+FHemHjksQT8PgJFybGhKQMFhg7N4EOkYJdfibieACw60Qkljfrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GQBTV5apUoNhqJeQQWMFVr+h9r6HSkwkDNEkif0DRXY=; b=coQt8T86xJULk3p5KKiPONL+l14W/vuZODnI8tcz0N4nAe5T0AyYLlvKuVUaevT0xwi9vAwkk4Ygv/3FDQji6UnL0GDCatNcIt3U4x58zWkcjfvlgNczD/kAd+bIYs+xXEqtCiLQiikQppuXqRQ8wj6OGm3ELuF05RdwQKNopYvXZIiWWAgM1ijbmEfjnnhH5Q6prBsaAX0S8Nvifa9Ogb+RGDEFGCa0lJr2CPcUigPZhQuzA9yI/x0Rsyl8a+X3Kn+58pvt2yLLyrZy+3J0qjferZHuXPOcMF9uDv2I8nSVYs3RoDdY97Ovx+g6kpDKMLElDl2OTisrS8HeOL9utw== Received: from CH2PR20CA0012.namprd20.prod.outlook.com (2603:10b6:610:58::22) by PH7PR12MB6977.namprd12.prod.outlook.com (2603:10b6:510:1b7::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.30; Wed, 26 Jun 2024 00:29:35 +0000 Received: from CH1PEPF0000AD80.namprd04.prod.outlook.com (2603:10b6:610:58:cafe::f2) by CH2PR20CA0012.outlook.office365.com (2603:10b6:610:58::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.38 via Frontend Transport; Wed, 26 Jun 2024 00:29:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH1PEPF0000AD80.mail.protection.outlook.com (10.167.244.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:34 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:26 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:25 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:25 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 02/10] hw/arm/virt: Add iommufd link to virt-machine Date: Tue, 25 Jun 2024 17:28:29 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD80:EE_|PH7PR12MB6977:EE_ X-MS-Office365-Filtering-Correlation-Id: f2ef3f70-9a0b-4f64-499c-08dc95770e19 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: BsAadwv4wnk9OBj4NwcyATi69CGgqHuAB9GSorgTpuPwfgxdV5pifkpLzxR0XzluHinqdAS5/g0gqe4+f3H2A2IuvknIO/lgRSU7tlTaFnkPo6AStXPwYy5jlKJSAFpmTxfM/Hz2M8Kb6qeBqUkuAdoEdYDtfaWOVVAz3zp2PQNt5qAAxNpXbBuCa6HyOBy1vTMuxpFPPmY5naVBTy2qkXIfTVW+9kyE6PmbylylkIoIsY/rIaAtanhhT8jabJkXDXpDyaJQ5ku0QGj0YdHGXTUxOAfMxnr2YycWTKEdwsK414xXuTb2lT67P6tvBEvSTidWrhIKGB2mYF6XuLQrgnbOtHwJf89YNrvdxbiuh+zTRJAVtMeMv51vG0A0HYUPpCFAIc+Mato90LGsc/M2nANlHn4qr8V5TNG8eDfaDXysXsU+8blYi3iuT37MAUKRX9MnIO776HMGQvLdwj4qbEjakZUNvTGj5Pxeu5AQmVV58PsiU8YpRv5MdBGmnk6/sXNJXgHd5VLcp5n31BAMX96KE0IEKSOSFfQGv617DuUymSIZalxmNl3y9srjGENrvyovPGUsu5TQHg34utsRjT96NBX8XP68nevy2+hMABFBOCY2/nSQYERVu5LJKylY/FQagM1kuCwuuUgN3t2K9L8DbABonR2znPI1jTyrjUtoawnSBK4MNq/ZzSCrgAXqAdXxtqFTjcE3rqAPSO3xeq2aGQHASenqEBmoTp7VpqK4TJ++5iRuVMKHpBfsogj3xHvix9hGiB3NnBSC2Mrs+LTygVJx4o2uuctf4erlaesDWOH35TXz+eTMu0qinPwijIuCfUH7KxfJw+scI7H+vRpraZPaEoDdJGchkAf9t9uvIrK0u7V5sQMG2eh0ctY7d87ujsZ4lrjFQm8YIYjk3XRXK+Zle/rwfPxDFNYk7+bEPwcSY+XHvmA0jeIIFIPGoiABh6YRHtk2pPuHXnRaU7cAipKuz02yvlNTtF1dLklQLfsMCXHovckhbBNuZf3cmn3dLTBg89wvB8LXaHO55prF8OOP+e6Mi1HBCDAf8ERarvS/kwdOwv87P+LAyv6ESRl/uRyjqKcz9qZjKMb8FSPh+o8hqOG5Swg5Hy3R9l5npTef/qEboYN7NHvkoScvoCcoK6i5XiZS818/a6EfJx3RlNv7mS23kfrbF8bn2es1xqb/3Sk0NP1RjyGWZa4MgqLGzSA6WsRGCibyoZkgl0zp6TP2CEiyb9OgMXYTAjZdo0ceAqxLZa5vnehT4a06ZaTZmVfoQ+VoedxjiOwp0v6P9bPE/l7K/3iHO0bAvKkKHyUy+0Kup0ADtANHLTtbnVnWrHucAMzk+3MkUwI4aK0mr8DLR9WKTiwjrD/BDmvSw7LdZV5hfZc53w+bQPhSuHPc4zx1KOD9EtDURoZf/A== X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:34.8234 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f2ef3f70-9a0b-4f64-499c-08dc95770e19 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD80.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6977 Received-SPF: softfail client-ip=2a01:111:f403:2412::631; envelope-from=nicolinc@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, T_SPF_HELO_TEMPERROR=0.01, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A nested SMMU must use iommufd ioctls to communicate with the host-level SMMU instance for 2-stage translation support. Add an iommufd link to the ARM virt-machine, allowing QEMU command to pass in an iommufd object. Signed-off-by: Nicolin Chen --- hw/arm/virt.c | 14 ++++++++++++++ include/hw/arm/virt.h | 2 ++ 2 files changed, 16 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 78af2d2195..71093d7c60 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1404,6 +1404,13 @@ static void create_smmu(const VirtMachineState *vms, object_property_set_link(OBJECT(dev), "primary-bus", OBJECT(bus), &error_abort); + + if (vms->iommu == VIRT_IOMMU_NESTED_SMMUV3) { + g_assert(vms->iommufd); + object_property_set_link(OBJECT(dev), "iommufd", OBJECT(vms->iommufd), + &error_abort); + object_property_set_bool(OBJECT(dev), "nested", true, &error_abort); + } sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal); sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, base); for (i = 0; i < NUM_SMMU_IRQS; i++) { @@ -3114,6 +3121,13 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) "Set GIC version. " "Valid values are 2, 3, 4, host and max"); + object_class_property_add_link(oc, "iommufd", TYPE_IOMMUFD_BACKEND, + offsetof(VirtMachineState, iommufd), + object_property_allow_set_link, + OBJ_PROP_LINK_STRONG); + object_class_property_set_description(oc, "iommufd", + "Set the IOMMUFD handler from \"-iommufd\""); + object_class_property_add_str(oc, "iommu", virt_get_iommu, virt_set_iommu); object_class_property_set_description(oc, "iommu", "Set the IOMMU type. " diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 7df0813e28..d5cbce1a30 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -36,6 +36,7 @@ #include "hw/arm/boot.h" #include "hw/arm/bsa.h" #include "hw/block/flash.h" +#include "sysemu/iommufd.h" #include "sysemu/kvm.h" #include "hw/intc/arm_gicv3_common.h" #include "qom/object.h" @@ -154,6 +155,7 @@ struct VirtMachineState { bool dtb_randomness; OnOffAuto acpi; VirtGICType gic_version; + IOMMUFDBackend *iommufd; VirtIOMMUType iommu; bool default_bus_bypass_iommu; VirtMSIControllerType msi_controller; From patchwork Wed Jun 26 00:28:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C18DDC2BBCA for ; Wed, 26 Jun 2024 03:08:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMJ0w-0008CJ-3S; Tue, 25 Jun 2024 23:08:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMJ0u-0008Ba-Aa for qemu-devel@nongnu.org; Tue, 25 Jun 2024 23:08:04 -0400 Received: from mail-dm6nam12on20600.outbound.protection.outlook.com ([2a01:111:f403:2417::600] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMJ0s-0006HF-Bq for qemu-devel@nongnu.org; Tue, 25 Jun 2024 23:08:04 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lQpUWcYsTqYUMnXMxg7nDwYgnGCBub8wUrq6C+tdHPldTDAIaOfyNepsgmyiH0sy62X5QeW502Dn5wt2j5tqlKas72ILykKkFEIG6FBO8Wlet3YkbavSB57T05D+VbKoizsiiojQMFIoJ30PC/HAsBW9eYCzIkUQnKJp6LN5RXvQEMqB/jHW6ZSbV4L+Aj5lr3P4x7m2ESYQUOf3+BbHjS/XqZtruxSt4VPI0YAjQM035aHbRqROasOp042N+dYVEALf6UrwaHBmekk6VJNdG7GwnFKtfbSHtZ6hV7KNKnOeSnmtPkVRi3WeH4gIYV680qIu4yL+mVelv8gTr9lOOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4Va3s+7gnprUj+TGnEupq8zdY2m42H+UaaRV6cxAQFw=; b=NhWct2g5FUhdFrojE0ide86VsWZHS6sohVeb1u47njU0XmQKb9VyhH7dv4TJK05J7gvp2SLasVu5OoCppl7I099ama+XWBqTdw2JmKCeIo9rv9H5V4Va7V3RokCdabzSQH8/8KezIcqv04Y0pKY/JEDMCApXWcIhhaDTLkQcFWalI6fpkqrn+tiV1Eoa06uLm0vV0caxzXB0h+ntHCCbyR3NoK7m6mSVTWMsALUDx6TjtdQfdLTDYoArewAf/Gl6+rMfZQx3f17cYW/+7/NwOzPXytdAvg/tjcsFTL356KUauLPFjIU6xYGmN5O5nmJ8iFWC3JuDG0o83ZLGvr5tXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4Va3s+7gnprUj+TGnEupq8zdY2m42H+UaaRV6cxAQFw=; b=VO7rBSo2idipVwimQOPFO4KwcGshAHk/PmRxO6JFqV2jlUeM5yTL/eOSaGgc6dtYuPsrh8kQSoY44WOz0ONB8XtEjUBQsvNThFHkC6PXONLPru9Ts0S7TuukPIwVa9BIt+I/bdtGntZODXJtQNia/j9QEbHw6O3BgPpCfrxigwPkwch8S5y2SnefmM1b9yaW/3tJu+as0lQzfmM3/G6TAkQfo7g1Z2+P+/Zk0QBG5XtOD36ffIZOpX0jBKSuIEN8jBkTqwjSjGhS9caqNveiBshzNV2yzpj11Yl33v3I8qREjMDNd63pJUNYrClfBwg9vmj3xr62g+pFBlqcC4MuUw== Received: from CH2PR15CA0020.namprd15.prod.outlook.com (2603:10b6:610:51::30) by CH3PR12MB8185.namprd12.prod.outlook.com (2603:10b6:610:123::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.32; Wed, 26 Jun 2024 00:29:36 +0000 Received: from CH1PEPF0000AD81.namprd04.prod.outlook.com (2603:10b6:610:51:cafe::17) by CH2PR15CA0020.outlook.office365.com (2603:10b6:610:51::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7719.22 via Frontend Transport; Wed, 26 Jun 2024 00:29:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH1PEPF0000AD81.mail.protection.outlook.com (10.167.244.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:36 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:27 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:26 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:25 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 03/10] hw/arm/virt: Get the number of host-level SMMUv3 instances Date: Tue, 25 Jun 2024 17:28:30 -0700 Message-ID: <886883c4cb43117ef26e6c9434247b75cd827f31.1719361174.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD81:EE_|CH3PR12MB8185:EE_ X-MS-Office365-Filtering-Correlation-Id: e4176d2e-2eb2-4127-b7e4-08dc95770f20 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: MYczyI3t1jNqYZr48bHtUKt6okTBpV69GpnZLQVqL0p44rkz7xG7Xek+NTTIw1QeojpBUIxDQRIZ45bcjbIljOjWHBbYi6XdnASz4jP+dn024/rscYrcsgdaoRzNESgxtjBR0sFy5k5kRgLG6sAAZ9DH8Gb62453iI1NLve0QP6VyoQDkZhMI2/GLQZ00Hp0zxsVFuYiaiLZ4HBJIEMF8EcDuo5kmPWwLTOYeq+yDQsFSREzdWa+dG3UlPTbJOtOFbNV8rcGmu8KihQFJMj7gwj18UlAzSPQtS02E+51zmwBAtcPv/mTO1NIkZdgihXcjnBjP4AESaFnK04VvUtzG4X/WZ2zDq471WZVH28NfSJklopd/ig9PrGcKcR4b/EUeJyHdsHlfbz/12MHQLcjlsD8FMOvyi/jS8t9QZzhGRw9kD2EQqzPOt2vP59X8qkhc9ZoGLtBZwEvzHnpsr4JzXuvRRkiOHNTU+siJndeEhdRNG4TpwmC0Ydqrpo4EOyGpmqFIK/7h63kRvIvf2+odO05ip3qqZzHlUTnA4H4FBU2LgKR5Hbqt7pspuVqC4K60MiRa1JCS+ghR2hKP03O/UqIAe6TOdRAuSUT1T1GsfC3CCYWu94YImmOzdTvN/oYE9EJe5BHmAUkG3stPYOu5iwEItZnNVmLsyvpU2ZXxCc5oi53DOoAqjAqvC65qOnDc+UejtQFQrWCXNLKuWEjPZdxn0WW3th6J0kMc4oFgFi+sZ7W/kKK8DOQI/94KtUogfIIHA08wJU4IvWOMGZkM0TnDLT7QwZBuhxsSspvzxloXOAwYQLqSPtrVuw0BHYItBBAFT0K5v2Cw2vppJEpb648PA124QVrEv399VbYErwzNzqdQ4iAeq8L6w0ZZeb9u2DAikLFZiO9tcdHQdfjhc/4n2d1mDgw0CL60cnKGeuQEo5b8RIgDNhk+pjoSnUxLTdfUowCH1cubAFMtiOcGwarrqP2XMLWbhDGieYmTtEk84IjfECh4MjQ/vaoavJvhm6jlmDm4TgxN8aRwk+LRii5xqDnAU5skYZKqjAuYkGiE4ITl2iuH/QEmYjsYrP8MOK21IZxS8yYAwZCO1WnmjK7oyCg5G+c7KsIU1U8o8nu8nwKY/84F3A8cfYwx4zWso+8QeM+AoH4fuV1rNMuHVuyGvG3+EmTZD8H8ddsCgFrTkF7j72iBO/DA6BI/lTQa/ETsvhZG6+uI8TO8gYwnElmGbICnKcmObliIj8ttNdF/HJqHLX2JtZjzTVZiGqoxTn9NmVg0eVq3BqrvEvpfgrSx8Y9H1aUfiJnpx2cZ7pOI97yItsXuMahIQyA1rhQJn+BUrqLXnHGn21CHWCVihpWoA2EhbQXGBs2n+nA4/jhrISc9/+CXUoiaw0uKon/+gD443Q8Zd0NWMnAL5El+CLyGa2djmH0eXq8SJygDNFaAkq124fFyLmMlgzkHXZ2 X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:36.5654 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e4176d2e-2eb2-4127-b7e4-08dc95770f20 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD81.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8185 Received-SPF: softfail client-ip=2a01:111:f403:2417::600; envelope-from=nicolinc@nvidia.com; helo=NAM12-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.151, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Nested SMMUv3 feature requires the support/presence of host-level SMMUv3 instance(s). Add a helper to read the sysfs for the number of instances. Log them in a vms list using a new struct VirtNestedSmmu. This will be used by a following patch to assign a passthrough device to corresponding nested SMMUv3 instance. Signed-off-by: Nicolin Chen --- hw/arm/virt.c | 35 +++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 8 ++++++++ 2 files changed, 43 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 71093d7c60..be3d8b0ce6 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2668,6 +2668,37 @@ static char *virt_get_iommu(Object *obj, Error **errp) } } +static int virt_get_num_nested_smmus(VirtMachineState *vms, Error **errp) +{ + VirtNestedSmmu *nested_smmu; + struct dirent *dent; + DIR *dir = NULL; + int num = 0; + + dir = opendir("/sys/class/iommu"); + if (!dir) { + error_setg_errno(errp, errno, "couldn't open /sys/class/iommu"); + return 0; + } + + while ((dent = readdir(dir))) { + if (!strncmp(dent->d_name, "smmu3.0x", 7)) { + nested_smmu = g_new0(VirtNestedSmmu, 1); + nested_smmu->index = num; + nested_smmu->smmu_node = g_strdup(dent->d_name); + QLIST_INSERT_HEAD(&vms->nested_smmu_list, nested_smmu, next); + num++; + } + } + + if (num == 0) { + error_setg_errno(errp, errno, + "couldn't find any SMMUv3 HW to setup nesting"); + } + + return num; +} + static void virt_set_iommu(Object *obj, const char *value, Error **errp) { VirtMachineState *vms = VIRT_MACHINE(obj); @@ -2676,6 +2707,7 @@ static void virt_set_iommu(Object *obj, const char *value, Error **errp) vms->iommu = VIRT_IOMMU_SMMUV3; } else if (!strcmp(value, "nested-smmuv3")) { vms->iommu = VIRT_IOMMU_NESTED_SMMUV3; + vms->num_nested_smmus = virt_get_num_nested_smmus(vms, errp); } else if (!strcmp(value, "none")) { vms->iommu = VIRT_IOMMU_NONE; } else { @@ -3232,6 +3264,9 @@ static void virt_instance_init(Object *obj) /* The default root bus is attached to iommu by default */ vms->default_bus_bypass_iommu = false; + /* Default disallows nested SMMU instantiation */ + vms->num_nested_smmus = 0; + /* Default disallows RAS instantiation */ vms->ras = false; diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index d5cbce1a30..e0c07527c4 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -135,6 +135,12 @@ struct VirtMachineClass { bool no_ns_el2_virt_timer_irq; }; +typedef struct VirtNestedSmmu { + int index; + char *smmu_node; + QLIST_ENTRY(VirtNestedSmmu) next; +} VirtNestedSmmu; + struct VirtMachineState { MachineState parent; Notifier machine_done; @@ -153,6 +159,7 @@ struct VirtMachineState { bool ras; bool mte; bool dtb_randomness; + int num_nested_smmus; OnOffAuto acpi; VirtGICType gic_version; IOMMUFDBackend *iommufd; @@ -178,6 +185,7 @@ struct VirtMachineState { char *oem_id; char *oem_table_id; bool ns_el2_virt_timer_irq; + QLIST_HEAD(, VirtNestedSmmu) nested_smmu_list; }; #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM) From patchwork Wed Jun 26 00:28:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C0DDBC2BBCA for ; Wed, 26 Jun 2024 03:07:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMIyn-0006XO-G8; Tue, 25 Jun 2024 23:05:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMIye-0006W5-7y; Tue, 25 Jun 2024 23:05:44 -0400 Received: from mail-dm6nam11on20601.outbound.protection.outlook.com ([2a01:111:f403:2415::601] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMIyV-0005tO-TC; Tue, 25 Jun 2024 23:05:43 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bb49AKlO24mv5L5u0DOyIbwFLDVOId3rrcLlpE6aQbKlCgZgrGL1jc+VXky3mChEp8MLU34e2kM65z9YMmgkjT8/GHrLPvLKhrTvyTsBHttYa/d2KsVF7X0I7MOUqOV38kxLgyyS4Wiy3WMDamLxv4ew2XmJp7M6hXzSD5ArDAFX6RKe8KSFXnSPcmtVqq6rC5kUndAvqgVYDiFv2jt1t3xhApzmMKg/TyRzNOLQK1zk/QktCqS5cCPTGQqp275TXgJZ7upQWpRYgJ3cmGhX3Kj5TO8SAYO/xDrXMe/FOoljndl0CtufY8ULipi1PRjMrYz3Ja8UNgAOj9PXTdAtjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TZsvW0ATL8cXuMa4JZCuaddpeOb7hodr5FwswBZEZSY=; b=i/6sZFndFg1nHordd0YkERjEMoWgNuBdfGKQwCsRWHl9AJhwewv6A79p6XjzndM5fr0vncgShqJI9gmG+Eec/QSqL5LD8hKp2sc7X+/mOBNenPrUGb0t8Axy68Z+Eyx6MTJaj8ouia78iA/o8+5IE3ncnVTW2fqZ67PEPuHtwWLBntlrv3byIj53HXUVJH2lqcPbiW+FRSws9eiBdNuyc4gitxYJSxvzRAoS4gAVB9oGNiiJMtInBR5kqp3orT1qlxTi5tprsYgJ91RDNjZX6Y+jijgDF4zEdxUg06i9v+rV7a5x2Z6h2QyyBvhLihqXZmr8Os3BZx6GOzMKiEdPSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TZsvW0ATL8cXuMa4JZCuaddpeOb7hodr5FwswBZEZSY=; b=E6qWP2lcOTSqAQxPYg3I+1YMrmYW2SR4YQCINjiSemdzxCgDqqoqWQ/f9tE2qOQCsOrnHsUjmFmL6ymDtdwibCcPAFZgKpPXToL/mrJgtXe6EcVVZWvDN15k9TZ6KENalckE2rVy3P34NwJGf8h96ZfQz8HN7QsKQb0vvIsU1hnRNk9qdvSB3p3Ng0wYuMyntyB+yZZ6nmi3PbZBKa1vz2ZbmYDu4hPQw1Nqi9rZ17ryzC3bw2pnfAz/jccL5m0MMs1dHVEjRG7tN4FVgjm3FP1om8GFGI3cFhKnojKPA0xm2U5cs1nO0VmIN2QPTzt3N0goxo+bkNgdHbN7r20ggw== Received: from CH2PR15CA0019.namprd15.prod.outlook.com (2603:10b6:610:51::29) by DS0PR12MB6582.namprd12.prod.outlook.com (2603:10b6:8:d2::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.32; Wed, 26 Jun 2024 00:29:37 +0000 Received: from CH1PEPF0000AD81.namprd04.prod.outlook.com (2603:10b6:610:51:cafe::bd) by CH2PR15CA0019.outlook.office365.com (2603:10b6:610:51::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.38 via Frontend Transport; Wed, 26 Jun 2024 00:29:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH1PEPF0000AD81.mail.protection.outlook.com (10.167.244.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:37 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:27 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:27 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:27 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 04/10] hw/arm/virt: Add an SMMU_IO_LEN macro Date: Tue, 25 Jun 2024 17:28:31 -0700 Message-ID: <37eabf1aadd02ab0af854ff993a952af56fef6cc.1719361174.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD81:EE_|DS0PR12MB6582:EE_ X-MS-Office365-Filtering-Correlation-Id: f1d2f29f-fd99-464b-65f6-08dc95770f6f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: a6wHIzp9z0EvIIFnnFKENuKaKl3jHn7QezQ8mOjU5RKsBlMuT4UxUGpR1AJU8FTxv3EePrgfy32uBo3j3+GEp3IgjANjXvrNzKFBhsoObcXyMHR1Li2QEincke+g6FpfwIhiV0wGAFi+HcxiZMMLAf3e5GaykGVh5jrSP6zv04PmFvJZVvHfK/rcDpRMSa7H/63zUNTR88OyJFc8JWfIZ2OHc2/swQgjCXIyCOPMsM6Bzc7l265BMd9FhIiOVxZxusNi7qA+YYCdRoD9xBZRlUWbX6gh+lAQBTXaBbUplw6LMgM5JZ3H5fAcE/w71Dt12jVlf0PN6Vca04wGwxOgIfRNlmPaogIAKk30kZXvJGKWCl9i4Dd0J5jYEkD/pQtoU2hU39WNmJxC3tGe0DkvOEfEDfVRhYsBmqGH7TjsgLD8OLLeDUE+kq16GUZKvrelvn9p3o32jE38bR4BxlOPVnH1cJQsoUpxIju6NsCKpDftruogo8SKHri9W8huej2FLWbxvQ9O4qX5+AEbPkAy1byzSxZpiFtfMsFS2xYYtSIok6pytKCotuuD609ZkoXd+bmUMCu0s73APTcVBZfSjJ4Gz1h4/2nn+WacfpOS6U1Ina5ozEmnpJeZbeHvzRnUmzsaytr5aXxDx5gcgwGGc6uEKC4xM+ppIc6Q5GyTDeYDBJ7x+aVvQ36mxZ+aufqWWdug7dKc7sl7vcLjoHRhltXoS+HwQvzU8mjwmOajD2C314ByZyPko//LPMRBjYfNuwO5OWbWXGlliPy7Upm+aEdvmoahDccTNp090i6K0kPX6EmwNlpvNg1DyqZaNzgttj2GU2MMvXHQ9xQGADoeNSy/gsgHk2zxRSmpOSvcUuCbPPiYL5QT0Nn7lKgXDWBpXPzo79DUF6+QPQJX2N6f30atSoWRnW6hvzSkkQ5Lhyp9neLnPvj5dBZGuvKAApH2ckUPrwq14rCB8plBP9IP2RQIeFyO276rDt+5xQeuwGTWCNx04qaeLE6nud8MfvpTv+4Jwh5E9uO77sY3E5RTLqlKMrdoGFqhWOrDVtX+S9x0/jdQ2Vm/v6dOEygjP+LxlrRb8eD7DKNpsqm14W4WfqcNABt9iZekGqnnC3oqRJQ1LccgCtQBTPlaonkVTlknbP09deV7PmG75M3WoEYlM88KQ4wnKhkBkEj1P+5jaLMPyDWQioI6mzyykygVQ77qgxz+jAuXGOYl0vcbiI+rnAv+AhS1W4M+iNRl/ihlR+fpOMHiUwfb0x1NdJYemhil6s53KDUDtR1HzJjGb3EJrqDo9m472iK43zWGWQ9G3bkk7LTr1p1XFLTS4voTPhks/ubF4knW59CXuBiIli/gFFUZz85ZB/79NcCIn1VN153CIVxqf6NujhQHbzx+HlKZv1tJzN7gVFRUQGqKW1EFmz/1i+1xEWHS5Vrlph/gNrbF0HopSlTVgWZBZ6N9U7p2 X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:37.0810 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f1d2f29f-fd99-464b-65f6-08dc95770f6f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD81.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6582 Received-SPF: softfail client-ip=2a01:111:f403:2415::601; envelope-from=nicolinc@nvidia.com; helo=NAM11-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.151, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A following patch will add a new MMIO region for nested SMMU instances. This macro will be repeatedly used to set offsets and MMIO sizes in both virt and virt-acpi-build. Signed-off-by: Nicolin Chen --- hw/arm/virt.c | 2 +- include/hw/arm/virt.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index be3d8b0ce6..ef59c79ca3 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -170,7 +170,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_FW_CFG] = { 0x09020000, 0x00000018 }, [VIRT_GPIO] = { 0x09030000, 0x00001000 }, [VIRT_SECURE_UART] = { 0x09040000, 0x00001000 }, - [VIRT_SMMU] = { 0x09050000, 0x00020000 }, + [VIRT_SMMU] = { 0x09050000, SMMU_IO_LEN }, [VIRT_PCDIMM_ACPI] = { 0x09070000, MEMORY_HOTPLUG_IO_LEN }, [VIRT_ACPI_GED] = { 0x09080000, ACPI_GED_EVT_SEL_LEN }, [VIRT_NVDIMM_ACPI] = { 0x09090000, NVDIMM_ACPI_IO_LEN}, diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index e0c07527c4..b35c4f62d7 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -48,6 +48,9 @@ /* See Linux kernel arch/arm64/include/asm/pvclock-abi.h */ #define PVTIME_SIZE_PER_CPU 64 +/* MMIO region size for SMMUv3 */ +#define SMMU_IO_LEN (0x20000) + enum { VIRT_FLASH, VIRT_MEM, From patchwork Wed Jun 26 00:28:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 462D5C2BBCA for ; Wed, 26 Jun 2024 02:37:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMIAz-0005WS-UC; Tue, 25 Jun 2024 22:14:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMIAK-0005TC-5h for qemu-devel@nongnu.org; Tue, 25 Jun 2024 22:13:50 -0400 Received: from mail-sn1nam02on20604.outbound.protection.outlook.com ([2a01:111:f400:7ea9::604] helo=NAM02-SN1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMI9a-0004Pm-4B for qemu-devel@nongnu.org; Tue, 25 Jun 2024 22:13:29 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UA/VWhyhc7/tTdmctfMHxQgPo2oyDO0x7XGzJXOPpFLlFWXRwLMlWJgCARxeVda+0aU5YWwGtgeuw/kIhxcVV76mERPyG8Ll9Txuj878AOmrhlLvXpBl3g0spMP7A4UQCmoE9+3uaDeojGE8l9jjrETt2iapakofiOP0zFg+G63E5SpuB5yXcyUZwC6rbPrLk7vob4XfuBREJi+vRpav1zMvDp8xDAv6KuBsVN4Cnb50Ib5a06P4m6zk2xhYH1+b7/P2P8RLrTWtX2okjKg0fxyVNsUrc7Fu1qwkpidbDQ3tDaU2GDpamVgGJwhoGEy+nG0KNzo8A6zY/Bgrnbln0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VT9dhE5C30C/kMiM4NyBOMn7CmmxC/qeuX97X8XzTHA=; b=KF5vI8vgRDVDfD/T1aXfSfJUyu0yN030p7DbW/FIU7VumIzobwTgmrZwgtCezE6fRYDz8psIkHAQvfVM4h+sSZ0Ho9zu0DmNM8GtfcCt10bzhIxnCHSgMuAlblfI0aNg0Vo0wZ58Rf/0s0uNruoHFGxNFoPa46FZdzu86RUlU41VdqeVEnnwqLVROHpe9hRc1IZkltnn+Jlzmlj+J/Z1A+BeLlMr6awmaYlFb8OMOj90DNpwDkrlufDs1mXOc+C5HQl3VNVrxyBSMOmixpQ7GMM56B+u8CcD1+nN2fBzzQiSNQqrfrwyJSAvzD6t+Q9g2OULqBUcINKF9k8ZnEEKCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VT9dhE5C30C/kMiM4NyBOMn7CmmxC/qeuX97X8XzTHA=; b=b9YsiDsXFQG6zcFs0+Ipkj4wEmJvyzGUJkv076VeV2gNHpIzdKvSbAjX6DzudWfRDlVD7XOkj50Iql6dcz8BdJEeEQ+pgpK/g/FA/EP2InXgS12Jxlm5IfjSVkEzU2L6T2idL/Ik0NMB/28cjsns8IqB2etKZAX44hEWokc4id7Bn43jNLEYQ3wVI66Z508ebhyx0mpm46d3fGzpy4Z9cgZzd6E4KnacD34UEHX0eivxArvKF3XMHRCfbiHgmpjVZ1vIJVXKfjKAT50u9w0S/XS47WZHYhv6Vg2JJyE63txYy+nhBlRjDoWAjg0QUScn81RX29n3c5OntT8DWJYMEw== Received: from BN0PR07CA0029.namprd07.prod.outlook.com (2603:10b6:408:141::11) by MN6PR12MB8513.namprd12.prod.outlook.com (2603:10b6:208:472::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.26; Wed, 26 Jun 2024 00:29:37 +0000 Received: from BN3PEPF0000B071.namprd04.prod.outlook.com (2603:10b6:408:141:cafe::40) by BN0PR07CA0029.outlook.office365.com (2603:10b6:408:141::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7719.22 via Frontend Transport; Wed, 26 Jun 2024 00:29:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BN3PEPF0000B071.mail.protection.outlook.com (10.167.243.116) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:37 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:28 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:28 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:27 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 05/10] hw/arm/virt: Add VIRT_NESTED_SMMU Date: Tue, 25 Jun 2024 17:28:32 -0700 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B071:EE_|MN6PR12MB8513:EE_ X-MS-Office365-Filtering-Correlation-Id: a7eb4f23-0abe-415f-adfa-08dc95770f95 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: 8tzZCxKKHnT5xI4q+9gDUgpLbQI5kYRrya2sVdHmmCMK8FKMKyMuAMgK608KvL1NUpB99zRhd6O8wJafrhM19YcWScjeJvPXEq2+AMzxHMHgcEdW+iWIcs+zbqBZFkWvev2tA7mihAUkR0UvU+OOaAUmjsJUg7EtgkmrAvTyV4XADpo+ulmpYKpbNmwSU7Iapq8IgZsn6OsbOq5YGsOlBHj0I2ODu/oiTiXqhiMS4IFT/4k7k0FkKfNISgK/KmD0a+bKoeUNstRb35MEEj2heKgMp8McYROW+VSUD7zW3CxarLv3UIM5nJezjVpt/vZWhPcI6sCkoCdZs14DgkKGyeRIrWmeRqqxWEimveQ/IdcRPZ0b4Yq+By8UC59ZWBZbCs0fmIup5ROqH76KKwD1v3qP/XQ7YvmippIwDbgLU2DQ0hl16+AZf/+r3kQ3D4lxTkpoKHAD66EIKAPLwcGPKtrrNy2Edb8EFDI6lNIxfrDnaDvGQ/q0oMAdjYY1L4PDeLPAxaa10rFUy96LeXVDiQ6abFf31SntjSrHlJ5+jsa+D7QZejWlYx/i4TPTUPppk905cDuvWU42RF4UfTpWr1gUm+nEar5Q4f9HcKZOvT2g+KDzDj7oyMJf/I0ZCtn7roiXfI+nx1ylXiioEnMWrFdah1GuVQQDsxpLxlZgJ8DUEJ75sdNpJEZQy20ICrXd1ANTIhw34Iymg0Y1xGuoZ9A+m2U/bx1zWGgmyxZCrtXFSlCq6JLTDjZ9JwhtgM6XlRQIK4HONItYLQm6w2DqMzb67DLAb7/UhdDOS8Q+EM4QqWk9NwwoeSM2R3mYdEPS6FJaQF2IMYU9VALtdDEkX1NfbAwbUkbA6EoUHHWubBEWJxQWoczSb55Bs+TIvSsnCOWrvuhNt0AUcPl4O9i+Idcw/VfmJ1+0066n947SA7Godbl/Ny7khhzEQsqM/Licc9Xyn1fLQUmNyfgVRKdI03yFuXexS1vs5OPNati1axh3D7PALOY4HMtrbszXLoIh/CAkCAzxjcYUe+hS9EroXZAO8DWtuVOl0kyQxoVdaZDXXgHLkAf5X1jNbrprSAmw7Kk9y0uUvorL4ZbgXM7ebHzwviFe0t3aWBau989vThpZPx1bTqLDkWa7jOJXgQI2tysfVGL2A1i8edpQbIvZSmTLX6NHB7NYKhhtxe1DWh2L62Re1wWF0Rhn/m9DOCLg0ZUDxcj9MrGGhofjYti8imSEii9HdQ3j8eFr4ca4v7FtMZRr/fJ5mheuBkRtD+p/N8KaXTAhoXdqw7jFckzd6MHY6Fy/lyFtN16sbhMSwTnDwHlrW6P4fSkmqYvsoEUCMiaAO5jZuiyZjgVa0O3cRHaqy3jm8lVxaqpgs/OlAadjE2y3KGmVtq4c5DBVw9mLQbQLjliY6C1Xom8/uyEfrSjXX1RaB974q3Qa1KKrOc96KHSW249eV7l3kOecjDWC X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:37.2851 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a7eb4f23-0abe-415f-adfa-08dc95770f95 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B071.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR12MB8513 Received-SPF: softfail client-ip=2a01:111:f400:7ea9::604; envelope-from=nicolinc@nvidia.com; helo=NAM02-SN1-obe.outbound.protection.outlook.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, KHOP_HELO_FCRDNS=0.4, T_SPF_HELO_TEMPERROR=0.01, T_SPF_TEMPERROR=0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org VIRT_SMMU can be used as an emulated SMMU, i.e. iommu=smmuv3. However, the MMIO space for VIRT_SMMU isn't large enough to accommodate multiple nested SMMUv3 instances. Add a new VIRT_NESTED_SMMU to separate MMIO space. Meanwhile, a nested SMMUv3 could only work with a vfio-pci device that is physically associated to a host-level SMMUv3 instance, meaning that such a passthrough deivce must be linked to a nesteed SMMU corresponding to the underlying host-level SMMUv3 instance: HW SMMU0 <-----------------> vSMMU0 (nested) ^ ^ | | ----> PCI dev0 & PCI dev1 <---- HW SMMU1 <-----------------> vSMMU1 (nested) ^ ^ | | ----> PCI dev2 & PCI dev3 <---- Note that, on the other hand, an emulated deivce should not be associated to a nested SMMU. To prepare for that, create a PCIe Expander Bridge per nested SMMU as a docker for pci-vfio devices. It eases the QEMU code to build ID mappings in the IORT. Signed-off-by: Nicolin Chen --- hw/arm/virt.c | 110 +++++++++++++++++++++++++++++++++++++----- include/hw/arm/virt.h | 17 +++++++ 2 files changed, 116 insertions(+), 11 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index ef59c79ca3..a54332fca8 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -55,6 +55,7 @@ #include "qemu/bitops.h" #include "qemu/error-report.h" #include "qemu/module.h" +#include "hw/pci/pci_bus.h" #include "hw/pci-host/gpex.h" #include "hw/virtio/virtio-pci.h" #include "hw/core/sysbus-fdt.h" @@ -83,6 +84,7 @@ #include "hw/virtio/virtio-md-pci.h" #include "hw/virtio/virtio-iommu.h" #include "hw/char/pl011.h" +#include "qemu/config-file.h" #include "qemu/guest-random.h" static GlobalProperty arm_virt_compat[] = { @@ -177,6 +179,7 @@ static const MemMapEntry base_memmap[] = { [VIRT_PVTIME] = { 0x090a0000, 0x00010000 }, [VIRT_SECURE_GPIO] = { 0x090b0000, 0x00001000 }, [VIRT_MMIO] = { 0x0a000000, 0x00000200 }, + [VIRT_NESTED_SMMU] = { 0x0b000000, 0x01000000 }, /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ [VIRT_PLATFORM_BUS] = { 0x0c000000, 0x02000000 }, [VIRT_SECURE_MEM] = { 0x0e000000, 0x01000000 }, @@ -221,7 +224,8 @@ static const int a15irqmap[] = { [VIRT_MMIO] = 16, /* ...to 16 + NUM_VIRTIO_TRANSPORTS - 1 */ [VIRT_GIC_V2M] = 48, /* ...to 48 + NUM_GICV2M_SPIS - 1 */ [VIRT_SMMU] = 74, /* ...to 74 + NUM_SMMU_IRQS - 1 */ - [VIRT_PLATFORM_BUS] = 112, /* ...to 112 + PLATFORM_BUS_NUM_IRQS -1 */ + [VIRT_PLATFORM_BUS] = 112, /* ...to 112 + PLATFORM_BUS_NUM_IRQS -1 (179) */ + [VIRT_NESTED_SMMU] = 200, /* Keep it at the end of list */ }; static void create_randomness(MachineState *ms, const char *node) @@ -1383,21 +1387,19 @@ static void create_pcie_irq_map(const MachineState *ms, 0x7 /* PCI irq */); } -static void create_smmu(const VirtMachineState *vms, - PCIBus *bus) +static DeviceState *_create_smmu(const VirtMachineState *vms, PCIBus *bus, + hwaddr base, hwaddr size, int irq, + uint32_t smmu_phandle) { char *node; const char compat[] = "arm,smmu-v3"; - int irq = vms->irqmap[VIRT_SMMU]; int i; - hwaddr base = vms->memmap[VIRT_SMMU].base; - hwaddr size = vms->memmap[VIRT_SMMU].size; const char irq_names[] = "eventq\0priq\0cmdq-sync\0gerror"; DeviceState *dev; MachineState *ms = MACHINE(vms); - if (!virt_has_smmuv3(vms) || !vms->iommu_phandle) { - return; + if (!virt_has_smmuv3(vms) || !smmu_phandle) { + return NULL; } dev = qdev_new(TYPE_ARM_SMMUV3); @@ -1436,8 +1438,31 @@ static void create_smmu(const VirtMachineState *vms, qemu_fdt_setprop_cell(ms->fdt, node, "#iommu-cells", 1); - qemu_fdt_setprop_cell(ms->fdt, node, "phandle", vms->iommu_phandle); + qemu_fdt_setprop_cell(ms->fdt, node, "phandle", smmu_phandle); g_free(node); + + return dev; +} + +static DeviceState *create_smmu(const VirtMachineState *vms, PCIBus *bus) +{ + hwaddr base = vms->memmap[VIRT_SMMU].base; + hwaddr size = vms->memmap[VIRT_SMMU].size; + int irq = vms->irqmap[VIRT_SMMU]; + + return _create_smmu(vms, bus, base, size, irq, vms->iommu_phandle); +} + +static DeviceState *create_nested_smmu(const VirtMachineState *vms, PCIBus *bus, + int i) +{ + hwaddr base = vms->memmap[VIRT_NESTED_SMMU].base + i * SMMU_IO_LEN; + int irq = vms->irqmap[VIRT_SMMU] + i * NUM_SMMU_IRQS; + hwaddr size = SMMU_IO_LEN; + DeviceState *dev; + + dev = _create_smmu(vms, bus, base, size, irq, vms->nested_smmu_phandle[i]); + return dev; } static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) @@ -1466,6 +1491,48 @@ static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) bdf + 1, vms->iommu_phandle, bdf + 1, 0xffff - bdf); } +/* + * FIXME this is used to reverse for hotplug devices, yet it could result in a + * big waste of PCI bus numbners. + */ +#define MAX_DEV_PER_NESTED_SMMU (4) + +static PCIBus *create_pcie_expander_bridge(VirtMachineState *vms, uint8_t idx) +{ + /* Total = PXB + MAX_DEV_PER_NESTED_SMMU x (Rootport + device) */ + const uint8_t total_bus_nums = 1 + MAX_DEV_PER_NESTED_SMMU * 2; + uint8_t bus_nr = PCI_BUS_MAX - + (vms->num_nested_smmus - idx) * total_bus_nums; + + VirtNestedSmmu *nested_smmu = find_nested_smmu_by_index(vms, idx); + char *name_pxb = g_strdup_printf("pxb_for_smmu.%d", idx); + PCIBus *bus = vms->bus; + DeviceState *dev; + + /* Create an expander bridge */ + dev = qdev_new("pxb-pcie"); + if (!qdev_set_id(dev, name_pxb, &error_fatal)) { + return NULL; + } + + qdev_prop_set_uint8(dev, "bus_nr", bus_nr); + qdev_prop_set_uint16(dev, "numa_node", idx); + qdev_realize_and_unref(dev, BUS(bus), &error_fatal); + + /* Get the pxb bus */ + QLIST_FOREACH(bus, &bus->child, sibling) { + if (pci_bus_num(bus) == bus_nr) { + break; + } + } + g_assert(bus && pci_bus_num(bus) == bus_nr); + nested_smmu->pci_bus = bus; + nested_smmu->reserved_bus_nums = total_bus_nums; + + /* Return the pxb bus */ + return bus; +} + static void create_pcie(VirtMachineState *vms) { hwaddr base_mmio = vms->memmap[VIRT_PCIE_MMIO].base; @@ -1580,12 +1647,33 @@ static void create_pcie(VirtMachineState *vms) qemu_fdt_setprop_cell(ms->fdt, nodename, "#interrupt-cells", 1); create_pcie_irq_map(ms, vms->gic_phandle, irq, nodename); - if (vms->iommu) { + /* Build PCI Expander Bridge + Root Port from the top of PCI_BUS_MAX */ + if (vms->num_nested_smmus) { + /* VIRT_NESTED_SMMU must hold all vSMMUs */ + g_assert(vms->num_nested_smmus <= + vms->memmap[VIRT_NESTED_SMMU].size / SMMU_IO_LEN); + + vms->nested_smmu_phandle = g_new0(uint32_t, vms->num_nested_smmus); + + for (i = 0; i < vms->num_nested_smmus; i++) { + DeviceState *smmu_dev; + PCIBus *pxb_bus; + + pxb_bus = create_pcie_expander_bridge(vms, i); + g_assert(pxb_bus); + + vms->nested_smmu_phandle[i] = qemu_fdt_alloc_phandle(ms->fdt); + smmu_dev = create_nested_smmu(vms, pxb_bus, i); + g_assert(smmu_dev); + + qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map", 0x0, + vms->nested_smmu_phandle[i], 0x0, 0x10000); + } + } else if (vms->iommu) { vms->iommu_phandle = qemu_fdt_alloc_phandle(ms->fdt); switch (vms->iommu) { case VIRT_IOMMU_SMMUV3: - case VIRT_IOMMU_NESTED_SMMUV3: create_smmu(vms, vms->bus); qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map", 0x0, vms->iommu_phandle, 0x0, 0x10000); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index b35c4f62d7..0a3f1ab8b5 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -63,6 +63,7 @@ enum { VIRT_GIC_ITS, VIRT_GIC_REDIST, VIRT_SMMU, + VIRT_NESTED_SMMU, VIRT_UART, VIRT_MMIO, VIRT_RTC, @@ -141,6 +142,8 @@ struct VirtMachineClass { typedef struct VirtNestedSmmu { int index; char *smmu_node; + PCIBus *pci_bus; + int reserved_bus_nums; QLIST_ENTRY(VirtNestedSmmu) next; } VirtNestedSmmu; @@ -179,6 +182,7 @@ struct VirtMachineState { uint32_t gic_phandle; uint32_t msi_phandle; uint32_t iommu_phandle; + uint32_t *nested_smmu_phandle; int psci_conduit; hwaddr highest_gpa; DeviceState *gic; @@ -229,4 +233,17 @@ static inline bool virt_has_smmuv3(const VirtMachineState *vms) vms->iommu == VIRT_IOMMU_NESTED_SMMUV3; } +static inline VirtNestedSmmu * +find_nested_smmu_by_index(VirtMachineState *vms, int index) +{ + VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { + if (nested_smmu->index == index) { + return nested_smmu; + } + } + return NULL; +} + #endif /* QEMU_ARM_VIRT_H */ From patchwork Wed Jun 26 00:28:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712144 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 64400C2BBCA for ; Wed, 26 Jun 2024 00:59:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMGqk-0007qB-2s; Tue, 25 Jun 2024 20:49:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGqR-0007mw-NQ for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:49:07 -0400 Received: from mail-dm6nam11on20601.outbound.protection.outlook.com ([2a01:111:f403:2415::601] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGpy-0008Nq-H9 for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:49:04 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UIHMQY+oc9UPETFdJMzr7ijr+JemxvnxtGxQxph1fVJSM4+PPtUAyWvdVvCwfi6yAgR/47k0t5haG60DtsYaWNVfBqVo6Cc+hRWlQmmTz4uKPmnVHRLEJP0xCFDaONh747KqNC/a6zPqcth8UAk39UpO5lhS1YnwHpF41+3anY9H9lmAivt52IC7ZdBSfypXBr1vbf7XTUZXAxVCRoKiluqNihnlGEtVCH9EpQDE+C8SvsHs/OkoBT0APLetlgHOnpy2TPvszjlpLEtQXWp6DUP92oIsXiCqU3cl6cIXsJJXaFLE0VAAl+krDGrL05QeoJ65pFsV79H5C6N8sVJtXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BKGAh2ATfoLK95hp8Z/cu4KOvl4dtRq0vlHJL2bcDTM=; b=lofQvF7JGhQvjMuZ4+cwDKaNGIaqlY43WqQfkxHcBqzopNt3bC04ZNGRuwHQgKhOIMTo3DkANFZ1QFO+E3lr3pAYso3ufJDtj7SyEK60TAyVWHNwb5gtyx6tTMSdO64+T/2E6eJ18NB9P+ndjCFWSz42K8qQtYMLqqsgM/TeA7SMjulf1bqUfimqRMoX3tkBqb9hWA7OCXIlMwaPy+JqZ3h8IDbLnNHwymfwzcPVynjiAGJPeyvkMRdohEWnnjDGOvwUtoVI7FMRizyufKd7fNOjsAv2I2+hxRzTJqtVd/eZPyH+hSXck/CiY+OGlCeLMT1cf8SAwTqvlkEusHpnTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BKGAh2ATfoLK95hp8Z/cu4KOvl4dtRq0vlHJL2bcDTM=; b=lqj4sI4yuN3WSvwZCvp9c5DEXS3/E5J2JU3Z1J0EWnRGQDvD4IGEM4AedW7rwpBRe65VAOxOlZSfVz/pPDdL3XHV9bLZahQti9zapX3OCgjNmsJT8Eku7SueiqP9KsBJWnjsXVbvb4DuMFyzeEt/5Src1BzcgFpxPZgZdyGlGrL+dndgGA40gdschMUdO4GIvuCReR5vejHjKIqwAtdxtmAWaOeSoEfPHDJF0dmrPkIKiYCYBDZEKBZLtyziHNCfnLuhA08baVN89auH9fJKiSAUKbTTwBBLHishr8KwhlDmkpbkzKCL1GNOJnPyyNKNiJ9sUaxtMPSJRBVFw5VEMA== Received: from BN9PR03CA0936.namprd03.prod.outlook.com (2603:10b6:408:108::11) by PH0PR12MB7813.namprd12.prod.outlook.com (2603:10b6:510:286::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.32; Wed, 26 Jun 2024 00:29:39 +0000 Received: from BN3PEPF0000B075.namprd04.prod.outlook.com (2603:10b6:408:108:cafe::88) by BN9PR03CA0936.outlook.office365.com (2603:10b6:408:108::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.38 via Frontend Transport; Wed, 26 Jun 2024 00:29:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BN3PEPF0000B075.mail.protection.outlook.com (10.167.243.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:38 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:29 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:29 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:28 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 06/10] hw/arm/virt: Assign vfio-pci devices to nested SMMUs Date: Tue, 25 Jun 2024 17:28:33 -0700 Message-ID: <67c6311756de2a6e827e3dd0563f939dcf334418.1719361174.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B075:EE_|PH0PR12MB7813:EE_ X-MS-Office365-Filtering-Correlation-Id: a3299bac-6fc8-43da-2099-08dc95771097 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: 71hlhyHHRosO4ciB93LxsiVk548v2TJavPB/gFRnm00Z7BuDmiDgLYGPF9XXIFaivCcCf94MCiLIa3ePd8d+RB5L60haJ8eTVTOzulpn4omJvYj3QLaFLSe8ypPWvXZ0C9N8Tfj0eZH0MuYSsFFJ1zXsC0QeAF0yZ0HY/wIN/sHxkAeEIMYBX/ou68j23STxLxArnhd3o0PnPSOJhu1r1K3rCKgbVvtBevZptRKqZ+RokWkgq/QO1knw/zX55j3kLzlF68NDHVtwkzpBVSV8O1F3Y6+GwEYQCF245ZCgeKWfDHbkxq5YUZDk3VvTmHZ2BA3/wo8fFmJ/aGEY2brrejJMzeZou9YQ9/lsEPQK0SHU+e7LWAh7I8Jh/gxeLftksalydSpjJX0wdSrvIflA7HyY3SPNWmzUBghHYBi8K28DIUbnn4y0lsN69pn0nk5i8vvXC9oQn/ywRIq2zVVvVzsEHdygEvW0FVaiH5Tvs1IepH6eN5Ja8X/uPSDyi0TfOij7zoKYUL9YYuUW1deUv/bsT/+M4XQpOG2XxU9s4nfPtuhfQc5MzM11M/IQIyViV0HCY2ZOz1aSsE9a28Siy98GrfvXHwud23af3KHyNlw60sw4AnQghY2jinYCOuDTGgEqt16Z5yGNYY564MIm65lodXkxTuH8JPHo4Tfy4CBLmmqPWL9D7hS2jQurOlGb6PSp3IOYZyis/yF/5VI/bdiewAytq6k33VzbDcy2dvbkudQpBQ+Z6b6LkYWvlyybYChwz/5gHZ/Vb7i8XdSn+Gv6gHevMprv3f9IUWndMXUDPtg/8sGudJ2PiGFyF6pBxYx9uDbrmnb1Gwb7/MBS7hAIbXrgWyW3L0A+a/sT4PCJpksd2VFoQkVtQYC8r9yYXd4uTRs8n5bvG+2d83foZ8RiSjVc7tuFiKXppSS16GGvmujL0Rx10o2guxh2WIgmKPUljimkEpLjcCDbiLedBC0M1RoykT4aVt9aU+RlJD3nK9MCXz+V2dulgS3fbo3MMNtaZDhJs1I9Xenh8Bpv+5KpRw//0OudKuvaNSYbeQ3HlVujI45IJxJDbaId1fwADc9wzZ/yj/4kbw17RSDV5t2dWtlmlS4CDSgDF1mbQ+UzvtdvZ4sGg2LdPfb/+m78ZUyjUw5BRpbE68csU8LB4UKpf20dTnFYbhDwUO7GVt+AujY93z7T2I53t66saBCxMFUrzIJpkvH6CAI/dHJH1UPRWbLQTNXQyJIWr9cajlHpAA8Hs8iPjnA6YYcsk3lhHzNAtJmYANr8j2TJfVBniC4oEh7XCmNKwDdIweofhht66KXqFh/pTDYxtnTPp95EwnmGANR1m4dPP489yuYbmAnX+4EanoNAWVegm2rxVWBXTwSoPyi248BRauepeOnoFnEe7xo4purM/SHQlmNxkwbRGgEnUJginLVQi+7eiANLlR2vkn1sr6wneetxq/DD X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:38.9723 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a3299bac-6fc8-43da-2099-08dc95771097 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B075.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7813 Received-SPF: softfail client-ip=2a01:111:f403:2415::601; envelope-from=nicolinc@nvidia.com; helo=NAM11-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FORGED_SPF_HELO=1, SPF_HELO_PASS=-0.001, T_SPF_TEMPERROR=0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org With iommu=nested-smmuv3, there could be multiple nested SMMU instances in the vms. A passthrough device must to look up for its iommu handler in its sysfs node, and then link to the nested SMMU instance created for the same iommu handler. This isn't easy to do. Add an auto-assign piece after all vSMMU backed pxb buses are created. It loops the existing input devices, and sets/replaces their pci bus numbers with a newly created pcie-root-port to the pxb bus. Note that this is not an ideal solution to handle hot plug device. Signed-off-by: Nicolin Chen --- hw/arm/virt.c | 110 ++++++++++++++++++++++++++++++++++++++++++ include/hw/arm/virt.h | 13 +++++ 2 files changed, 123 insertions(+) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index a54332fca8..3610f53304 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -38,6 +38,7 @@ #include "hw/arm/primecell.h" #include "hw/arm/virt.h" #include "hw/block/flash.h" +#include "hw/vfio/pci.h" #include "hw/vfio/vfio-calxeda-xgmac.h" #include "hw/vfio/vfio-amd-xgbe.h" #include "hw/display/ramfb.h" @@ -1491,6 +1492,112 @@ static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) bdf + 1, vms->iommu_phandle, bdf + 1, 0xffff - bdf); } +static char *create_new_pcie_port(VirtNestedSmmu *nested_smmu, Error **errp) +{ + uint32_t port_nr = nested_smmu->pci_bus->qbus.num_children; + uint32_t chassis_nr = UINT8_MAX - nested_smmu->index; + uint32_t bus_nr = pci_bus_num(nested_smmu->pci_bus); + DeviceState *dev; + char *name_port; + + /* Create a root port */ + dev = qdev_new("pcie-root-port"); + name_port = g_strdup_printf("smmu_bus0x%x_port%d", bus_nr, port_nr); + + if (!qdev_set_id(dev, name_port, &error_fatal)) { + /* FIXME retry with a different port num? */ + error_setg(errp, "Could not set pcie-root-port ID %s", name_port); + g_free(name_port); + g_free(dev); + return NULL; + } + qdev_prop_set_uint32(dev, "chassis", chassis_nr); + qdev_prop_set_uint32(dev, "slot", port_nr); + qdev_prop_set_uint64(dev, "io-reserve", 0); + qdev_realize_and_unref(dev, BUS(nested_smmu->pci_bus), &error_fatal); + return name_port; +} + +static int assign_nested_smmu(void *opaque, QemuOpts *opts, Error **errp) +{ + VirtMachineState *vms = (VirtMachineState *)opaque; + const char *sysfsdev = qemu_opt_get(opts, "sysfsdev"); + const char *iommufd = qemu_opt_get(opts, "iommufd"); + const char *driver = qemu_opt_get(opts, "driver"); + const char *host = qemu_opt_get(opts, "host"); + const char *bus = qemu_opt_get(opts, "bus"); + VirtNestedSmmu *nested_smmu; + char *link_iommu; + char *dir_iommu; + char *smmu_node; + char *name_port; + int ret = 0; + + if (!iommufd || !driver) { + return 0; + } + if (!sysfsdev && !host) { + return 0; + } + if (strncmp(driver, TYPE_VFIO_PCI, strlen(TYPE_VFIO_PCI))) { + return 0; + } + /* If the device wants to attach to the default bus, do not reassign it */ + if (bus && !strncmp(bus, "pcie.0", strlen(bus))) { + return 0; + } + + if (sysfsdev) { + link_iommu = g_strdup_printf("%s/iommu", sysfsdev); + } else { + link_iommu = g_strdup_printf("/sys/bus/pci/devices/%s/iommu", host); + } + + dir_iommu = realpath(link_iommu, NULL); + if (!dir_iommu) { + error_setg(errp, "Could not get the real path for iommu link: %s", + link_iommu); + ret = -EINVAL; + goto free_link; + } + + smmu_node = g_path_get_basename(dir_iommu); + if (!smmu_node) { + error_setg(errp, "Could not get SMMU node name for iommu at: %s", + dir_iommu); + ret = -EINVAL; + goto free_dir; + } + + nested_smmu = find_nested_smmu_by_sysfs(vms, smmu_node); + if (!nested_smmu) { + error_setg(errp, "Could not find any detected SMMU matching node: %s", + smmu_node); + ret = -EINVAL; + goto free_node; + } + + name_port = create_new_pcie_port(nested_smmu, errp); + if (!name_port) { + ret = -EBUSY; + goto free_node; + } + + qemu_opt_set(opts, "bus", name_port, &error_fatal); + if (bus) { + error_report("overriding PCI bus %s to %s for device %s [%s]", + bus, name_port, host, sysfsdev); + } + +free_node: + free(smmu_node); +free_dir: + free(dir_iommu); +free_link: + free(link_iommu); + return ret; +} + /* * FIXME this is used to reverse for hotplug devices, yet it could result in a * big waste of PCI bus numbners. @@ -1669,6 +1776,9 @@ static void create_pcie(VirtMachineState *vms) qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map", 0x0, vms->nested_smmu_phandle[i], 0x0, 0x10000); } + + qemu_opts_foreach(qemu_find_opts("device"), + assign_nested_smmu, vms, &error_fatal); } else if (vms->iommu) { vms->iommu_phandle = qemu_fdt_alloc_phandle(ms->fdt); diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 0a3f1ab8b5..dfbc4bba3c 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -246,4 +246,17 @@ find_nested_smmu_by_index(VirtMachineState *vms, int index) return NULL; } +static inline VirtNestedSmmu * +find_nested_smmu_by_sysfs(VirtMachineState *vms, char *node) +{ + VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { + if (!strncmp(nested_smmu->smmu_node, node, strlen(node))) { + return nested_smmu; + } + } + return NULL; +} + #endif /* QEMU_ARM_VIRT_H */ From patchwork Wed Jun 26 00:28:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7667EC2BBCA for ; Wed, 26 Jun 2024 02:48:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMGaf-00032T-W7; Tue, 25 Jun 2024 20:32:54 -0400 Received: from [2001:470:142:3::10] (helo=eggs.gnu.org) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGaP-0002ym-PW for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:32:41 -0400 Received: from mail-dm6nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:2417::601] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGZR-0005AS-Rf for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:32:18 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AwinHG9r4rMTlQxLRH8Sm8vskCZmMm4jLFhL/1iKVI15q3Jjvk/+V/GKr4OMGiOSfBkNJ0StZPzcw6WShQbbBa1P8NknEI4+TU4D6m4uM4L0I+o+Nr3vm2um82Y3BGetGyO/BMUrgP4P5Rgcy+FZsnFoXJ4CdZ++IIT8SlSOmgqVY8VrcwwhQICJy94BO/3ZfhekqNGPCUBXMVMktXytANv8UjzjE0qxTWHJ+sSUFq/Rh1myAhWeWVHiXjpA/5V2Dzq/z99Q4XfxE/rR3e7X8prTsKdggev021eEsRkpiEVGk55jpGgUsw22+QQKkp2ciomfnXJGfuUe7Vz120mR+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=st6cXlXTtG4KWRzuy7Kgk9SpM3l4nRJKRVnsNJCtIGs=; b=kgSpCjH0pusS3lw3AfxGladS1eKmOfbO+RNuqsIdV5rrTLeHeTP+2NK1sub1l1gLS0pQkIkY6/DtxcQu43gvr5NeGA3bMR8rYrybUSMhDeihWNOgjbuYBszIpco3hAzyp2MKIIWiMUH7w+WGFeJUhjUsJkopn898fVf256U8BUK75pLGKJ7wdiDGijqO+cj6HP07EmPMWYLi3iFWKIdDoPEjb8az3AVWM9SfpyUzcMrAS6NfGd1NboWdrHOIMfgsHYTfHIHe+asqtZ86I8dv7Ka3LAzK68HnIy+ZgQ4GcV7hJ+0XasstL3E0zOyobkyzH5sBl4K51+CmjECNl2c8lQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=st6cXlXTtG4KWRzuy7Kgk9SpM3l4nRJKRVnsNJCtIGs=; b=Tv4PpvP6ahFkQSMl7YXS34j50mQ+f7J3AdFf0k0a/h8cfnT7aKkXUkhLFFZdxaiozVINyXfJhhMFDsPYteVkOKAktSVO92lP2xQj3AWFQ/56kWdSbZqyeRjZw3rwcDFKJ84BHymWGxyqRF53BL7XYw15NgxDDu/aDTFF+DWMtniL713RnBS7V19Ij86fSpEnW+4h1f477JjQwmMQKu6dFM1CUI7gMq8DGm8n2+W6ZbaeS5z2fsM+muj6bF6VssjohKv4vbVqiaesTytJh7BQV2YHkEpxLhR2HjWgjMa5mfKStpjJFAxlxj5oNdbIh8abFInfCwt+zYBLYXvA2+YkvQ== Received: from BN9PR03CA0949.namprd03.prod.outlook.com (2603:10b6:408:108::24) by IA1PR12MB7541.namprd12.prod.outlook.com (2603:10b6:208:42f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.30; Wed, 26 Jun 2024 00:29:40 +0000 Received: from BN3PEPF0000B075.namprd04.prod.outlook.com (2603:10b6:408:108:cafe::c5) by BN9PR03CA0949.outlook.office365.com (2603:10b6:408:108::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.38 via Frontend Transport; Wed, 26 Jun 2024 00:29:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BN3PEPF0000B075.mail.protection.outlook.com (10.167.243.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:39 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:30 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:30 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:29 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 07/10] hw/arm/virt: Bypass iommu for default PCI bus Date: Tue, 25 Jun 2024 17:28:34 -0700 Message-ID: <27c5765e7a945de1c10269f05ca3347b5d62eedb.1719361174.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B075:EE_|IA1PR12MB7541:EE_ X-MS-Office365-Filtering-Correlation-Id: 6453cc0c-adfa-4dab-a3e4-08dc9577111c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: C8onOuZJcVof3n1G/tZBkqCJMvQgpMjjfp7FIQRJ7wGK8HRFqk7eXLE5fPCugKZAC4daPDnX0tIiRqru8pHjyq+WtV5X2DMapd90QreDmIWMErGM+XD1fLA9F4iee9rfddq3ryZoiIPW+FdPWVW4X5xNnzWaOdIBTzs2idYkI+tM7go6QmqRdz9W+cZt9kXsJMwApqAVDCBK632RdPqJtrY7Flxbx+dU2ASjKtj1QuHvdX71FSr/uUkVlAM7Fz0tvuiVnCResHdh8rkoMrCF75j/zhQ2kdXDaApnmS+n8CSPWZOWkKwy9UxH2VFtanoQ8u3yBLjqfmB5VQbwW0NeA4ColQXzOxgPLAYqa3nYDAB4ScckUWtuw0Ow8SY6txCahRtYgPyqN6NXrrIpEKsZNAeNBBtF1W5viKg1aWjhcW9lTiwOCXjUICfjD3xvsgzKlb70Ms83gsWMymuowIKRwgvjkeGqAZYC/Wpd3En0efJ7Qfm+zjTZfZRFsyBJkqrbj74AZAE0rv3dKLL3x5j/NNmWMC6SRI/qSqIHSqOwVmwgGEprJ0VNs9IJCrrPQR+ju67xVlNZkF0dH/w0meX5vONZ44UHClVcXOeO3WUj7y6Mi1AlSJFfgCR4sFTwhxKYIfPS71dfAQ007g3+beZBcBYZEHGEzenk9Mp/4E34IcABT37QelCN7VG32l4kIe7vtaQ53G3f06UQNkqg1NXM1iOuZYdwkI6wJUAGSmnHZgn/tt6M2JskbHJ12yMwutsRIId15H2cP1qt2lJwFpyVdZYhgyebvx0CSlocspzK++T2+6aRTfiEu6lSYH4m/IsJLL1RSafaDBqtP1En/GU4aaaXrqZMxD9Ih8f3yZRNHI4/OWL6ADnCO+pcoNGU6D1byDSyQxj1GYdDikJz5UHwC0AlJYTQb/S84CXeaMiZWofinMiQQ7UiyFj9btUFKpaIGhRlo7fMBbKN9W4BcVMiOeyYlAn1VcrRPDapV5KvgYUh9qEeyhXzrEa/iitleF1V6nzfyqVnKY2gevZyfVJ8QoZD7Oes0ws+9/v5Pn9U5KT4p9TPF+htr9lfCU0K0DCc6ifcfnuqjev0+15dSWmH9YAhsVNpw8uiTQLMlLheEAqaQeMoMm1NwAjQ/c1QcuFu+8zdLYHoC755+ilEVeiB7U4MGa/EQbsJyJjKcFpz4f9HhCRotJ9bircNto2PtwtXW6vBMqzb8HpsyjuhpPIdYNTZ+UrZVbEqchrJPODDv0stsEVn9jl7rQu+Z9WO9mkYdooY0ekGpj95yJ91Ip7S3d5Udv5ox5nFEW8u5x/c9h2l8tye4S//PQ6KdCrGpwgM/uJ62ZiGjfUeVte2LVqP1rf7IBhWx17TtEtws0+pS5SdSIIAqGZRzCjIf0DfHtkI/2DXLr7LpulXKgHUe4xjcQ== X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:39.8316 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6453cc0c-adfa-4dab-a3e4-08dc9577111c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B075.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7541 Received-SPF: softfail client-ip=2a01:111:f403:2417::601; envelope-from=nicolinc@nvidia.com; helo=NAM12-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, KHOP_HELO_FCRDNS=0.4, T_SPF_HELO_TEMPERROR=0.01, T_SPF_TEMPERROR=0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Now, all passthrough devices that should benefit from the nested SMMUv3 feature are assigned to dedicated pxb buses. So, the default PCI bus can be only used by emulated devices. In theory, these emualted devices can be still attached to an emualted SMMUv3 instance, yet there is no gain doing that. Set the default PCI bus to bypass iommu, for the maximum performance. Signed-off-by: Nicolin Chen --- hw/arm/virt.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 3610f53304..5e91dc8c3d 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1708,7 +1708,8 @@ static void create_pcie(VirtMachineState *vms) } pci = PCI_HOST_BRIDGE(dev); - pci->bypass_iommu = vms->default_bus_bypass_iommu; + /* Default bus used by emulated devices does not go through nested SMMUs */ + pci->bypass_iommu = vms->default_bus_bypass_iommu || vms->num_nested_smmus; vms->bus = pci->bus; if (vms->bus) { pci_init_nic_devices(pci->bus, mc->default_nic); From patchwork Wed Jun 26 00:28:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0F5BFC2BBCA for ; Wed, 26 Jun 2024 01:00:35 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMGrc-0008LR-3E; Tue, 25 Jun 2024 20:50:22 -0400 Received: from [2001:470:142:3::10] (helo=eggs.gnu.org) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGrG-0008AT-R4 for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:50:12 -0400 Received: from mail-mw2nam10on20607.outbound.protection.outlook.com ([2a01:111:f403:2412::607] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGqO-00004w-Hy for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:49:43 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hLwwTNf31EI5BVzUMV6hyEnbwzR9TD//RC/qYs0MumfAfdj2poecoV0JNB3lpMiQxytm9MEuG1KYzpJ5KMpDab3l822mnhUSw48Q9dGpUF/0B2hoXwFqyciFt0FRU4zE5sCZkWGLkt4T1BWnHQkicgv9yUDnt0kqwyKYnWuzxbgrnBqgYmgMThWHdXF/28tGPAQi2B/l7OJVklfOmxkMhOryunVX9EMQ7Fj19ZbN6IJ2kihObU5H1VbrXVCU6sUb7Xoy2I62foU0V/bRpw2jO1T6BudTRX4KOeX9y71CoXfw5/emfFB5jDmUZT1Z1QNtD/6rv1Ga1nB2CW3CN0WCEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6wOvi1OGUotDm+vB1MY5ShyuvtP9VXlclIHrs5TuR3U=; b=Ql20/zTc1ZU4QbA51pNoL/t8dLyNrsrJE8m1di5nZo95yCP5JiwXOt7PfwXIHh/CndAvdaGxc3LsP61T6/DNr+PnpNOp8KiGESblb78ZIecvt3vD+74zSpD5hmH+C+1R58oREf1r2tEmlV/U9yBnjEKMykO7emguOQbcDb/sbMKodR7m6PHQooTNcynEmHv+zui7BTa2DMx/r/ZJn6bxlPXwgG2w1VDDCZlakqh29UWEJ6OEoCu7Aq4RE59bUSIo0r9BvjeCpfQ3fOQAdkD+Xso4Qq9YQ8+MCze02OvKDRVAlJ/J/nEpzdg/rmNNA+WG7NmKo0fK4uAKg+lfQ97jlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6wOvi1OGUotDm+vB1MY5ShyuvtP9VXlclIHrs5TuR3U=; b=Tkb+d1FSUGocRUqUNEou00p1TATj+dOT5/F11h5RcbQPrKsphmJcIHUSek2HziLcp+nYn+MvMnU2K5HBUnulVjKg2gL8tn59/+QesG21fFfiBlrZDE1+ZQDEAiolOiT1IFVsm9lQaWOm8wv2QrvWMXXCXkF0DKZmfd87bcXKA9qkUL/Aq5on++yRE/4d391kRkTcUZVv8EK9JyRIS7BBw+uCUocoVR2KnCwE5NRcvx+lDWuPWZGOCzUufpvkFqdcSZyldEEyZdNWjOiytP1NecuJVPnpdekJH/iWRX/s08GGauB9w6ZDMBdScio6x19Aj0EL33wMb0Es7bXjR5OlHw== Received: from CH2PR20CA0005.namprd20.prod.outlook.com (2603:10b6:610:58::15) by SN7PR12MB8002.namprd12.prod.outlook.com (2603:10b6:806:34b::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.29; Wed, 26 Jun 2024 00:29:43 +0000 Received: from CH1PEPF0000AD80.namprd04.prod.outlook.com (2603:10b6:610:58:cafe::c2) by CH2PR20CA0005.outlook.office365.com (2603:10b6:610:58::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.38 via Frontend Transport; Wed, 26 Jun 2024 00:29:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH1PEPF0000AD80.mail.protection.outlook.com (10.167.244.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:42 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:31 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:31 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:30 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 08/10] hw/arm/virt-acpi-build: Handle reserved bus number of pxb buses Date: Tue, 25 Jun 2024 17:28:35 -0700 Message-ID: <2f5791d4148859a71bc1eae041cf6c7d478b2cb2.1719361174.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD80:EE_|SN7PR12MB8002:EE_ X-MS-Office365-Filtering-Correlation-Id: 0b133c44-5cb1-47ef-2b50-08dc957712a7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: ITGw7nyVO0foD6UuBse2ibnHs5SVTLMJL07fOvmyTpjdpRehWdNDWKlVgn64jDimucQvfYtGSwqYBoxODKnjbxoes3apj/qALPyKUtiV9pSSY4yAN2X0YbKAI8aAzc1HliRX1IChgy6bNuW5sQ2LQIJuB5Ogv3LaB1dSGA5KqDo5luV/WtvtMjDQlaZx35uwiEk6Pi3e96O+s3WLPSU1j8viiAAMh+zC0kGg9LQb/cf+45EUfHOVCHmdXLL4gwsN287i+mHMgm1AJTQ7A4wZ0FBPcCojYU1VA/scZ4aL+V4COgN1WP3fCfRnrraNSiqkbWfXcKc7TLgS5FbMiREIIU2rrWNYMBGBXIjHYB/YB0FbglRIjvHwOpoDxMuYFptrhYO+u8ohTLjxqY0+afRHHiK0IhmlKZkFeCPMNHTn6IprGPSecPVN5eWGUg6wVM7huU53Ort1X/j7kPuMGKcHTSpx7iYlFb2LcBRijukHeQN+VndwR+Z4VByELt+Xr23RDsB1N0qVRi3zS43Gr0NccE1I56l4/juUeRWR7mgFK0tJKiNJNIZl8n7f0EDjyDiYn+3vbejOjH0x4fJxbTdQX4zz12kkSHf30EZpnp09El71gsisUUsDT/vfrzOxxm0cXbp7cJWoemKXnAGyDCawgZfL36OS772M+Nsvt2wdBG+om/vCIIEKWu9gUAhxKd7PoGLA6FAfmHzoLfp0P5rzR64cI8rQbbBsTKryQ6Bd/3PWIMMiOSc+kdpJrzDHKxVJd9qyB+a+fC07Sh8NLFBvceo15UL5H2uxDZ1cDKcbQcCJavqDEuiFr3O1YnhzPaotAXbERDpNKe10N3Bcz04BQZnVXv2jNYxr6PMLNONAWZm8ycrrwzxp+rrFQ1iJQAvU0Sdrl6wgMsgrz1PZKq+P//YsTEAQmq43kqYIYShWy0vy+ilcuYH8ABAs6fZnA+bpRErz9MxFVMuTz6LZNaZ+dlSQpkD1HBb2SXcpJY2+z1NrGmcngGC6upMbWHMlGJ0/+W5XE9kswUqwcg3W3Dwh5udcOb2bRCDPQnMLSphW/Jzl9JFKC7b2lDU0eAYk0jeqmLHznScyyllp6CWTek22M00UgqBO7oEEuH23FCz9Z5gi2s00mpICjCmwTMz6kdW11MOaQ268g29oL0gwvpIpgjx1/d9W27UibBQXXC9HxOgt0IrR47p91HmGyRFD/uqtzJywYmVNNg8A1SHr5SQO1j+uyk6UJZ04lEEObdrGSxqMsBIJZJRlsRqUxb0QJ3FcOTNwYuf3DtFFvZEMa4lIVMWOhVEkOHpqSzND9pOExndyg38XiE4ywZ0YGGP0rOMVjMqFEmxPmDP6TY5/3a7tydp0xWgHy9RaLKtGMOqh77/T0Jl3dzbFF+WINeun9nwBkW+BczMwXyBDqk5wAtqJrA== X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:42.4641 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0b133c44-5cb1-47ef-2b50-08dc957712a7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD80.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8002 Received-SPF: softfail client-ip=2a01:111:f403:2412::607; envelope-from=nicolinc@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, T_SPF_HELO_TEMPERROR=0.01, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Each pxb bus created for a nested SMMU has a reserved bus number, allowing a hotplug device to attach to the bus in a later stage. Read it out to apply to the id_count calculation. Signed-off-by: Nicolin Chen --- hw/arm/virt-acpi-build.c | 28 ++++++++++++++++++++++++---- include/hw/arm/virt.h | 13 +++++++++++++ 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index d5e72800f6..91f53f90ca 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -236,6 +236,12 @@ build_iort_id_mapping(GArray *table_data, uint32_t input_base, build_append_int_noprefix(table_data, flags, 4); /* Flags */ } +struct AcpiIortIdMappingVM { + VirtMachineState *vms; + GArray *smmu_idmaps; +}; +typedef struct AcpiIortIdMappingVM AcpiIortIdMappingVM; + struct AcpiIortIdMapping { uint32_t input_base; uint32_t id_count; @@ -246,21 +252,34 @@ typedef struct AcpiIortIdMapping AcpiIortIdMapping; static int iort_host_bridges(Object *obj, void *opaque) { - GArray *idmap_blob = opaque; + AcpiIortIdMappingVM *idmap_vm = opaque; + VirtMachineState *vms = idmap_vm->vms; if (object_dynamic_cast(obj, TYPE_PCI_HOST_BRIDGE)) { PCIBus *bus = PCI_HOST_BRIDGE(obj)->bus; if (bus && !pci_bus_bypass_iommu(bus)) { + VirtNestedSmmu *nested_smmu = find_nested_smmu_by_pci_bus(vms, bus); int min_bus, max_bus; - pci_bus_range(bus, &min_bus, &max_bus); + if (vms->iommu == VIRT_IOMMU_NESTED_SMMUV3) { + /* PCI host bridge hehind a nested SMMU has reserved buses */ + if (nested_smmu) { + min_bus = pci_bus_num(nested_smmu->pci_bus); + max_bus = min_bus + nested_smmu->reserved_bus_nums - 1; + } else { + /* Not connected to a nested SMMU */ + return 0; + } + } else { + pci_bus_range(bus, &min_bus, &max_bus); + } AcpiIortIdMapping idmap = { .input_base = min_bus << 8, .id_count = (max_bus - min_bus + 1) << 8, }; - g_array_append_val(idmap_blob, idmap); + g_array_append_val(idmap_vm->smmu_idmaps, idmap); } } @@ -331,6 +350,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) uint32_t id = 0; GArray *smmu_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping)); GArray *its_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping)); + AcpiIortIdMappingVM idmap_vm = { .vms = vms, .smmu_idmaps = smmu_idmaps, }; AcpiTable table = { .sig = "IORT", .rev = 5, .oem_id = vms->oem_id, .oem_table_id = vms->oem_table_id }; @@ -341,7 +361,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) AcpiIortIdMapping next_range = {0}; object_child_foreach_recursive(object_get_root(), - iort_host_bridges, smmu_idmaps); + iort_host_bridges, &idmap_vm); nb_nodes = 3; /* RC, ITS, SMMUv3 */ diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index dfbc4bba3c..7ac392eb88 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -259,4 +259,17 @@ find_nested_smmu_by_sysfs(VirtMachineState *vms, char *node) return NULL; } +static inline VirtNestedSmmu * +find_nested_smmu_by_pci_bus(VirtMachineState *vms, PCIBus *pci_bus) +{ + VirtNestedSmmu *nested_smmu; + + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { + if (nested_smmu->pci_bus == pci_bus) { + return nested_smmu; + } + } + return NULL; +} + #endif /* QEMU_ARM_VIRT_H */ From patchwork Wed Jun 26 00:28:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5EB72C30653 for ; Wed, 26 Jun 2024 01:01:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMGrt-0008AM-85; Tue, 25 Jun 2024 20:50:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGq6-0007bf-DU for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:49:47 -0400 Received: from mail-mw2nam10on2060d.outbound.protection.outlook.com ([2a01:111:f403:2412::60d] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMGp5-0008P0-8f for qemu-devel@nongnu.org; Tue, 25 Jun 2024 20:48:31 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mZznCwzvT1WhPZWlypTI8piZCf6tOY3s891wKxRCtkw1OBAZWDqS5x16wpNtPIsF9b+EpEFKuUasrj/3Z5OawEngnqaea4Z93J3dHfa1eJak1OS8QEOPq8gL6hZm9G8mo1l4SV24OuRkLzycyB6+nVR7taHJSz0OuewiMvBRaB/Z6KTi9HfXRDtjfFxbRu7rc9BKAWpEQLSaA4o9chV24PvgH7WInC+KJMsION8JsBk99tgQSaRHZ8a4igUuEZeQbRkHaggnENK5POY9FjKqUEpCQFd/XQAuQUHg+oCjre+WcvRGg7Z+A/4mscss59l7E1BaEFEv4Nl0qqDibz65Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9vBeQ2ESg7mBeqhgmI+k3egLqqRyw3wevs4tS2Tw3Uk=; b=JE67/ZkcKp28Bg3a77tNu88yy7ZE8dgPuCAzQ7dOCyKQsCuk7mMT72dF5AG2/JWgCOtV08tty6ghKeC2FrwNonGIbmNVBbpxHu1B5HT7d8ms+X6xgVDds2dJAoHs7+yccTafHim5Z3NEsS1e5+RCBUuw7MZZY2Ay/wvptex7oE+gd9qYv+YQFxC2xTQIasX+dz2s4Rc719mR7Y6/vtumUUOrkilvW2p2nZ2j0FK/oqDptJ9ahn9g6aa53KWl2Xk6/rAFHQSFWp4RNSJBiJCozyRstkuqlTdZzKj25pRscvyhREfwk+7HrT5gi9HpEEew50ftdrYZ5GuTcX09YFhMFg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9vBeQ2ESg7mBeqhgmI+k3egLqqRyw3wevs4tS2Tw3Uk=; b=KTDSk3KaB6HEPO76lOqU+X5IivS/kMP7nublGtJQBsMvUbQKKO8tprDhKcF2+EC8uCbb3GWCURQZKSojFT/jZSuZwoekmnFTmHWsZXDbBSkpGu2PA3FE3QQieomiUVRHaAhv21Z59t3d3v6oHPo+MvP564j7D23WvpTMDHtGYngxkst9au9aqKEHuAKWl8psXUHFRwMpR3owv9hZUBEIyPA0GXuwvm1k7DWbkBd5Ph1/CTYX9XKlOASXbejpCTPe42Zv7l3xfmaYwhUTsPik1YuKuoyI/QsZWmYeDpR4MvJwSk9DR05NSSKoEXFOI7/Jk26PebMgdCTRevJUKHNt7Q== Received: from BN9PR03CA0934.namprd03.prod.outlook.com (2603:10b6:408:108::9) by IA0PR12MB8745.namprd12.prod.outlook.com (2603:10b6:208:48d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.32; Wed, 26 Jun 2024 00:29:43 +0000 Received: from BN3PEPF0000B075.namprd04.prod.outlook.com (2603:10b6:408:108:cafe::83) by BN9PR03CA0934.outlook.office365.com (2603:10b6:408:108::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7719.22 via Frontend Transport; Wed, 26 Jun 2024 00:29:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BN3PEPF0000B075.mail.protection.outlook.com (10.167.243.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:43 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:32 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:32 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:31 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 09/10] hw/arm/virt-acpi-build: Build IORT with multiple SMMU nodes Date: Tue, 25 Jun 2024 17:28:36 -0700 Message-ID: <074b51a5a42875304d7d8d223d0bc644fa7a4736.1719361174.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B075:EE_|IA0PR12MB8745:EE_ X-MS-Office365-Filtering-Correlation-Id: 0a97331d-57c5-44d5-c533-08dc95771345 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|82310400024|36860700011|1800799022|376012|7416012; X-Microsoft-Antispam-Message-Info: nGXMnlqXDhq5ATrnMoHz1WHN0DYTzUSCHVMZS/fxlUxP7DTrL6vJeAGpqJpoRyoSCTwtl8P7Hx9xPPytg1QqrzrInxxjxGL2KvToSucgy333t7iZC192y3GpHKyIoMtixWmm3+6OuzD0r7mY58+RNSs0zTeijWiuaTyyhA0EsyrHUfUDSfIvj60dE+HaOe3Npphs+Q9vM/NVIycz6CADMyRemCA8awnhju9X0LBjfDyk/toR9z5M6oiYb2SHLl0fqdIfU0xOggfuczmGeGiO4wzw3oUv6yzTGJ0fQmEPlenO9Ol6lVD5fC7bGABzBtSZiPqXHaVaMddl7gHRFVirl/oJ6WFD8SBbJt8E+EK582SW0t4QrJQwTIl+eJ5EpHfD18pgEJQ+T38SurdzaTgNiqB5u+lUXAy97C31Cl6hXSY94FZRbhzZifMSGseyIfniHs5PtxaFj4IXCYraU4htigLVdlmrNOtWEbTzKGhX9a9dSyI5qUqlezsMrvHlRF9xS5qslq65/sbDFAtOU+rizyqkl0Nj00lIglrpp1Bhq+DOqiIl4Hhcefcn+pUyZCzprPq6p55C1gkpdbhJdalSmHG2ELcsPsk81Ly9MnpM/97vIuxNbXkgNmsZ1tGNotUSKYhBQTUat1Q5mtTl0DTDkID/zQYI8qtKjV5pK7yNLZAfImszJ1ffV8MRpWL2/OMJ/6Re0xirQPDn7jjLn3ZuOYyx46hxAy+H+7DdnwDz8E0WwKCMvVa9X1+BYd4ePJ7QS1Z7N3iGF/38gwvb7bNDP5UpWZqEbdiPrACmEu8H5ZIqE8ZCm/o0cav415eiewuR5d92HeMjIIXrDzgmPzzaPvTm3RPb7PIPfzlL/k+etAOpQI/wjx2mDN+t1vrRS9Y/pLC0N9aY/gbpIzZP2Bzr47VyGBdY3TB1a+Z04WVpSOaEPRjtv0qsAlXNFIeyqfPxr4J1CH7hOdZP5NGIlvR7c4WzLQzJfFAwSOv1eM1xvzbhFYN4Eh992dIWRit+Dvxdgupj4V7CpxpS39znNjc+tZHGioDkOgihProUuTw8gr9TEOMizhpl32Oh6heY0SPS40l4dGlDKtVQJs3CiEzF2HuRCT8NTNP4f8V1R9KY6K69K6l6+WaIuuZCXXpc3kgLF0nrFdVNjPKk+0OenpGy1X6lX0h6drdxRizOmkmLIbRRZ2wMkn1u1KEQrJea4jY+ii4EKBcp96O5wtJ2oZN9SpsJAB3uMRlGzRrYlA+BzIhdwiIpaTPAvtUwqlhU9uskdzIN3yc9DEdAzz8niSpLUv/MaAd0CvIxDPJPec6gWHwMiPfv0eoXFdUTW2G8wfX3dqzrjqKmTYJDideOOYDaegYbhaHAKIDoQnDL37s33qvh/19bx6Lz2KcVH5p6FI4l/5rlcz5P4F5aejbTizOnRGFFdBRbs3q5rTO+GmcBIQkJC8/YV14lsh4fFZn750B8 X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230038)(82310400024)(36860700011)(1800799022)(376012)(7416012); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:43.4879 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0a97331d-57c5-44d5-c533-08dc95771345 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B075.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8745 Received-SPF: softfail client-ip=2a01:111:f403:2412::60d; envelope-from=nicolinc@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, SPF_PASS=-0.001, T_SPF_HELO_TEMPERROR=0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org There can be multiple PCI buses behind different SMMU nodes. And each pair should be associated in the IORT table too when building the ID mappings. Create multiple SMMU nodes if needed, store their offsets in an array. Signed-off-by: Nicolin Chen --- hw/arm/virt-acpi-build.c | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 91f53f90ca..6d8b9aea42 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -295,7 +295,7 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b) } static void -build_iort_rmr_nodes(GArray *table_data, GArray *smmu_idmaps, int smmu_offset, uint32_t *id) { +build_iort_rmr_nodes(GArray *table_data, GArray *smmu_idmaps, size_t *smmu_offset, uint32_t *id) { AcpiIortIdMapping *range; int i; @@ -323,7 +323,7 @@ build_iort_rmr_nodes(GArray *table_data, GArray *smmu_idmaps, int smmu_offset, u build_append_int_noprefix(table_data, 1 , 4); /* Reference to Memory Range Descriptors */ build_append_int_noprefix(table_data, 28 + ID_MAPPING_ENTRY_SIZE, 4); - build_iort_id_mapping(table_data, bdf, range->id_count, smmu_offset, 1); + build_iort_id_mapping(table_data, bdf, range->id_count, smmu_offset[i], 1); /* Table 19 Memory Range Descriptor */ @@ -345,25 +345,42 @@ static void build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) { int i, nb_nodes, rc_mapping_count; - size_t node_size, smmu_offset = 0; + size_t node_size, *smmu_offset; AcpiIortIdMapping *idmap; uint32_t id = 0; GArray *smmu_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping)); GArray *its_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping)); AcpiIortIdMappingVM idmap_vm = { .vms = vms, .smmu_idmaps = smmu_idmaps, }; + int irq_offset = NUM_SMMU_IRQS; + hwaddr offset = SMMU_IO_LEN; + int irq, num_smmus = 0; + hwaddr base; AcpiTable table = { .sig = "IORT", .rev = 5, .oem_id = vms->oem_id, .oem_table_id = vms->oem_table_id }; /* Table 2 The IORT */ acpi_table_begin(&table, table_data); + if (vms->num_nested_smmus) { + irq = vms->irqmap[VIRT_NESTED_SMMU] + ARM_SPI_BASE; + base = vms->memmap[VIRT_NESTED_SMMU].base; + num_smmus = vms->num_nested_smmus; + } else if (virt_has_smmuv3(vms)) { + irq = vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE; + base = vms->memmap[VIRT_SMMU].base; + num_smmus = 1; + } + smmu_offset = g_new0(size_t, num_smmus); + + nb_nodes = 2; /* RC, ITS */ + nb_nodes += num_smmus; /* SMMU nodes */ + if (virt_has_smmuv3(vms)) { AcpiIortIdMapping next_range = {0}; object_child_foreach_recursive(object_get_root(), iort_host_bridges, &idmap_vm); - nb_nodes = 3; /* RC, ITS, SMMUv3 */ /* Sort the smmu idmap by input_base */ g_array_sort(smmu_idmaps, iort_idmap_compare); @@ -394,7 +411,6 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) rc_mapping_count = smmu_idmaps->len + its_idmaps->len; } else { - nb_nodes = 2; /* RC, ITS */ rc_mapping_count = 1; } /* Number of IORT Nodes */ @@ -416,10 +432,9 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) /* GIC ITS Identifier Array */ build_append_int_noprefix(table_data, 0 /* MADT translation_id */, 4); - if (virt_has_smmuv3(vms)) { - int irq = vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE; + for (i = 0; i < num_smmus; i++) { + smmu_offset[i] = table_data->len - table.table_offset; - smmu_offset = table_data->len - table.table_offset; /* Table 9 SMMUv3 Format */ build_append_int_noprefix(table_data, 4 /* SMMUv3 */, 1); /* Type */ node_size = SMMU_V3_ENTRY_SIZE + ID_MAPPING_ENTRY_SIZE; @@ -430,12 +445,13 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) /* Reference to ID Array */ build_append_int_noprefix(table_data, SMMU_V3_ENTRY_SIZE, 4); /* Base address */ - build_append_int_noprefix(table_data, vms->memmap[VIRT_SMMU].base, 8); + build_append_int_noprefix(table_data, base + i * offset, 8); /* Flags */ build_append_int_noprefix(table_data, 1 /* COHACC Override */, 4); build_append_int_noprefix(table_data, 0, 4); /* Reserved */ build_append_int_noprefix(table_data, 0, 8); /* VATOS address */ /* Model */ + irq += irq_offset; build_append_int_noprefix(table_data, 0 /* Generic SMMU-v3 */, 4); build_append_int_noprefix(table_data, irq, 4); /* Event */ build_append_int_noprefix(table_data, irq + 1, 4); /* PRI */ @@ -487,7 +503,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) range = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i); /* output IORT node is the smmuv3 node */ build_iort_id_mapping(table_data, range->input_base, - range->id_count, smmu_offset, 0); + range->id_count, smmu_offset[i], 0); } /* bypassed RIDs connect to ITS group node directly: RC -> ITS */ From patchwork Wed Jun 26 00:28:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolin Chen X-Patchwork-Id: 13712185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56632C2BBCA for ; Wed, 26 Jun 2024 01:47:46 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sMHb5-0004ZS-19; Tue, 25 Jun 2024 21:37:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMHYV-00041s-Ix for qemu-devel@nongnu.org; Tue, 25 Jun 2024 21:36:57 -0400 Received: from mail-dm6nam04on20601.outbound.protection.outlook.com ([2a01:111:f403:2409::601] helo=NAM04-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sMH8O-0005DY-Il for qemu-devel@nongnu.org; Tue, 25 Jun 2024 21:08:09 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ENT7zgplO1Ji/Fh4HXy0ggtPEaraClBUVwBuTNh3MpDBYtsPodE16DeEfYGaE3NpekW/JvSJO81Xl+titNfD3TEUc/T3zwzHhDnrL6KRvYP9lR4uPHkI8qTklLjYMtD9pt5014bYQP+4TztlBUnCVvERwXJcn6Rnj9geUdSGvXvnb/sNlzJNaPSS8TV9bXzInP+Pe44uESm4prNah8cyT7IhrHARc033Qk3gSjSO6QxuEgNnHxjfq+kTTNnovblEZ89VyEnuv4gtxr+TAsh+5fUyO6UWsFXEsTe0uY+HR3/BsDgwW5GXVtbyIOFfRTnwI2y4tVy75bwZk5inESnevw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Pejh5GTymrIgruXb5nqDuJ1Mr+SdU80MfVDak+pNDGc=; b=C8prnXspoBg5aln9P9109iO2e2oV7fatsq8eoREE9m/D/j//Nf2ioQyj1JCroobgEdSGU+U5AqWd5veV/ED+venboR22FJeWjFraBq7q2uheIt2+wPa1cX2NMyqAUhNyax8UKFHrWNoB5nHa61wA0mE07RK/EQBJ1BTM15bjm+6DlOo8nmthJYfW5d9MjRvPu7EuPZbhjh9tkmLXiEW39j65jkWW0121yZWqYPzOcgPncJ+6SFVFAly867c6KRs3P5v9T8u0nAfN7zmquCnCcOgU1/lMOdJW/a2pT5N3vGRSqZWqwM1aI0QkzWelvCYptVwK5+VVSHTXjU28XFekMQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=linaro.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Pejh5GTymrIgruXb5nqDuJ1Mr+SdU80MfVDak+pNDGc=; b=qPDMGD83W9wXBLs51nGkMyxPdjBgBFlORjCBfEQ0u6+4T3HUZ+KU8ZtNx9CY/qBmFVNLwutHgTnhzOHVvgqBlmUmXVYz2BjeWcBmtrfytlXVbqxGGEIMAH5SngDBiOQN1LNyCp8XCNufPE0ugE+X7wfH0EkzfsGKflNEX7ZDHaxpvb9sKQhQgi1CX96/3bJNOD3UBgHyCj1BK7h2yc0VoGyGsn5gVZOuWRf5r8seeb5tuJT9eg8xKdsV/4qIKaGTSeHJOXjJZ1ywLLJxA/rxC27UcU6Eod7bKRVLcMIjUfJq+7gMJL/6RpqVV0aNXlDRXesfHY52Eo9kJ+mCAKatCg== Received: from CH2PR15CA0025.namprd15.prod.outlook.com (2603:10b6:610:51::35) by CH2PR12MB4037.namprd12.prod.outlook.com (2603:10b6:610:7a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.30; Wed, 26 Jun 2024 00:29:43 +0000 Received: from CH1PEPF0000AD81.namprd04.prod.outlook.com (2603:10b6:610:51:cafe::a2) by CH2PR15CA0025.outlook.office365.com (2603:10b6:610:51::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7719.22 via Frontend Transport; Wed, 26 Jun 2024 00:29:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CH1PEPF0000AD81.mail.protection.outlook.com (10.167.244.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.15 via Frontend Transport; Wed, 26 Jun 2024 00:29:43 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:33 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 25 Jun 2024 17:29:32 -0700 Received: from Asurada-Nvidia.nvidia.com (10.127.8.10) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 25 Jun 2024 17:29:32 -0700 From: Nicolin Chen To: , , , , , , CC: , , , , Subject: [PATCH RFCv1 10/10] hw/arm/virt-acpi-build: Enable ATS for nested SMMUv3 Date: Tue, 25 Jun 2024 17:28:37 -0700 Message-ID: <228a33507c1dc46862f61e32a6482aa0b05b4ce8.1719361174.git.nicolinc@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD81:EE_|CH2PR12MB4037:EE_ X-MS-Office365-Filtering-Correlation-Id: 4a0403e9-c344-47a3-a10c-08dc95771313 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230038|36860700011|376012|7416012|1800799022|82310400024; X-Microsoft-Antispam-Message-Info: H0A3BzAy9bv2UWqdV6V9YIfANFD8UPtAVw6wCzGT6LZYlsk/mULwGXOuADEK2Vk4RFcGdJfIZQyQ8lgdQgyY1K6TZezWS2emJIOPtcZBu16CJNNkTC+O0gSmTq1Y0SShsBpXWQ6CN+FzpazPOs+HnkxcQS+k6Wk7nkXMVVrHX+kXQshfMVgLK42an1/DEM+ex6Cf9xNJts8OfYgpN7I+/nmVaMuhV293CQW9Quw12DYzdGlwI0HBSHSzoFPEbNclTFK0gKrKk6+vlU428jHKteC1DkI7flDdX5qqeExJ9ec+AQMyD6Jm55uznsEzqH6QuogiHLc8sFmn0WOBTLwK02yenq3pjZAzfRoBqJqr/IU+nkEeoPfKI/WZAu2WkH2hyhLttDVNl3vxndx1ifyQNZIG8HAIaom6vXwUyrrhxT72fg60InwWx8vWdC83NWIVC7+uis3jN8MqYD5+Zp7tmQ3/TCj4J4xyuSM2ikVIjZ6BU9r6O3USrUY/TKmOFK6booD5og9bYG15p5KiBtDPuLcITxn+uWGynmODpLinaAMCO1JQ3jxdDBVudJezYCsZqjItetkkz+S7JunJDFcM6+Ldtm+oSVNsSxPAOumuwRrvPM+uG6Wk5PsU9oZ7svR3X1lUReJ/uPbQYro9NkY7XdFzn1Ovd/IxPJbbO20C7ScRzdrkEcOLWth9K9EcOYjZizvbrjuV3jNyE4/LK2m84P4xf68n8sMIfrinZc1DZaIjf8Sr1/RVCYb4hnopVGz9YAHfqHaVncaPYCSX64nWgE0+DOZZAfsyUfC/E1n1fQor2KjCQigETZeGMn7vb/IUSsIdDhrEYMBzpOGC5piqL8vCIBeQYaEXaTJqv/QGDZZZf5om2X0wKO65QDcLgLKvM/5oK5hDes/Cf5pFiJeKYmaQV0/W5HMfhAY/XhTUyL+w3OjpIJDRDIW4metmfMFpBrfY4tp8M0xH4cQcAbodXmWKB6nT/ONyoh1j0/YzzTtSd3HYTzYbD8YrclL+asYwit9qjFVZ3ontA1Z0oGlBhhd2NJ8/SBTXY4g8nLmKvwS31zkPlRy7gHRmBz1olCuLuC/zJqCQzoisnR6JyIsUrE4PG22NtLCGmEoZhLjyrGVH844Rsw0asgKLl5r2kPuvOUR84dAcS4SYyx+y17jRIACcfwVSZpSNUN6x2APIy9crQd3aWoAWzviqGp92mG0Ezp4lWCpi9E7oMt5169PAwQ70lB5eiRhhvKxzapgzvp/TdquWcYuJVhF9+oiQLLQJWNvHEALnF7pxP5/tf0ocVqjPrk4wLhwUQ+L+I9Y9nlGLyjCwPT6XVWPLtKPwUjEIZ1yz2AnMDQAaiiqn5w9qfR5eZdJbZ71B4NIUFmov0CVJdkou8dESofLizl8NrNwsYCZx4CIZ7EGVcxHJ0MrV8g== X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230038)(36860700011)(376012)(7416012)(1800799022)(82310400024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2024 00:29:43.1905 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4a0403e9-c344-47a3-a10c-08dc95771313 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD81.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4037 Received-SPF: softfail client-ip=2a01:111:f403:2409::601; envelope-from=nicolinc@nvidia.com; helo=NAM04-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, KHOP_HELO_FCRDNS=0.4, T_SPF_HELO_TEMPERROR=0.01, T_SPF_TEMPERROR=0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org For a nested SMMUv3, the ATS capaiblity is decided by the underlying HW, and then reflected in the IDR0 register of the vSMMU. The IORT on the other hand could allow it to be always enabled, relying on the guest-level SMMU kernel driver to disable ATS feature if the ATS bit isn't set in IDR0. Signed-off-by: Nicolin Chen --- hw/arm/virt-acpi-build.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index 6d8b9aea42..c4cf1caf22 100644 --- a/hw/arm/virt-acpi-build.c +++ b/hw/arm/virt-acpi-build.c @@ -485,7 +485,11 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms) /* Table 15 Memory Access Flags */ build_append_int_noprefix(table_data, 0x3 /* CCA = CPM = DACS = 1 */, 1); - build_append_int_noprefix(table_data, 0, 4); /* ATS Attribute */ + if (vms->iommu == VIRT_IOMMU_NESTED_SMMUV3) { + build_append_int_noprefix(table_data, 1, 4); /* ATS Attribute */ + } else { + build_append_int_noprefix(table_data, 0, 4); /* ATS Attribute */ + } /* MCFG pci_segment */ build_append_int_noprefix(table_data, 0, 4); /* PCI Segment number */