From patchwork Wed Jul 19 19:33:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2BAA8C001DF for ; Wed, 19 Jul 2023 19:34:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jHTSIwunSYoUFpEbOopdJM4e4aiNxVOc43PMFYor5Q8=; b=jHIRWU4zTJ45/+ SGRFpYh4xyMAXAzgAfrUxRSqo6C1vYzos1CoXLq0QEfRaVRd913PEp0hpvxxk1svn1BrZVYRtHT4O WlUoPDPehevnv4NQfX+ydB5qZmEDK3h9YCjWHkVAZX9ENG+1kWSky9zWCShFVNVwOPuflCp9eoCcZ 39wKpoRiqF3uW+AXZCLdCetGjdSNO+h0BMLaXgLsuy121EDz+/E5QJ0oivMK8G7XeFMotZqwRs2zX 8zX09xwrEyjG9gx5RA+EPmECZSP/8ejbja5Ha3zSSKNAsSVrpQsRnR3n3K3mRtvFESvqmOK7Qunys ZbsrKFjgaO99UaCIwcog==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwW-008h8d-0G; Wed, 19 Jul 2023 19:34:36 +0000 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwN-008h32-0Z for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:32 +0000 Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-666eef03ebdso4913832b3a.1 for ; Wed, 19 Jul 2023 12:34:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795265; x=1690400065; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=meqWXhdfKEKZKYicyMNqHYo0zICg0jYUZDszzytI9Ig=; b=GYabIW+6yBdQjFnsR3CSUSv+qtyaegN11YyMqc4Az8i62to6h3HDu1ysn9Hq3ZAXPy yx8h33oS6rRkY8kLOhQvaz/pCrELj5Fq1Gsqpy2L4VatmxKrOf3wopuLpnBlPJchk316 wmnXiczpngs/MXJ9MZbP7TGKakBDEAasItLCEZ8IoUAyeFhT358i5X+pW8khBT0aFdkE 9ECA0BjSP9gqf/+XwakNlbBmcgD6abRXahdn+3mAP6WlDeIxwFniw6MbkcD2lrS4ubF8 RmrPvhGv9H5qGqQ4yD6/60YLHVll5MPBJ4RecaAwOqwAr0Ouqd/b7ZAPONss5cixAYkO HbIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795265; x=1690400065; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=meqWXhdfKEKZKYicyMNqHYo0zICg0jYUZDszzytI9Ig=; b=GN0xahVMwtl4cskybqJXRt3MzeykJ0OgOsw/SplUeuPkFuNa+qAt/3F2MRZxj5yVWv SOesmvnoUz2+QvhpR7QH826n+wMsIZJ11UKB69MLog+o8IypFE0whix7FZSWaRl9CasK tbzKvRNE1RCMWLewc1vO8ov0rOey7OMNJbQTnjYC/LmFjLBzDLEtAUn3cyTZykE0mFtS O4C+HPL1iFYlj+0/5nyIAYy95bmhvxRJTVyOjisz9UIIuRTavFJZRA8uWZBnPzfG/L4Q TIy9phcffd/9bDcf6Zht7wmfQKL58lAf463wFXoVkjgJaWEgMs9LDZXjoMlTSUq+MmoI YSAw== X-Gm-Message-State: ABy/qLayikk2GAgTOBn1bG2l0tPz19cxM1sT9mxoQlmWAvMuTKtez9bM AuYagWAiU0aTczEihKI90jhs/g== X-Google-Smtp-Source: APBJJlFwPtPknF3KPLq7jqBZDFj4kU7NtOMirpwbw3MoG5ptF0O5Zr0wJ8bfbWk+IKeTOeoecXyh/A== X-Received: by 2002:a17:90a:c20d:b0:263:f5b7:bc05 with SMTP id e13-20020a17090ac20d00b00263f5b7bc05mr14724194pjt.14.1689795265069; Wed, 19 Jul 2023 12:34:25 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:24 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 01/11] RISC-V: drivers/iommu: Add RISC-V IOMMU - Ziommu support. Date: Wed, 19 Jul 2023 12:33:45 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123427_269914_0B8F1552 X-CRM114-Status: GOOD ( 18.42 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The patch introduces skeleton IOMMU device driver implementation as defined by RISC-V IOMMU Architecture Specification, Version 1.0 [1], with minimal support for pass-through mapping, basic initialization and bindings for platform and PCIe hardware implementations. Series of patches following specification evolution has been reorganized to provide functional separation of implemented blocks, compliant with ratified specification. This and following patch series includes code contributed by: Nick Kossifidis (iommu-platform device, number of specification clarification and bugfixes and readability improvements), Sebastien Boeuf (page table creation, ATS/PGR flow). Complete history can be found at the maintainer's repository branch [2]. Device driver enables RISC-V 32/64 support for memory translation for DMA capable PCI and platform devices, multilevel device directory table, process directory, shared virtual address support, wired and message signaled interrupt for translation I/O fault, page request interface and command processing. Matching RISCV-V IOMMU device emulation implementation is available for QEMU project, along with educational device extensions for PASID ATS/PRI support [3]. References: - [1] https://github.com/riscv-non-isa/riscv-iommu - [2] https://github.com/tjeznach/linux/tree/tjeznach/riscv-iommu - [3] https://github.com/tjeznach/qemu/tree/tjeznach/riscv-iommu Co-developed-by: Nick Kossifidis Signed-off-by: Nick Kossifidis Co-developed-by: Sebastien Boeuf Signed-off-by: Sebastien Boeuf Signed-off-by: Tomasz Jeznach --- drivers/iommu/Kconfig | 1 + drivers/iommu/Makefile | 2 +- drivers/iommu/riscv/Kconfig | 22 + drivers/iommu/riscv/Makefile | 1 + drivers/iommu/riscv/iommu-bits.h | 704 +++++++++++++++++++++++++++ drivers/iommu/riscv/iommu-pci.c | 134 +++++ drivers/iommu/riscv/iommu-platform.c | 94 ++++ drivers/iommu/riscv/iommu.c | 660 +++++++++++++++++++++++++ drivers/iommu/riscv/iommu.h | 115 +++++ 9 files changed, 1732 insertions(+), 1 deletion(-) create mode 100644 drivers/iommu/riscv/Kconfig create mode 100644 drivers/iommu/riscv/Makefile create mode 100644 drivers/iommu/riscv/iommu-bits.h create mode 100644 drivers/iommu/riscv/iommu-pci.c create mode 100644 drivers/iommu/riscv/iommu-platform.c create mode 100644 drivers/iommu/riscv/iommu.c create mode 100644 drivers/iommu/riscv/iommu.h diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 2b12b583ef4b..36fcc6fd5b4e 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -187,6 +187,7 @@ config MSM_IOMMU source "drivers/iommu/amd/Kconfig" source "drivers/iommu/intel/Kconfig" source "drivers/iommu/iommufd/Kconfig" +source "drivers/iommu/riscv/Kconfig" config IRQ_REMAP bool "Support for Interrupt Remapping" diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 769e43d780ce..8f57110a9fb1 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -1,5 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += amd/ intel/ arm/ iommufd/ +obj-y += amd/ intel/ arm/ iommufd/ riscv/ obj-$(CONFIG_IOMMU_API) += iommu.o obj-$(CONFIG_IOMMU_API) += iommu-traces.o obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o diff --git a/drivers/iommu/riscv/Kconfig b/drivers/iommu/riscv/Kconfig new file mode 100644 index 000000000000..01d4043849d4 --- /dev/null +++ b/drivers/iommu/riscv/Kconfig @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: GPL-2.0-only +# RISC-V IOMMU support + +config RISCV_IOMMU + bool "RISC-V IOMMU driver" + depends on RISCV + select IOMMU_API + select IOMMU_DMA + select IOMMU_SVA + select IOMMU_IOVA + select IOMMU_IO_PGTABLE + select IOASID + select PCI_MSI + select PCI_ATS + select PCI_PRI + select PCI_PASID + select MMU_NOTIFIER + help + Support for devices following RISC-V IOMMU specification. + + If unsure, say N here. + diff --git a/drivers/iommu/riscv/Makefile b/drivers/iommu/riscv/Makefile new file mode 100644 index 000000000000..38730c11e4a8 --- /dev/null +++ b/drivers/iommu/riscv/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-pci.o iommu-platform.o \ No newline at end of file diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h new file mode 100644 index 000000000000..b2946793a73d --- /dev/null +++ b/drivers/iommu/riscv/iommu-bits.h @@ -0,0 +1,704 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright © 2022-2023 Rivos Inc. + * Copyright © 2023 FORTH-ICS/CARV + * Copyright © 2023 RISC-V IOMMU Task Group + * + * RISC-V Ziommu - Register Layout and Data Structures. + * + * Based on the 'RISC-V IOMMU Architecture Specification', Version 1.0 + * Published at https://github.com/riscv-non-isa/riscv-iommu + * + */ + +#ifndef _RISCV_IOMMU_BITS_H_ +#define _RISCV_IOMMU_BITS_H_ + +#include +#include +#include + +/* + * Chapter 5: Memory Mapped register interface + */ + +/* Common field positions */ +#define RISCV_IOMMU_PPN_FIELD GENMASK_ULL(53, 10) +#define RISCV_IOMMU_QUEUE_LOGSZ_FIELD GENMASK_ULL(4, 0) +#define RISCV_IOMMU_QUEUE_INDEX_FIELD GENMASK_ULL(31, 0) +#define RISCV_IOMMU_QUEUE_ENABLE BIT(0) +#define RISCV_IOMMU_QUEUE_INTR_ENABLE BIT(1) +#define RISCV_IOMMU_QUEUE_MEM_FAULT BIT(8) +#define RISCV_IOMMU_QUEUE_OVERFLOW BIT(9) +#define RISCV_IOMMU_QUEUE_ACTIVE BIT(16) +#define RISCV_IOMMU_QUEUE_BUSY BIT(17) + +#define RISCV_IOMMU_ATP_PPN_FIELD GENMASK_ULL(43, 0) +#define RISCV_IOMMU_ATP_MODE_FIELD GENMASK_ULL(63, 60) + +/* 5.3 IOMMU Capabilities (64bits) */ +#define RISCV_IOMMU_REG_CAP 0x0000 +#define RISCV_IOMMU_CAP_VERSION GENMASK_ULL(7, 0) +#define RISCV_IOMMU_CAP_S_SV32 BIT_ULL(8) +#define RISCV_IOMMU_CAP_S_SV39 BIT_ULL(9) +#define RISCV_IOMMU_CAP_S_SV48 BIT_ULL(10) +#define RISCV_IOMMU_CAP_S_SV57 BIT_ULL(11) +#define RISCV_IOMMU_CAP_SVPBMT BIT_ULL(15) +#define RISCV_IOMMU_CAP_G_SV32 BIT_ULL(16) +#define RISCV_IOMMU_CAP_G_SV39 BIT_ULL(17) +#define RISCV_IOMMU_CAP_G_SV48 BIT_ULL(18) +#define RISCV_IOMMU_CAP_G_SV57 BIT_ULL(19) +#define RISCV_IOMMU_CAP_MSI_FLAT BIT_ULL(22) +#define RISCV_IOMMU_CAP_MSI_MRIF BIT_ULL(23) +#define RISCV_IOMMU_CAP_AMO BIT_ULL(24) +#define RISCV_IOMMU_CAP_ATS BIT_ULL(25) +#define RISCV_IOMMU_CAP_T2GPA BIT_ULL(26) +#define RISCV_IOMMU_CAP_END BIT_ULL(27) +#define RISCV_IOMMU_CAP_IGS GENMASK_ULL(29, 28) +#define RISCV_IOMMU_CAP_HPM BIT_ULL(30) +#define RISCV_IOMMU_CAP_DBG BIT_ULL(31) +#define RISCV_IOMMU_CAP_PAS GENMASK_ULL(37, 32) +#define RISCV_IOMMU_CAP_PD8 BIT_ULL(38) +#define RISCV_IOMMU_CAP_PD17 BIT_ULL(39) +#define RISCV_IOMMU_CAP_PD20 BIT_ULL(40) + +#define RISCV_IOMMU_CAP_VERSION_VER_MASK 0xF0 +#define RISCV_IOMMU_CAP_VERSION_REV_MASK 0x0F + +/** + * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings + * @RISCV_IOMMU_CAP_IGS_MSI: I/O MMU supports only MSI generation + * @RISCV_IOMMU_CAP_IGS_WSI: I/O MMU supports only Wired-Signaled interrupt + * @RISCV_IOMMU_CAP_IGS_BOTH: I/O MMU supports both MSI and WSI generation + * @RISCV_IOMMU_CAP_IGS_RSRV: Reserved for standard use + */ +enum riscv_iommu_igs_settings { + RISCV_IOMMU_CAP_IGS_MSI = 0, + RISCV_IOMMU_CAP_IGS_WSI = 1, + RISCV_IOMMU_CAP_IGS_BOTH = 2, + RISCV_IOMMU_CAP_IGS_RSRV = 3 +}; + +/* 5.4 Features control register (32bits) */ +#define RISCV_IOMMU_REG_FCTL 0x0008 +#define RISCV_IOMMU_FCTL_BE BIT(0) +#define RISCV_IOMMU_FCTL_WSI BIT(1) +#define RISCV_IOMMU_FCTL_GXL BIT(2) + +/* 5.5 Device-directory-table pointer (64bits) */ +#define RISCV_IOMMU_REG_DDTP 0x0010 +#define RISCV_IOMMU_DDTP_MODE GENMASK_ULL(3, 0) +#define RISCV_IOMMU_DDTP_BUSY BIT_ULL(4) +#define RISCV_IOMMU_DDTP_PPN RISCV_IOMMU_PPN_FIELD + +/** + * enum riscv_iommu_ddtp_modes - I/O MMU translation modes + * @RISCV_IOMMU_DDTP_MODE_OFF: No inbound transactions allowed + * @RISCV_IOMMU_DDTP_MODE_BARE: Pass-through mode + * @RISCV_IOMMU_DDTP_MODE_1LVL: One-level DDT + * @RISCV_IOMMU_DDTP_MODE_2LVL: Two-level DDT + * @RISCV_IOMMU_DDTP_MODE_3LVL: Three-level DDT + */ +enum riscv_iommu_ddtp_modes { + RISCV_IOMMU_DDTP_MODE_OFF = 0, + RISCV_IOMMU_DDTP_MODE_BARE = 1, + RISCV_IOMMU_DDTP_MODE_1LVL = 2, + RISCV_IOMMU_DDTP_MODE_2LVL = 3, + RISCV_IOMMU_DDTP_MODE_3LVL = 4, + RISCV_IOMMU_DDTP_MODE_MAX = 4 +}; + +/* 5.6 Command Queue Base (64bits) */ +#define RISCV_IOMMU_REG_CQB 0x0018 +#define RISCV_IOMMU_CQB_ENTRIES RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_CQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.7 Command Queue head (32bits) */ +#define RISCV_IOMMU_REG_CQH 0x0020 +#define RISCV_IOMMU_CQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.8 Command Queue tail (32bits) */ +#define RISCV_IOMMU_REG_CQT 0x0024 +#define RISCV_IOMMU_CQT_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.9 Fault Queue Base (64bits) */ +#define RISCV_IOMMU_REG_FQB 0x0028 +#define RISCV_IOMMU_FQB_ENTRIES RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_FQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.10 Fault Queue Head (32bits) */ +#define RISCV_IOMMU_REG_FQH 0x0030 +#define RISCV_IOMMU_FQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.11 Fault Queue tail (32bits) */ +#define RISCV_IOMMU_REG_FQT 0x0034 +#define RISCV_IOMMU_FQT_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.12 Page Request Queue base (64bits) */ +#define RISCV_IOMMU_REG_PQB 0x0038 +#define RISCV_IOMMU_PQB_ENTRIES RISCV_IOMMU_QUEUE_LOGSZ_FIELD +#define RISCV_IOMMU_PQB_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.13 Page Request Queue head (32bits) */ +#define RISCV_IOMMU_REG_PQH 0x0040 +#define RISCV_IOMMU_PQH_INDEX RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.14 Page Request Queue tail (32bits) */ +#define RISCV_IOMMU_REG_PQT 0x0044 +#define RISCV_IOMMU_PQT_INDEX_MASK RISCV_IOMMU_QUEUE_INDEX_FIELD + +/* 5.15 Command Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_CQCSR 0x0048 +#define RISCV_IOMMU_CQCSR_CQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_CQCSR_CIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_CQCSR_CQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_CQCSR_CMD_TO BIT(9) +#define RISCV_IOMMU_CQCSR_CMD_ILL BIT(10) +#define RISCV_IOMMU_CQCSR_FENCE_W_IP BIT(11) +#define RISCV_IOMMU_CQCSR_CQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_CQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + +/* 5.16 Fault Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_FQCSR 0x004C +#define RISCV_IOMMU_FQCSR_FQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_FQCSR_FIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_FQCSR_FQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_FQCSR_FQOF RISCV_IOMMU_QUEUE_OVERFLOW +#define RISCV_IOMMU_FQCSR_FQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_FQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + +/* 5.17 Page Request Queue CSR (32bits) */ +#define RISCV_IOMMU_REG_PQCSR 0x0050 +#define RISCV_IOMMU_PQCSR_PQEN RISCV_IOMMU_QUEUE_ENABLE +#define RISCV_IOMMU_PQCSR_PIE RISCV_IOMMU_QUEUE_INTR_ENABLE +#define RISCV_IOMMU_PQCSR_PQMF RISCV_IOMMU_QUEUE_MEM_FAULT +#define RISCV_IOMMU_PQCSR_PQOF RISCV_IOMMU_QUEUE_OVERFLOW +#define RISCV_IOMMU_PQCSR_PQON RISCV_IOMMU_QUEUE_ACTIVE +#define RISCV_IOMMU_PQCSR_BUSY RISCV_IOMMU_QUEUE_BUSY + +/* 5.18 Interrupt Pending Status (32bits) */ +#define RISCV_IOMMU_REG_IPSR 0x0054 + +#define RISCV_IOMMU_INTR_CQ 0 +#define RISCV_IOMMU_INTR_FQ 1 +#define RISCV_IOMMU_INTR_PM 2 +#define RISCV_IOMMU_INTR_PQ 3 +#define RISCV_IOMMU_INTR_COUNT 4 + +#define RISCV_IOMMU_IPSR_CIP BIT(RISCV_IOMMU_INTR_CQ) +#define RISCV_IOMMU_IPSR_FIP BIT(RISCV_IOMMU_INTR_FQ) +#define RISCV_IOMMU_IPSR_PMIP BIT(RISCV_IOMMU_INTR_PM) +#define RISCV_IOMMU_IPSR_PIP BIT(RISCV_IOMMU_INTR_PQ) + +/* 5.19 Performance monitoring counter overflow status (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTOVF 0x0058 +#define RISCV_IOMMU_IOCOUNTOVF_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTOVF_HPM GENMASK_ULL(31, 1) + +/* 5.20 Performance monitoring counter inhibits (32bits) */ +#define RISCV_IOMMU_REG_IOCOUNTINH 0x005C +#define RISCV_IOMMU_IOCOUNTINH_CY BIT(0) +#define RISCV_IOMMU_IOCOUNTINH_HPM GENMASK(31, 1) + +/* 5.21 Performance monitoring cycles counter (64bits) */ +#define RISCV_IOMMU_REG_IOHPMCYCLES 0x0060 +#define RISCV_IOMMU_IOHPMCYCLES_COUNTER GENMASK_ULL(62, 0) +#define RISCV_IOMMU_IOHPMCYCLES_OVF BIT_ULL(63) + +/* 5.22 Performance monitoring event counters (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMCTR_BASE 0x0068 +#define RISCV_IOMMU_REG_IOHPMCTR(_n) (RISCV_IOMMU_REG_IOHPMCTR_BASE + (_n * 0x8)) + +/* 5.23 Performance monitoring event selectors (31 * 64bits) */ +#define RISCV_IOMMU_REG_IOHPMEVT_BASE 0x0160 +#define RISCV_IOMMU_REG_IOHPMEVT(_n) (RISCV_IOMMU_REG_IOHPMEVT_BASE + (_n * 0x8)) +#define RISCV_IOMMU_IOHPMEVT_EVENT_ID GENMASK_ULL(14, 0) +#define RISCV_IOMMU_IOHPMEVT_DMASK BIT_ULL(15) +#define RISCV_IOMMU_IOHPMEVT_PID_PSCID GENMASK_ULL(35, 16) +#define RISCV_IOMMU_IOHPMEVT_DID_GSCID GENMASK_ULL(59, 36) +#define RISCV_IOMMU_IOHPMEVT_PV_PSCV BIT_ULL(60) +#define RISCV_IOMMU_IOHPMEVT_DV_GSCV BIT_ULL(61) +#define RISCV_IOMMU_IOHPMEVT_IDT BIT_ULL(62) +#define RISCV_IOMMU_IOHPMEVT_OF BIT_ULL(63) + +/** + * enum riscv_iommu_hpmevent_id - Performance-monitoring event identifier + * + * @RISCV_IOMMU_HPMEVENT_INVALID: Invalid event, do not count + * @RISCV_IOMMU_HPMEVENT_URQ: Untranslated requests + * @RISCV_IOMMU_HPMEVENT_TRQ: Translated requests + * @RISCV_IOMMU_HPMEVENT_ATS_RQ: ATS translation requests + * @RISCV_IOMMU_HPMEVENT_TLB_MISS: TLB misses + * @RISCV_IOMMU_HPMEVENT_DD_WALK: Device directory walks + * @RISCV_IOMMU_HPMEVENT_PD_WALK: Process directory walks + * @RISCV_IOMMU_HPMEVENT_S_VS_WALKS: S/VS-Stage page table walks + * @RISCV_IOMMU_HPMEVENT_G_WALKS: G-Stage page table walks + * @RISCV_IOMMU_HPMEVENT_MAX: Value to denote maximum Event IDs + */ +enum riscv_iommu_hpmevent_id { + RISCV_IOMMU_HPMEVENT_INVALID = 0, + RISCV_IOMMU_HPMEVENT_URQ = 1, + RISCV_IOMMU_HPMEVENT_TRQ = 2, + RISCV_IOMMU_HPMEVENT_ATS_RQ = 3, + RISCV_IOMMU_HPMEVENT_TLB_MISS = 4, + RISCV_IOMMU_HPMEVENT_DD_WALK = 5, + RISCV_IOMMU_HPMEVENT_PD_WALK = 6, + RISCV_IOMMU_HPMEVENT_S_VS_WALKS = 7, + RISCV_IOMMU_HPMEVENT_G_WALKS = 8, + RISCV_IOMMU_HPMEVENT_MAX = 9 +}; + +/* 5.24 Translation request IOVA (64bits) */ +#define RISCV_IOMMU_REG_TR_REQ_IOVA 0x0258 +#define RISCV_IOMMU_TR_REQ_IOVA_VPN GENMASK_ULL(63, 12) + +/* 5.25 Translation request control (64bits) */ +#define RISCV_IOMMU_REG_TR_REQ_CTL 0x0260 +#define RISCV_IOMMU_TR_REQ_CTL_GO_BUSY BIT_ULL(0) +#define RISCV_IOMMU_TR_REQ_CTL_PRIV BIT_ULL(1) +#define RISCV_IOMMU_TR_REQ_CTL_EXE BIT_ULL(2) +#define RISCV_IOMMU_TR_REQ_CTL_NW BIT_ULL(3) +#define RISCV_IOMMU_TR_REQ_CTL_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_TR_REQ_CTL_PV BIT_ULL(32) +#define RISCV_IOMMU_TR_REQ_CTL_DID GENMASK_ULL(63, 40) + +/* 5.26 Translation request response (64bits) */ +#define RISCV_IOMMU_REG_TR_RESPONSE 0x0268 +#define RISCV_IOMMU_TR_RESPONSE_FAULT BIT_ULL(0) +#define RISCV_IOMMU_TR_RESPONSE_PBMT GENMASK_ULL(8, 7) +#define RISCV_IOMMU_TR_RESPONSE_SZ BIT_ULL(9) +#define RISCV_IOMMU_TR_RESPONSE_PPN RISCV_IOMMU_PPN_FIELD + +/* 5.27 Interrupt cause to vector (64bits) */ +#define RISCV_IOMMU_REG_IVEC 0x02F8 +#define RISCV_IOMMU_IVEC_CIV GENMASK_ULL(3, 0) +#define RISCV_IOMMU_IVEC_FIV GENMASK_ULL(7, 4) +#define RISCV_IOMMU_IVEC_PMIV GENMASK_ULL(11, 8) +#define RISCV_IOMMU_IVEC_PIV GENMASK_ULL(15,12) + +/* 5.28 MSI Configuration table (32 * 64bits) */ +#define RISCV_IOMMU_REG_MSI_CONFIG 0x0300 +#define RISCV_IOMMU_REG_MSI_ADDR(_n) (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10)) +#define RISCV_IOMMU_MSI_ADDR GENMASK_ULL(55, 2) +#define RISCV_IOMMU_REG_MSI_DATA(_n) (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10) + 0x08) +#define RISCV_IOMMU_MSI_DATA GENMASK_ULL(31, 0) +#define RISCV_IOMMU_REG_MSI_VEC_CTL(_n) (RISCV_IOMMU_REG_MSI_CONFIG + (_n * 0x10) + 0x0C) +#define RISCV_IOMMU_MSI_VEC_CTL_M BIT_ULL(0) + +#define RISCV_IOMMU_REG_SIZE 0x1000 + +/* + * Chapter 2: Data structures + */ + +/* + * Device Directory Table macros for non-leaf nodes + */ +#define RISCV_IOMMU_DDTE_VALID BIT_ULL(0) +#define RISCV_IOMMU_DDTE_PPN RISCV_IOMMU_PPN_FIELD + +/** + * struct riscv_iommu_dc - Device Context + * @tc: Translation Control + * @iohgatp: I/O Hypervisor guest address translation and protection + * (Second stage context) + * @ta: Translation Attributes + * @fsc: First stage context + * @msiptpt: MSI page table pointer + * @msi_addr_mask: MSI address mask + * @msi_addr_pattern: MSI address pattern + * + * This structure is used for leaf nodes on the Device Directory Table, + * in case RISCV_IOMMU_CAP_MSI_FLAT is not set, the bottom 4 fields are + * not present and are skipped with pointer arithmetic to avoid + * casting, check out riscv_iommu_get_dc(). + * See section 2.1 for more details + */ +struct riscv_iommu_dc { + u64 tc; + u64 iohgatp; + u64 ta; + u64 fsc; + u64 msiptp; + u64 msi_addr_mask; + u64 msi_addr_pattern; + u64 _reserved; +}; + +/* Translation control fields */ +#define RISCV_IOMMU_DC_TC_V BIT_ULL(0) +#define RISCV_IOMMU_DC_TC_EN_ATS BIT_ULL(1) +#define RISCV_IOMMU_DC_TC_EN_PRI BIT_ULL(2) +#define RISCV_IOMMU_DC_TC_T2GPA BIT_ULL(3) +#define RISCV_IOMMU_DC_TC_DTF BIT_ULL(4) +#define RISCV_IOMMU_DC_TC_PDTV BIT_ULL(5) +#define RISCV_IOMMU_DC_TC_PRPR BIT_ULL(6) +#define RISCV_IOMMU_DC_TC_GADE BIT_ULL(7) +#define RISCV_IOMMU_DC_TC_SADE BIT_ULL(8) +#define RISCV_IOMMU_DC_TC_DPE BIT_ULL(9) +#define RISCV_IOMMU_DC_TC_SBE BIT_ULL(10) +#define RISCV_IOMMU_DC_TC_SXL BIT_ULL(11) + +/* Second-stage (aka G-stage) context fields */ +#define RISCV_IOMMU_DC_IOHGATP_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_IOHGATP_GSCID GENMASK_ULL(59, 44) +#define RISCV_IOMMU_DC_IOHGATP_MODE RISCV_IOMMU_ATP_MODE_FIELD + +/** + * enum riscv_iommu_dc_iohgatp_modes - Guest address translation/protection modes + * @RISCV_IOMMU_DC_IOHGATP_MODE_BARE: No translation/protection + * @RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4: Sv32x4 (2-bit extension of Sv32), when fctl.GXL == 1 + * @RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4: Sv39x4 (2-bit extension of Sv39), when fctl.GXL == 0 + * @RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4: Sv48x4 (2-bit extension of Sv48), when fctl.GXL == 0 + * @RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4: Sv57x4 (2-bit extension of Sv57), when fctl.GXL == 0 + */ +enum riscv_iommu_dc_iohgatp_modes { + RISCV_IOMMU_DC_IOHGATP_MODE_BARE = 0, + RISCV_IOMMU_DC_IOHGATP_MODE_SV32X4 = 8, + RISCV_IOMMU_DC_IOHGATP_MODE_SV39X4 = 8, + RISCV_IOMMU_DC_IOHGATP_MODE_SV48X4 = 9, + RISCV_IOMMU_DC_IOHGATP_MODE_SV57X4 = 10 +}; + +/* Translation attributes fields */ +#define RISCV_IOMMU_DC_TA_PSCID GENMASK_ULL(31,12) + +/* First-stage context fields */ +#define RISCV_IOMMU_DC_FSC_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_FSC_MODE RISCV_IOMMU_ATP_MODE_FIELD + +/** + * enum riscv_iommu_dc_fsc_atp_modes - First stage address translation/protection modes + * @RISCV_IOMMU_DC_FSC_MODE_BARE: No translation/protection + * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32: Sv32, when dc.tc.SXL == 1 + * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39: Sv39, when dc.tc.SXL == 0 + * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48: Sv48, when dc.tc.SXL == 0 + * @RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57: Sv57, when dc.tc.SXL == 0 + * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8: 1lvl PDT, 8bit process ids + * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17: 2lvl PDT, 17bit process ids + * @RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20: 3lvl PDT, 20bit process ids + * + * FSC holds IOSATP when RISCV_IOMMU_DC_TC_PDTV is 0 and PDTP otherwise. + * IOSATP controls the first stage address translation (same as the satp register on + * the RISC-V MMU), and PDTP holds the process directory table, used to select a + * first stage page table based on a process id (for devices that support multiple + * process ids). + */ +enum riscv_iommu_dc_fsc_atp_modes { + RISCV_IOMMU_DC_FSC_MODE_BARE = 0, + RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV32 = 8, + RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV39 = 8, + RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV48 = 9, + RISCV_IOMMU_DC_FSC_IOSATP_MODE_SV57 = 10, + RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8 = 1, + RISCV_IOMMU_DC_FSC_PDTP_MODE_PD17 = 2, + RISCV_IOMMU_DC_FSC_PDTP_MODE_PD20 = 3 +}; + +/* MSI page table pointer */ +#define RISCV_IOMMU_DC_MSIPTP_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_DC_MSIPTP_MODE RISCV_IOMMU_ATP_MODE_FIELD +#define RISCV_IOMMU_DC_MSIPTP_MODE_OFF 0 +#define RISCV_IOMMU_DC_MSIPTP_MODE_FLAT 1 + +/* MSI address mask */ +#define RISCV_IOMMU_DC_MSI_ADDR_MASK GENMASK_ULL(51, 0) + +/* MSI address pattern */ +#define RISCV_IOMMU_DC_MSI_PATTERN GENMASK_ULL(51, 0) + +/** + * struct riscv_iommu_pc - Process Context + * @ta: Translation Attributes + * @fsc: First stage context + * + * This structure is used for leaf nodes on the Process Directory Table + * See section 2.3 for more details + */ +struct riscv_iommu_pc { + u64 ta; + u64 fsc; +}; + +/* Translation attributes fields */ +#define RISCV_IOMMU_PC_TA_V BIT_ULL(0) +#define RISCV_IOMMU_PC_TA_ENS BIT_ULL(1) +#define RISCV_IOMMU_PC_TA_SUM BIT_ULL(2) +#define RISCV_IOMMU_PC_TA_PSCID GENMASK_ULL(31, 12) + +/* First stage context fields */ +#define RISCV_IOMMU_PC_FSC_PPN RISCV_IOMMU_ATP_PPN_FIELD +#define RISCV_IOMMU_PC_FSC_MODE RISCV_IOMMU_ATP_MODE_FIELD + +/* + * Chapter 3: In-memory queue interface + */ + +/** + * struct riscv_iommu_cmd - Generic I/O MMU command structure + * @dword0: Includes the opcode and the function identifier + * @dword1: Opcode specific data + * + * The commands are interpreted as two 64bit fields, where the first + * 7bits of the first field are the opcode which also defines the + * command's format, followed by a 3bit field that specifies the + * function invoked by that command, and the rest is opcode-specific. + * This is a generic struct which will be populated differently + * according to each command. For more infos on the commands and + * the command queue check section 3.1. + */ +struct riscv_iommu_command { + u64 dword0; + u64 dword1; +}; + +/* Fields on dword0, common for all commands */ +#define RISCV_IOMMU_CMD_OPCODE GENMASK_ULL(6, 0) +#define RISCV_IOMMU_CMD_FUNC GENMASK_ULL(9, 7) + +/* 3.1.1 I/O MMU Page-table cache invalidation */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IOTINVAL_OPCODE 1 +#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA 0 +#define RISCV_IOMMU_CMD_IOTINVAL_FUNC_GVMA 1 +#define RISCV_IOMMU_CMD_IOTINVAL_AV BIT_ULL(10) +#define RISCV_IOMMU_CMD_IOTINVAL_PSCID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_IOTINVAL_PSCV BIT_ULL(32) +#define RISCV_IOMMU_CMD_IOTINVAL_GV BIT_ULL(33) +#define RISCV_IOMMU_CMD_IOTINVAL_GSCID GENMASK_ULL(59, 44) +/* dword1 is the address, 4K-alligned and shifted to the right by + * two bits. */ + +/* 3.1.2 I/O MMU Command Queue Fences */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IOFENCE_OPCODE 2 +#define RISCV_IOMMU_CMD_IOFENCE_FUNC_C 0 +#define RISCV_IOMMU_CMD_IOFENCE_AV BIT_ULL(10) +#define RISCV_IOMMU_CMD_IOFENCE_WSI BIT_ULL(11) +#define RISCV_IOMMU_CMD_IOFENCE_PR BIT_ULL(12) +#define RISCV_IOMMU_CMD_IOFENCE_PW BIT_ULL(13) +#define RISCV_IOMMU_CMD_IOFENCE_DATA GENMASK_ULL(63, 32) +/* dword1 is the address, word-size alligned and shifted to the + * right by two bits. */ + +/* 3.1.3 I/O MMU Directory cache invalidation */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_IODIR_OPCODE 3 +#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT 0 +#define RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT 1 +#define RISCV_IOMMU_CMD_IODIR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_IODIR_DV BIT_ULL(33) +#define RISCV_IOMMU_CMD_IODIR_DID GENMASK_ULL(63, 40) +/* dword1 is reserved for standard use */ + +/* 3.1.4 I/O MMU PCIe ATS */ +/* Fields on dword0 */ +#define RISCV_IOMMU_CMD_ATS_OPCODE 4 +#define RISCV_IOMMU_CMD_ATS_FUNC_INVAL 0 +#define RISCV_IOMMU_CMD_ATS_FUNC_PRGR 1 +#define RISCV_IOMMU_CMD_ATS_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_CMD_ATS_PV BIT_ULL(32) +#define RISCV_IOMMU_CMD_ATS_DSV BIT_ULL(33) +#define RISCV_IOMMU_CMD_ATS_RID GENMASK_ULL(55, 40) +#define RISCV_IOMMU_CMD_ATS_DSEG GENMASK_ULL(63, 56) +/* dword1 is the ATS payload, two different payload types for INVAL and PRGR */ + +/* ATS.INVAL payload*/ +#define RISCV_IOMMU_CMD_ATS_INVAL_G BIT_ULL(0) +/* Bits 1 - 10 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_INVAL_S BIT_ULL(11) +#define RISCV_IOMMU_CMD_ATS_INVAL_UADDR GENMASK_ULL(63, 12) + +/* ATS.PRGR payload */ +/* Bits 0 - 31 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_PRGR_PRG_INDEX GENMASK_ULL(40, 32) +/* Bits 41 - 43 are zeroed */ +#define RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE GENMASK_ULL(47, 44) +#define RISCV_IOMMU_CMD_ATS_PRGR_DST_ID GENMASK_ULL(63, 48) + +/** + * struct riscv_iommu_fq_record - Fault/Event Queue Record + * @hdr: Header, includes fault/event cause, PID/DID, transaction type etc + * @_reserved: Low 32bits for custom use, high 32bits for standard use + * @iotval: Transaction-type/cause specific format + * @iotval2: Cause specific format + * + * The fault/event queue reports events and failures raised when + * processing transactions. Each record is a 32byte structure where + * the first dword has a fixed format for providing generic infos + * regarding the fault/event, and two more dwords are there for + * fault/event-specific information. For more details see section + * 3.2. + */ +struct riscv_iommu_fq_record { + u64 hdr; + u64 _reserved; + u64 iotval; + u64 iotval2; +}; + +/* Fields on header */ +#define RISCV_IOMMU_FQ_HDR_CAUSE GENMASK_ULL(11, 0) +#define RISCV_IOMMU_FQ_HDR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_FQ_HDR_PV BIT_ULL(32) +#define RISCV_IOMMU_FQ_HDR_PRIV BIT_ULL(33) +#define RISCV_IOMMU_FQ_HDR_TTYPE GENMASK_ULL(39, 34) +#define RISCV_IOMMU_FQ_HDR_DID GENMASK_ULL(63, 40) + +/** + * enum riscv_iommu_fq_causes - Fault/event cause values + * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT: Instruction access fault + * @RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED: Read address misaligned + * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT: Read load fault + * @RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED: Write/AMO address misaligned + * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT: Write/AMO access fault + * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S: Instruction page fault + * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S: Read page fault + * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S: Write/AMO page fault + * @RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS: Instruction guest page fault + * @RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS: Read guest page fault + * @RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS: Write/AMO guest page fault + * @RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED: All inbound transactions disallowed + * @RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT: DDT entry load access fault + * @RISCV_IOMMU_FQ_CAUSE_DDT_INVALID: DDT entry invalid + * @RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED: DDT entry misconfigured + * @RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED: Transaction type disallowed + * @RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT: MSI PTE load access fault + * @RISCV_IOMMU_FQ_CAUSE_MSI_INVALID: MSI PTE invalid + * @RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED: MSI PTE misconfigured + * @RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT: MRIF access fault + * @RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT: PDT entry load access fault + * @RISCV_IOMMU_FQ_CAUSE_PDT_INVALID: PDT entry invalid + * @RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED: PDT entry misconfigured + * @RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED: DDT data corruption + * @RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED: PDT data corruption + * @RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED: MSI page table data corruption + * @RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED: MRIF data corruption + * @RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR: Internal data path error + * @RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT: IOMMU MSI write access fault + * @RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED: First/second stage page table data corruption + * + * Values are on table 11 of the spec, encodings 275 - 2047 are reserved for standard + * use, and 2048 - 4095 for custom use. + */ +enum riscv_iommu_fq_causes { + RISCV_IOMMU_FQ_CAUSE_INST_FAULT = 1, + RISCV_IOMMU_FQ_CAUSE_RD_ADDR_MISALIGNED = 4, + RISCV_IOMMU_FQ_CAUSE_RD_FAULT = 5, + RISCV_IOMMU_FQ_CAUSE_WR_ADDR_MISALIGNED = 6, + RISCV_IOMMU_FQ_CAUSE_WR_FAULT = 7, + RISCV_IOMMU_FQ_CAUSE_INST_FAULT_S = 12, + RISCV_IOMMU_FQ_CAUSE_RD_FAULT_S = 13, + RISCV_IOMMU_FQ_CAUSE_WR_FAULT_S = 15, + RISCV_IOMMU_FQ_CAUSE_INST_FAULT_VS = 20, + RISCV_IOMMU_FQ_CAUSE_RD_FAULT_VS = 21, + RISCV_IOMMU_FQ_CAUSE_WR_FAULT_VS = 23, + RISCV_IOMMU_FQ_CAUSE_DMA_DISABLED = 256, + RISCV_IOMMU_FQ_CAUSE_DDT_LOAD_FAULT = 257, + RISCV_IOMMU_FQ_CAUSE_DDT_INVALID = 258, + RISCV_IOMMU_FQ_CAUSE_DDT_MISCONFIGURED = 259, + RISCV_IOMMU_FQ_CAUSE_TTYPE_BLOCKED = 260, + RISCV_IOMMU_FQ_CAUSE_MSI_LOAD_FAULT = 261, + RISCV_IOMMU_FQ_CAUSE_MSI_INVALID = 262, + RISCV_IOMMU_FQ_CAUSE_MSI_MISCONFIGURED = 263, + RISCV_IOMMU_FQ_CAUSE_MRIF_FAULT = 264, + RISCV_IOMMU_FQ_CAUSE_PDT_LOAD_FAULT = 265, + RISCV_IOMMU_FQ_CAUSE_PDT_INVALID = 266, + RISCV_IOMMU_FQ_CAUSE_PDT_MISCONFIGURED = 267, + RISCV_IOMMU_FQ_CAUSE_DDT_CORRUPTED = 268, + RISCV_IOMMU_FQ_CAUSE_PDT_CORRUPTED = 269, + RISCV_IOMMU_FQ_CAUSE_MSI_PT_CORRUPTED = 270, + RISCV_IOMMU_FQ_CAUSE_MRIF_CORRUIPTED = 271, + RISCV_IOMMU_FQ_CAUSE_INTERNAL_DP_ERROR = 272, + RISCV_IOMMU_FQ_CAUSE_MSI_WR_FAULT = 273, + RISCV_IOMMU_FQ_CAUSE_PT_CORRUPTED = 274 +}; + +/** + * enum riscv_iommu_fq_ttypes: Fault/event transaction types + * @RISCV_IOMMU_FQ_TTYPE_NONE: None. Fault not caused by an inbound transaction. + * @RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH: Instruction fetch from untranslated address + * @RISCV_IOMMU_FQ_TTYPE_UADDR_RD: Read from untranslated address + * @RISCV_IOMMU_FQ_TTYPE_UADDR_WR: Write/AMO to untranslated address + * @RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH: Instruction fetch from translated address + * @RISCV_IOMMU_FQ_TTYPE_TADDR_RD: Read from translated address + * @RISCV_IOMMU_FQ_TTYPE_TADDR_WR: Write/AMO to translated address + * @RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ: PCIe ATS translation request + * @RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ: PCIe message request + * + * Values are on table 12 of the spec, type 4 and 10 - 31 are reserved for standard use + * and 31 - 63 for custom use. + */ +enum riscv_iommu_fq_ttypes { + RISCV_IOMMU_FQ_TTYPE_NONE = 0, + RISCV_IOMMU_FQ_TTYPE_UADDR_INST_FETCH = 1, + RISCV_IOMMU_FQ_TTYPE_UADDR_RD = 2, + RISCV_IOMMU_FQ_TTYPE_UADDR_WR = 3, + RISCV_IOMMU_FQ_TTYPE_TADDR_INST_FETCH = 5, + RISCV_IOMMU_FQ_TTYPE_TADDR_RD = 6, + RISCV_IOMMU_FQ_TTYPE_TADDR_WR = 7, + RISCV_IOMMU_FQ_TTYPE_PCIE_ATS_REQ = 8, + RISCV_IOMMU_FW_TTYPE_PCIE_MSG_REQ = 9, +}; + +/** + * struct riscv_iommu_pq_record - PCIe Page Request record + * @hdr: Header, includes PID, DID etc + * @payload: Holds the page address, request group and permission bits + * + * For more infos on the PCIe Page Request queue see chapter 3.3. + */ +struct riscv_iommu_pq_record { + u64 hdr; + u64 payload; +}; + +/* Header fields */ +#define RISCV_IOMMU_PREQ_HDR_PID GENMASK_ULL(31, 12) +#define RISCV_IOMMU_PREQ_HDR_PV BIT_ULL(32) +#define RISCV_IOMMU_PREQ_HDR_PRIV BIT_ULL(33) +#define RISCV_IOMMU_PREQ_HDR_EXEC BIT_ULL(34) +#define RISCV_IOMMU_PREQ_HDR_DID GENMASK_ULL(63, 40) + +/* Payload fields */ +#define RISCV_IOMMU_PREQ_PAYLOAD_R BIT_ULL(0) +#define RISCV_IOMMU_PREQ_PAYLOAD_W BIT_ULL(1) +#define RISCV_IOMMU_PREQ_PAYLOAD_L BIT_ULL(2) +#define RISCV_IOMMU_PREQ_PAYLOAD_M GENMASK_ULL(2, 0) /* Mask of RWL for convenience */ +#define RISCV_IOMMU_PREQ_PRG_INDEX GENMASK_ULL(11, 3) +#define RISCV_IOMMU_PREQ_UADDR GENMASK_ULL(63, 12) + +/** + * struct riscv_iommu_msi_pte - MSI Page Table Entry + * @pte: MSI PTE + * @mrif_info: Memory-resident interrupt file info + * + * The MSI Page Table is used for virtualizing MSIs, so that when + * a device sends an MSI to a guest, the IOMMU can reroute it + * by translating the MSI address, either to a guest interrupt file + * or a memory resident interrupt file (MRIF). Note that this page table + * is an array of MSI PTEs, not a multi-level pt, each entry + * is a leaf entry. For more infos check out the the AIA spec, chapter 9.5. + * + * Also in basic mode the mrif_info field is ignored by the IOMMU and can + * be used by software, any other reserved fields on pte must be zeroed-out + * by software. + */ +struct riscv_iommu_msi_pte { + u64 pte; + u64 mrif_info; +}; + +/* Fields on pte */ +#define RISCV_IOMMU_MSI_PTE_V BIT_ULL(0) +#define RISCV_IOMMU_MSI_PTE_M GENMASK_ULL(2, 1) +#define RISCV_IOMMU_MSI_PTE_MRIF_ADDR GENMASK_ULL(53, 7) /* When M == 1 (MRIF mode) */ +#define RISCV_IOMMU_MSI_PTE_PPN RISCV_IOMMU_PPN_FIELD /* When M == 3 (basic mode) */ +#define RISCV_IOMMU_MSI_PTE_C BIT_ULL(63) + +/* Fields on mrif_info */ +#define RISCV_IOMMU_MSI_MRIF_NID GENMASK_ULL(9, 0) +#define RISCV_IOMMU_MSI_MRIF_NPPN RISCV_IOMMU_PPN_FIELD +#define RISCV_IOMMU_MSI_MRIF_NID_MSB BIT_ULL(60) + +#endif /* _RISCV_IOMMU_BITS_H_ */ diff --git a/drivers/iommu/riscv/iommu-pci.c b/drivers/iommu/riscv/iommu-pci.c new file mode 100644 index 000000000000..c91f963d7a29 --- /dev/null +++ b/drivers/iommu/riscv/iommu-pci.c @@ -0,0 +1,134 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* + * Copyright © 2022-2023 Rivos Inc. + * Copyright © 2023 FORTH-ICS/CARV + * + * RISCV IOMMU as a PCIe device + * + * Authors + * Tomasz Jeznach + * Nick Kossifidis + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "iommu.h" + +/* Rivos Inc. assigned PCI Vendor and Device IDs */ +#ifndef PCI_VENDOR_ID_RIVOS +#define PCI_VENDOR_ID_RIVOS 0x1efd +#endif + +#ifndef PCI_DEVICE_ID_RIVOS_IOMMU +#define PCI_DEVICE_ID_RIVOS_IOMMU 0xedf1 +#endif + +static int riscv_iommu_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) +{ + struct device *dev = &pdev->dev; + struct riscv_iommu_device *iommu; + int ret; + + ret = pci_enable_device_mem(pdev); + if (ret < 0) + return ret; + + ret = pci_request_mem_regions(pdev, KBUILD_MODNAME); + if (ret < 0) + goto fail; + + ret = -ENOMEM; + + iommu = devm_kzalloc(dev, sizeof(*iommu), GFP_KERNEL); + if (!iommu) + goto fail; + + if (!(pci_resource_flags(pdev, 0) & IORESOURCE_MEM)) + goto fail; + + if (pci_resource_len(pdev, 0) < RISCV_IOMMU_REG_SIZE) + goto fail; + + iommu->reg_phys = pci_resource_start(pdev, 0); + if (!iommu->reg_phys) + goto fail; + + iommu->reg = devm_ioremap(dev, iommu->reg_phys, RISCV_IOMMU_REG_SIZE); + if (!iommu->reg) + goto fail; + + iommu->dev = dev; + dev_set_drvdata(dev, iommu); + + dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); + pci_set_master(pdev); + + ret = riscv_iommu_init(iommu); + if (!ret) + return ret; + + fail: + pci_clear_master(pdev); + pci_release_regions(pdev); + pci_disable_device(pdev); + /* Note: devres_release_all() will release iommu and iommu->reg */ + return ret; +} + +static void riscv_iommu_pci_remove(struct pci_dev *pdev) +{ + riscv_iommu_remove(dev_get_drvdata(&pdev->dev)); + pci_clear_master(pdev); + pci_release_regions(pdev); + pci_disable_device(pdev); +} + +static int riscv_iommu_suspend(struct device *dev) +{ + dev_warn(dev, "RISC-V IOMMU PM not implemented"); + return -ENODEV; +} + +static int riscv_iommu_resume(struct device *dev) +{ + dev_warn(dev, "RISC-V IOMMU PM not implemented"); + return -ENODEV; +} + +static DEFINE_SIMPLE_DEV_PM_OPS(riscv_iommu_pm_ops, riscv_iommu_suspend, + riscv_iommu_resume); + +static const struct pci_device_id riscv_iommu_pci_tbl[] = { + {PCI_VENDOR_ID_RIVOS, PCI_DEVICE_ID_RIVOS_IOMMU, + PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0}, + {0,} +}; + +MODULE_DEVICE_TABLE(pci, riscv_iommu_pci_tbl); + +static const struct of_device_id riscv_iommu_of_match[] = { + {.compatible = "riscv,pci-iommu",}, + {}, +}; + +MODULE_DEVICE_TABLE(of, riscv_iommu_of_match); + +static struct pci_driver riscv_iommu_pci_driver = { + .name = KBUILD_MODNAME, + .id_table = riscv_iommu_pci_tbl, + .probe = riscv_iommu_pci_probe, + .remove = riscv_iommu_pci_remove, + .driver = { + .pm = pm_sleep_ptr(&riscv_iommu_pm_ops), + .of_match_table = riscv_iommu_of_match, + }, +}; + +module_driver(riscv_iommu_pci_driver, pci_register_driver, pci_unregister_driver); diff --git a/drivers/iommu/riscv/iommu-platform.c b/drivers/iommu/riscv/iommu-platform.c new file mode 100644 index 000000000000..e4e8ca6711e7 --- /dev/null +++ b/drivers/iommu/riscv/iommu-platform.c @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * RISC-V IOMMU as a platform device + * + * Copyright © 2023 FORTH-ICS/CARV + * + * Author: Nick Kossifidis + */ + +#include +#include +#include +#include + +#include "iommu-bits.h" +#include "iommu.h" + +static int riscv_iommu_platform_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct riscv_iommu_device *iommu = NULL; + struct resource *res = NULL; + int ret = 0; + + iommu = devm_kzalloc(dev, sizeof(*iommu), GFP_KERNEL); + if (!iommu) + return -ENOMEM; + + iommu->dev = dev; + dev_set_drvdata(dev, iommu); + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + if (!res) { + dev_err(dev, "could not find resource for register region\n"); + return -EINVAL; + } + + iommu->reg = devm_platform_get_and_ioremap_resource(pdev, 0, &res); + if (IS_ERR(iommu->reg)) { + ret = dev_err_probe(dev, PTR_ERR(iommu->reg), + "could not map register region\n"); + goto fail; + }; + + iommu->reg_phys = res->start; + + ret = -ENODEV; + + /* Sanity check: Did we get the whole register space ? */ + if ((res->end - res->start + 1) < RISCV_IOMMU_REG_SIZE) { + dev_err(dev, "device region smaller than register file (0x%llx)\n", + res->end - res->start); + goto fail; + } + + dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); + + return riscv_iommu_init(iommu); + + fail: + /* Note: devres_release_all() will release iommu and iommu->reg */ + return ret; +}; + +static void riscv_iommu_platform_remove(struct platform_device *pdev) +{ + riscv_iommu_remove(dev_get_drvdata(&pdev->dev)); +}; + +static void riscv_iommu_platform_shutdown(struct platform_device *pdev) +{ + return; +}; + +static const struct of_device_id riscv_iommu_of_match[] = { + {.compatible = "riscv,iommu",}, + {}, +}; + +MODULE_DEVICE_TABLE(of, riscv_iommu_of_match); + +static struct platform_driver riscv_iommu_platform_driver = { + .driver = { + .name = "riscv,iommu", + .of_match_table = riscv_iommu_of_match, + .suppress_bind_attrs = true, + }, + .probe = riscv_iommu_platform_probe, + .remove_new = riscv_iommu_platform_remove, + .shutdown = riscv_iommu_platform_shutdown, +}; + +module_driver(riscv_iommu_platform_driver, platform_driver_register, + platform_driver_unregister); diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c new file mode 100644 index 000000000000..8c236242e2cc --- /dev/null +++ b/drivers/iommu/riscv/iommu.c @@ -0,0 +1,660 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * IOMMU API for RISC-V architected Ziommu implementations. + * + * Copyright © 2022-2023 Rivos Inc. + * Copyright © 2023 FORTH-ICS/CARV + * + * Authors + * Tomasz Jeznach + * Nick Kossifidis + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../dma-iommu.h" +#include "../iommu-sva.h" +#include "iommu.h" + +#include +#include + +MODULE_DESCRIPTION("IOMMU driver for RISC-V architected Ziommu implementations"); +MODULE_AUTHOR("Tomasz Jeznach "); +MODULE_AUTHOR("Nick Kossifidis "); +MODULE_ALIAS("riscv-iommu"); +MODULE_LICENSE("GPL v2"); + +/* Global IOMMU params. */ +static int ddt_mode = RISCV_IOMMU_DDTP_MODE_BARE; +module_param(ddt_mode, int, 0644); +MODULE_PARM_DESC(ddt_mode, "Device Directory Table mode."); + +/* IOMMU PSCID allocation namespace. */ +#define RISCV_IOMMU_MAX_PSCID (1U << 20) +static DEFINE_IDA(riscv_iommu_pscids); + +/* 1 second */ +#define RISCV_IOMMU_TIMEOUT riscv_timebase + +/* RISC-V IOMMU PPN <> PHYS address conversions, PHYS <=> PPN[53:10] */ +#define phys_to_ppn(va) (((va) >> 2) & (((1ULL << 44) - 1) << 10)) +#define ppn_to_phys(pn) (((pn) << 2) & (((1ULL << 44) - 1) << 12)) + +#define iommu_domain_to_riscv(iommu_domain) \ + container_of(iommu_domain, struct riscv_iommu_domain, domain) + +#define iommu_device_to_riscv(iommu_device) \ + container_of(iommu_device, struct riscv_iommu, iommu) + +static const struct iommu_domain_ops riscv_iommu_domain_ops; +static const struct iommu_ops riscv_iommu_ops; + +/* + * Register device for IOMMU tracking. + */ +static void riscv_iommu_add_device(struct riscv_iommu_device *iommu, struct device *dev) +{ + struct riscv_iommu_endpoint *ep, *rb_ep; + struct rb_node **new_node, *parent_node = NULL; + + mutex_lock(&iommu->eps_mutex); + + ep = dev_iommu_priv_get(dev); + + new_node = &(iommu->eps.rb_node); + while (*new_node) { + rb_ep = rb_entry(*new_node, struct riscv_iommu_endpoint, node); + parent_node = *new_node; + if (rb_ep->devid > ep->devid) { + new_node = &((*new_node)->rb_left); + } else if (rb_ep->devid < ep->devid) { + new_node = &((*new_node)->rb_right); + } else { + dev_warn(dev, "device %u already in the tree\n", ep->devid); + break; + } + } + + rb_link_node(&ep->node, parent_node, new_node); + rb_insert_color(&ep->node, &iommu->eps); + + mutex_unlock(&iommu->eps_mutex); +} + +/* + * Endpoint management + */ + +static int riscv_iommu_of_xlate(struct device *dev, struct of_phandle_args *args) +{ + return iommu_fwspec_add_ids(dev, args->args, 1); +} + +static bool riscv_iommu_capable(struct device *dev, enum iommu_cap cap) +{ + switch (cap) { + case IOMMU_CAP_CACHE_COHERENCY: + case IOMMU_CAP_PRE_BOOT_PROTECTION: + return true; + + default: + break; + } + + return false; +} + +static struct iommu_device *riscv_iommu_probe_device(struct device *dev) +{ + struct riscv_iommu_device *iommu; + struct riscv_iommu_endpoint *ep; + struct iommu_fwspec *fwspec; + + fwspec = dev_iommu_fwspec_get(dev); + if (!fwspec || fwspec->ops != &riscv_iommu_ops || + !fwspec->iommu_fwnode || !fwspec->iommu_fwnode->dev) + return ERR_PTR(-ENODEV); + + iommu = dev_get_drvdata(fwspec->iommu_fwnode->dev); + if (!iommu) + return ERR_PTR(-ENODEV); + + if (dev_iommu_priv_get(dev)) + return &iommu->iommu; + + ep = kzalloc(sizeof(*ep), GFP_KERNEL); + if (!ep) + return ERR_PTR(-ENOMEM); + + mutex_init(&ep->lock); + INIT_LIST_HEAD(&ep->domain); + + if (dev_is_pci(dev)) { + ep->devid = pci_dev_id(to_pci_dev(dev)); + ep->domid = pci_domain_nr(to_pci_dev(dev)->bus); + } else { + /* TODO: Make this generic, for now hardcode domain id to 0 */ + ep->devid = fwspec->ids[0]; + ep->domid = 0; + } + + ep->iommu = iommu; + ep->dev = dev; + + dev_info(iommu->dev, "adding device to iommu with devid %i in domain %i\n", + ep->devid, ep->domid); + + dev_iommu_priv_set(dev, ep); + riscv_iommu_add_device(iommu, dev); + + return &iommu->iommu; +} + +static void riscv_iommu_probe_finalize(struct device *dev) +{ + set_dma_ops(dev, NULL); + iommu_setup_dma_ops(dev, 0, U64_MAX); +} + +static void riscv_iommu_release_device(struct device *dev) +{ + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + struct riscv_iommu_device *iommu = ep->iommu; + + dev_info(dev, "device with devid %i released\n", ep->devid); + + mutex_lock(&ep->lock); + list_del(&ep->domain); + mutex_unlock(&ep->lock); + + /* Remove endpoint from IOMMU tracking structures */ + mutex_lock(&iommu->eps_mutex); + rb_erase(&ep->node, &iommu->eps); + mutex_unlock(&iommu->eps_mutex); + + set_dma_ops(dev, NULL); + dev_iommu_priv_set(dev, NULL); + + kfree(ep); +} + +static struct iommu_group *riscv_iommu_device_group(struct device *dev) +{ + if (dev_is_pci(dev)) + return pci_device_group(dev); + return generic_device_group(dev); +} + +static void riscv_iommu_get_resv_regions(struct device *dev, struct list_head *head) +{ + iommu_dma_get_resv_regions(dev, head); +} + +/* + * Domain management + */ + +static struct iommu_domain *riscv_iommu_domain_alloc(unsigned type) +{ + struct riscv_iommu_domain *domain; + + if (type != IOMMU_DOMAIN_IDENTITY && + type != IOMMU_DOMAIN_BLOCKED) + return NULL; + + domain = kzalloc(sizeof(*domain), GFP_KERNEL); + if (!domain) + return NULL; + + mutex_init(&domain->lock); + INIT_LIST_HEAD(&domain->endpoints); + + domain->domain.ops = &riscv_iommu_domain_ops; + domain->mode = RISCV_IOMMU_DC_FSC_MODE_BARE; + domain->pscid = ida_alloc_range(&riscv_iommu_pscids, 1, + RISCV_IOMMU_MAX_PSCID, GFP_KERNEL); + + printk("domain type %x alloc %u\n", type, domain->pscid); + + return &domain->domain; +} + +static void riscv_iommu_domain_free(struct iommu_domain *iommu_domain) +{ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + + if (!list_empty(&domain->endpoints)) { + pr_warn("IOMMU domain is not empty!\n"); + } + + if (domain->pgd_root) + free_pages((unsigned long)domain->pgd_root, 0); + + if ((int)domain->pscid > 0) + ida_free(&riscv_iommu_pscids, domain->pscid); + + printk("domain free %u\n", domain->pscid); + + kfree(domain); +} + +static int riscv_iommu_domain_finalize(struct riscv_iommu_domain *domain, + struct riscv_iommu_device *iommu) +{ + struct iommu_domain_geometry *geometry; + + /* Domain assigned to another iommu */ + if (domain->iommu && domain->iommu != iommu) + return -EINVAL; + /* Domain already initialized */ + else if (domain->iommu) + return 0; + + /* + * TODO: Before using VA_BITS and satp_mode here, verify they + * are supported by the iommu, through the capabilities register. + */ + + geometry = &domain->domain.geometry; + + /* + * Note: RISC-V Privilege spec mandates that virtual addresses + * need to be sign-extended, so if (VA_BITS - 1) is set, all + * bits >= VA_BITS need to also be set or else we'll get a + * page fault. However the code that creates the mappings + * above us (e.g. iommu_dma_alloc_iova()) won't do that for us + * for now, so we'll end up with invalid virtual addresses + * to map. As a workaround until we get this sorted out + * limit the available virtual addresses to VA_BITS - 1. + */ + geometry->aperture_start = 0; + geometry->aperture_end = DMA_BIT_MASK(VA_BITS - 1); + geometry->force_aperture = true; + + domain->iommu = iommu; + + if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) + return 0; + + /* TODO: Fix this for RV32 */ + domain->mode = satp_mode >> 60; + domain->pgd_root = (pgd_t *) __get_free_pages(GFP_KERNEL | __GFP_ZERO, 0); + + if (!domain->pgd_root) + return -ENOMEM; + + return 0; +} + +static int riscv_iommu_attach_dev(struct iommu_domain *iommu_domain, struct device *dev) +{ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + int ret; + + /* PSCID not valid */ + if ((int)domain->pscid < 0) + return -ENOMEM; + + mutex_lock(&domain->lock); + mutex_lock(&ep->lock); + + if (!list_empty(&ep->domain)) { + dev_warn(dev, "endpoint already attached to a domain. dropping\n"); + list_del_init(&ep->domain); + } + + /* allocate root pages, initialize io-pgtable ops, etc. */ + ret = riscv_iommu_domain_finalize(domain, ep->iommu); + if (ret < 0) { + dev_err(dev, "can not finalize domain: %d\n", ret); + mutex_unlock(&ep->lock); + mutex_unlock(&domain->lock); + return ret; + } + + if (ep->iommu->ddt_mode != RISCV_IOMMU_DDTP_MODE_BARE || + domain->domain.type != IOMMU_DOMAIN_IDENTITY) { + dev_warn(dev, "domain type %d not supported\n", + domain->domain.type); + return -ENODEV; + } + + list_add_tail(&ep->domain, &domain->endpoints); + mutex_unlock(&ep->lock); + mutex_unlock(&domain->lock); + + dev_info(dev, "domain type %d attached w/ PSCID %u\n", + domain->domain.type, domain->pscid); + + return 0; +} + +static void riscv_iommu_flush_iotlb_range(struct iommu_domain *iommu_domain, + unsigned long *start, unsigned long *end, + size_t *pgsize) +{ + /* Command interface not implemented */ +} + +static void riscv_iommu_flush_iotlb_all(struct iommu_domain *iommu_domain) +{ + riscv_iommu_flush_iotlb_range(iommu_domain, NULL, NULL, NULL); +} + +static void riscv_iommu_iotlb_sync(struct iommu_domain *iommu_domain, + struct iommu_iotlb_gather *gather) +{ + riscv_iommu_flush_iotlb_range(iommu_domain, &gather->start, &gather->end, + &gather->pgsize); +} + +static void riscv_iommu_iotlb_sync_map(struct iommu_domain *iommu_domain, + unsigned long iova, size_t size) +{ + unsigned long end = iova + size - 1; + /* + * Given we don't know the page size used by this range, we assume the + * smallest page size to ensure all possible entries are flushed from + * the IOATC. + */ + size_t pgsize = PAGE_SIZE; + riscv_iommu_flush_iotlb_range(iommu_domain, &iova, &end, &pgsize); +} + +static int riscv_iommu_map_pages(struct iommu_domain *iommu_domain, + unsigned long iova, phys_addr_t phys, + size_t pgsize, size_t pgcount, int prot, + gfp_t gfp, size_t *mapped) +{ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + + if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) { + *mapped = pgsize * pgcount; + return 0; + } + + return -ENODEV; +} + +static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain, + unsigned long iova, size_t pgsize, + size_t pgcount, struct iommu_iotlb_gather *gather) +{ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + + if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) + return pgsize * pgcount; + + return 0; +} + +static phys_addr_t riscv_iommu_iova_to_phys(struct iommu_domain *iommu_domain, + dma_addr_t iova) +{ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + + if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) + return (phys_addr_t) iova; + + return 0; +} + +/* + * Translation mode setup + */ + +static u64 riscv_iommu_get_ddtp(struct riscv_iommu_device *iommu) +{ + u64 ddtp; + cycles_t end_cycles = RISCV_IOMMU_TIMEOUT + get_cycles(); + + /* Wait for DDTP.BUSY to be cleared and return latest value */ + do { + ddtp = riscv_iommu_readq(iommu, RISCV_IOMMU_REG_DDTP); + if (!(ddtp & RISCV_IOMMU_DDTP_BUSY)) + break; + cpu_relax(); + } while (get_cycles() < end_cycles); + + return ddtp; +} + +static void riscv_iommu_ddt_cleanup(struct riscv_iommu_device *iommu) +{ + /* TODO: teardown whole device directory tree. */ + if (iommu->ddtp) { + if (iommu->ddtp_in_iomem) { + iounmap((void *)iommu->ddtp); + } else + free_page(iommu->ddtp); + iommu->ddtp = 0; + } +} + +static int riscv_iommu_enable(struct riscv_iommu_device *iommu, unsigned requested_mode) +{ + struct device *dev = iommu->dev; + u64 ddtp = 0; + u64 ddtp_paddr = 0; + unsigned mode = requested_mode; + unsigned mode_readback = 0; + + ddtp = riscv_iommu_get_ddtp(iommu); + if (ddtp & RISCV_IOMMU_DDTP_BUSY) + return -EBUSY; + + /* Disallow state transtion from xLVL to xLVL. */ + switch (FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp)) { + case RISCV_IOMMU_DDTP_MODE_BARE: + case RISCV_IOMMU_DDTP_MODE_OFF: + break; + default: + if ((mode != RISCV_IOMMU_DDTP_MODE_BARE) + && (mode != RISCV_IOMMU_DDTP_MODE_OFF)) + return -EINVAL; + break; + } + + retry: + switch (mode) { + case RISCV_IOMMU_DDTP_MODE_BARE: + case RISCV_IOMMU_DDTP_MODE_OFF: + riscv_iommu_ddt_cleanup(iommu); + ddtp = FIELD_PREP(RISCV_IOMMU_DDTP_MODE, mode); + break; + case RISCV_IOMMU_DDTP_MODE_1LVL: + case RISCV_IOMMU_DDTP_MODE_2LVL: + case RISCV_IOMMU_DDTP_MODE_3LVL: + if (!iommu->ddtp) { + /* + * We haven't initialized ddtp yet, since it's WARL make + * sure that we don't have a hardwired PPN field there + * that points to i/o memory instead. + */ + riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP, 0); + ddtp = riscv_iommu_get_ddtp(iommu); + ddtp_paddr = ppn_to_phys(ddtp); + if (ddtp_paddr) { + dev_warn(dev, "ddtp at 0x%llx\n", ddtp_paddr); + iommu->ddtp = + (unsigned long)ioremap(ddtp_paddr, PAGE_SIZE); + iommu->ddtp_in_iomem = true; + } else { + iommu->ddtp = get_zeroed_page(GFP_KERNEL); + } + } + if (!iommu->ddtp) + return -ENOMEM; + + ddtp = FIELD_PREP(RISCV_IOMMU_DDTP_MODE, mode) | + phys_to_ppn(__pa(iommu->ddtp)); + + break; + default: + return -EINVAL; + } + + riscv_iommu_writeq(iommu, RISCV_IOMMU_REG_DDTP, ddtp); + ddtp = riscv_iommu_get_ddtp(iommu); + if (ddtp & RISCV_IOMMU_DDTP_BUSY) { + dev_warn(dev, "timeout when setting ddtp (ddt mode: %i)\n", mode); + return -EBUSY; + } + + mode_readback = FIELD_GET(RISCV_IOMMU_DDTP_MODE, ddtp); + dev_info(dev, "mode_readback: %i, mode: %i\n", mode_readback, mode); + if (mode_readback != mode) { + /* + * Mode field is WARL, an I/O MMU may support a subset of + * directory table levels in which case if we tried to set + * an unsupported number of levels we'll readback either + * a valid xLVL or off/bare. If we got off/bare, try again + * with a smaller xLVL. + */ + if (mode_readback < RISCV_IOMMU_DDTP_MODE_1LVL && + mode > RISCV_IOMMU_DDTP_MODE_1LVL) { + mode--; + goto retry; + } + + /* + * We tried all supported xLVL modes and still got off/bare instead, + * an I/O MMU must support at least one supported xLVL mode so something + * went very wrong. + */ + if (mode_readback < RISCV_IOMMU_DDTP_MODE_1LVL && + mode == RISCV_IOMMU_DDTP_MODE_1LVL) + goto fail; + + /* + * We tried setting off or bare and got something else back, something + * went very wrong since off/bare is always legal. + */ + if (mode < RISCV_IOMMU_DDTP_MODE_1LVL) + goto fail; + + /* + * We tried setting an xLVL mode but got another xLVL mode that + * we don't support (e.g. a custom one). + */ + if (mode_readback > RISCV_IOMMU_DDTP_MODE_MAX) + goto fail; + + /* We tried setting an xLVL mode but got another supported xLVL mode */ + mode = mode_readback; + } + + if (mode != requested_mode) + dev_warn(dev, "unsupported DDT mode requested (%i), using %i instead\n", + requested_mode, mode); + + iommu->ddt_mode = mode; + dev_info(dev, "ddt_mode: %i\n", iommu->ddt_mode); + return 0; + + fail: + dev_err(dev, "failed to set DDT mode, tried: %i and got %i\n", mode, + mode_readback); + riscv_iommu_ddt_cleanup(iommu); + return -EINVAL; +} + +/* + * Common I/O MMU driver probe/teardown + */ + +static const struct iommu_domain_ops riscv_iommu_domain_ops = { + .free = riscv_iommu_domain_free, + .attach_dev = riscv_iommu_attach_dev, + .map_pages = riscv_iommu_map_pages, + .unmap_pages = riscv_iommu_unmap_pages, + .iova_to_phys = riscv_iommu_iova_to_phys, + .iotlb_sync = riscv_iommu_iotlb_sync, + .iotlb_sync_map = riscv_iommu_iotlb_sync_map, + .flush_iotlb_all = riscv_iommu_flush_iotlb_all, +}; + +static const struct iommu_ops riscv_iommu_ops = { + .owner = THIS_MODULE, + .pgsize_bitmap = SZ_4K | SZ_2M | SZ_512M, + .capable = riscv_iommu_capable, + .domain_alloc = riscv_iommu_domain_alloc, + .probe_device = riscv_iommu_probe_device, + .probe_finalize = riscv_iommu_probe_finalize, + .release_device = riscv_iommu_release_device, + .device_group = riscv_iommu_device_group, + .get_resv_regions = riscv_iommu_get_resv_regions, + .of_xlate = riscv_iommu_of_xlate, + .default_domain_ops = &riscv_iommu_domain_ops, +}; + +void riscv_iommu_remove(struct riscv_iommu_device *iommu) +{ + iommu_device_unregister(&iommu->iommu); + riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_OFF); +} + +int riscv_iommu_init(struct riscv_iommu_device *iommu) +{ + struct device *dev = iommu->dev; + u32 fctl = 0; + int ret; + + iommu->eps = RB_ROOT; + + fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL); + +#ifdef CONFIG_CPU_BIG_ENDIAN + if (!(cap & RISCV_IOMMU_CAP_END)) { + dev_err(dev, "IOMMU doesn't support Big Endian\n"); + return -EIO; + } else if (!(fctl & RISCV_IOMMU_FCTL_BE)) { + fctl |= FIELD_PREP(RISCV_IOMMU_FCTL_BE, 1); + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_FCTL, fctl); + } +#endif + + /* Clear any pending interrupt flag. */ + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, + RISCV_IOMMU_IPSR_CIP | + RISCV_IOMMU_IPSR_FIP | + RISCV_IOMMU_IPSR_PMIP | RISCV_IOMMU_IPSR_PIP); + spin_lock_init(&iommu->cq_lock); + mutex_init(&iommu->eps_mutex); + + ret = riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_BARE); + + if (ret) { + dev_err(dev, "cannot enable iommu device (%d)\n", ret); + goto fail; + } + + ret = iommu_device_register(&iommu->iommu, &riscv_iommu_ops, dev); + if (ret) { + dev_err(dev, "cannot register iommu interface (%d)\n", ret); + iommu_device_sysfs_remove(&iommu->iommu); + goto fail; + } + + return 0; + fail: + riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_OFF); + return ret; +} diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h new file mode 100644 index 000000000000..7baefd3630b3 --- /dev/null +++ b/drivers/iommu/riscv/iommu.h @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright © 2022-2023 Rivos Inc. + * Copyright © 2023 FORTH-ICS/CARV + * + * RISC-V Ziommu - IOMMU Interface Specification. + * + * Authors + * Tomasz Jeznach + * Nick Kossifidis + */ + +#ifndef _RISCV_IOMMU_H_ +#define _RISCV_IOMMU_H_ + +#include +#include +#include +#include +#include +#include +#include + +#include "iommu-bits.h" + +#define IOMMU_PAGE_SIZE_4K BIT_ULL(12) +#define IOMMU_PAGE_SIZE_2M BIT_ULL(21) +#define IOMMU_PAGE_SIZE_1G BIT_ULL(30) +#define IOMMU_PAGE_SIZE_512G BIT_ULL(39) + +struct riscv_iommu_device { + struct iommu_device iommu; /* iommu core interface */ + struct device *dev; /* iommu hardware */ + + /* hardware control register space */ + void __iomem *reg; + resource_size_t reg_phys; + + /* IRQs for the various queues */ + int irq_cmdq; + int irq_fltq; + int irq_pm; + int irq_priq; + + /* supported and enabled hardware capabilities */ + u64 cap; + + /* global lock, to be removed */ + spinlock_t cq_lock; + + /* device directory table root pointer and mode */ + unsigned long ddtp; + unsigned ddt_mode; + bool ddtp_in_iomem; + + /* Connected end-points */ + struct rb_root eps; + struct mutex eps_mutex; +}; + +struct riscv_iommu_domain { + struct iommu_domain domain; + + struct list_head endpoints; + struct mutex lock; + struct riscv_iommu_device *iommu; + + unsigned mode; /* RIO_ATP_MODE_* enum */ + unsigned pscid; /* RISC-V IOMMU PSCID */ + + pgd_t *pgd_root; /* page table root pointer */ +}; + +/* Private dev_iommu_priv object, device-domain relationship. */ +struct riscv_iommu_endpoint { + struct device *dev; /* platform or PCI endpoint device */ + unsigned devid; /* PCI bus:device:function number */ + unsigned domid; /* PCI domain number, segment */ + struct rb_node node; /* device tracking node (lookup by devid) */ + struct riscv_iommu_device *iommu; /* parent iommu device */ + + struct mutex lock; + struct list_head domain; /* endpoint attached managed domain */ +}; + +/* Helper functions and macros */ + +static inline u32 riscv_iommu_readl(struct riscv_iommu_device *iommu, + unsigned offset) +{ + return readl_relaxed(iommu->reg + offset); +} + +static inline void riscv_iommu_writel(struct riscv_iommu_device *iommu, + unsigned offset, u32 val) +{ + writel_relaxed(val, iommu->reg + offset); +} + +static inline u64 riscv_iommu_readq(struct riscv_iommu_device *iommu, + unsigned offset) +{ + return readq_relaxed(iommu->reg + offset); +} + +static inline void riscv_iommu_writeq(struct riscv_iommu_device *iommu, + unsigned offset, u64 val) +{ + writeq_relaxed(val, iommu->reg + offset); +} + +int riscv_iommu_init(struct riscv_iommu_device *iommu); +void riscv_iommu_remove(struct riscv_iommu_device *iommu); + +#endif From patchwork Wed Jul 19 19:33:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59274C04A94 for ; Wed, 19 Jul 2023 19:34:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=R/Jzjw8SSe18rpuaYQhKMXwSp3v7ZaVZA2QqFL1HtJg=; b=SrUEmZwuTNPax6 MHfnh0zopLdqzRwa8Zmrv3up9v3lkqt7k0b8VQ4bFw/lDNqS/8GKeR74VGuD5jJYx0SnRrCu71byv YWGr6o5Q+LjS+ZI8qwVaMQQABO+qceYxI0k5KIlnEnGyz1hzZZV0ldBqo7IisMfaui5CtLB2i6fxj rVdfYhLHYlnVC3sUowO+HkUY4nVqZEoVWoBgRD6ENiVuLPr25+Hx0BlGzJ0FppZtUTVpd0zPUMlkE 3jFUb2pjYUFr0YREYIf9qFoJwlZE7BgkuiUgIc8iXNSWDmITg45jb9RfFHZCwoiNNzuK0qPNN2JSL x1Fj59jElfvSxlN1wKlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwV-008h82-0c; Wed, 19 Jul 2023 19:34:35 +0000 Received: from mail-oo1-xc2a.google.com ([2607:f8b0:4864:20::c2a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwQ-008h4J-0h for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:32 +0000 Received: by mail-oo1-xc2a.google.com with SMTP id 006d021491bc7-563439ea4a2so67794eaf.0 for ; Wed, 19 Jul 2023 12:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795266; x=1690400066; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lKs32IhtjLCxiTXuNk8Uk4pJZW2VlUAJQXvCnEO8tTw=; b=ttEo92CKB+yR7JU7SvNGV8Atfkln1N918fvtYmF+Wgf2jy4/UvkxLvZRx27dLz5xpa MK28QZNyj3okLeYSDgB53bNVDZxBUGsJIzl7b0oHpt1mjYQlvG742DV2OfZqfp/toq6S cqv66fUUijUqZgzvJveQtk6qJyo4+Its4OkrqFa3ow6PzmvaLPj3RDllFbjds40oEybI 2GhbBCEJRA/5XrGCEgfKz/bCXBbANcUZY6dwuHG8oPTtVgjEK2yl85TLej5y5liOEynz +B83Dx+fGKsqspEcpyTw+uOBVovf/8uysb74AEM26+yzu0N1vlrrsUTsxZJlKdR5cWqB AE5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795266; x=1690400066; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lKs32IhtjLCxiTXuNk8Uk4pJZW2VlUAJQXvCnEO8tTw=; b=W5mw/W/vxE3MGaCaUif2em8Mv4ZowLbugsYM+tFvcGXG3fYfYP1KFIx0o9uf/co8am B7VfWxuruqv3NihQiNmvey2tH1K+Gm1WgWMxTDI+Y13gVeVCyKQjzV5A6CSsY5w/zB6J hkhsXuLHULjbpFcoSZKBLdMRyjFYv5u4AwfASJUgd9Oi6vGjxrBtYY3ntftvQjHX/jrx msXhbDvg3gUoQthkWuJjuQ1dN3TNCGeDVz1YnQ0IscUvG3rsQzcU05lWsPNbyCWEUANx jiSWCVtsD7Rq20Py0ujCzNMOCtsbQi3Rlv/S01jZclSRJ5AgLxQYy5o8e+OMaXm9b/AB xKZQ== X-Gm-Message-State: ABy/qLZQDe/8a3omVdKE2jhvSwJcKd5yWiLKI+gWvppYDMKkRBC7BhXQ Lwc+ONAKfqZGBE6KxSblU5Hmiw== X-Google-Smtp-Source: APBJJlF46NxrjlcdpKJudq/OXjPgbmDMiFkMVyFxOnpopja4waUL727ujepZ1ywnk72M22OszAadog== X-Received: by 2002:a05:6808:30a4:b0:3a4:f7b:4490 with SMTP id bl36-20020a05680830a400b003a40f7b4490mr21501434oib.57.1689795266359; Wed, 19 Jul 2023 12:34:26 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:26 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 02/11] RISC-V: arch/riscv/config: enable RISC-V IOMMU support Date: Wed, 19 Jul 2023 12:33:46 -0700 Message-Id: <961d125558137f7cb960de65e5f71da5d299d3bc.1689792825.git.tjeznach@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123430_300555_0E7C6AE1 X-CRM114-Status: UNSURE ( 7.96 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org --- arch/riscv/configs/defconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/riscv/configs/defconfig b/arch/riscv/configs/defconfig index 0a0107460a5c..1a0c3b24329f 100644 --- a/arch/riscv/configs/defconfig +++ b/arch/riscv/configs/defconfig @@ -178,6 +178,7 @@ CONFIG_VIRTIO_PCI=y CONFIG_VIRTIO_BALLOON=y CONFIG_VIRTIO_INPUT=y CONFIG_VIRTIO_MMIO=y +CONFIG_RISCV_IOMMU=y CONFIG_SUN8I_DE2_CCU=m CONFIG_SUN50I_IOMMU=y CONFIG_RPMSG_CHAR=y From patchwork Wed Jul 19 19:33:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB7F0C001B0 for ; Wed, 19 Jul 2023 19:34:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Gj+uY7TwAY5og1H/cPM3XtoFNn0HN16wXZvUeBzMK+4=; b=TQRC9zDn7x6s/E 8e34HtVZHtzBaYs2Xq4dFfj9/dZRIFT5GNI8HkRHyxYXjbkigvsMjZShBNhdnVR3umJmOpi9EPJsQ kzC8ES5Rbb46IbyQROR1JFTjlgp9Uo+cp8iKzWDF0dZqhCSKG9cq0eY4JLNEpn8V+aPg9tHKha0PV wfIyJfo39rXdqQS8Nuh4X+IafLA6/bm34pnrsDxTJcStiMsIOmLtaBERjc95i+CLzI4xd2b00QxO3 EeJNL9HY91VcylN3mGRxZyIbfpq5Afg9XVD/0RobwJHjTIUFJHRaEE/f3tJHJA7Kq/evbk+GsFej0 yH7sHd3pw1/E0VfOqyag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwR-008h5o-2j; Wed, 19 Jul 2023 19:34:31 +0000 Received: from mail-io1-xd2e.google.com ([2607:f8b0:4864:20::d2e]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwO-008h48-34 for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:30 +0000 Received: by mail-io1-xd2e.google.com with SMTP id ca18e2360f4ac-78372b895d6so391519439f.2 for ; Wed, 19 Jul 2023 12:34:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795267; x=1690400067; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FdGmPDy4n8Vb9yckrY3NHJ0yplI6UyLJ02ulp/zmXZw=; b=HecAfKZg9Xu/PFX1+zVYZYlqqow6E/dqLCp2s59n3O/xfMNhZ1IcSBmsZ+yJK5yNZL N5QknaUrOuOpLSTFmj+MWb/R4/rP7abzfMWRnDK/dpQ/I6huV/G8S6SoVAYQf3jyjBYc ITXB1yxmxSgIKH+s1sMPFwsyhlwsCD3w0so/PM12hwEOfcikG3bfcZdbu0ypKBEuN/gu VP71K10oPMLxWNldnN5H5JKrGbSGgWpQLr4SMMleYtLvyes4e4RlNw7G1q4GorTf0DJt gAqzhs9DEql29OqfU1pSrU/R0JUUn/T1cffMhVSuj+e5CHeakg/+oKqiTz3fVOG0oVX+ 1rXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795267; x=1690400067; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FdGmPDy4n8Vb9yckrY3NHJ0yplI6UyLJ02ulp/zmXZw=; b=eLlfdOagTDfAuUQG+u9xgyKsOYt59jhXwUAY95BFHoF+1F5K2/JSL4/0cod4GYj/2X vIFUtnyjfw+wpp8RoWFBTJrW8bJOejtvZ1WOgHyqDqFViqayiN3Lsvo79BDJISJhPAnv 0zBy3smKa5tqT+Lb+fgxw6Mes7SOnS6qAcos13XEHPB4wNRaSyI4Bu8cqsAu4w3OhMeK v+gKElqOx2K5Y1orQe1tuvUgSTtB5qkRj3tMyYFtaXrwfqrFYPcGHXv8HLFhk6C7OH7c NzVMjaANlPaxxeA2H9H6fqVt6D6Bp2gpDc1WTH37/JQg8RqMCar/JvY2YPiEf/UajWf4 gosw== X-Gm-Message-State: ABy/qLZfQjXO9coX1TieLvAIB7jD4GaLC/qKm1f93DjNZryUQ6vjmave lO/xr1O78g1u4RfG/2/dYlalbg== X-Google-Smtp-Source: APBJJlGN7srrpW1Dfo4Ygw07TXS+Dfssh8mV+rp8AEjMaeoQuXFsWsSaJe/QFh3kOeDRzzfvovNgSA== X-Received: by 2002:a05:6e02:12ed:b0:348:936b:ae12 with SMTP id l13-20020a056e0212ed00b00348936bae12mr3737637iln.11.1689795267467; Wed, 19 Jul 2023 12:34:27 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:27 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 03/11] dt-bindings: Add RISC-V IOMMU bindings Date: Wed, 19 Jul 2023 12:33:47 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123429_019016_E137ECA8 X-CRM114-Status: GOOD ( 18.24 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Anup Patel We add DT bindings document for RISC-V IOMMU platform and PCI devices defined by the RISC-V IOMMU specification. Signed-off-by: Anup Patel --- .../bindings/iommu/riscv,iommu.yaml | 146 ++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 Documentation/devicetree/bindings/iommu/riscv,iommu.yaml diff --git a/Documentation/devicetree/bindings/iommu/riscv,iommu.yaml b/Documentation/devicetree/bindings/iommu/riscv,iommu.yaml new file mode 100644 index 000000000000..8a9aedb61768 --- /dev/null +++ b/Documentation/devicetree/bindings/iommu/riscv,iommu.yaml @@ -0,0 +1,146 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/iommu/riscv,iommu.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: RISC-V IOMMU Implementation + +maintainers: + - Tomasz Jeznach + +description: + The RISC-V IOMMU specificaiton defines an IOMMU for RISC-V platforms + which can be a regular platform device or a PCI device connected to + the host root port. + + The RISC-V IOMMU provides two stage translation, device directory table, + command queue and fault reporting as wired interrupt or MSIx event for + both PCI and platform devices. + + Visit https://github.com/riscv-non-isa/riscv-iommu for more details. + +properties: + compatible: + oneOf: + - description: RISC-V IOMMU as a platform device + items: + - enum: + - vendor,chip-iommu + - const: riscv,iommu + + - description: RISC-V IOMMU as a PCI device connected to root port + items: + - enum: + - vendor,chip-pci-iommu + - const: riscv,pci-iommu + + reg: + maxItems: 1 + description: + For RISC-V IOMMU as a platform device, this represents the MMIO base + address of registers. + + For RISC-V IOMMU as a PCI device, this represents the PCI-PCI bridge + details as described in Documentation/devicetree/bindings/pci/pci.txt + + '#iommu-cells': + const: 2 + description: | + Each IOMMU specifier represents the base device ID and number of + device IDs. + + interrupts: + minItems: 1 + maxItems: 16 + description: + The presence of this property implies that given RISC-V IOMMU uses + wired interrupts to notify the RISC-V HARTS (or CPUs). + + msi-parent: + description: + The presence of this property implies that given RISC-V IOMMU uses + MSIx to notify the RISC-V HARTs (or CPUs). This property should be + considered only when the interrupts property is absent. + + dma-coherent: + description: + Present if page table walks and DMA accessed made by the RISC-V IOMMU + are cache coherent with the CPU. + + power-domains: + maxItems: 1 + +required: + - compatible + - reg + - '#iommu-cells' + +additionalProperties: false + +examples: + - | + /* Example 1 (IOMMU platform device with wired interrupts) */ + immu1: iommu@1bccd000 { + compatible = "vendor,chip-iommu", "riscv,iommu"; + reg = <0x1bccd000 0x1000>; + interrupt-parent = <&aplic_smode>; + interrupts = <32 4>, <33 4>, <34 4>, <35 4>; + #iommu-cells = <2>; + }; + + /* Device with two IOMMU device IDs, 0 and 7 */ + master1 { + iommus = <&immu1 0 1>, <&immu1 7 1>; + }; + + - | + /* Example 2 (IOMMU platform device with MSIs) */ + immu2: iommu@1bcdd000 { + compatible = "vendor,chip-iommu", "riscv,iommu"; + reg = <0x1bccd000 0x1000>; + msi-parent = <&imsics_smode>; + #iommu-cells = <2>; + }; + + bus { + #address-cells = <2>; + #size-cells = <2>; + + /* Device with IOMMU device IDs ranging from 32 to 64 */ + master1 { + iommus = <&immu2 32 32>; + }; + + pcie@40000000 { + compatible = "pci-host-cam-generic"; + device_type = "pci"; + #address-cells = <3>; + #size-cells = <2>; + bus-range = <0x0 0x1>; + + /* CPU_PHYSICAL(2) SIZE(2) */ + reg = <0x0 0x40000000 0x0 0x1000000>; + + /* BUS_ADDRESS(3) CPU_PHYSICAL(2) SIZE(2) */ + ranges = <0x01000000 0x0 0x01000000 0x0 0x01000000 0x0 0x00010000>, + <0x02000000 0x0 0x41000000 0x0 0x41000000 0x0 0x3f000000>; + + #interrupt-cells = <0x1>; + + /* PCI_DEVICE(3) INT#(1) CONTROLLER(PHANDLE) CONTROLLER_DATA(2) */ + interrupt-map = < 0x0 0x0 0x0 0x1 &aplic_smode 0x4 0x1>, + < 0x800 0x0 0x0 0x1 &aplic_smode 0x5 0x1>, + <0x1000 0x0 0x0 0x1 &aplic_smode 0x6 0x1>, + <0x1800 0x0 0x0 0x1 &aplic_smode 0x7 0x1>; + + /* PCI_DEVICE(3) INT#(1) */ + interrupt-map-mask = <0xf800 0x0 0x0 0x7>; + + msi-parent = <&imsics_smode>; + + /* Devices with bus number 0-127 are mastered via immu2 */ + iommu-map = <0x0000 &immu2 0x0000 0x8000>; + }; + }; +... From patchwork Wed Jul 19 19:33:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A40FAC001DE for ; Wed, 19 Jul 2023 19:34:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UD3d7EtnEPkbfrUKfXGdxlRNbJZ+vhW6mOV4xWVYuek=; b=q0l8TGDx3KHpeI E2Aji9axcuMTo14ZLfxY6xrdn3KmeJmpwg3x1GaL+WeuCi4nocztQRDUI/1mzsjGTPW8pLZZ4M3s/ 2K1wnuaZcJaQR3KoWoaaQoDJJgfa5GRI9BgRKb3Vmio+ofJOv8PRZ6ervpYZrhJjzmL9Q5Vz7BkjJ D1iUWVwaMWJeLnkJ2TCcMKSRzqqmnLEiPVl1YwmyIiyI+8SBoqQVs9dyqoIFqGFi2waetSMIkKnf5 btued2GBKQfq/LGKucGszCC08a9Ww+SnlzzKGrpM8jaXjUA4Ph6zQ8CkddUhVmobHkMygBGVTvcO2 Smqve2gUtJmEVddm73BQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwU-008h7a-1q; Wed, 19 Jul 2023 19:34:34 +0000 Received: from mail-il1-x134.google.com ([2607:f8b0:4864:20::134]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwQ-008h4U-02 for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:31 +0000 Received: by mail-il1-x134.google.com with SMTP id e9e14a558f8ab-3466725f0beso548745ab.1 for ; Wed, 19 Jul 2023 12:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795268; x=1690400068; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BsGoT51dWJmzQKPxb3tGZnxKgH62aNnVUMM3GK3B70I=; b=25R+yCvMKbfvQR87JPugzrtI45C63mU3sviX8G0Vy3Wvo5ueLJ6LoiuXaMFt1wKE/9 J2h7M7wC9mw8pp112AUlWRQnBXuQYeNqWCUrHT4s/bSKKKVibQt5pMn4JpFlbK212NTU Sh9aDJz8I0G32u8J5s8SbpYs7u5AYuv0KdPzsQzp4A4tsU6FfR5KRmDtU19hydphE8fS Dvwd2miaFaVXcHQq7Pao/6dE0JkI4ofWpL3JzgHPGwqTAgtKWllYKsvDJSJBHZ4k4ptP KcOM7qqQwWLW6QhyEFO3/HD64siCzcAvSJ8/yq3MiSdTk8+aGW6aGukJ4KThrK/kYnKa PovA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795268; x=1690400068; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BsGoT51dWJmzQKPxb3tGZnxKgH62aNnVUMM3GK3B70I=; b=N9KVQ+TirieHWBSHQaJGM2wwcXO/v+IOb0fh517Fwy9B9AlvyYIVc5xGikm1Jj3dRU vC2nymg04+nekEag8SvARy8aN12b9VQr5bJ6N7mupAZV3GpNFmmILJFXdm2UrL7F6D+p weYqXZO4bbWMBOx60BsNzXzI+3lZsRTYtuNKvrxo9CCAQYeTrJuIOTYS8FOafgNKbLad 44VBURNJcgC7Er3CUTh+ECzGEC/P6OEtbxM4lnyJCyXqfypy7uzSiJe04RylEN7YoZSe hjonCXsjrTNxcxNIyPQV32uzM9vC1+SsICcBaFOOm7L+5IYiZnQHE9rlyrJT3Ea8SqYv iPdQ== X-Gm-Message-State: ABy/qLZomngTdfTMM2FGOlQ491DS4PnBlmsJRhAviFjAeiTFrJUZ4UYw KOdaONSE2vI0k6BjFOQ8X1Cgqg== X-Google-Smtp-Source: APBJJlG+MFbJEKkJ1B9LQsQ5MgT1jSSRYuMm6X1iFGTwoFDCvk/QYGkIlda8SXG94SgiZ9wVyGnQjg== X-Received: by 2002:a05:6e02:1c49:b0:346:e850:6cc8 with SMTP id d9-20020a056e021c4900b00346e8506cc8mr3677361ilg.9.1689795268696; Wed, 19 Jul 2023 12:34:28 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:28 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 04/11] MAINTAINERS: Add myself for RISC-V IOMMU driver Date: Wed, 19 Jul 2023 12:33:48 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123430_057117_0C594C13 X-CRM114-Status: UNSURE ( 8.25 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Signed-off-by: Tomasz Jeznach --- MAINTAINERS | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index aee340630eca..d28b1b99f4c6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -18270,6 +18270,13 @@ F: arch/riscv/ N: riscv K: riscv +RISC-V IOMMU +M: Tomasz Jeznach +L: linux-riscv@lists.infradead.org +S: Maintained +F: Documentation/devicetree/bindings/iommu/riscv,iommu.yaml +F: drivers/iommu/riscv/ + RISC-V MICROCHIP FPGA SUPPORT M: Conor Dooley M: Daire McNamara From patchwork Wed Jul 19 19:33:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CFABBC001E0 for ; Wed, 19 Jul 2023 19:34:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Kvn+qvm+cw20i9IPlhIv+bcwMY1ovbcA+/ElNfN6kVE=; b=GFuMsSZEe5PBGA kbMZmXY00N8d9D4noGxbGQGK5FZT36Dmt5+mf3ri9zXKls+N59qylG2R9KkaWBavXcoPXA7ps5o5Z 2PjQqrzck64/q72VggkzlDtQ+2Wc4BV/t7o/xwPjQjf4/vOdd6b7wlFUkDGJcihZ2VqmUANdnCfra UtJHhBifQ+XULp4nMHoXtkCqQpgpGfxAvphknnboIYHQgAmmR8hvRtw58pRBRZhBBfy/4A92VIr09 G5QUToOE+w14ckHqYZra8t7lSk/F8vPff+2bUuJ2Uo9EWkaJhF2aTPjlHdRYWBK34y0bV+949w4k4 Gpnym3PubMgaZxAJjB/g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwX-008h9R-01; Wed, 19 Jul 2023 19:34:37 +0000 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwR-008h5H-0N for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:33 +0000 Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-55b0e7efb1cso4219726a12.1 for ; Wed, 19 Jul 2023 12:34:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795270; x=1690400070; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kazktXpEq6ieqwYBLKPipn4wLh0FPr3HRxxmkK9WPZw=; b=UveCNGN/BmDS/dpiK9E9WFCdSGeEIQLJPq+TntSzb0OVJUKw8457QVlJpeuQBkGvPA RRHvJ5DPD+YjGNmWNyhCdSuuVl3VMSI9ofTQbKQwSNcK0e0Vg9o0L2GIs2aMxrWtZOox kU5On+QttGg8berHD3o13+eKyaPVp79IJuAcTNbIymf/uJKx8aPzDCZvoII2/6G6/+ck D1BftMOLEl51OKc99Xxz5oRFJQFX2KzO62i0MeapKueO7YOgnefuGlVIlaTcZHcaIQfM gmdcVwEUQKTJx3z1JyyRfqHKURuY6kO2Aj2t4Tqhfy7BXNJZtPpPHZ3GxSNpwxmRozHd nqxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795270; x=1690400070; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kazktXpEq6ieqwYBLKPipn4wLh0FPr3HRxxmkK9WPZw=; b=g+/vq016vz4yU84Zh7DDSdWUd24Leg6anWsOMkgh03vCHC8s0utu8kwr3cxd0E5xoE 0JHvrlPclQ1WzrH57nuIl/gQYF8pOfclfgBSNLbgZbkqVzxt+rNYJ5ZgSP5QrvlgFt3U BtGFvmWYUlGOPRtKF1s5W2i7GzoMsyp09msQmyrlhcoJO3XR4I2n/SA4isJlI/lp4BsT XOHzpcM158Xfxq9EgYNgaVvdD9+ZG8AU/n1n3FxMjllGXoR/IQeQB7XjAVnRlHLLW4hl XwLd4yAwSlzcGPd+1fEv6Z87cADREIQgBoA0qD+pa8Hes2BE9Nj5VzIym786EGbwOPyy jx1Q== X-Gm-Message-State: ABy/qLZL7PjrcdaglPYoX1kbDxtLYjHu1UiSC9C0qw8DcQdWGoYCRDVN /5HiPKpXLww3lZyNWU72XvPg8g== X-Google-Smtp-Source: APBJJlE778Ah24ZWQjDXEFU6ITYD0j+QG6da7Iiv5Pd0d2PSojZrjWDn9oS4U2WcdSjIz8eWHa04MA== X-Received: by 2002:a17:90a:901:b0:263:fc43:5f39 with SMTP id n1-20020a17090a090100b00263fc435f39mr169347pjn.13.1689795269929; Wed, 19 Jul 2023 12:34:29 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:29 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 05/11] RISC-V: drivers/iommu/riscv: Add sysfs interface Date: Wed, 19 Jul 2023 12:33:49 -0700 Message-Id: <610abe685f90870be52bc7c2ca45ab5235bd8eb4.1689792825.git.tjeznach@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123431_180699_4F8C0A0A X-CRM114-Status: GOOD ( 16.50 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Enable sysfs debug / visibility interface providing restricted access to hardware registers. Signed-off-by: Tomasz Jeznach --- drivers/iommu/riscv/Makefile | 2 +- drivers/iommu/riscv/iommu-sysfs.c | 183 ++++++++++++++++++++++++++++++ drivers/iommu/riscv/iommu.c | 7 ++ drivers/iommu/riscv/iommu.h | 2 + 4 files changed, 193 insertions(+), 1 deletion(-) create mode 100644 drivers/iommu/riscv/iommu-sysfs.c diff --git a/drivers/iommu/riscv/Makefile b/drivers/iommu/riscv/Makefile index 38730c11e4a8..9523eb053cfc 100644 --- a/drivers/iommu/riscv/Makefile +++ b/drivers/iommu/riscv/Makefile @@ -1 +1 @@ -obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-pci.o iommu-platform.o \ No newline at end of file +obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-pci.o iommu-platform.o iommu-sysfs.o \ No newline at end of file diff --git a/drivers/iommu/riscv/iommu-sysfs.c b/drivers/iommu/riscv/iommu-sysfs.c new file mode 100644 index 000000000000..f038ea8445c5 --- /dev/null +++ b/drivers/iommu/riscv/iommu-sysfs.c @@ -0,0 +1,183 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * IOMMU API for RISC-V architected Ziommu implementations. + * + * Copyright © 2022-2023 Rivos Inc. + * + * Author: Tomasz Jeznach + */ + +#include +#include +#include +#include +#include +#include + +#include "iommu.h" + +#define sysfs_dev_to_iommu(dev) \ + container_of(dev_get_drvdata(dev), struct riscv_iommu_device, iommu) + +static ssize_t address_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct riscv_iommu_device *iommu = sysfs_dev_to_iommu(dev); + return sprintf(buf, "%llx\n", iommu->reg_phys); +} + +static DEVICE_ATTR_RO(address); + +#define ATTR_RD_REG32(name, offset) \ + ssize_t reg_ ## name ## _show(struct device *dev, \ + struct device_attribute *attr, char *buf) \ +{ \ + struct riscv_iommu_device *iommu = sysfs_dev_to_iommu(dev); \ + return sprintf(buf, "0x%x\n", \ + riscv_iommu_readl(iommu, offset)); \ +} + +#define ATTR_RD_REG64(name, offset) \ + ssize_t reg_ ## name ## _show(struct device *dev, \ + struct device_attribute *attr, char *buf) \ +{ \ + struct riscv_iommu_device *iommu = sysfs_dev_to_iommu(dev); \ + return sprintf(buf, "0x%llx\n", \ + riscv_iommu_readq(iommu, offset)); \ +} + +#define ATTR_WR_REG32(name, offset) \ + ssize_t reg_ ## name ## _store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ +{ \ + struct riscv_iommu_device *iommu = sysfs_dev_to_iommu(dev); \ + unsigned long val; \ + int ret; \ + ret = kstrtoul(buf, 0, &val); \ + if (ret) \ + return ret; \ + riscv_iommu_writel(iommu, offset, val); \ + return len; \ +} + +#define ATTR_WR_REG64(name, offset) \ + ssize_t reg_ ## name ## _store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ +{ \ + struct riscv_iommu_device *iommu = sysfs_dev_to_iommu(dev); \ + unsigned long long val; \ + int ret; \ + ret = kstrtoull(buf, 0, &val); \ + if (ret) \ + return ret; \ + riscv_iommu_writeq(iommu, offset, val); \ + return len; \ +} + +#define ATTR_RO_REG32(name, offset) \ +static ATTR_RD_REG32(name, offset); \ +static DEVICE_ATTR_RO(reg_ ## name) + +#define ATTR_RW_REG32(name, offset) \ +static ATTR_RD_REG32(name, offset); \ +static ATTR_WR_REG32(name, offset); \ +static DEVICE_ATTR_RW(reg_ ## name) + +#define ATTR_RO_REG64(name, offset) \ +static ATTR_RD_REG64(name, offset); \ +static DEVICE_ATTR_RO(reg_ ## name) + +#define ATTR_RW_REG64(name, offset) \ +static ATTR_RD_REG64(name, offset); \ +static ATTR_WR_REG64(name, offset); \ +static DEVICE_ATTR_RW(reg_ ## name) + +ATTR_RO_REG64(cap, RISCV_IOMMU_REG_CAP); +ATTR_RO_REG64(fctl, RISCV_IOMMU_REG_FCTL); +ATTR_RO_REG32(cqh, RISCV_IOMMU_REG_CQH); +ATTR_RO_REG32(cqt, RISCV_IOMMU_REG_CQT); +ATTR_RO_REG32(cqcsr, RISCV_IOMMU_REG_CQCSR); +ATTR_RO_REG32(fqh, RISCV_IOMMU_REG_FQH); +ATTR_RO_REG32(fqt, RISCV_IOMMU_REG_FQT); +ATTR_RO_REG32(fqcsr, RISCV_IOMMU_REG_FQCSR); +ATTR_RO_REG32(pqh, RISCV_IOMMU_REG_PQH); +ATTR_RO_REG32(pqt, RISCV_IOMMU_REG_PQT); +ATTR_RO_REG32(pqcsr, RISCV_IOMMU_REG_PQCSR); +ATTR_RO_REG32(ipsr, RISCV_IOMMU_REG_IPSR); +ATTR_RO_REG32(ivec, RISCV_IOMMU_REG_IVEC); +ATTR_RW_REG64(tr_iova, RISCV_IOMMU_REG_TR_REQ_IOVA); +ATTR_RW_REG64(tr_ctrl, RISCV_IOMMU_REG_TR_REQ_CTL); +ATTR_RW_REG64(tr_response, RISCV_IOMMU_REG_TR_RESPONSE); +ATTR_RW_REG32(iocntovf, RISCV_IOMMU_REG_IOCOUNTOVF); +ATTR_RW_REG32(iocntinh, RISCV_IOMMU_REG_IOCOUNTINH); +ATTR_RW_REG64(iohpmcycles, RISCV_IOMMU_REG_IOHPMCYCLES); +ATTR_RW_REG64(iohpmevt_1, RISCV_IOMMU_REG_IOHPMEVT(0)); +ATTR_RW_REG64(iohpmevt_2, RISCV_IOMMU_REG_IOHPMEVT(1)); +ATTR_RW_REG64(iohpmevt_3, RISCV_IOMMU_REG_IOHPMEVT(2)); +ATTR_RW_REG64(iohpmevt_4, RISCV_IOMMU_REG_IOHPMEVT(3)); +ATTR_RW_REG64(iohpmevt_5, RISCV_IOMMU_REG_IOHPMEVT(4)); +ATTR_RW_REG64(iohpmevt_6, RISCV_IOMMU_REG_IOHPMEVT(5)); +ATTR_RW_REG64(iohpmevt_7, RISCV_IOMMU_REG_IOHPMEVT(6)); +ATTR_RW_REG64(iohpmctr_1, RISCV_IOMMU_REG_IOHPMCTR(0)); +ATTR_RW_REG64(iohpmctr_2, RISCV_IOMMU_REG_IOHPMCTR(1)); +ATTR_RW_REG64(iohpmctr_3, RISCV_IOMMU_REG_IOHPMCTR(2)); +ATTR_RW_REG64(iohpmctr_4, RISCV_IOMMU_REG_IOHPMCTR(3)); +ATTR_RW_REG64(iohpmctr_5, RISCV_IOMMU_REG_IOHPMCTR(4)); +ATTR_RW_REG64(iohpmctr_6, RISCV_IOMMU_REG_IOHPMCTR(5)); +ATTR_RW_REG64(iohpmctr_7, RISCV_IOMMU_REG_IOHPMCTR(6)); + +static struct attribute *riscv_iommu_attrs[] = { + &dev_attr_address.attr, + &dev_attr_reg_cap.attr, + &dev_attr_reg_fctl.attr, + &dev_attr_reg_cqh.attr, + &dev_attr_reg_cqt.attr, + &dev_attr_reg_cqcsr.attr, + &dev_attr_reg_fqh.attr, + &dev_attr_reg_fqt.attr, + &dev_attr_reg_fqcsr.attr, + &dev_attr_reg_pqh.attr, + &dev_attr_reg_pqt.attr, + &dev_attr_reg_pqcsr.attr, + &dev_attr_reg_ipsr.attr, + &dev_attr_reg_ivec.attr, + &dev_attr_reg_tr_iova.attr, + &dev_attr_reg_tr_ctrl.attr, + &dev_attr_reg_tr_response.attr, + &dev_attr_reg_iocntovf.attr, + &dev_attr_reg_iocntinh.attr, + &dev_attr_reg_iohpmcycles.attr, + &dev_attr_reg_iohpmctr_1.attr, + &dev_attr_reg_iohpmevt_1.attr, + &dev_attr_reg_iohpmctr_2.attr, + &dev_attr_reg_iohpmevt_2.attr, + &dev_attr_reg_iohpmctr_3.attr, + &dev_attr_reg_iohpmevt_3.attr, + &dev_attr_reg_iohpmctr_4.attr, + &dev_attr_reg_iohpmevt_4.attr, + &dev_attr_reg_iohpmctr_5.attr, + &dev_attr_reg_iohpmevt_5.attr, + &dev_attr_reg_iohpmctr_6.attr, + &dev_attr_reg_iohpmevt_6.attr, + &dev_attr_reg_iohpmctr_7.attr, + &dev_attr_reg_iohpmevt_7.attr, + NULL, +}; + +static struct attribute_group riscv_iommu_group = { + .name = "riscv-iommu", + .attrs = riscv_iommu_attrs, +}; + +const struct attribute_group *riscv_iommu_groups[] = { + &riscv_iommu_group, + NULL, +}; + +int riscv_iommu_sysfs_add(struct riscv_iommu_device *iommu) { + return iommu_device_sysfs_add(&iommu->iommu, NULL, + riscv_iommu_groups, "riscv-iommu@%llx", iommu->reg_phys); +} + diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 8c236242e2cc..31dc3c458e13 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -608,6 +608,7 @@ static const struct iommu_ops riscv_iommu_ops = { void riscv_iommu_remove(struct riscv_iommu_device *iommu) { iommu_device_unregister(&iommu->iommu); + iommu_device_sysfs_remove(&iommu->iommu); riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_OFF); } @@ -646,6 +647,12 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) goto fail; } + ret = riscv_iommu_sysfs_add(iommu); + if (ret) { + dev_err(dev, "cannot register sysfs interface (%d)\n", ret); + goto fail; + } + ret = iommu_device_register(&iommu->iommu, &riscv_iommu_ops, dev); if (ret) { dev_err(dev, "cannot register iommu interface (%d)\n", ret); diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 7baefd3630b3..7dc9baa59a50 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -112,4 +112,6 @@ static inline void riscv_iommu_writeq(struct riscv_iommu_device *iommu, int riscv_iommu_init(struct riscv_iommu_device *iommu); void riscv_iommu_remove(struct riscv_iommu_device *iommu); +int riscv_iommu_sysfs_add(struct riscv_iommu_device *iommu); + #endif From patchwork Wed Jul 19 19:33:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C00D0C001B0 for ; Wed, 19 Jul 2023 19:34:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wa9NTR5mwq5q8LRFDuWTtNqIqwQP4mcxU15sZl+gh8Y=; b=GCy1tQ87tpOHG4 xKuNol/b5vSarN+DwCUymhsHBJbFafP9aDn28VuI3OmY+ljbhDsB20fn6w4vhG5KxCNx2E2XU+Pav hZdFMoqAAzK4YH/JfM2cTyR7n3s4gB2ewef4nwELJ9ZYx4UcTPXD0lmGLrfWDzedw4mhYONR12tiK WTdRtXOmCUze5VWPTZuy/r+KatRWhLNLH2/QPatFdzZjIe/xiknl7E/ITILMtkX2A6wTWmxMTnxme PsBcGrrSY+HuuDaKDbV5zshUfKjQ3ICDBJi3civt5Y6dvjuIHMpgXdr8CzauSWwOnhD+Yagj3ycft FPTcC4m+1w2nQoKVgq2A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwd-008hEN-2k; Wed, 19 Jul 2023 19:34:43 +0000 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwT-008h5n-08 for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:36 +0000 Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-557790487feso5518995a12.0 for ; Wed, 19 Jul 2023 12:34:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795271; x=1690400071; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XoFjJfAz3I5O9im0/Hv4bgNnfpgtHpDRgG5sHeV+rkY=; b=menGXr1L20hgVjukm6kJfV2kYWPxzyXWIjqYOpyOwtJsCx7tgEAT282qiKkFfTSZUV IZqbr3NEJ8oMbRazXGBgXLlLhMhI9oPhk359wklRwaBvGYkacApS744cTpm7Bq8fuZbj t7he6jgi+rcU1C6yCc8lxuw0fQFeBhC2v6GeuB0dCV/ZU7DOJ6VikLS3ADKutnXdZEqU 4fbgKNppYwZzEMY7ROCHCJ2U2i//hTBYyNJBwNlrTGU75p4RJmeiGvyLk0z7KZiAAsii 2cE9Rl5aIZ9ul3BHByBpBYZokJm10pT6ByI3JjxccAUoDRMpHAz76Ls2fCTsLD3lqJxN HQmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795271; x=1690400071; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XoFjJfAz3I5O9im0/Hv4bgNnfpgtHpDRgG5sHeV+rkY=; b=VRHlL94K5T8Y6k6CrC1p/qdKu6oY90sUMvqsaC5/cJuKIwEpiLT81glRi2v1NjHWBF fzlXvO5/xglmvw4FXIzGhs6Y+RlqIKsHXwVf/q/4CpN8BrIXd6Smu8t5j9n8+ot4MwFj Lt7JCtlRB3UxhCDnWWyf8fx1x7jjRdl+VcJKt/hVa5nAEVQHlw5nzvaQYBLURMhe18L5 DYshpPFghN3jYeB+QiUSprLI7ughcFWqpzzTsoB9RiSSQHaxdXnxf1PZeyB6vj4FyQqx SrO7IbwqyWWLzw6KJhbLs1BmB4foghBP1HMB0L+WN/8vV4g3hwi+EbIWt5bIL0F3xyEU kuCQ== X-Gm-Message-State: ABy/qLaXkTJhofUjFLTSNmPz8zTXGU5XzluTqoWulb26lLAfPUvjAwH6 IX2t6jm5LRhJAB0c7//vAo7HXQ== X-Google-Smtp-Source: APBJJlEArSDTyheryXvHOREVem7n8Y7EJuc7QR8Czd8lOHO6HKchLcwflBgO7Oix1Pe7e17OND59ZA== X-Received: by 2002:a17:90a:68c8:b0:261:2a59:dc38 with SMTP id q8-20020a17090a68c800b002612a59dc38mr147665pjj.25.1689795271180; Wed, 19 Jul 2023 12:34:31 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:30 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 06/11] RISC-V: drivers/iommu/riscv: Add command, fault, page-req queues Date: Wed, 19 Jul 2023 12:33:50 -0700 Message-Id: <1fd79e5c53d9d6ed2264f60dd4261f293cc00472.1689792825.git.tjeznach@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123433_123361_27549213 X-CRM114-Status: GOOD ( 26.75 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Enables message or wire signal interrupts for PCIe and platforms devices. Co-developed-by: Nick Kossifidis Signed-off-by: Nick Kossifidis Signed-off-by: Tomasz Jeznach --- drivers/iommu/riscv/iommu-pci.c | 72 ++++ drivers/iommu/riscv/iommu-platform.c | 66 +++ drivers/iommu/riscv/iommu.c | 604 ++++++++++++++++++++++++++- drivers/iommu/riscv/iommu.h | 28 ++ 4 files changed, 769 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/riscv/iommu-pci.c b/drivers/iommu/riscv/iommu-pci.c index c91f963d7a29..9ea0647f7b92 100644 --- a/drivers/iommu/riscv/iommu-pci.c +++ b/drivers/iommu/riscv/iommu-pci.c @@ -34,6 +34,7 @@ static int riscv_iommu_pci_probe(struct pci_dev *pdev, const struct pci_device_i { struct device *dev = &pdev->dev; struct riscv_iommu_device *iommu; + u64 icvec; int ret; ret = pci_enable_device_mem(pdev); @@ -67,14 +68,84 @@ static int riscv_iommu_pci_probe(struct pci_dev *pdev, const struct pci_device_i iommu->dev = dev; dev_set_drvdata(dev, iommu); + /* Check device reported capabilities. */ + iommu->cap = riscv_iommu_readq(iommu, RISCV_IOMMU_REG_CAP); + + /* The PCI driver only uses MSIs, make sure the IOMMU supports this */ + switch (FIELD_GET(RISCV_IOMMU_CAP_IGS, iommu->cap)) { + case RISCV_IOMMU_CAP_IGS_MSI: + case RISCV_IOMMU_CAP_IGS_BOTH: + break; + default: + dev_err(dev, "unable to use message-signaled interrupts\n"); + ret = -ENODEV; + goto fail; + } + dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); pci_set_master(pdev); + /* Allocate and assign IRQ vectors for the various events */ + ret = pci_alloc_irq_vectors(pdev, 1, RISCV_IOMMU_INTR_COUNT, PCI_IRQ_MSIX); + if (ret < 0) { + dev_err(dev, "unable to allocate irq vectors\n"); + goto fail; + } + + ret = -ENODEV; + + iommu->irq_cmdq = msi_get_virq(dev, RISCV_IOMMU_INTR_CQ); + if (!iommu->irq_cmdq) { + dev_warn(dev, "no MSI vector %d for the command queue\n", + RISCV_IOMMU_INTR_CQ); + goto fail; + } + + iommu->irq_fltq = msi_get_virq(dev, RISCV_IOMMU_INTR_FQ); + if (!iommu->irq_fltq) { + dev_warn(dev, "no MSI vector %d for the fault/event queue\n", + RISCV_IOMMU_INTR_FQ); + goto fail; + } + + if (iommu->cap & RISCV_IOMMU_CAP_HPM) { + iommu->irq_pm = msi_get_virq(dev, RISCV_IOMMU_INTR_PM); + if (!iommu->irq_pm) { + dev_warn(dev, + "no MSI vector %d for performance monitoring\n", + RISCV_IOMMU_INTR_PM); + goto fail; + } + } + + if (iommu->cap & RISCV_IOMMU_CAP_ATS) { + iommu->irq_priq = msi_get_virq(dev, RISCV_IOMMU_INTR_PQ); + if (!iommu->irq_priq) { + dev_warn(dev, + "no MSI vector %d for page-request queue\n", + RISCV_IOMMU_INTR_PQ); + goto fail; + } + } + + /* Set simple 1:1 mapping for MSI vectors */ + icvec = FIELD_PREP(RISCV_IOMMU_IVEC_CIV, RISCV_IOMMU_INTR_CQ) | + FIELD_PREP(RISCV_IOMMU_IVEC_FIV, RISCV_IOMMU_INTR_FQ); + + if (iommu->cap & RISCV_IOMMU_CAP_HPM) + icvec |= FIELD_PREP(RISCV_IOMMU_IVEC_PMIV, RISCV_IOMMU_INTR_PM); + + if (iommu->cap & RISCV_IOMMU_CAP_ATS) + icvec |= FIELD_PREP(RISCV_IOMMU_IVEC_PIV, RISCV_IOMMU_INTR_PQ); + + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IVEC, icvec); + ret = riscv_iommu_init(iommu); if (!ret) return ret; fail: + pci_free_irq_vectors(pdev); pci_clear_master(pdev); pci_release_regions(pdev); pci_disable_device(pdev); @@ -85,6 +156,7 @@ static int riscv_iommu_pci_probe(struct pci_dev *pdev, const struct pci_device_i static void riscv_iommu_pci_remove(struct pci_dev *pdev) { riscv_iommu_remove(dev_get_drvdata(&pdev->dev)); + pci_free_irq_vectors(pdev); pci_clear_master(pdev); pci_release_regions(pdev); pci_disable_device(pdev); diff --git a/drivers/iommu/riscv/iommu-platform.c b/drivers/iommu/riscv/iommu-platform.c index e4e8ca6711e7..35935d3c7ef4 100644 --- a/drivers/iommu/riscv/iommu-platform.c +++ b/drivers/iommu/riscv/iommu-platform.c @@ -20,6 +20,8 @@ static int riscv_iommu_platform_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct riscv_iommu_device *iommu = NULL; struct resource *res = NULL; + u32 fctl = 0; + int irq = 0; int ret = 0; iommu = devm_kzalloc(dev, sizeof(*iommu), GFP_KERNEL); @@ -53,6 +55,70 @@ static int riscv_iommu_platform_probe(struct platform_device *pdev) goto fail; } + iommu->cap = riscv_iommu_readq(iommu, RISCV_IOMMU_REG_CAP); + + /* For now we only support WSIs until we have AIA support */ + ret = FIELD_GET(RISCV_IOMMU_CAP_IGS, iommu->cap); + if (ret == RISCV_IOMMU_CAP_IGS_MSI) { + dev_err(dev, "IOMMU only supports MSIs\n"); + goto fail; + } + + /* Parse IRQ assignment */ + irq = platform_get_irq_byname_optional(pdev, "cmdq"); + if (irq > 0) + iommu->irq_cmdq = irq; + else { + dev_err(dev, "no IRQ provided for the command queue\n"); + goto fail; + } + + irq = platform_get_irq_byname_optional(pdev, "fltq"); + if (irq > 0) + iommu->irq_fltq = irq; + else { + dev_err(dev, "no IRQ provided for the fault/event queue\n"); + goto fail; + } + + if (iommu->cap & RISCV_IOMMU_CAP_HPM) { + irq = platform_get_irq_byname_optional(pdev, "pm"); + if (irq > 0) + iommu->irq_pm = irq; + else { + dev_err(dev, "no IRQ provided for performance monitoring\n"); + goto fail; + } + } + + if (iommu->cap & RISCV_IOMMU_CAP_ATS) { + irq = platform_get_irq_byname_optional(pdev, "priq"); + if (irq > 0) + iommu->irq_priq = irq; + else { + dev_err(dev, "no IRQ provided for the page-request queue\n"); + goto fail; + } + } + + /* Make sure fctl.WSI is set */ + fctl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FCTL); + fctl |= RISCV_IOMMU_FCTL_WSI; + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_FCTL, fctl); + + /* Parse Queue lengts */ + ret = of_property_read_u32(pdev->dev.of_node, "cmdq_len", &iommu->cmdq_len); + if (!ret) + dev_info(dev, "command queue length set to %i\n", iommu->cmdq_len); + + ret = of_property_read_u32(pdev->dev.of_node, "fltq_len", &iommu->fltq_len); + if (!ret) + dev_info(dev, "fault/event queue length set to %i\n", iommu->fltq_len); + + ret = of_property_read_u32(pdev->dev.of_node, "priq_len", &iommu->priq_len); + if (!ret) + dev_info(dev, "page request queue length set to %i\n", iommu->priq_len); + dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); return riscv_iommu_init(iommu); diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 31dc3c458e13..5c4cf9875302 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -45,6 +45,18 @@ static int ddt_mode = RISCV_IOMMU_DDTP_MODE_BARE; module_param(ddt_mode, int, 0644); MODULE_PARM_DESC(ddt_mode, "Device Directory Table mode."); +static int cmdq_length = 1024; +module_param(cmdq_length, int, 0644); +MODULE_PARM_DESC(cmdq_length, "Command queue length."); + +static int fltq_length = 1024; +module_param(fltq_length, int, 0644); +MODULE_PARM_DESC(fltq_length, "Fault queue length."); + +static int priq_length = 1024; +module_param(priq_length, int, 0644); +MODULE_PARM_DESC(priq_length, "Page request interface queue length."); + /* IOMMU PSCID allocation namespace. */ #define RISCV_IOMMU_MAX_PSCID (1U << 20) static DEFINE_IDA(riscv_iommu_pscids); @@ -65,6 +77,497 @@ static DEFINE_IDA(riscv_iommu_pscids); static const struct iommu_domain_ops riscv_iommu_domain_ops; static const struct iommu_ops riscv_iommu_ops; +/* + * Common queue management routines + */ + +/* Note: offsets are the same for all queues */ +#define Q_HEAD(q) ((q)->qbr + (RISCV_IOMMU_REG_CQH - RISCV_IOMMU_REG_CQB)) +#define Q_TAIL(q) ((q)->qbr + (RISCV_IOMMU_REG_CQT - RISCV_IOMMU_REG_CQB)) + +static unsigned riscv_iommu_queue_consume(struct riscv_iommu_device *iommu, + struct riscv_iommu_queue *q, unsigned *ready) +{ + u32 tail = riscv_iommu_readl(iommu, Q_TAIL(q)); + *ready = q->lui; + + BUG_ON(q->cnt <= tail); + if (q->lui <= tail) + return tail - q->lui; + return q->cnt - q->lui; +} + +static void riscv_iommu_queue_release(struct riscv_iommu_device *iommu, + struct riscv_iommu_queue *q, unsigned count) +{ + q->lui = (q->lui + count) & (q->cnt - 1); + riscv_iommu_writel(iommu, Q_HEAD(q), q->lui); +} + +static u32 riscv_iommu_queue_ctrl(struct riscv_iommu_device *iommu, + struct riscv_iommu_queue *q, u32 val) +{ + cycles_t end_cycles = RISCV_IOMMU_TIMEOUT + get_cycles(); + + riscv_iommu_writel(iommu, q->qcr, val); + do { + val = riscv_iommu_readl(iommu, q->qcr); + if (!(val & RISCV_IOMMU_QUEUE_BUSY)) + break; + cpu_relax(); + } while (get_cycles() < end_cycles); + + return val; +} + +static void riscv_iommu_queue_free(struct riscv_iommu_device *iommu, + struct riscv_iommu_queue *q) +{ + size_t size = q->len * q->cnt; + + riscv_iommu_queue_ctrl(iommu, q, 0); + + if (q->base) { + if (q->in_iomem) + iounmap(q->base); + else + dmam_free_coherent(iommu->dev, size, q->base, q->base_dma); + } + if (q->irq) + free_irq(q->irq, q); +} + +static irqreturn_t riscv_iommu_cmdq_irq_check(int irq, void *data); +static irqreturn_t riscv_iommu_cmdq_process(int irq, void *data); +static irqreturn_t riscv_iommu_fltq_irq_check(int irq, void *data); +static irqreturn_t riscv_iommu_fltq_process(int irq, void *data); +static irqreturn_t riscv_iommu_priq_irq_check(int irq, void *data); +static irqreturn_t riscv_iommu_priq_process(int irq, void *data); + +static int riscv_iommu_queue_init(struct riscv_iommu_device *iommu, int queue_id) +{ + struct device *dev = iommu->dev; + struct riscv_iommu_queue *q = NULL; + size_t queue_size = 0; + irq_handler_t irq_check; + irq_handler_t irq_process; + const char *name; + int count = 0; + int irq = 0; + unsigned order = 0; + u64 qbr_val = 0; + u64 qbr_readback = 0; + u64 qbr_paddr = 0; + int ret = 0; + + switch (queue_id) { + case RISCV_IOMMU_COMMAND_QUEUE: + q = &iommu->cmdq; + q->len = sizeof(struct riscv_iommu_command); + count = iommu->cmdq_len; + irq = iommu->irq_cmdq; + irq_check = riscv_iommu_cmdq_irq_check; + irq_process = riscv_iommu_cmdq_process; + q->qbr = RISCV_IOMMU_REG_CQB; + q->qcr = RISCV_IOMMU_REG_CQCSR; + name = "cmdq"; + break; + case RISCV_IOMMU_FAULT_QUEUE: + q = &iommu->fltq; + q->len = sizeof(struct riscv_iommu_fq_record); + count = iommu->fltq_len; + irq = iommu->irq_fltq; + irq_check = riscv_iommu_fltq_irq_check; + irq_process = riscv_iommu_fltq_process; + q->qbr = RISCV_IOMMU_REG_FQB; + q->qcr = RISCV_IOMMU_REG_FQCSR; + name = "fltq"; + break; + case RISCV_IOMMU_PAGE_REQUEST_QUEUE: + q = &iommu->priq; + q->len = sizeof(struct riscv_iommu_pq_record); + count = iommu->priq_len; + irq = iommu->irq_priq; + irq_check = riscv_iommu_priq_irq_check; + irq_process = riscv_iommu_priq_process; + q->qbr = RISCV_IOMMU_REG_PQB; + q->qcr = RISCV_IOMMU_REG_PQCSR; + name = "priq"; + break; + default: + dev_err(dev, "invalid queue interrupt index in queue_init!\n"); + return -EINVAL; + } + + /* Polling not implemented */ + if (!irq) + return -ENODEV; + + /* Allocate queue in memory and set the base register */ + order = ilog2(count); + do { + queue_size = q->len * (1ULL << order); + q->base = dmam_alloc_coherent(dev, queue_size, &q->base_dma, GFP_KERNEL); + if (q->base || queue_size < PAGE_SIZE) + break; + + order--; + } while (1); + + if (!q->base) { + dev_err(dev, "failed to allocate %s queue (cnt: %u)\n", name, count); + return -ENOMEM; + } + + q->cnt = 1ULL << order; + + qbr_val = phys_to_ppn(q->base_dma) | + FIELD_PREP(RISCV_IOMMU_QUEUE_LOGSZ_FIELD, order - 1); + + riscv_iommu_writeq(iommu, q->qbr, qbr_val); + + /* + * Queue base registers are WARL, so it's possible that whatever we wrote + * there was illegal/not supported by the hw in which case we need to make + * sure we set a supported PPN and/or queue size. + */ + qbr_readback = riscv_iommu_readq(iommu, q->qbr); + if (qbr_readback == qbr_val) + goto irq; + + dmam_free_coherent(dev, queue_size, q->base, q->base_dma); + + /* Get supported queue size */ + order = FIELD_GET(RISCV_IOMMU_QUEUE_LOGSZ_FIELD, qbr_readback) + 1; + q->cnt = 1ULL << order; + queue_size = q->len * q->cnt; + + /* + * In case we also failed to set PPN, it means the field is hardcoded and the + * queue resides in I/O memory instead, so get its physical address and + * ioremap it. + */ + qbr_paddr = ppn_to_phys(qbr_readback); + if (qbr_paddr != q->base_dma) { + dev_info(dev, + "hardcoded ppn in %s base register, using io memory for the queue\n", + name); + dev_info(dev, "queue length for %s set to %i\n", name, q->cnt); + q->in_iomem = true; + q->base = ioremap(qbr_paddr, queue_size); + if (!q->base) { + dev_err(dev, "failed to map %s queue (cnt: %u)\n", name, q->cnt); + return -ENOMEM; + } + q->base_dma = qbr_paddr; + } else { + /* + * We only failed to set the queue size, re-try to allocate memory with + * the queue size supported by the hw. + */ + dev_info(dev, "hardcoded queue size in %s base register\n", name); + dev_info(dev, "retrying with queue length: %i\n", q->cnt); + q->base = dmam_alloc_coherent(dev, queue_size, &q->base_dma, GFP_KERNEL); + if (!q->base) { + dev_err(dev, "failed to allocate %s queue (cnt: %u)\n", + name, q->cnt); + return -ENOMEM; + } + } + + qbr_val = phys_to_ppn(q->base_dma) | + FIELD_PREP(RISCV_IOMMU_QUEUE_LOGSZ_FIELD, order - 1); + riscv_iommu_writeq(iommu, q->qbr, qbr_val); + + /* Final check to make sure hw accepted our write */ + qbr_readback = riscv_iommu_readq(iommu, q->qbr); + if (qbr_readback != qbr_val) { + dev_err(dev, "failed to set base register for %s\n", name); + goto fail; + } + + irq: + if (request_threaded_irq(irq, irq_check, irq_process, IRQF_ONESHOT | IRQF_SHARED, + dev_name(dev), q)) { + dev_err(dev, "fail to request irq %d for %s\n", irq, name); + goto fail; + } + + q->irq = irq; + + /* Note: All RIO_xQ_EN/IE fields are in the same offsets */ + ret = + riscv_iommu_queue_ctrl(iommu, q, + RISCV_IOMMU_QUEUE_ENABLE | + RISCV_IOMMU_QUEUE_INTR_ENABLE); + if (ret & RISCV_IOMMU_QUEUE_BUSY) { + dev_err(dev, "%s init timeout\n", name); + ret = -EBUSY; + goto fail; + } + + return 0; + + fail: + riscv_iommu_queue_free(iommu, q); + return 0; +} + +/* + * I/O MMU Command queue chapter 3.1 + */ + +static inline void riscv_iommu_cmd_inval_vma(struct riscv_iommu_command *cmd) +{ + cmd->dword0 = + FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, + RISCV_IOMMU_CMD_IOTINVAL_OPCODE) | FIELD_PREP(RISCV_IOMMU_CMD_FUNC, + RISCV_IOMMU_CMD_IOTINVAL_FUNC_VMA); + cmd->dword1 = 0; +} + +static inline void riscv_iommu_cmd_inval_set_addr(struct riscv_iommu_command *cmd, + u64 addr) +{ + cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV; + cmd->dword1 = addr; +} + +static inline void riscv_iommu_cmd_inval_set_pscid(struct riscv_iommu_command *cmd, + unsigned pscid) +{ + cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_PSCID, pscid) | + RISCV_IOMMU_CMD_IOTINVAL_PSCV; +} + +static inline void riscv_iommu_cmd_inval_set_gscid(struct riscv_iommu_command *cmd, + unsigned gscid) +{ + cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_GSCID, gscid) | + RISCV_IOMMU_CMD_IOTINVAL_GV; +} + +static inline void riscv_iommu_cmd_iofence(struct riscv_iommu_command *cmd) +{ + cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IOFENCE_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IOFENCE_FUNC_C); + cmd->dword1 = 0; +} + +static inline void riscv_iommu_cmd_iofence_set_av(struct riscv_iommu_command *cmd, + u64 addr, u32 data) +{ + cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IOFENCE_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IOFENCE_FUNC_C) | + FIELD_PREP(RISCV_IOMMU_CMD_IOFENCE_DATA, data) | RISCV_IOMMU_CMD_IOFENCE_AV; + cmd->dword1 = (addr >> 2); +} + +static inline void riscv_iommu_cmd_iodir_inval_ddt(struct riscv_iommu_command *cmd) +{ + cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IODIR_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_DDT); + cmd->dword1 = 0; +} + +static inline void riscv_iommu_cmd_iodir_inval_pdt(struct riscv_iommu_command *cmd) +{ + cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_IODIR_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_IODIR_FUNC_INVAL_PDT); + cmd->dword1 = 0; +} + +static inline void riscv_iommu_cmd_iodir_set_did(struct riscv_iommu_command *cmd, + unsigned devid) +{ + cmd->dword0 |= + FIELD_PREP(RISCV_IOMMU_CMD_IODIR_DID, devid) | RISCV_IOMMU_CMD_IODIR_DV; +} + +/* TODO: Convert into lock-less MPSC implementation. */ +static bool riscv_iommu_post_sync(struct riscv_iommu_device *iommu, + struct riscv_iommu_command *cmd, bool sync) +{ + u32 head, tail, next, last; + unsigned long flags; + + spin_lock_irqsave(&iommu->cq_lock, flags); + head = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_CQH) & (iommu->cmdq.cnt - 1); + tail = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_CQT) & (iommu->cmdq.cnt - 1); + last = iommu->cmdq.lui; + if (tail != last) { + spin_unlock_irqrestore(&iommu->cq_lock, flags); + /* + * FIXME: This is a workaround for dropped MMIO writes/reads on QEMU platform. + * While debugging of the problem is still ongoing, this provides + * a simple impolementation of try-again policy. + * Will be changed to lock-less algorithm in the feature. + */ + dev_dbg(iommu->dev, "IOMMU CQT: %x != %x (1st)\n", last, tail); + spin_lock_irqsave(&iommu->cq_lock, flags); + tail = + riscv_iommu_readl(iommu, RISCV_IOMMU_REG_CQT) & (iommu->cmdq.cnt - 1); + last = iommu->cmdq.lui; + if (tail != last) { + spin_unlock_irqrestore(&iommu->cq_lock, flags); + dev_dbg(iommu->dev, "IOMMU CQT: %x != %x (2nd)\n", last, tail); + spin_lock_irqsave(&iommu->cq_lock, flags); + } + } + + next = (last + 1) & (iommu->cmdq.cnt - 1); + if (next != head) { + struct riscv_iommu_command *ptr = iommu->cmdq.base; + ptr[last] = *cmd; + wmb(); + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_CQT, next); + iommu->cmdq.lui = next; + } + + spin_unlock_irqrestore(&iommu->cq_lock, flags); + + if (sync && head != next) { + cycles_t start_time = get_cycles(); + while (1) { + last = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_CQH) & + (iommu->cmdq.cnt - 1); + if (head < next && last >= next) + break; + if (head > next && last < head && last >= next) + break; + if (RISCV_IOMMU_TIMEOUT < (get_cycles() - start_time)) { + dev_err(iommu->dev, "IOFENCE TIMEOUT\n"); + return false; + } + cpu_relax(); + } + } + + return next != head; +} + +static bool riscv_iommu_post(struct riscv_iommu_device *iommu, + struct riscv_iommu_command *cmd) +{ + return riscv_iommu_post_sync(iommu, cmd, false); +} + +static bool riscv_iommu_iofence_sync(struct riscv_iommu_device *iommu) +{ + struct riscv_iommu_command cmd; + riscv_iommu_cmd_iofence(&cmd); + return riscv_iommu_post_sync(iommu, &cmd, true); +} + +/* Command queue primary interrupt handler */ +static irqreturn_t riscv_iommu_cmdq_irq_check(int irq, void *data) +{ + struct riscv_iommu_queue *q = (struct riscv_iommu_queue *)data; + struct riscv_iommu_device *iommu = + container_of(q, struct riscv_iommu_device, cmdq); + u32 ipsr = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_IPSR); + if (ipsr & RISCV_IOMMU_IPSR_CIP) + return IRQ_WAKE_THREAD; + return IRQ_NONE; +} + +/* Command queue interrupt hanlder thread function */ +static irqreturn_t riscv_iommu_cmdq_process(int irq, void *data) +{ + struct riscv_iommu_queue *q = (struct riscv_iommu_queue *)data; + struct riscv_iommu_device *iommu; + unsigned ctrl; + + iommu = container_of(q, struct riscv_iommu_device, cmdq); + + /* Error reporting, clear error reports if any. */ + ctrl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_CQCSR); + if (ctrl & (RISCV_IOMMU_CQCSR_CQMF | + RISCV_IOMMU_CQCSR_CMD_TO | RISCV_IOMMU_CQCSR_CMD_ILL)) { + riscv_iommu_queue_ctrl(iommu, &iommu->cmdq, ctrl); + dev_warn_ratelimited(iommu->dev, + "Command queue error: fault: %d tout: %d err: %d\n", + !!(ctrl & RISCV_IOMMU_CQCSR_CQMF), + !!(ctrl & RISCV_IOMMU_CQCSR_CMD_TO), + !!(ctrl & RISCV_IOMMU_CQCSR_CMD_ILL)); + } + + /* Clear fault interrupt pending. */ + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, RISCV_IOMMU_IPSR_CIP); + + return IRQ_HANDLED; +} + +/* + * Fault/event queue, chapter 3.2 + */ + +static void riscv_iommu_fault_report(struct riscv_iommu_device *iommu, + struct riscv_iommu_fq_record *event) +{ + unsigned err, devid; + + err = FIELD_GET(RISCV_IOMMU_FQ_HDR_CAUSE, event->hdr); + devid = FIELD_GET(RISCV_IOMMU_FQ_HDR_DID, event->hdr); + + dev_warn_ratelimited(iommu->dev, + "Fault %d devid: %d" " iotval: %llx iotval2: %llx\n", err, + devid, event->iotval, event->iotval2); +} + +/* Fault/event queue primary interrupt handler */ +static irqreturn_t riscv_iommu_fltq_irq_check(int irq, void *data) +{ + struct riscv_iommu_queue *q = (struct riscv_iommu_queue *)data; + struct riscv_iommu_device *iommu = + container_of(q, struct riscv_iommu_device, fltq); + u32 ipsr = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_IPSR); + if (ipsr & RISCV_IOMMU_IPSR_FIP) + return IRQ_WAKE_THREAD; + return IRQ_NONE; +} + +/* Fault queue interrupt hanlder thread function */ +static irqreturn_t riscv_iommu_fltq_process(int irq, void *data) +{ + struct riscv_iommu_queue *q = (struct riscv_iommu_queue *)data; + struct riscv_iommu_device *iommu; + struct riscv_iommu_fq_record *events; + unsigned cnt, len, idx, ctrl; + + iommu = container_of(q, struct riscv_iommu_device, fltq); + events = (struct riscv_iommu_fq_record *)q->base; + + /* Error reporting, clear error reports if any. */ + ctrl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_FQCSR); + if (ctrl & (RISCV_IOMMU_FQCSR_FQMF | RISCV_IOMMU_FQCSR_FQOF)) { + riscv_iommu_queue_ctrl(iommu, &iommu->fltq, ctrl); + dev_warn_ratelimited(iommu->dev, + "Fault queue error: fault: %d full: %d\n", + !!(ctrl & RISCV_IOMMU_FQCSR_FQMF), + !!(ctrl & RISCV_IOMMU_FQCSR_FQOF)); + } + + /* Clear fault interrupt pending. */ + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, RISCV_IOMMU_IPSR_FIP); + + /* Report fault events. */ + do { + cnt = riscv_iommu_queue_consume(iommu, q, &idx); + if (!cnt) + break; + for (len = 0; len < cnt; idx++, len++) + riscv_iommu_fault_report(iommu, &events[idx]); + riscv_iommu_queue_release(iommu, q, cnt); + } while (1); + + return IRQ_HANDLED; +} + +/* + * Page request queue, chapter 3.3 + */ + /* * Register device for IOMMU tracking. */ @@ -97,6 +600,54 @@ static void riscv_iommu_add_device(struct riscv_iommu_device *iommu, struct devi mutex_unlock(&iommu->eps_mutex); } +/* Page request interface queue primary interrupt handler */ +static irqreturn_t riscv_iommu_priq_irq_check(int irq, void *data) +{ + struct riscv_iommu_queue *q = (struct riscv_iommu_queue *)data; + struct riscv_iommu_device *iommu = + container_of(q, struct riscv_iommu_device, priq); + u32 ipsr = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_IPSR); + if (ipsr & RISCV_IOMMU_IPSR_PIP) + return IRQ_WAKE_THREAD; + return IRQ_NONE; +} + +/* Page request interface queue interrupt hanlder thread function */ +static irqreturn_t riscv_iommu_priq_process(int irq, void *data) +{ + struct riscv_iommu_queue *q = (struct riscv_iommu_queue *)data; + struct riscv_iommu_device *iommu; + struct riscv_iommu_pq_record *requests; + unsigned cnt, idx, ctrl; + + iommu = container_of(q, struct riscv_iommu_device, priq); + requests = (struct riscv_iommu_pq_record *)q->base; + + /* Error reporting, clear error reports if any. */ + ctrl = riscv_iommu_readl(iommu, RISCV_IOMMU_REG_PQCSR); + if (ctrl & (RISCV_IOMMU_PQCSR_PQMF | RISCV_IOMMU_PQCSR_PQOF)) { + riscv_iommu_queue_ctrl(iommu, &iommu->priq, ctrl); + dev_warn_ratelimited(iommu->dev, + "Page request queue error: fault: %d full: %d\n", + !!(ctrl & RISCV_IOMMU_PQCSR_PQMF), + !!(ctrl & RISCV_IOMMU_PQCSR_PQOF)); + } + + /* Clear page request interrupt pending. */ + riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, RISCV_IOMMU_IPSR_PIP); + + /* Process page requests. */ + do { + cnt = riscv_iommu_queue_consume(iommu, q, &idx); + if (!cnt) + break; + dev_warn(iommu->dev, "unexpected %u page requests\n", cnt); + riscv_iommu_queue_release(iommu, q, cnt); + } while (1); + + return IRQ_HANDLED; +} + /* * Endpoint management */ @@ -350,7 +901,29 @@ static void riscv_iommu_flush_iotlb_range(struct iommu_domain *iommu_domain, unsigned long *start, unsigned long *end, size_t *pgsize) { - /* Command interface not implemented */ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + struct riscv_iommu_command cmd; + unsigned long iova; + + if (domain->mode == RISCV_IOMMU_DC_FSC_MODE_BARE) + return; + + /* Domain not attached to an IOMMU! */ + BUG_ON(!domain->iommu); + + riscv_iommu_cmd_inval_vma(&cmd); + riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid); + + if (start && end && pgsize) { + /* Cover only the range that is needed */ + for (iova = *start; iova <= *end; iova += *pgsize) { + riscv_iommu_cmd_inval_set_addr(&cmd, iova); + riscv_iommu_post(domain->iommu, &cmd); + } + } else { + riscv_iommu_post(domain->iommu, &cmd); + } + riscv_iommu_iofence_sync(domain->iommu); } static void riscv_iommu_flush_iotlb_all(struct iommu_domain *iommu_domain) @@ -610,6 +1183,9 @@ void riscv_iommu_remove(struct riscv_iommu_device *iommu) iommu_device_unregister(&iommu->iommu); iommu_device_sysfs_remove(&iommu->iommu); riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_OFF); + riscv_iommu_queue_free(iommu, &iommu->cmdq); + riscv_iommu_queue_free(iommu, &iommu->fltq); + riscv_iommu_queue_free(iommu, &iommu->priq); } int riscv_iommu_init(struct riscv_iommu_device *iommu) @@ -632,6 +1208,16 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) } #endif + /* + * Assign queue lengths from module parameters if not already + * set on the device tree. + */ + if (!iommu->cmdq_len) + iommu->cmdq_len = cmdq_length; + if (!iommu->fltq_len) + iommu->fltq_len = fltq_length; + if (!iommu->priq_len) + iommu->priq_len = priq_length; /* Clear any pending interrupt flag. */ riscv_iommu_writel(iommu, RISCV_IOMMU_REG_IPSR, RISCV_IOMMU_IPSR_CIP | @@ -639,7 +1225,20 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) RISCV_IOMMU_IPSR_PMIP | RISCV_IOMMU_IPSR_PIP); spin_lock_init(&iommu->cq_lock); mutex_init(&iommu->eps_mutex); + ret = riscv_iommu_queue_init(iommu, RISCV_IOMMU_COMMAND_QUEUE); + if (ret) + goto fail; + ret = riscv_iommu_queue_init(iommu, RISCV_IOMMU_FAULT_QUEUE); + if (ret) + goto fail; + if (!(iommu->cap & RISCV_IOMMU_CAP_ATS)) + goto no_ats; + + ret = riscv_iommu_queue_init(iommu, RISCV_IOMMU_PAGE_REQUEST_QUEUE); + if (ret) + goto fail; + no_ats: ret = riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_BARE); if (ret) { @@ -663,5 +1262,8 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) return 0; fail: riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_OFF); + riscv_iommu_queue_free(iommu, &iommu->priq); + riscv_iommu_queue_free(iommu, &iommu->fltq); + riscv_iommu_queue_free(iommu, &iommu->cmdq); return ret; } diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 7dc9baa59a50..04148a2a8ffd 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -28,6 +28,24 @@ #define IOMMU_PAGE_SIZE_1G BIT_ULL(30) #define IOMMU_PAGE_SIZE_512G BIT_ULL(39) +struct riscv_iommu_queue { + dma_addr_t base_dma; /* ring buffer bus address */ + void *base; /* ring buffer pointer */ + size_t len; /* single item length */ + u32 cnt; /* items count */ + u32 lui; /* last used index, consumer/producer share */ + unsigned qbr; /* queue base register offset */ + unsigned qcr; /* queue control and status register offset */ + int irq; /* registered interrupt number */ + bool in_iomem; /* indicates queue data are in I/O memory */ +}; + +enum riscv_queue_ids { + RISCV_IOMMU_COMMAND_QUEUE = 0, + RISCV_IOMMU_FAULT_QUEUE = 1, + RISCV_IOMMU_PAGE_REQUEST_QUEUE = 2 +}; + struct riscv_iommu_device { struct iommu_device iommu; /* iommu core interface */ struct device *dev; /* iommu hardware */ @@ -42,6 +60,11 @@ struct riscv_iommu_device { int irq_pm; int irq_priq; + /* Queue lengths */ + int cmdq_len; + int fltq_len; + int priq_len; + /* supported and enabled hardware capabilities */ u64 cap; @@ -53,6 +76,11 @@ struct riscv_iommu_device { unsigned ddt_mode; bool ddtp_in_iomem; + /* hardware queues */ + struct riscv_iommu_queue cmdq; + struct riscv_iommu_queue fltq; + struct riscv_iommu_queue priq; + /* Connected end-points */ struct rb_root eps; struct mutex eps_mutex; From patchwork Wed Jul 19 19:33:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E13D8C001DC for ; Wed, 19 Jul 2023 19:34:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=C/v+LNst2FS2JX+DkU75evRhJNoB29vtwLLRSJhwDHo=; b=HxbfVbn67pL0yH 3ujM7J6Kz1zUhUqeenJW6krjpv7icaxsl9LaZqCMkN5tBnWgpU3ehgzX8hrBKMx7XQM92wGvEsdvY Udwu3NMVuEXcvHNPHa7SqdmDjwPLsCTb1W17WjmUuAVjZuF5gx1aB/sRZQh39/ZbKnS1jPa98lWVa pbMpnBypFMMfUQrOlM8pCJxmZqn5GXBRb0nTT5EQDkzOtlQE+Kc4NYzsriH87Jru76Z0xbqy1AbJ5 GTz8C2E0qcEx2fSbXXBvadJ4kUXaoYyJU0czdM+kpqgudRnLYeqw1usriKsaycKT7+icNPmWz7g1S yAlJe3NpsoRp3X1vgxyg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwd-008hDO-03; Wed, 19 Jul 2023 19:34:43 +0000 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwT-008h6l-2m for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:35 +0000 Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-55b83ae9be2so5501346a12.2 for ; Wed, 19 Jul 2023 12:34:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795272; x=1690400072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5i4Hac8zNCva6KUF+QJNd5Gpa8QthSQdhYqfDg3A/Yw=; b=dQXEnazYB2vuwxM3Chz9h8ooEIEl1UekfaSytU+WdBmWuSL9V63l5RoT3DGf6pjwHl weGKiimrDhgSSn0fm13pJgSoFM5Rr+lBybpKdsxGNuLhzr4v0smdnfzN0Z0tvwCFQpyZ eLGmG/VSiiQyRofCAaXzqwSfS34WKPoX16qQgnezgzBpHIl/C5girTKlxssW25rnCAmo osvYK0DeFq9ywfVGbudJlnavXdQpRSMW5ahMxJfxNAUCHJSY+TjkL6KBKNGAnumhvx8G 9wPBFLPhIjl0gkzvfeWyvcyp0ews1NrpuCb4dzbvGp92JdrwsP4aULpZ0LTTGUGJVRRs aDnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795272; x=1690400072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5i4Hac8zNCva6KUF+QJNd5Gpa8QthSQdhYqfDg3A/Yw=; b=LUf59Xws92znI/yBCIv0Z7eVbHpSbSy2YkJMxBb1agppUab+Yzy7AJALmt1zm3Wd/l vxMLl52DXljMRqubxR0klojpBdAZqdu2HPQ1Pvu2vStn7GQtzaDGBu4XI8S/Y2s3x74M t85lZjL4G3HJD/S4Hak9u5MDywIA+Ikcs9TkBSDGn7FHQ03gXxogJDnHXtOlhQLSI8pd ntcNfKwZloB3BSw1NRWYhwx6aNlM3ZGwUL91hR5/6fIzL+Rt52U4RrrJtFD84SHjsxMP yL1kWF+bP/zZkmQbg+qsjggGUw3+/+r796KdY9t1115hgKVulup6edDV23efZHJC9SLT 8qBw== X-Gm-Message-State: ABy/qLZX5p7MjYL8MvXrhOgoSlQq+AqGNZqOUb3we0X+HzPJZ/F2zSS/ lJcJ9wh40dAknPu3dOse9RWDEQ== X-Google-Smtp-Source: APBJJlF3WkEbRRR9gTWmIaq5El2jlzM9pGhayk4YtU6NIiU4+Fk9kbGI6fRUQkRmlf3TAy8N8NPqjQ== X-Received: by 2002:a17:90a:6709:b0:24e:4b1c:74d2 with SMTP id n9-20020a17090a670900b0024e4b1c74d2mr113588pjj.32.1689795272577; Wed, 19 Jul 2023 12:34:32 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:32 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 07/11] RISC-V: drivers/iommu/riscv: Add device context support Date: Wed, 19 Jul 2023 12:33:51 -0700 Message-Id: <702670f5be2b641a231f2dda84be12024afd2002.1689792825.git.tjeznach@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123433_900409_74434D80 X-CRM114-Status: GOOD ( 25.03 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Introduces per device translation context, with 1,2 or 3 tree level device tree structures. Signed-off-by: Tomasz Jeznach --- drivers/iommu/riscv/iommu.c | 163 ++++++++++++++++++++++++++++++++++-- drivers/iommu/riscv/iommu.h | 1 + 2 files changed, 158 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 5c4cf9875302..9ee7d2b222b5 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -41,7 +41,7 @@ MODULE_ALIAS("riscv-iommu"); MODULE_LICENSE("GPL v2"); /* Global IOMMU params. */ -static int ddt_mode = RISCV_IOMMU_DDTP_MODE_BARE; +static int ddt_mode = RISCV_IOMMU_DDTP_MODE_3LVL; module_param(ddt_mode, int, 0644); MODULE_PARM_DESC(ddt_mode, "Device Directory Table mode."); @@ -452,6 +452,14 @@ static bool riscv_iommu_post(struct riscv_iommu_device *iommu, return riscv_iommu_post_sync(iommu, cmd, false); } +static bool riscv_iommu_iodir_inv_devid(struct riscv_iommu_device *iommu, unsigned devid) +{ + struct riscv_iommu_command cmd; + riscv_iommu_cmd_iodir_inval_ddt(&cmd); + riscv_iommu_cmd_iodir_set_did(&cmd, devid); + return riscv_iommu_post(iommu, &cmd); +} + static bool riscv_iommu_iofence_sync(struct riscv_iommu_device *iommu) { struct riscv_iommu_command cmd; @@ -671,6 +679,94 @@ static bool riscv_iommu_capable(struct device *dev, enum iommu_cap cap) return false; } +/* TODO: implement proper device context management, e.g. teardown flow */ + +/* Lookup or initialize device directory info structure. */ +static struct riscv_iommu_dc *riscv_iommu_get_dc(struct riscv_iommu_device *iommu, + unsigned devid) +{ + const bool base_format = !(iommu->cap & RISCV_IOMMU_CAP_MSI_FLAT); + unsigned depth = iommu->ddt_mode - RISCV_IOMMU_DDTP_MODE_1LVL; + u8 ddi_bits[3] = { 0 }; + u64 *ddtp = NULL, ddt; + + if (iommu->ddt_mode == RISCV_IOMMU_DDTP_MODE_OFF || + iommu->ddt_mode == RISCV_IOMMU_DDTP_MODE_BARE) + return NULL; + + /* Make sure the mode is valid */ + if (iommu->ddt_mode > RISCV_IOMMU_DDTP_MODE_MAX) + return NULL; + + /* + * Device id partitioning for base format: + * DDI[0]: bits 0 - 6 (1st level) (7 bits) + * DDI[1]: bits 7 - 15 (2nd level) (9 bits) + * DDI[2]: bits 16 - 23 (3rd level) (8 bits) + * + * For extended format: + * DDI[0]: bits 0 - 5 (1st level) (6 bits) + * DDI[1]: bits 6 - 14 (2nd level) (9 bits) + * DDI[2]: bits 15 - 23 (3rd level) (9 bits) + */ + if (base_format) { + ddi_bits[0] = 7; + ddi_bits[1] = 7 + 9; + ddi_bits[2] = 7 + 9 + 8; + } else { + ddi_bits[0] = 6; + ddi_bits[1] = 6 + 9; + ddi_bits[2] = 6 + 9 + 9; + } + + /* Make sure device id is within range */ + if (devid >= (1 << ddi_bits[depth])) + return NULL; + + /* Get to the level of the non-leaf node that holds the device context */ + for (ddtp = (u64 *) iommu->ddtp; depth-- > 0;) { + const int split = ddi_bits[depth]; + /* + * Each non-leaf node is 64bits wide and on each level + * nodes are indexed by DDI[depth]. + */ + ddtp += (devid >> split) & 0x1FF; + + retry: + /* + * Check if this node has been populated and if not + * allocate a new level and populate it. + */ + ddt = READ_ONCE(*ddtp); + if (ddt & RISCV_IOMMU_DDTE_VALID) { + ddtp = __va(ppn_to_phys(ddt)); + } else { + u64 old, new = get_zeroed_page(GFP_KERNEL); + if (!new) + return NULL; + + old = cmpxchg64_relaxed(ddtp, ddt, + phys_to_ppn(__pa(new)) | + RISCV_IOMMU_DDTE_VALID); + + if (old != ddt) { + free_page(new); + goto retry; + } + + ddtp = (u64 *) new; + } + } + + /* + * Grab the node that matches DDI[depth], note that when using base + * format the device context is 4 * 64bits, and the extended format + * is 8 * 64bits, hence the (3 - base_format) below. + */ + ddtp += (devid & ((64 << base_format) - 1)) << (3 - base_format); + return (struct riscv_iommu_dc *)ddtp; +} + static struct iommu_device *riscv_iommu_probe_device(struct device *dev) { struct riscv_iommu_device *iommu; @@ -708,6 +804,9 @@ static struct iommu_device *riscv_iommu_probe_device(struct device *dev) ep->iommu = iommu; ep->dev = dev; + /* Initial DC pointer can be NULL if IOMMU is configured in OFF or BARE mode */ + ep->dc = riscv_iommu_get_dc(iommu, ep->devid); + dev_info(iommu->dev, "adding device to iommu with devid %i in domain %i\n", ep->devid, ep->domid); @@ -734,6 +833,16 @@ static void riscv_iommu_release_device(struct device *dev) list_del(&ep->domain); mutex_unlock(&ep->lock); + if (ep->dc) { + // this should be already done by domain detach. + ep->dc->tc = 0ULL; + wmb(); + ep->dc->fsc = 0ULL; + ep->dc->iohgatp = 0ULL; + wmb(); + riscv_iommu_iodir_inv_devid(iommu, ep->devid); + } + /* Remove endpoint from IOMMU tracking structures */ mutex_lock(&iommu->eps_mutex); rb_erase(&ep->node, &iommu->eps); @@ -853,11 +962,21 @@ static int riscv_iommu_domain_finalize(struct riscv_iommu_domain *domain, return 0; } +static u64 riscv_iommu_domain_atp(struct riscv_iommu_domain *domain) +{ + u64 atp = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, domain->mode); + if (domain->mode != RISCV_IOMMU_DC_FSC_MODE_BARE) + atp |= FIELD_PREP(RISCV_IOMMU_DC_FSC_PPN, virt_to_pfn(domain->pgd_root)); + return atp; +} + static int riscv_iommu_attach_dev(struct iommu_domain *iommu_domain, struct device *dev) { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + struct riscv_iommu_dc *dc = ep->dc; int ret; + u64 val; /* PSCID not valid */ if ((int)domain->pscid < 0) @@ -880,17 +999,44 @@ static int riscv_iommu_attach_dev(struct iommu_domain *iommu_domain, struct devi return ret; } - if (ep->iommu->ddt_mode != RISCV_IOMMU_DDTP_MODE_BARE || - domain->domain.type != IOMMU_DOMAIN_IDENTITY) { - dev_warn(dev, "domain type %d not supported\n", - domain->domain.type); + if (ep->iommu->ddt_mode == RISCV_IOMMU_DDTP_MODE_BARE && + domain->domain.type == IOMMU_DOMAIN_IDENTITY) { + dev_info(dev, "domain type %d attached w/ PSCID %u\n", + domain->domain.type, domain->pscid); + return 0; + } + + if (!dc) { return -ENODEV; } + /* + * S-Stage translation table. G-Stage remains unmodified (BARE). + */ + val = FIELD_PREP(RISCV_IOMMU_DC_TA_PSCID, domain->pscid); + + dc->ta = cpu_to_le64(val); + dc->fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + + wmb(); + + /* Mark device context as valid, synchronise device context cache. */ + val = RISCV_IOMMU_DC_TC_V; + + if (ep->iommu->cap & RISCV_IOMMU_CAP_AMO) { + val |= RISCV_IOMMU_DC_TC_GADE | + RISCV_IOMMU_DC_TC_SADE; + } + + dc->tc = cpu_to_le64(val); + wmb(); + list_add_tail(&ep->domain, &domain->endpoints); mutex_unlock(&ep->lock); mutex_unlock(&domain->lock); + riscv_iommu_iodir_inv_devid(ep->iommu, ep->devid); + dev_info(dev, "domain type %d attached w/ PSCID %u\n", domain->domain.type, domain->pscid); @@ -1239,7 +1385,12 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) goto fail; no_ats: - ret = riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_BARE); + if (iommu_default_passthrough()) { + dev_info(dev, "iommu set to passthrough mode\n"); + ret = riscv_iommu_enable(iommu, RISCV_IOMMU_DDTP_MODE_BARE); + } else { + ret = riscv_iommu_enable(iommu, ddt_mode); + } if (ret) { dev_err(dev, "cannot enable iommu device (%d)\n", ret); diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 04148a2a8ffd..9140df71e17b 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -105,6 +105,7 @@ struct riscv_iommu_endpoint { unsigned devid; /* PCI bus:device:function number */ unsigned domid; /* PCI domain number, segment */ struct rb_node node; /* device tracking node (lookup by devid) */ + struct riscv_iommu_dc *dc; /* device context pointer */ struct riscv_iommu_device *iommu; /* parent iommu device */ struct mutex lock; From patchwork Wed Jul 19 19:33:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D05D7C001DE for ; Wed, 19 Jul 2023 19:34:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WBFQYmezSQWXA+h+Mn64NBe3mE/K9sKXbX14xGWdnqM=; b=EKxARHpIFHTzv1 ailqhYuHCkUN2ej81emMXdXIQl87uMgKc3g1x2tFoaS+PXpNmfXjUfWfedA5k+VsZBYhHsGpsfUc0 bas4mVx4j5D07kbDYqKU9pnYgk+MwVpJ7vu0wbeK2fm1ejsPY+SPC8i354/LRXeS+8S+k1nhWBsOX ylNlhjw6Wy1qs81UcLWUAdIQFgRI9ZP5Eb8JiYS1H0zI2HajBy6l6ZZM5Zu/Jn51b5v42cNK6ue/Y nhIljfSfVhpPs9lXnO7DbLx41/Nor+h5q5iZ1HiVED+RmQ8yGkaM7QYJZVlptRrSoyIcnEaybrbtl 6qko24iIY46AlK78EZSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwf-008hFK-0G; Wed, 19 Jul 2023 19:34:45 +0000 Received: from mail-il1-x12a.google.com ([2607:f8b0:4864:20::12a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwV-008h7U-0M for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:37 +0000 Received: by mail-il1-x12a.google.com with SMTP id e9e14a558f8ab-34637362297so481175ab.2 for ; Wed, 19 Jul 2023 12:34:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795274; x=1690400074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ESdqCz79AqYqq1R9n18M3NCf1nQwZKZlVuGXa3nv+14=; b=0/zDuCQOfMpFDxXbQVlQLcyvvXi/F8tdUpaBbm0sOShcpabIGwxFQxtzVEr6CW5qZT qLbEmOcaWG4nRaIYFf85eKLGTmVEzllTL2dt/Cl4yPvlaPQFwO0TejYCG8CLe78e3kbN pXyzNrJP3bpLJwoU9eXFJDywKPwsYzPEdR37h2TijJfLOsLCGuux78HgzhgpSJpXRzTO Ex3WxHyQTOGbS9HPiK4OAs3jBRNV9KbyL9ndEWKjcGnYuFPj3ueMAs5r6pW2Dj0/P2/P sdIYKZUimNe0mipfF1+98uu6OSj0POrVFQnzB4Pr/mmEPgi/qd92+xDuTPyD7xNGJveX 5f5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795274; x=1690400074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ESdqCz79AqYqq1R9n18M3NCf1nQwZKZlVuGXa3nv+14=; b=F1ZknmxxkW+f2ikQrDf36kHZrFImMIPPfneOKturZSm1ZYA8xPtiIi47koOjzibrS2 JVzQDxErOLqPJ6+9jY7J0NL1lRrDd9nRJrkK2NP/rjh4DIgqziahGaVCvTuqkzBV42Y1 MVXSom0XVKrP/WkkGNj4eUHuWQ8/RMrmeGpGXEcODxdfEbACZuyiolpJU4q4xQiM4ngg aSstaaQufxwidL4YUKicC+a9zfjxy/zjNoPas+ZHD7tHB8K/22y+jT9xdbh3L9btfGvI Y5sYYC00D+o7hrczjq+HFW5sDpJOSPl1/Dad1TQ8PtUoFBsr1xWvwe5jRG93lM8quuof 6XJw== X-Gm-Message-State: ABy/qLaonlignTwoHXrhhHbJVXNqZ3sHJeTCkGTXYAjRhBFX3v7CFvhO lAyEnSgNQpMIORbdvm4IQ510FA== X-Google-Smtp-Source: APBJJlGbMjLqbUUww9MTTDncszL/stgB6E5Ra1/NBZzu4U8A8mtdW8ttUdgoJDZmlnbMHfcfKTQ/8Q== X-Received: by 2002:a92:cd8f:0:b0:345:dc7a:7c66 with SMTP id r15-20020a92cd8f000000b00345dc7a7c66mr7383304ilb.27.1689795273798; Wed, 19 Jul 2023 12:34:33 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:33 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 08/11] RISC-V: drivers/iommu/riscv: Add page table support Date: Wed, 19 Jul 2023 12:33:52 -0700 Message-Id: <2d5242e79a98dc75cd8fa0fefdb4e3ad37a920ba.1689792825.git.tjeznach@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123435_157381_AFFCAC9F X-CRM114-Status: GOOD ( 25.90 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Introduces I/O page level translation services, with 4K, 2M, 1G page size support and enables page level iommu_map/unmap domain interfaces. Co-developed-by: Sebastien Boeuf Signed-off-by: Sebastien Boeuf Signed-off-by: Tomasz Jeznach --- drivers/iommu/io-pgtable.c | 3 + drivers/iommu/riscv/Makefile | 2 +- drivers/iommu/riscv/io_pgtable.c | 266 +++++++++++++++++++++++++++++++ drivers/iommu/riscv/iommu.c | 40 +++-- drivers/iommu/riscv/iommu.h | 1 + include/linux/io-pgtable.h | 2 + 6 files changed, 297 insertions(+), 17 deletions(-) create mode 100644 drivers/iommu/riscv/io_pgtable.c diff --git a/drivers/iommu/io-pgtable.c b/drivers/iommu/io-pgtable.c index b843fcd365d2..c4807175934f 100644 --- a/drivers/iommu/io-pgtable.c +++ b/drivers/iommu/io-pgtable.c @@ -32,6 +32,9 @@ io_pgtable_init_table[IO_PGTABLE_NUM_FMTS] = { [AMD_IOMMU_V1] = &io_pgtable_amd_iommu_v1_init_fns, [AMD_IOMMU_V2] = &io_pgtable_amd_iommu_v2_init_fns, #endif +#ifdef CONFIG_RISCV_IOMMU + [RISCV_IOMMU] = &io_pgtable_riscv_init_fns, +#endif }; struct io_pgtable_ops *alloc_io_pgtable_ops(enum io_pgtable_fmt fmt, diff --git a/drivers/iommu/riscv/Makefile b/drivers/iommu/riscv/Makefile index 9523eb053cfc..13af452c3052 100644 --- a/drivers/iommu/riscv/Makefile +++ b/drivers/iommu/riscv/Makefile @@ -1 +1 @@ -obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-pci.o iommu-platform.o iommu-sysfs.o \ No newline at end of file +obj-$(CONFIG_RISCV_IOMMU) += iommu.o iommu-pci.o iommu-platform.o iommu-sysfs.o io_pgtable.o diff --git a/drivers/iommu/riscv/io_pgtable.c b/drivers/iommu/riscv/io_pgtable.c new file mode 100644 index 000000000000..b6e603e6726e --- /dev/null +++ b/drivers/iommu/riscv/io_pgtable.c @@ -0,0 +1,266 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright © 2022-2023 Rivos Inc. + * + * RISC-V IOMMU page table allocator. + * + * Authors: + * Tomasz Jeznach + * Sebastien Boeuf + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "iommu.h" + +#define io_pgtable_to_domain(x) \ + container_of((x), struct riscv_iommu_domain, pgtbl) + +#define io_pgtable_ops_to_domain(x) \ + io_pgtable_to_domain(container_of((x), struct io_pgtable, ops)) + +static inline size_t get_page_size(size_t size) +{ + if (size >= IOMMU_PAGE_SIZE_512G) + return IOMMU_PAGE_SIZE_512G; + + if (size >= IOMMU_PAGE_SIZE_1G) + return IOMMU_PAGE_SIZE_1G; + + if (size >= IOMMU_PAGE_SIZE_2M) + return IOMMU_PAGE_SIZE_2M; + + return IOMMU_PAGE_SIZE_4K; +} + +static void riscv_iommu_pt_walk_free(pmd_t * ptp, unsigned shift, bool root) +{ + pmd_t *pte, *pt_base; + int i; + + if (shift == PAGE_SHIFT) + return; + + if (root) + pt_base = ptp; + else + pt_base = + (pmd_t *) pfn_to_virt(__page_val_to_pfn(pmd_val(*ptp))); + + /* Recursively free all sub page table pages */ + for (i = 0; i < PTRS_PER_PMD; i++) { + pte = pt_base + i; + if (pmd_present(*pte) && !pmd_leaf(*pte)) + riscv_iommu_pt_walk_free(pte, shift - 9, false); + } + + /* Now free the current page table page */ + if (!root && pmd_present(*pt_base)) + free_page((unsigned long)pt_base); +} + +static void riscv_iommu_free_pgtable(struct io_pgtable *iop) +{ + struct riscv_iommu_domain *domain = io_pgtable_to_domain(iop); + riscv_iommu_pt_walk_free((pmd_t *) domain->pgd_root, PGDIR_SHIFT, true); +} + +static pte_t *riscv_iommu_pt_walk_alloc(pmd_t * ptp, unsigned long iova, + unsigned shift, bool root, + size_t pgsize, + unsigned long (*pd_alloc)(gfp_t), + gfp_t gfp) +{ + pmd_t *pte; + unsigned long pfn; + + if (root) + pte = ptp + ((iova >> shift) & (PTRS_PER_PMD - 1)); + else + pte = (pmd_t *) pfn_to_virt(__page_val_to_pfn(pmd_val(*ptp))) + + ((iova >> shift) & (PTRS_PER_PMD - 1)); + + if ((1ULL << shift) <= pgsize) { + if (pmd_present(*pte) && !pmd_leaf(*pte)) + riscv_iommu_pt_walk_free(pte, shift - 9, false); + return (pte_t *) pte; + } + + if (pmd_none(*pte)) { + pfn = pd_alloc ? virt_to_pfn(pd_alloc(gfp)) : 0; + if (!pfn) + return NULL; + set_pmd(pte, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE)); + } + + return riscv_iommu_pt_walk_alloc(pte, iova, shift - 9, false, + pgsize, pd_alloc, gfp); +} + +static pte_t *riscv_iommu_pt_walk_fetch(pmd_t * ptp, + unsigned long iova, unsigned shift, + bool root) +{ + pmd_t *pte; + + if (root) + pte = ptp + ((iova >> shift) & (PTRS_PER_PMD - 1)); + else + pte = (pmd_t *) pfn_to_virt(__page_val_to_pfn(pmd_val(*ptp))) + + ((iova >> shift) & (PTRS_PER_PMD - 1)); + + if (pmd_leaf(*pte)) + return (pte_t *) pte; + else if (pmd_none(*pte)) + return NULL; + else if (shift == PAGE_SHIFT) + return NULL; + + return riscv_iommu_pt_walk_fetch(pte, iova, shift - 9, false); +} + +static int riscv_iommu_map_pages(struct io_pgtable_ops *ops, + unsigned long iova, phys_addr_t phys, + size_t pgsize, size_t pgcount, int prot, + gfp_t gfp, size_t *mapped) +{ + struct riscv_iommu_domain *domain = io_pgtable_ops_to_domain(ops); + size_t size = 0; + size_t page_size = get_page_size(pgsize); + pte_t *pte; + pte_t pte_val; + pgprot_t pte_prot; + + if (domain->domain.type == IOMMU_DOMAIN_BLOCKED) + return -ENODEV; + + if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) { + *mapped = pgsize * pgcount; + return 0; + } + + pte_prot = (prot & IOMMU_WRITE) ? + __pgprot(_PAGE_BASE | _PAGE_READ | _PAGE_WRITE | _PAGE_DIRTY) : + __pgprot(_PAGE_BASE | _PAGE_READ); + + while (pgcount--) { + pte = + riscv_iommu_pt_walk_alloc((pmd_t *) domain->pgd_root, iova, + PGDIR_SHIFT, true, page_size, + get_zeroed_page, gfp); + if (!pte) { + *mapped = size; + return -ENOMEM; + } + + pte_val = pfn_pte(phys_to_pfn(phys), pte_prot); + + set_pte(pte, pte_val); + + size += page_size; + iova += page_size; + phys += page_size; + } + + *mapped = size; + return 0; +} + +static size_t riscv_iommu_unmap_pages(struct io_pgtable_ops *ops, + unsigned long iova, size_t pgsize, + size_t pgcount, + struct iommu_iotlb_gather *gather) +{ + struct riscv_iommu_domain *domain = io_pgtable_ops_to_domain(ops); + size_t size = 0; + size_t page_size = get_page_size(pgsize); + pte_t *pte; + + if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) + return pgsize * pgcount; + + while (pgcount--) { + pte = riscv_iommu_pt_walk_fetch((pmd_t *) domain->pgd_root, + iova, PGDIR_SHIFT, true); + if (!pte) + return size; + + set_pte(pte, __pte(0)); + + iommu_iotlb_gather_add_page(&domain->domain, gather, iova, + pgsize); + + size += page_size; + iova += page_size; + } + + return size; +} + +static phys_addr_t riscv_iommu_iova_to_phys(struct io_pgtable_ops *ops, + unsigned long iova) +{ + struct riscv_iommu_domain *domain = io_pgtable_ops_to_domain(ops); + pte_t *pte; + + if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) + return (phys_addr_t) iova; + + pte = riscv_iommu_pt_walk_fetch((pmd_t *) domain->pgd_root, + iova, PGDIR_SHIFT, true); + if (!pte || !pte_present(*pte)) + return 0; + + return (pfn_to_phys(pte_pfn(*pte)) | (iova & PAGE_MASK)); +} + +static void riscv_iommu_tlb_inv_all(void *cookie) +{ +} + +static void riscv_iommu_tlb_inv_walk(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ +} + +static void riscv_iommu_tlb_add_page(struct iommu_iotlb_gather *gather, + unsigned long iova, size_t granule, + void *cookie) +{ +} + +static const struct iommu_flush_ops riscv_iommu_flush_ops = { + .tlb_flush_all = riscv_iommu_tlb_inv_all, + .tlb_flush_walk = riscv_iommu_tlb_inv_walk, + .tlb_add_page = riscv_iommu_tlb_add_page, +}; + +/* NOTE: cfg should point to riscv_iommu_domain structure member pgtbl.cfg */ +static struct io_pgtable *riscv_iommu_alloc_pgtable(struct io_pgtable_cfg *cfg, + void *cookie) +{ + struct io_pgtable *iop = container_of(cfg, struct io_pgtable, cfg); + + cfg->pgsize_bitmap = SZ_4K | SZ_2M | SZ_1G; + cfg->ias = 57; // va mode, SvXX -> ias + cfg->oas = 57; // pa mode, or SvXX+4 -> oas + cfg->tlb = &riscv_iommu_flush_ops; + + iop->ops.map_pages = riscv_iommu_map_pages; + iop->ops.unmap_pages = riscv_iommu_unmap_pages; + iop->ops.iova_to_phys = riscv_iommu_iova_to_phys; + + return iop; +} + +struct io_pgtable_init_fns io_pgtable_riscv_init_fns = { + .alloc = riscv_iommu_alloc_pgtable, + .free = riscv_iommu_free_pgtable, +}; diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 9ee7d2b222b5..2ef6952a2109 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -807,7 +807,7 @@ static struct iommu_device *riscv_iommu_probe_device(struct device *dev) /* Initial DC pointer can be NULL if IOMMU is configured in OFF or BARE mode */ ep->dc = riscv_iommu_get_dc(iommu, ep->devid); - dev_info(iommu->dev, "adding device to iommu with devid %i in domain %i\n", + dev_dbg(iommu->dev, "adding device to iommu with devid %i in domain %i\n", ep->devid, ep->domid); dev_iommu_priv_set(dev, ep); @@ -874,7 +874,10 @@ static struct iommu_domain *riscv_iommu_domain_alloc(unsigned type) { struct riscv_iommu_domain *domain; - if (type != IOMMU_DOMAIN_IDENTITY && + if (type != IOMMU_DOMAIN_DMA && + type != IOMMU_DOMAIN_DMA_FQ && + type != IOMMU_DOMAIN_UNMANAGED && + type != IOMMU_DOMAIN_IDENTITY && type != IOMMU_DOMAIN_BLOCKED) return NULL; @@ -890,7 +893,7 @@ static struct iommu_domain *riscv_iommu_domain_alloc(unsigned type) domain->pscid = ida_alloc_range(&riscv_iommu_pscids, 1, RISCV_IOMMU_MAX_PSCID, GFP_KERNEL); - printk("domain type %x alloc %u\n", type, domain->pscid); + printk("domain alloc %u\n", domain->pscid); return &domain->domain; } @@ -903,6 +906,9 @@ static void riscv_iommu_domain_free(struct iommu_domain *iommu_domain) pr_warn("IOMMU domain is not empty!\n"); } + if (domain->pgtbl.cookie) + free_io_pgtable_ops(&domain->pgtbl.ops); + if (domain->pgd_root) free_pages((unsigned long)domain->pgd_root, 0); @@ -959,6 +965,9 @@ static int riscv_iommu_domain_finalize(struct riscv_iommu_domain *domain, if (!domain->pgd_root) return -ENOMEM; + if (!alloc_io_pgtable_ops(RISCV_IOMMU, &domain->pgtbl.cfg, domain)) + return -ENOMEM; + return 0; } @@ -1006,9 +1015,8 @@ static int riscv_iommu_attach_dev(struct iommu_domain *iommu_domain, struct devi return 0; } - if (!dc) { + if (!dc) return -ENODEV; - } /* * S-Stage translation table. G-Stage remains unmodified (BARE). @@ -1104,12 +1112,11 @@ static int riscv_iommu_map_pages(struct iommu_domain *iommu_domain, { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); - if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) { - *mapped = pgsize * pgcount; - return 0; - } + if (!domain->pgtbl.ops.map_pages) + return -ENODEV; - return -ENODEV; + return domain->pgtbl.ops.map_pages(&domain->pgtbl.ops, iova, phys, + pgsize, pgcount, prot, gfp, mapped); } static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain, @@ -1118,10 +1125,11 @@ static size_t riscv_iommu_unmap_pages(struct iommu_domain *iommu_domain, { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); - if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) - return pgsize * pgcount; + if (!domain->pgtbl.ops.unmap_pages) + return 0; - return 0; + return domain->pgtbl.ops.unmap_pages(&domain->pgtbl.ops, iova, pgsize, + pgcount, gather); } static phys_addr_t riscv_iommu_iova_to_phys(struct iommu_domain *iommu_domain, @@ -1129,10 +1137,10 @@ static phys_addr_t riscv_iommu_iova_to_phys(struct iommu_domain *iommu_domain, { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); - if (domain->domain.type == IOMMU_DOMAIN_IDENTITY) - return (phys_addr_t) iova; + if (!domain->pgtbl.ops.iova_to_phys) + return 0; - return 0; + return domain->pgtbl.ops.iova_to_phys(&domain->pgtbl.ops, iova); } /* diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 9140df71e17b..fe32a4eff14e 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -88,6 +88,7 @@ struct riscv_iommu_device { struct riscv_iommu_domain { struct iommu_domain domain; + struct io_pgtable pgtbl; struct list_head endpoints; struct mutex lock; diff --git a/include/linux/io-pgtable.h b/include/linux/io-pgtable.h index 1b7a44b35616..8dd9d3a28e3a 100644 --- a/include/linux/io-pgtable.h +++ b/include/linux/io-pgtable.h @@ -19,6 +19,7 @@ enum io_pgtable_fmt { AMD_IOMMU_V2, APPLE_DART, APPLE_DART2, + RISCV_IOMMU, IO_PGTABLE_NUM_FMTS, }; @@ -258,5 +259,6 @@ extern struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns; extern struct io_pgtable_init_fns io_pgtable_amd_iommu_v1_init_fns; extern struct io_pgtable_init_fns io_pgtable_amd_iommu_v2_init_fns; extern struct io_pgtable_init_fns io_pgtable_apple_dart_init_fns; +extern struct io_pgtable_init_fns io_pgtable_riscv_init_fns; #endif /* __IO_PGTABLE_H */ From patchwork Wed Jul 19 19:33:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E3C6C001DF for ; Wed, 19 Jul 2023 19:34:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mT442EjZlzix0sfCO96+h/bxl6Jw++5HYIxY0HRn7TE=; b=f6YPKaSW1WOKGb jbzxyR+/GpO+psThToQNHJJGyTFJNSel2KvTn7bN6siM2TXTDMV6sqQDUagoGDlm38Eb1Qa4rLAfJ jrKIT9RDwGTajqrSwbWhTZ90wUryGFj2EBkLgUm6WRmG/xFVvY9UBf6KQmjbOUAN1ctRr4AFNGzEX lzOmNBY3WM7eZPst7byNJ94IA5Q6TMy0zHPW/SCHOcOX9vYIQxkvvswj2BiCdajHtolKALQKBncmH 3FguNuU7QvJRifJlN0dkmnhX9NjHXMKSgsNYdhljaBEps0tzO6Fdy9QisLOBpAeJTcw0XuXGIHtKe RyJczQu3h6tw0qzJI+IQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwh-008hH0-0B; Wed, 19 Jul 2023 19:34:47 +0000 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwW-008h8L-16 for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:39 +0000 Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-55bac17b442so5458597a12.3 for ; Wed, 19 Jul 2023 12:34:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795275; x=1690400075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+/UAMiG1bQ6endV/IctuiWSddw0eWce+t6nTi33PhHs=; b=uvzyT/t99XEHRT8eL7RJR0m76yg5f9RxGZ/C6x+D7eM4Ic/Aab/9jt080pS0Yg6Swd pD+EdJ9OzlqBLs3SI2UABL/9GJd17J1YbhEYvxTLoEOJvRQF3VXnnPJ/k3imRAqummcB P6dBsDv0DZ9RtShHrRSULl/u5jmXh4EwY5GThy6qZPvYdG2Lp9bobsd1XejSRPETm3VC FUXK1WxWwh7F/h1cYYs/6bZ/Dg5/DlEJ7iUNEabWeVWRQLaRjT9snM98ZuQLE3urOh/U op0WKqGfVvQzAeGXfywvLHRsP4X2bwO+gddbNcGthPqwQShFsr0KqXj7fHJOx+sxJDus BLCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795275; x=1690400075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+/UAMiG1bQ6endV/IctuiWSddw0eWce+t6nTi33PhHs=; b=K7ac9MQy0qZC2kPSLW4tUgTyKuXNc3bnqEEYkEMWpr9w0hAEhqWCsZX/Sb9NtddNdd w2r1J64+VvGtmds/dyrGGjv5L8mur6iKmzDXTGNOhOyl9TVxwG7wjisoK+ABjbdLIseo AXvArKUreOrPYxAe0GB5Av2Wt2YxnepABV/QZ9GLzDMZTyyHtrz2MijJMo17UnZgINm3 20lb8JZVUl1reCkduRslG+j0i8THLNJdrqo4IFlhaogYqqlBgnonwY7m/TjxFQfJjfcZ E+WGwGjlz5afAG1CP6PnQptbm3VtpXFMFawrNQczhiaN2efHc/ymvHvF7J7B9UXtiCh8 VzNw== X-Gm-Message-State: ABy/qLaj000DiHC3zNbE1lpBIUJw3t+jAvAO45X8EC/hvz/ukyjBvBVH 1i1pvtHxH8+j4vebChTrw1CrxIx5VNEiKP2h5I458A== X-Google-Smtp-Source: APBJJlFdt1FEf1/IDn+Rsj6fa98EP/ATDEE3K8y66XqfzBdbBtpcJRUA58KvCvm3DFI5GVbWBlioeQ== X-Received: by 2002:a17:90a:a58f:b0:263:e814:5d0f with SMTP id b15-20020a17090aa58f00b00263e8145d0fmr16914579pjq.41.1689795275018; Wed, 19 Jul 2023 12:34:35 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:34 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 09/11] RISC-V: drivers/iommu/riscv: Add SVA with PASID/ATS/PRI support. Date: Wed, 19 Jul 2023 12:33:53 -0700 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123436_489805_E81ECFB7 X-CRM114-Status: GOOD ( 23.44 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Introduces SVA (Shared Virtual Address) for RISC-V IOMMU, with ATS/PRI services for capable devices. Co-developed-by: Sebastien Boeuf Signed-off-by: Sebastien Boeuf Signed-off-by: Tomasz Jeznach --- drivers/iommu/riscv/iommu.c | 601 +++++++++++++++++++++++++++++++++++- drivers/iommu/riscv/iommu.h | 14 + 2 files changed, 610 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 2ef6952a2109..6042c35be3ca 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -384,6 +384,89 @@ static inline void riscv_iommu_cmd_iodir_set_did(struct riscv_iommu_command *cmd FIELD_PREP(RISCV_IOMMU_CMD_IODIR_DID, devid) | RISCV_IOMMU_CMD_IODIR_DV; } +static inline void riscv_iommu_cmd_iodir_set_pid(struct riscv_iommu_command *cmd, + unsigned pasid) +{ + cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_IODIR_PID, pasid); +} + +static void riscv_iommu_cmd_ats_inval(struct riscv_iommu_command *cmd) +{ + cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_ATS_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_ATS_FUNC_INVAL); + cmd->dword1 = 0; +} + +static inline void riscv_iommu_cmd_ats_prgr(struct riscv_iommu_command *cmd) +{ + cmd->dword0 = FIELD_PREP(RISCV_IOMMU_CMD_OPCODE, RISCV_IOMMU_CMD_ATS_OPCODE) | + FIELD_PREP(RISCV_IOMMU_CMD_FUNC, RISCV_IOMMU_CMD_ATS_FUNC_PRGR); + cmd->dword1 = 0; +} + +static void riscv_iommu_cmd_ats_set_rid(struct riscv_iommu_command *cmd, u32 rid) +{ + cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_ATS_RID, rid); +} + +static void riscv_iommu_cmd_ats_set_pid(struct riscv_iommu_command *cmd, u32 pid) +{ + cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_ATS_PID, pid) | RISCV_IOMMU_CMD_ATS_PV; +} + +static void riscv_iommu_cmd_ats_set_dseg(struct riscv_iommu_command *cmd, u8 seg) +{ + cmd->dword0 |= FIELD_PREP(RISCV_IOMMU_CMD_ATS_DSEG, seg) | RISCV_IOMMU_CMD_ATS_DSV; +} + +static void riscv_iommu_cmd_ats_set_payload(struct riscv_iommu_command *cmd, u64 payload) +{ + cmd->dword1 = payload; +} + +/* Prepare the ATS invalidation payload */ +static unsigned long riscv_iommu_ats_inval_payload(unsigned long start, + unsigned long end, bool global_inv) +{ + size_t len = end - start + 1; + unsigned long payload = 0; + + /* + * PCI Express specification + * Section 10.2.3.2 Translation Range Size (S) Field + */ + if (len < PAGE_SIZE) + len = PAGE_SIZE; + else + len = __roundup_pow_of_two(len); + + payload = (start & ~(len - 1)) | (((len - 1) >> 12) << 11); + + if (global_inv) + payload |= RISCV_IOMMU_CMD_ATS_INVAL_G; + + return payload; +} + +/* Prepare the ATS invalidation payload for all translations to be invalidated. */ +static unsigned long riscv_iommu_ats_inval_all_payload(bool global_inv) +{ + unsigned long payload = GENMASK_ULL(62, 11); + + if (global_inv) + payload |= RISCV_IOMMU_CMD_ATS_INVAL_G; + + return payload; +} + +/* Prepare the ATS "Page Request Group Response" payload */ +static unsigned long riscv_iommu_ats_prgr_payload(u16 dest_id, u8 resp_code, u16 grp_idx) +{ + return FIELD_PREP(RISCV_IOMMU_CMD_ATS_PRGR_DST_ID, dest_id) | + FIELD_PREP(RISCV_IOMMU_CMD_ATS_PRGR_RESP_CODE, resp_code) | + FIELD_PREP(RISCV_IOMMU_CMD_ATS_PRGR_PRG_INDEX, grp_idx); +} + /* TODO: Convert into lock-less MPSC implementation. */ static bool riscv_iommu_post_sync(struct riscv_iommu_device *iommu, struct riscv_iommu_command *cmd, bool sync) @@ -460,6 +543,16 @@ static bool riscv_iommu_iodir_inv_devid(struct riscv_iommu_device *iommu, unsign return riscv_iommu_post(iommu, &cmd); } +static bool riscv_iommu_iodir_inv_pasid(struct riscv_iommu_device *iommu, + unsigned devid, unsigned pasid) +{ + struct riscv_iommu_command cmd; + riscv_iommu_cmd_iodir_inval_pdt(&cmd); + riscv_iommu_cmd_iodir_set_did(&cmd, devid); + riscv_iommu_cmd_iodir_set_pid(&cmd, pasid); + return riscv_iommu_post(iommu, &cmd); +} + static bool riscv_iommu_iofence_sync(struct riscv_iommu_device *iommu) { struct riscv_iommu_command cmd; @@ -467,6 +560,62 @@ static bool riscv_iommu_iofence_sync(struct riscv_iommu_device *iommu) return riscv_iommu_post_sync(iommu, &cmd, true); } +static void riscv_iommu_mm_invalidate(struct mmu_notifier *mn, + struct mm_struct *mm, unsigned long start, + unsigned long end) +{ + struct riscv_iommu_command cmd; + struct riscv_iommu_endpoint *endpoint; + struct riscv_iommu_domain *domain = + container_of(mn, struct riscv_iommu_domain, mn); + unsigned long iova; + /* + * The mm_types defines vm_end as the first byte after the end address, + * different from IOMMU subsystem using the last address of an address + * range. So do a simple translation here by updating what end means. + */ + unsigned long payload = riscv_iommu_ats_inval_payload(start, end - 1, true); + + riscv_iommu_cmd_inval_vma(&cmd); + riscv_iommu_cmd_inval_set_gscid(&cmd, 0); + riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid); + if (end > start) { + /* Cover only the range that is needed */ + for (iova = start; iova < end; iova += PAGE_SIZE) { + riscv_iommu_cmd_inval_set_addr(&cmd, iova); + riscv_iommu_post(domain->iommu, &cmd); + } + } else { + riscv_iommu_post(domain->iommu, &cmd); + } + + riscv_iommu_iofence_sync(domain->iommu); + + /* ATS invalidation for every device and for specific translation range. */ + list_for_each_entry(endpoint, &domain->endpoints, domain) { + if (!endpoint->pasid_enabled) + continue; + + riscv_iommu_cmd_ats_inval(&cmd); + riscv_iommu_cmd_ats_set_dseg(&cmd, endpoint->domid); + riscv_iommu_cmd_ats_set_rid(&cmd, endpoint->devid); + riscv_iommu_cmd_ats_set_pid(&cmd, domain->pasid); + riscv_iommu_cmd_ats_set_payload(&cmd, payload); + riscv_iommu_post(domain->iommu, &cmd); + } + riscv_iommu_iofence_sync(domain->iommu); +} + +static void riscv_iommu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) +{ + /* TODO: removed from notifier, cleanup PSCID mapping, flush IOTLB */ +} + +static const struct mmu_notifier_ops riscv_iommu_mmuops = { + .release = riscv_iommu_mm_release, + .invalidate_range = riscv_iommu_mm_invalidate, +}; + /* Command queue primary interrupt handler */ static irqreturn_t riscv_iommu_cmdq_irq_check(int irq, void *data) { @@ -608,6 +757,128 @@ static void riscv_iommu_add_device(struct riscv_iommu_device *iommu, struct devi mutex_unlock(&iommu->eps_mutex); } +/* + * Get device reference based on device identifier (requester id). + * Decrement reference count with put_device() call. + */ +static struct device *riscv_iommu_get_device(struct riscv_iommu_device *iommu, + unsigned devid) +{ + struct rb_node *node; + struct riscv_iommu_endpoint *ep; + struct device *dev = NULL; + + mutex_lock(&iommu->eps_mutex); + + node = iommu->eps.rb_node; + while (node && !dev) { + ep = rb_entry(node, struct riscv_iommu_endpoint, node); + if (ep->devid < devid) + node = node->rb_right; + else if (ep->devid > devid) + node = node->rb_left; + else + dev = get_device(ep->dev); + } + + mutex_unlock(&iommu->eps_mutex); + + return dev; +} + +static int riscv_iommu_ats_prgr(struct device *dev, struct iommu_page_response *msg) +{ + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + struct riscv_iommu_command cmd; + u8 resp_code; + unsigned long payload; + + switch (msg->code) { + case IOMMU_PAGE_RESP_SUCCESS: + resp_code = 0b0000; + break; + case IOMMU_PAGE_RESP_INVALID: + resp_code = 0b0001; + break; + case IOMMU_PAGE_RESP_FAILURE: + resp_code = 0b1111; + break; + } + payload = riscv_iommu_ats_prgr_payload(ep->devid, resp_code, msg->grpid); + + /* ATS Page Request Group Response */ + riscv_iommu_cmd_ats_prgr(&cmd); + riscv_iommu_cmd_ats_set_dseg(&cmd, ep->domid); + riscv_iommu_cmd_ats_set_rid(&cmd, ep->devid); + if (msg->flags & IOMMU_PAGE_RESP_PASID_VALID) + riscv_iommu_cmd_ats_set_pid(&cmd, msg->pasid); + riscv_iommu_cmd_ats_set_payload(&cmd, payload); + riscv_iommu_post(ep->iommu, &cmd); + + return 0; +} + +static void riscv_iommu_page_request(struct riscv_iommu_device *iommu, + struct riscv_iommu_pq_record *req) +{ + struct iommu_fault_event event = { 0 }; + struct iommu_fault_page_request *prm = &event.fault.prm; + int ret; + struct device *dev; + unsigned devid = FIELD_GET(RISCV_IOMMU_PREQ_HDR_DID, req->hdr); + + /* Ignore PGR Stop marker. */ + if ((req->payload & RISCV_IOMMU_PREQ_PAYLOAD_M) == RISCV_IOMMU_PREQ_PAYLOAD_L) + return; + + dev = riscv_iommu_get_device(iommu, devid); + if (!dev) { + /* TODO: Handle invalid page request */ + return; + } + + event.fault.type = IOMMU_FAULT_PAGE_REQ; + + if (req->payload & RISCV_IOMMU_PREQ_PAYLOAD_L) + prm->flags |= IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE; + if (req->payload & RISCV_IOMMU_PREQ_PAYLOAD_W) + prm->perm |= IOMMU_FAULT_PERM_WRITE; + if (req->payload & RISCV_IOMMU_PREQ_PAYLOAD_R) + prm->perm |= IOMMU_FAULT_PERM_READ; + + prm->grpid = FIELD_GET(RISCV_IOMMU_PREQ_PRG_INDEX, req->payload); + prm->addr = FIELD_GET(RISCV_IOMMU_PREQ_UADDR, req->payload) << PAGE_SHIFT; + + if (req->hdr & RISCV_IOMMU_PREQ_HDR_PV) { + prm->flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + /* TODO: where to find this bit */ + prm->flags |= IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID; + prm->pasid = FIELD_GET(RISCV_IOMMU_PREQ_HDR_PID, req->hdr); + } + + ret = iommu_report_device_fault(dev, &event); + if (ret) { + struct iommu_page_response resp = { + .grpid = prm->grpid, + .code = IOMMU_PAGE_RESP_FAILURE, + }; + if (prm->flags & IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID) { + resp.flags |= IOMMU_PAGE_RESP_PASID_VALID; + resp.pasid = prm->pasid; + } + riscv_iommu_ats_prgr(dev, &resp); + } + + put_device(dev); +} + +static int riscv_iommu_page_response(struct device *dev, + struct iommu_fault_event *evt, + struct iommu_page_response *msg) +{ + return riscv_iommu_ats_prgr(dev, msg); +} + /* Page request interface queue primary interrupt handler */ static irqreturn_t riscv_iommu_priq_irq_check(int irq, void *data) { @@ -626,7 +897,7 @@ static irqreturn_t riscv_iommu_priq_process(int irq, void *data) struct riscv_iommu_queue *q = (struct riscv_iommu_queue *)data; struct riscv_iommu_device *iommu; struct riscv_iommu_pq_record *requests; - unsigned cnt, idx, ctrl; + unsigned cnt, len, idx, ctrl; iommu = container_of(q, struct riscv_iommu_device, priq); requests = (struct riscv_iommu_pq_record *)q->base; @@ -649,7 +920,8 @@ static irqreturn_t riscv_iommu_priq_process(int irq, void *data) cnt = riscv_iommu_queue_consume(iommu, q, &idx); if (!cnt) break; - dev_warn(iommu->dev, "unexpected %u page requests\n", cnt); + for (len = 0; len < cnt; idx++, len++) + riscv_iommu_page_request(iommu, &requests[idx]); riscv_iommu_queue_release(iommu, q, cnt); } while (1); @@ -660,6 +932,169 @@ static irqreturn_t riscv_iommu_priq_process(int irq, void *data) * Endpoint management */ +/* Endpoint features/capabilities */ +static void riscv_iommu_disable_ep(struct riscv_iommu_endpoint *ep) +{ + struct pci_dev *pdev; + + if (!dev_is_pci(ep->dev)) + return; + + pdev = to_pci_dev(ep->dev); + + if (ep->pasid_enabled) { + pci_disable_ats(pdev); + pci_disable_pri(pdev); + pci_disable_pasid(pdev); + ep->pasid_enabled = false; + } +} + +static void riscv_iommu_enable_ep(struct riscv_iommu_endpoint *ep) +{ + int rc, feat, num; + struct pci_dev *pdev; + struct device *dev = ep->dev; + + if (!dev_is_pci(dev)) + return; + + if (!ep->iommu->iommu.max_pasids) + return; + + pdev = to_pci_dev(dev); + + if (!pci_ats_supported(pdev)) + return; + + if (!pci_pri_supported(pdev)) + return; + + feat = pci_pasid_features(pdev); + if (feat < 0) + return; + + num = pci_max_pasids(pdev); + if (!num) { + dev_warn(dev, "Can't enable PASID (num: %d)\n", num); + return; + } + + if (num > ep->iommu->iommu.max_pasids) + num = ep->iommu->iommu.max_pasids; + + rc = pci_enable_pasid(pdev, feat); + if (rc) { + dev_warn(dev, "Can't enable PASID (rc: %d)\n", rc); + return; + } + + rc = pci_reset_pri(pdev); + if (rc) { + dev_warn(dev, "Can't reset PRI (rc: %d)\n", rc); + pci_disable_pasid(pdev); + return; + } + + /* TODO: Get supported PRI queue length, hard-code to 32 entries */ + rc = pci_enable_pri(pdev, 32); + if (rc) { + dev_warn(dev, "Can't enable PRI (rc: %d)\n", rc); + pci_disable_pasid(pdev); + return; + } + + rc = pci_enable_ats(pdev, PAGE_SHIFT); + if (rc) { + dev_warn(dev, "Can't enable ATS (rc: %d)\n", rc); + pci_disable_pri(pdev); + pci_disable_pasid(pdev); + return; + } + + ep->pc = (struct riscv_iommu_pc *)get_zeroed_page(GFP_KERNEL); + if (!ep->pc) { + pci_disable_ats(pdev); + pci_disable_pri(pdev); + pci_disable_pasid(pdev); + return; + } + + ep->pasid_enabled = true; + ep->pasid_feat = feat; + ep->pasid_bits = ilog2(num); + + dev_dbg(ep->dev, "PASID/ATS support enabled, %d bits\n", ep->pasid_bits); +} + +static int riscv_iommu_enable_sva(struct device *dev) +{ + int ret; + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + + if (!ep || !ep->iommu || !ep->iommu->pq_work) + return -EINVAL; + + if (!ep->pasid_enabled) + return -ENODEV; + + ret = iopf_queue_add_device(ep->iommu->pq_work, dev); + if (ret) + return ret; + + return iommu_register_device_fault_handler(dev, iommu_queue_iopf, dev); +} + +static int riscv_iommu_disable_sva(struct device *dev) +{ + int ret; + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + + ret = iommu_unregister_device_fault_handler(dev); + if (!ret) + ret = iopf_queue_remove_device(ep->iommu->pq_work, dev); + + return ret; +} + +static int riscv_iommu_enable_iopf(struct device *dev) +{ + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + + if (ep && ep->pasid_enabled) + return 0; + + return -EINVAL; +} + +static int riscv_iommu_dev_enable_feat(struct device *dev, enum iommu_dev_features feat) +{ + switch (feat) { + case IOMMU_DEV_FEAT_IOPF: + return riscv_iommu_enable_iopf(dev); + + case IOMMU_DEV_FEAT_SVA: + return riscv_iommu_enable_sva(dev); + + default: + return -ENODEV; + } +} + +static int riscv_iommu_dev_disable_feat(struct device *dev, enum iommu_dev_features feat) +{ + switch (feat) { + case IOMMU_DEV_FEAT_IOPF: + return 0; + + case IOMMU_DEV_FEAT_SVA: + return riscv_iommu_disable_sva(dev); + + default: + return -ENODEV; + } +} + static int riscv_iommu_of_xlate(struct device *dev, struct of_phandle_args *args) { return iommu_fwspec_add_ids(dev, args->args, 1); @@ -812,6 +1247,7 @@ static struct iommu_device *riscv_iommu_probe_device(struct device *dev) dev_iommu_priv_set(dev, ep); riscv_iommu_add_device(iommu, dev); + riscv_iommu_enable_ep(ep); return &iommu->iommu; } @@ -843,6 +1279,8 @@ static void riscv_iommu_release_device(struct device *dev) riscv_iommu_iodir_inv_devid(iommu, ep->devid); } + riscv_iommu_disable_ep(ep); + /* Remove endpoint from IOMMU tracking structures */ mutex_lock(&iommu->eps_mutex); rb_erase(&ep->node, &iommu->eps); @@ -878,7 +1316,8 @@ static struct iommu_domain *riscv_iommu_domain_alloc(unsigned type) type != IOMMU_DOMAIN_DMA_FQ && type != IOMMU_DOMAIN_UNMANAGED && type != IOMMU_DOMAIN_IDENTITY && - type != IOMMU_DOMAIN_BLOCKED) + type != IOMMU_DOMAIN_BLOCKED && + type != IOMMU_DOMAIN_SVA) return NULL; domain = kzalloc(sizeof(*domain), GFP_KERNEL); @@ -906,6 +1345,9 @@ static void riscv_iommu_domain_free(struct iommu_domain *iommu_domain) pr_warn("IOMMU domain is not empty!\n"); } + if (domain->mn.ops && iommu_domain->mm) + mmu_notifier_unregister(&domain->mn, iommu_domain->mm); + if (domain->pgtbl.cookie) free_io_pgtable_ops(&domain->pgtbl.ops); @@ -1023,14 +1465,29 @@ static int riscv_iommu_attach_dev(struct iommu_domain *iommu_domain, struct devi */ val = FIELD_PREP(RISCV_IOMMU_DC_TA_PSCID, domain->pscid); - dc->ta = cpu_to_le64(val); - dc->fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + if (ep->pasid_enabled) { + ep->pc[0].ta = cpu_to_le64(val | RISCV_IOMMU_PC_TA_V); + ep->pc[0].fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + dc->ta = 0; + dc->fsc = cpu_to_le64(virt_to_pfn(ep->pc) | + FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8)); + } else { + dc->ta = cpu_to_le64(val); + dc->fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + } wmb(); /* Mark device context as valid, synchronise device context cache. */ val = RISCV_IOMMU_DC_TC_V; + if (ep->pasid_enabled) { + val |= RISCV_IOMMU_DC_TC_EN_ATS | + RISCV_IOMMU_DC_TC_EN_PRI | + RISCV_IOMMU_DC_TC_DPE | + RISCV_IOMMU_DC_TC_PDTV; + } + if (ep->iommu->cap & RISCV_IOMMU_CAP_AMO) { val |= RISCV_IOMMU_DC_TC_GADE | RISCV_IOMMU_DC_TC_SADE; @@ -1051,13 +1508,107 @@ static int riscv_iommu_attach_dev(struct iommu_domain *iommu_domain, struct devi return 0; } +static int riscv_iommu_set_dev_pasid(struct iommu_domain *iommu_domain, + struct device *dev, ioasid_t pasid) +{ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + u64 ta, fsc; + + if (!iommu_domain || !iommu_domain->mm) + return -EINVAL; + + /* Driver uses TC.DPE mode, PASID #0 is incorrect. */ + if (pasid == 0) + return -EINVAL; + + /* Incorrect domain identifier */ + if ((int)domain->pscid < 0) + return -ENOMEM; + + /* Process Context table should be set for pasid enabled endpoints. */ + if (!ep || !ep->pasid_enabled || !ep->dc || !ep->pc) + return -ENODEV; + + domain->pasid = pasid; + domain->iommu = ep->iommu; + domain->mn.ops = &riscv_iommu_mmuops; + + /* register mm notifier */ + if (mmu_notifier_register(&domain->mn, iommu_domain->mm)) + return -ENODEV; + + /* TODO: get SXL value for the process, use 32 bit or SATP mode */ + fsc = virt_to_pfn(iommu_domain->mm->pgd) | satp_mode; + ta = RISCV_IOMMU_PC_TA_V | FIELD_PREP(RISCV_IOMMU_PC_TA_PSCID, domain->pscid); + + fsc = le64_to_cpu(xchg_relaxed(&(ep->pc[pasid].fsc), cpu_to_le64(fsc))); + ta = le64_to_cpu(xchg_relaxed(&(ep->pc[pasid].ta), cpu_to_le64(ta))); + + wmb(); + + if (ta & RISCV_IOMMU_PC_TA_V) { + riscv_iommu_iodir_inv_pasid(ep->iommu, ep->devid, pasid); + riscv_iommu_iofence_sync(ep->iommu); + } + + dev_info(dev, "domain type %d attached w/ PSCID %u PASID %u\n", + domain->domain.type, domain->pscid, domain->pasid); + + return 0; +} + +static void riscv_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) +{ + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + struct riscv_iommu_command cmd; + unsigned long payload = riscv_iommu_ats_inval_all_payload(false); + u64 ta; + + /* invalidate TA.V */ + ta = le64_to_cpu(xchg_relaxed(&(ep->pc[pasid].ta), 0)); + + wmb(); + + dev_info(dev, "domain removed w/ PSCID %u PASID %u\n", + (unsigned)FIELD_GET(RISCV_IOMMU_PC_TA_PSCID, ta), pasid); + + /* 1. invalidate PDT entry */ + riscv_iommu_iodir_inv_pasid(ep->iommu, ep->devid, pasid); + + /* 2. invalidate all matching IOATC entries (if PASID was valid) */ + if (ta & RISCV_IOMMU_PC_TA_V) { + riscv_iommu_cmd_inval_vma(&cmd); + riscv_iommu_cmd_inval_set_gscid(&cmd, 0); + riscv_iommu_cmd_inval_set_pscid(&cmd, + FIELD_GET(RISCV_IOMMU_PC_TA_PSCID, ta)); + riscv_iommu_post(ep->iommu, &cmd); + } + + /* 3. Wait IOATC flush to happen */ + riscv_iommu_iofence_sync(ep->iommu); + + /* 4. ATS invalidation */ + riscv_iommu_cmd_ats_inval(&cmd); + riscv_iommu_cmd_ats_set_dseg(&cmd, ep->domid); + riscv_iommu_cmd_ats_set_rid(&cmd, ep->devid); + riscv_iommu_cmd_ats_set_pid(&cmd, pasid); + riscv_iommu_cmd_ats_set_payload(&cmd, payload); + riscv_iommu_post(ep->iommu, &cmd); + + /* 5. Wait DevATC flush to happen */ + riscv_iommu_iofence_sync(ep->iommu); +} + static void riscv_iommu_flush_iotlb_range(struct iommu_domain *iommu_domain, unsigned long *start, unsigned long *end, size_t *pgsize) { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); struct riscv_iommu_command cmd; + struct riscv_iommu_endpoint *endpoint; unsigned long iova; + unsigned long payload; if (domain->mode == RISCV_IOMMU_DC_FSC_MODE_BARE) return; @@ -1065,6 +1616,12 @@ static void riscv_iommu_flush_iotlb_range(struct iommu_domain *iommu_domain, /* Domain not attached to an IOMMU! */ BUG_ON(!domain->iommu); + if (start && end) { + payload = riscv_iommu_ats_inval_payload(*start, *end, true); + } else { + payload = riscv_iommu_ats_inval_all_payload(true); + } + riscv_iommu_cmd_inval_vma(&cmd); riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid); @@ -1078,6 +1635,20 @@ static void riscv_iommu_flush_iotlb_range(struct iommu_domain *iommu_domain, riscv_iommu_post(domain->iommu, &cmd); } riscv_iommu_iofence_sync(domain->iommu); + + /* ATS invalidation for every device and for every translation */ + list_for_each_entry(endpoint, &domain->endpoints, domain) { + if (!endpoint->pasid_enabled) + continue; + + riscv_iommu_cmd_ats_inval(&cmd); + riscv_iommu_cmd_ats_set_dseg(&cmd, endpoint->domid); + riscv_iommu_cmd_ats_set_rid(&cmd, endpoint->devid); + riscv_iommu_cmd_ats_set_pid(&cmd, domain->pasid); + riscv_iommu_cmd_ats_set_payload(&cmd, payload); + riscv_iommu_post(domain->iommu, &cmd); + } + riscv_iommu_iofence_sync(domain->iommu); } static void riscv_iommu_flush_iotlb_all(struct iommu_domain *iommu_domain) @@ -1310,6 +1881,7 @@ static int riscv_iommu_enable(struct riscv_iommu_device *iommu, unsigned request static const struct iommu_domain_ops riscv_iommu_domain_ops = { .free = riscv_iommu_domain_free, .attach_dev = riscv_iommu_attach_dev, + .set_dev_pasid = riscv_iommu_set_dev_pasid, .map_pages = riscv_iommu_map_pages, .unmap_pages = riscv_iommu_unmap_pages, .iova_to_phys = riscv_iommu_iova_to_phys, @@ -1326,9 +1898,13 @@ static const struct iommu_ops riscv_iommu_ops = { .probe_device = riscv_iommu_probe_device, .probe_finalize = riscv_iommu_probe_finalize, .release_device = riscv_iommu_release_device, + .remove_dev_pasid = riscv_iommu_remove_dev_pasid, .device_group = riscv_iommu_device_group, .get_resv_regions = riscv_iommu_get_resv_regions, .of_xlate = riscv_iommu_of_xlate, + .dev_enable_feat = riscv_iommu_dev_enable_feat, + .dev_disable_feat = riscv_iommu_dev_disable_feat, + .page_response = riscv_iommu_page_response, .default_domain_ops = &riscv_iommu_domain_ops, }; @@ -1340,6 +1916,7 @@ void riscv_iommu_remove(struct riscv_iommu_device *iommu) riscv_iommu_queue_free(iommu, &iommu->cmdq); riscv_iommu_queue_free(iommu, &iommu->fltq); riscv_iommu_queue_free(iommu, &iommu->priq); + iopf_queue_free(iommu->pq_work); } int riscv_iommu_init(struct riscv_iommu_device *iommu) @@ -1362,6 +1939,12 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) } #endif + if (iommu->cap & RISCV_IOMMU_CAP_PD20) + iommu->iommu.max_pasids = 1u << 20; + else if (iommu->cap & RISCV_IOMMU_CAP_PD17) + iommu->iommu.max_pasids = 1u << 17; + else if (iommu->cap & RISCV_IOMMU_CAP_PD8) + iommu->iommu.max_pasids = 1u << 8; /* * Assign queue lengths from module parameters if not already * set on the device tree. @@ -1387,6 +1970,13 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) goto fail; if (!(iommu->cap & RISCV_IOMMU_CAP_ATS)) goto no_ats; + /* PRI functionally depends on ATS’s capabilities. */ + iommu->pq_work = iopf_queue_alloc(dev_name(dev)); + if (!iommu->pq_work) { + dev_err(dev, "failed to allocate iopf queue\n"); + ret = -ENOMEM; + goto fail; + } ret = riscv_iommu_queue_init(iommu, RISCV_IOMMU_PAGE_REQUEST_QUEUE); if (ret) @@ -1424,5 +2014,6 @@ int riscv_iommu_init(struct riscv_iommu_device *iommu) riscv_iommu_queue_free(iommu, &iommu->priq); riscv_iommu_queue_free(iommu, &iommu->fltq); riscv_iommu_queue_free(iommu, &iommu->cmdq); + iopf_queue_free(iommu->pq_work); return ret; } diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index fe32a4eff14e..83e8d00fd0f8 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -17,9 +17,11 @@ #include #include #include +#include #include #include #include +#include #include "iommu-bits.h" @@ -76,6 +78,9 @@ struct riscv_iommu_device { unsigned ddt_mode; bool ddtp_in_iomem; + /* I/O page fault queue */ + struct iopf_queue *pq_work; + /* hardware queues */ struct riscv_iommu_queue cmdq; struct riscv_iommu_queue fltq; @@ -91,11 +96,14 @@ struct riscv_iommu_domain { struct io_pgtable pgtbl; struct list_head endpoints; + struct list_head notifiers; struct mutex lock; + struct mmu_notifier mn; struct riscv_iommu_device *iommu; unsigned mode; /* RIO_ATP_MODE_* enum */ unsigned pscid; /* RISC-V IOMMU PSCID */ + ioasid_t pasid; /* IOMMU_DOMAIN_SVA: Cached PASID */ pgd_t *pgd_root; /* page table root pointer */ }; @@ -107,10 +115,16 @@ struct riscv_iommu_endpoint { unsigned domid; /* PCI domain number, segment */ struct rb_node node; /* device tracking node (lookup by devid) */ struct riscv_iommu_dc *dc; /* device context pointer */ + struct riscv_iommu_pc *pc; /* process context root, valid if pasid_enabled is true */ struct riscv_iommu_device *iommu; /* parent iommu device */ struct mutex lock; struct list_head domain; /* endpoint attached managed domain */ + + /* end point info bits */ + unsigned pasid_bits; + unsigned pasid_feat; + bool pasid_enabled; }; /* Helper functions and macros */ From patchwork Wed Jul 19 19:33:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2F24C001DC for ; Wed, 19 Jul 2023 19:34:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sOYJB7aEM/QJl26jiwNSAYBj+KuVa7w6qqTgueqLEm8=; b=e9dzWCePuDh/Bk 7DwsM//7frZCloyd47iL+CUwb/Qa+jGeaQCVrUXkJfik2UiHTN+ANd6sobc9Y8givoX+qZssUKtRb GVxoEpl58a7OFTgdFI2AeylTgaKMpfNsIzVCU5ew4NN6B0OkrMIbzEaM6xWHEv5UMf4c8I7T71AJG X4dyjRfeyxms2CQQn89OrqAsLJpCXgVZt1o+chOdu2GzEr93khtfdC5gkJolmoIOCYNQJfz89dam6 eAU+goOFz6Kc4sH2MKhpEQ4BycydjakDaRzCs9gN1ls9py2I3r/TzHl2PoDndGceXTgAhLGPjF/C5 rFH9oClyJjPehMXfK0ZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwg-008hGQ-0h; Wed, 19 Jul 2023 19:34:46 +0000 Received: from mail-pj1-x1035.google.com ([2607:f8b0:4864:20::1035]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwX-008h9O-1t for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:39 +0000 Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-263121cd04eso4016127a91.2 for ; Wed, 19 Jul 2023 12:34:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795276; x=1690400076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PUXvXbLuP8wDy1OSCml3bGjGEs0jCX+1NqTNk6KKpl8=; b=yJG+r2hMaEM+aliN8AVg0otRKaB4Z8eB26rded4dmVlMEf7QPzs7oXMfpQmJMKL++k pDvxPqx/ZqxGapQ8WSH3D2kxHKnBm3x9Q5Nio4OahuUSf/B/GgPdLMDrg7z5Sq1B3ikz UeH/z3vVIqFc2rMrJ9TpbVfnlAuwcYXAo6tCGKDkTZV8bY/H0jQvKdPnhjLxXacZpTy/ iW900N9T4LgH9r16t39SqEYpRVL8dfRWtJHsro1fR6D0IcienUrOo4o6hxfggu3gqr6G 4XvpBJFV2BHbmMGsUPaauELDeBsOihKwyQFkYQ1vNuTpXydFhJ5707WI9dxod6GTdPZl oKJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795276; x=1690400076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PUXvXbLuP8wDy1OSCml3bGjGEs0jCX+1NqTNk6KKpl8=; b=DdN1OxMVQ8tKocCxNw0Ud/hGwp4q4gVREY2wcy7z4Qw6Zqc3ujkJxBW6Ih6C/LHlc4 TgOLIjVskhCsX9+/o35tjKQkTKAeOjK//skGsYzpQX+mnHEP/+t/PMS0hTuUtqjKK1/o qLqn9JbaKx6gAcKi78BqeU5/sYdp9nDQXQNnZEyG6yTG/KDujQyTNMnH6L4gtlsJ93as TSGFHJOfSVo8w1xQ0tg9YmxKJHAlEnvBJhkF+jvegDHX8Da2IFbOI+Msgbzefo5xlmi0 ydgceueMgxsAduAwXwsAwxq3Z+jRGuaillj1DCtpTxRWlTF5Fl4bGQxF+LB2cMWfG/lf Jfyw== X-Gm-Message-State: ABy/qLZbUI7qtGHAK/9U8ud/v+rEWXRSK0ecKLv7ePaRAmLj/mTl7CLX 3QKr1C4C8hXOhKP6ZCFIlg1FXw== X-Google-Smtp-Source: APBJJlGbDD5aTGEi40JcY4RNgasjfgy48ieoko/t1uTH9H4BvKdLkbsHdSPwZCOuiw6bztkQOdloJA== X-Received: by 2002:a17:90a:aa04:b0:263:f776:8ba3 with SMTP id k4-20020a17090aaa0400b00263f7768ba3mr197600pjq.9.1689795276298; Wed, 19 Jul 2023 12:34:36 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:35 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 10/11] RISC-V: drivers/iommu/riscv: Add MSI identity remapping Date: Wed, 19 Jul 2023 12:33:54 -0700 Message-Id: <660b7a8707e494a6bb2706e10569a7414c3640a7.1689792825.git.tjeznach@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123437_649505_B1606EC5 X-CRM114-Status: GOOD ( 18.76 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This change provides basic identity mapping support to excercise MSI_FLAT hardware capability. Signed-off-by: Tomasz Jeznach --- drivers/iommu/riscv/iommu.c | 81 +++++++++++++++++++++++++++++++++++++ drivers/iommu/riscv/iommu.h | 3 ++ 2 files changed, 84 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 6042c35be3ca..7b3e3e135cf6 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -61,6 +61,9 @@ MODULE_PARM_DESC(priq_length, "Page request interface queue length."); #define RISCV_IOMMU_MAX_PSCID (1U << 20) static DEFINE_IDA(riscv_iommu_pscids); +/* TODO: Enable MSI remapping */ +#define RISCV_IMSIC_BASE 0x28000000 + /* 1 second */ #define RISCV_IOMMU_TIMEOUT riscv_timebase @@ -932,6 +935,72 @@ static irqreturn_t riscv_iommu_priq_process(int irq, void *data) * Endpoint management */ +static int riscv_iommu_enable_ir(struct riscv_iommu_endpoint *ep) +{ + struct riscv_iommu_device *iommu = ep->iommu; + struct iommu_resv_region *entry; + struct irq_domain *msi_domain; + u64 val; + int i; + + /* Initialize MSI remapping */ + if (!ep->dc || !(iommu->cap & RISCV_IOMMU_CAP_MSI_FLAT)) + return 0; + + ep->msi_root = (struct riscv_iommu_msi_pte *)get_zeroed_page(GFP_KERNEL); + if (!ep->msi_root) + return -ENOMEM; + + for (i = 0; i < 256; i++) { + ep->msi_root[i].pte = RISCV_IOMMU_MSI_PTE_V | + FIELD_PREP(RISCV_IOMMU_MSI_PTE_M, 3) | + phys_to_ppn(RISCV_IMSIC_BASE + i * PAGE_SIZE); + } + + entry = iommu_alloc_resv_region(RISCV_IMSIC_BASE, PAGE_SIZE * 256, 0, + IOMMU_RESV_SW_MSI, GFP_KERNEL); + if (entry) + list_add_tail(&entry->list, &ep->regions); + + val = virt_to_pfn(ep->msi_root) | + FIELD_PREP(RISCV_IOMMU_DC_MSIPTP_MODE, RISCV_IOMMU_DC_MSIPTP_MODE_FLAT); + ep->dc->msiptp = cpu_to_le64(val); + + /* Single page of MSIPTP, 256 IMSIC files */ + ep->dc->msi_addr_mask = cpu_to_le64(255); + ep->dc->msi_addr_pattern = cpu_to_le64(RISCV_IMSIC_BASE >> 12); + wmb(); + + /* set msi domain for the device as isolated. hack. */ + msi_domain = dev_get_msi_domain(ep->dev); + if (msi_domain) { + msi_domain->flags |= IRQ_DOMAIN_FLAG_ISOLATED_MSI; + } + + dev_dbg(ep->dev, "RV-IR enabled\n"); + + ep->ir_enabled = true; + + return 0; +} + +static void riscv_iommu_disable_ir(struct riscv_iommu_endpoint *ep) +{ + if (!ep->ir_enabled) + return; + + ep->dc->msi_addr_pattern = 0ULL; + ep->dc->msi_addr_mask = 0ULL; + ep->dc->msiptp = 0ULL; + wmb(); + + dev_dbg(ep->dev, "RV-IR disabled\n"); + + free_pages((unsigned long)ep->msi_root, 0); + ep->msi_root = NULL; + ep->ir_enabled = false; +} + /* Endpoint features/capabilities */ static void riscv_iommu_disable_ep(struct riscv_iommu_endpoint *ep) { @@ -1226,6 +1295,7 @@ static struct iommu_device *riscv_iommu_probe_device(struct device *dev) mutex_init(&ep->lock); INIT_LIST_HEAD(&ep->domain); + INIT_LIST_HEAD(&ep->regions); if (dev_is_pci(dev)) { ep->devid = pci_dev_id(to_pci_dev(dev)); @@ -1248,6 +1318,7 @@ static struct iommu_device *riscv_iommu_probe_device(struct device *dev) dev_iommu_priv_set(dev, ep); riscv_iommu_add_device(iommu, dev); riscv_iommu_enable_ep(ep); + riscv_iommu_enable_ir(ep); return &iommu->iommu; } @@ -1279,6 +1350,7 @@ static void riscv_iommu_release_device(struct device *dev) riscv_iommu_iodir_inv_devid(iommu, ep->devid); } + riscv_iommu_disable_ir(ep); riscv_iommu_disable_ep(ep); /* Remove endpoint from IOMMU tracking structures */ @@ -1301,6 +1373,15 @@ static struct iommu_group *riscv_iommu_device_group(struct device *dev) static void riscv_iommu_get_resv_regions(struct device *dev, struct list_head *head) { + struct iommu_resv_region *entry, *new_entry; + struct riscv_iommu_endpoint *ep = dev_iommu_priv_get(dev); + + list_for_each_entry(entry, &ep->regions, list) { + new_entry = kmemdup(entry, sizeof(*entry), GFP_KERNEL); + if (new_entry) + list_add_tail(&new_entry->list, head); + } + iommu_dma_get_resv_regions(dev, head); } diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 83e8d00fd0f8..55418a1144fb 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -117,14 +117,17 @@ struct riscv_iommu_endpoint { struct riscv_iommu_dc *dc; /* device context pointer */ struct riscv_iommu_pc *pc; /* process context root, valid if pasid_enabled is true */ struct riscv_iommu_device *iommu; /* parent iommu device */ + struct riscv_iommu_msi_pte *msi_root; /* interrupt re-mapping */ struct mutex lock; struct list_head domain; /* endpoint attached managed domain */ + struct list_head regions; /* reserved regions, interrupt remapping window */ /* end point info bits */ unsigned pasid_bits; unsigned pasid_feat; bool pasid_enabled; + bool ir_enabled; }; /* Helper functions and macros */ From patchwork Wed Jul 19 19:33:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Jeznach X-Patchwork-Id: 13319432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EAF53C001B0 for ; Wed, 19 Jul 2023 19:34:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2AqZtNXb0BJjiV5MWiBoXEXyQWpjTxNY2Nm5kaIVIBw=; b=gmbhxgHdKXfpe4 tz/4ufzBhVbRvWW51MBTjBexJckVH2xMmOeVlqZ3m+3yPcitNkPZ11YBtjlR3nXSzR0bGJrN5pvEQ Zhz8r3f1dDqazL9Uz07dZXGhWMVacpBFlR3kyJzQBBcOhrVUVhj27UblTNqgp8LHTJrQFdY6UzLtn GmSrbwhUKxu4CE8/Oe8i+0f/IBN4/lulodearWAXR6cTdpoI6l2x1MQTJtlTaicOzNXX6SYUbRdpO yPQxi2vrxobVrKKY47CeQrnnzjrIftMBszMLsbdiM4GuhvpRtBE4fzGUvu7WV7X9aVVQho9A3OQn4 qfru0X6oLG2ULjMYx0Rw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwi-008hIJ-2d; Wed, 19 Jul 2023 19:34:49 +0000 Received: from mail-il1-x130.google.com ([2607:f8b0:4864:20::130]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMCwZ-008hAH-1a for linux-riscv@lists.infradead.org; Wed, 19 Jul 2023 19:34:41 +0000 Received: by mail-il1-x130.google.com with SMTP id e9e14a558f8ab-345df4e68afso556535ab.0 for ; Wed, 19 Jul 2023 12:34:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689795277; x=1690400077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cQtL8/41Z1V1hiq79fEomhJLs5bJqMNwqo2oB2sFh+M=; b=N+rgicSBNL326QCmnjhUOfpQ8izFbGdkgaThZ9XQu4Mk7iD0IHeKkXdtZ5VmwkR5S/ Rha2e6WKVkIXnSSSPhpJB5kwm5JJ8K6GHWb1/fSWu2MjSn1i8oKOPgc1WYSEqaXjg0bt BFjczOR5ZKrHW0axat2rmuT9KwZFaMslcXz3HWujeiLsrx5CKDzH7WulClxeSbbBaLMP BtB6PO2ogDbNr5NDZKO/s7PS8iWesDpwiw1+8ebka5fPlOPD45mP8UUxpsidA+uJ2WNV 00rqY9w3Bf+YceD4mU4wYnXuoqe2R9l40eJtVpzgc63P1VjrzeYo+3tJpu1Ir2bR9L50 KhPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689795277; x=1690400077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cQtL8/41Z1V1hiq79fEomhJLs5bJqMNwqo2oB2sFh+M=; b=T7tnaEje4FpQfJV2CofGdxLZYuN37IXEDnonNAkUWEru90Jhlbl2qPw6qTf6uRqcky eDFRA9fu2OdGH2BkEn+/R3CCsU1Gtrg1L+SNvYCjIyrvRBQ7HBAE9oMBViaRI+ltrVeh VF+S9F+Qz9FVzxtCvEIkbQMHvCvpOewTNQearFIw5wY7APfmH/2b8abY5D50BGNoddDw XfTo0vHQ5xdeY0ODAlIExXKVPg4RgY2h3w0HVpnUw6jle2IfF2wwkh3no7RLbmQ9wpDF 89ninBLdljW3U6UNBaGP7gicLItwYYelNCJ4Qu748KoBrKVUyRZC/z1aNcLTIzLGJ97E ouZA== X-Gm-Message-State: ABy/qLZH8QTK87Re8f3RM7hktvkopaBfHm1dmfKU/3S87Pn4h2Ft1+2s 8XLI441TRkUtzK3/LDMyYGkNJA== X-Google-Smtp-Source: APBJJlFzjlxmwn6yHr60YRnzS6QP9ZKpMDICdaPBklxkA0AGmYtcK5/R2NpPUsB4j8EU4mV73Zv3TQ== X-Received: by 2002:a05:6e02:1c8f:b0:345:6c70:a7cd with SMTP id w15-20020a056e021c8f00b003456c70a7cdmr3753958ill.1.1689795277679; Wed, 19 Jul 2023 12:34:37 -0700 (PDT) Received: from tjeznach.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id 23-20020a17090a031700b00264040322desm1591053pje.40.2023.07.19.12.34.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 12:34:37 -0700 (PDT) From: Tomasz Jeznach To: Joerg Roedel , Will Deacon , Robin Murphy , Paul Walmsley Subject: [PATCH 11/11] RISC-V: drivers/iommu/riscv: Add G-Stage translation support Date: Wed, 19 Jul 2023 12:33:55 -0700 Message-Id: <0c391072fe0be52b3bdf3d826e4313d960aecba0.1689792825.git.tjeznach@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230719_123439_536979_495F7025 X-CRM114-Status: GOOD ( 17.64 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Albert Ou , Tomasz Jeznach , linux@rivosinc.com, linux-kernel@vger.kernel.org, Sebastien Boeuf , iommu@lists.linux.dev, Palmer Dabbelt , Nick Kossifidis , linux-riscv@lists.infradead.org Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This change introduces 2nd stage translation configuration support, enabling nested translation for IOMMU hardware. Pending integration with VMM IOMMUFD interfaces to manage 1st stage translation and IOMMU virtialization interfaces. Signed-off-by: Tomasz Jeznach --- drivers/iommu/riscv/iommu.c | 58 ++++++++++++++++++++++++++++--------- drivers/iommu/riscv/iommu.h | 3 +- 2 files changed, 46 insertions(+), 15 deletions(-) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 7b3e3e135cf6..3ca2f0194d3c 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -1418,6 +1418,19 @@ static struct iommu_domain *riscv_iommu_domain_alloc(unsigned type) return &domain->domain; } +/* mark domain as second-stage translation */ +static int riscv_iommu_enable_nesting(struct iommu_domain *iommu_domain) +{ + struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); + + mutex_lock(&domain->lock); + if (list_empty(&domain->endpoints)) + domain->g_stage = true; + mutex_unlock(&domain->lock); + + return domain->g_stage ? 0 : -EBUSY; +} + static void riscv_iommu_domain_free(struct iommu_domain *iommu_domain) { struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain); @@ -1433,7 +1446,7 @@ static void riscv_iommu_domain_free(struct iommu_domain *iommu_domain) free_io_pgtable_ops(&domain->pgtbl.ops); if (domain->pgd_root) - free_pages((unsigned long)domain->pgd_root, 0); + free_pages((unsigned long)domain->pgd_root, domain->g_stage ? 2 : 0); if ((int)domain->pscid > 0) ida_free(&riscv_iommu_pscids, domain->pscid); @@ -1483,7 +1496,8 @@ static int riscv_iommu_domain_finalize(struct riscv_iommu_domain *domain, /* TODO: Fix this for RV32 */ domain->mode = satp_mode >> 60; - domain->pgd_root = (pgd_t *) __get_free_pages(GFP_KERNEL | __GFP_ZERO, 0); + domain->pgd_root = (pgd_t *) __get_free_pages(GFP_KERNEL | __GFP_ZERO, + domain->g_stage ? 2 : 0); if (!domain->pgd_root) return -ENOMEM; @@ -1499,6 +1513,8 @@ static u64 riscv_iommu_domain_atp(struct riscv_iommu_domain *domain) u64 atp = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, domain->mode); if (domain->mode != RISCV_IOMMU_DC_FSC_MODE_BARE) atp |= FIELD_PREP(RISCV_IOMMU_DC_FSC_PPN, virt_to_pfn(domain->pgd_root)); + if (domain->g_stage) + atp |= FIELD_PREP(RISCV_IOMMU_DC_IOHGATP_GSCID, domain->pscid); return atp; } @@ -1541,20 +1557,30 @@ static int riscv_iommu_attach_dev(struct iommu_domain *iommu_domain, struct devi if (!dc) return -ENODEV; - /* - * S-Stage translation table. G-Stage remains unmodified (BARE). - */ - val = FIELD_PREP(RISCV_IOMMU_DC_TA_PSCID, domain->pscid); - - if (ep->pasid_enabled) { - ep->pc[0].ta = cpu_to_le64(val | RISCV_IOMMU_PC_TA_V); - ep->pc[0].fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + if (domain->g_stage) { + /* + * Enable G-Stage translation with initial pass-through mode + * for S-Stage. VMM is responsible for more restrictive + * guest VA translation scheme configuration. + */ dc->ta = 0; - dc->fsc = cpu_to_le64(virt_to_pfn(ep->pc) | - FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8)); + dc->fsc = 0ULL; /* RISCV_IOMMU_DC_FSC_MODE_BARE */ ; + dc->iohgatp = cpu_to_le64(riscv_iommu_domain_atp(domain)); } else { - dc->ta = cpu_to_le64(val); - dc->fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + /* S-Stage translation table. G-Stage remains unmodified. */ + if (ep->pasid_enabled) { + val = FIELD_PREP(RISCV_IOMMU_DC_TA_PSCID, domain->pscid); + ep->pc[0].ta = cpu_to_le64(val | RISCV_IOMMU_PC_TA_V); + ep->pc[0].fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + dc->ta = 0; + val = FIELD_PREP(RISCV_IOMMU_DC_FSC_MODE, + RISCV_IOMMU_DC_FSC_PDTP_MODE_PD8); + dc->fsc = cpu_to_le64(val | virt_to_pfn(ep->pc)); + } else { + val = FIELD_PREP(RISCV_IOMMU_DC_TA_PSCID, domain->pscid); + dc->ta = cpu_to_le64(val); + dc->fsc = cpu_to_le64(riscv_iommu_domain_atp(domain)); + } } wmb(); @@ -1599,6 +1625,9 @@ static int riscv_iommu_set_dev_pasid(struct iommu_domain *iommu_domain, if (!iommu_domain || !iommu_domain->mm) return -EINVAL; + if (domain->g_stage) + return -EINVAL; + /* Driver uses TC.DPE mode, PASID #0 is incorrect. */ if (pasid == 0) return -EINVAL; @@ -1969,6 +1998,7 @@ static const struct iommu_domain_ops riscv_iommu_domain_ops = { .iotlb_sync = riscv_iommu_iotlb_sync, .iotlb_sync_map = riscv_iommu_iotlb_sync_map, .flush_iotlb_all = riscv_iommu_flush_iotlb_all, + .enable_nesting = riscv_iommu_enable_nesting, }; static const struct iommu_ops riscv_iommu_ops = { diff --git a/drivers/iommu/riscv/iommu.h b/drivers/iommu/riscv/iommu.h index 55418a1144fb..55e5aafea5bc 100644 --- a/drivers/iommu/riscv/iommu.h +++ b/drivers/iommu/riscv/iommu.h @@ -102,8 +102,9 @@ struct riscv_iommu_domain { struct riscv_iommu_device *iommu; unsigned mode; /* RIO_ATP_MODE_* enum */ - unsigned pscid; /* RISC-V IOMMU PSCID */ + unsigned pscid; /* RISC-V IOMMU PSCID / GSCID */ ioasid_t pasid; /* IOMMU_DOMAIN_SVA: Cached PASID */ + bool g_stage; /* 2nd stage translation domain */ pgd_t *pgd_root; /* page table root pointer */ };