From patchwork Fri Apr 23 09:51:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Kumar Gautam X-Patchwork-Id: 12220029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B21BC433ED for ; Fri, 23 Apr 2021 09:55:32 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C1BF613CB for ; Fri, 23 Apr 2021 09:55:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C1BF613CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=dG/xiP60/vdgVCC2sa3TvRbrKdTpdkeOEH6nxHcH4JA=; b=LelHfmaomiG1rAHdc4skXehNs1 owQufP5R6PvDsBYmTmFSrCQXeGVqC67jMb+tfJYhuiWny84f6o6P9S+yGxbQrVV1/cP+y6l9FyMS5 wikucG2+L9J5gFXl6OwzM4A0sOlVtUabHqSxblSw4pYa6fHdkPbNXZzhUtlipymh/xvILTh+nG3Ow JYzS/R1fht8cngPBT0eMFUP+7HTFqZ1yRN8undn2A6TjFJGDyPQeTXLpvaewyP/Zr+manq/E4i3Ra HgKJ0ug938XrtAMldgoDkVTPVTrajewkjfab/TYar9hwYM3dHFYZv9vXaEmXKBB+0EZdbgykPKfzc 7XKffzbQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lZsVG-001DVp-Mz; Fri, 23 Apr 2021 09:53:39 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lZsUE-001DGO-WE for linux-arm-kernel@desiato.infradead.org; Fri, 23 Apr 2021 09:52:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description; bh=QYvt3gMGtjtge1RA+CB4KiGB3/g77ZX8A8fk4SNdQ34=; b=d++KYfrFo9MwQm6BaX5mHqAj1D 9DaJ7ZmzCJLYWYPXaZBRd0jWWBaCUnTPh3izl0DIpkcTGgHF4FvUV6tVTTXLLV6KDU+hQ691XorHO l4AtN+I64iTR5K1kBlV/QD49BmD2FbCBNhzoLE7eKIs3VfsBrYBksM7bsE4cr1ZFYucCjuINd1CDU 5XqYSjbPP4yQbeb42lzx8CgBczxD0tviS9ZGW5md1WdYcahRwBJAFiS6hIGd1srqVzH69/gJcNR4o 9BjX3EcIUmQD6lQ18bJdW7ZNGo2b0jag0fuSVPS1I3NtnbKt7bGijxiUXJ9yZP66c9tLG4QVmtskU 8kcbuVOg==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lZsUB-00EIiF-Mx for linux-arm-kernel@lists.infradead.org; Fri, 23 Apr 2021 09:52:33 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2AA06161B; Fri, 23 Apr 2021 02:52:31 -0700 (PDT) Received: from usa.arm.com (a074945.blr.arm.com [10.162.16.71]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 030173F774; Fri, 23 Apr 2021 02:52:26 -0700 (PDT) From: Vivek Gautam To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, virtualization@lists.linux-foundation.org Cc: joro@8bytes.org, will.deacon@arm.com, mst@redhat.com, robin.murphy@arm.com, jean-philippe@linaro.org, eric.auger@redhat.com, kevin.tian@intel.com, jacob.jun.pan@linux.intel.com, yi.l.liu@intel.com, Lorenzo.Pieralisi@arm.com, shameerali.kolothum.thodi@huawei.com, Vivek Gautam Subject: [PATCH RFC v1 07/11] iommu/arm-smmu-v3: Move shared context descriptor code to cd-lib Date: Fri, 23 Apr 2021 15:21:43 +0530 Message-Id: <20210423095147.27922-8-vivek.gautam@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210423095147.27922-1-vivek.gautam@arm.com> References: <20210423095147.27922-1-vivek.gautam@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210423_025231_868601_269E71CC X-CRM114-Status: GOOD ( 22.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation to adding shared descriptor allocation and free pasid table ops, pull out the alloc_shared_cd and free_shared_cd helpers from arm-smmu-v3 and place them in arm-smmu-v3-cd-lib library driver. Signed-off-by: Vivek Gautam --- .../arm/arm-smmu-v3/arm-smmu-v3-cd-lib.c | 149 ++++++++++++++++++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 147 ----------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 12 ++ drivers/iommu/iommu-pasid-table.h | 1 + 4 files changed, 162 insertions(+), 147 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-cd-lib.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-cd-lib.c index ea94f57ad261..537b7c784d40 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-cd-lib.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-cd-lib.c @@ -6,9 +6,11 @@ */ #include +#include #include "arm-smmu-v3.h" #include "../../iommu-pasid-table.h" +#include "../../io-pgtable-arm.h" static int arm_smmu_alloc_cd_leaf_table(struct device *dev, struct arm_smmu_l1_ctx_desc *l1_desc) @@ -278,6 +280,153 @@ static bool arm_smmu_free_asid(struct xarray *xa, void *cookie_cd) return free; } +/* + * Check if the CPU ASID is available on the SMMU side. If a private context + * descriptor is using it, try to replace it. + */ +static struct arm_smmu_ctx_desc * +arm_smmu_share_asid(struct mm_struct *mm, u16 asid) +{ + int ret; + u32 new_asid; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain; + struct iommu_pasid_table *tbl; + + cd = xa_load(&arm_smmu_asid_xa, asid); + if (!cd) + return NULL; + + if (cd->mm) { + if (WARN_ON(cd->mm != mm)) + return ERR_PTR(-EINVAL); + /* All devices bound to this mm use the same cd struct. */ + refcount_inc(&cd->refs); + return cd; + } + + smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu = smmu_domain->smmu; + tbl = smmu_domain->tbl; + + ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + if (ret) + return ERR_PTR(-ENOSPC); + /* + * Race with unmap: TLB invalidations will start targeting the new ASID, + * which isn't assigned yet. We'll do an invalidate-all on the old ASID + * later, so it doesn't matter. + */ + cd->asid = new_asid; + /* + * Update ASID and invalidate CD in all associated masters. There will + * be some overlap between use of both ASIDs, until we invalidate the + * TLB. + */ + ret = iommu_psdtable_write(tbl, &tbl->cfg, 0, cd); + if (ret) + return ERR_PTR(-ENOSYS); + + /* Invalidate TLB entries previously associated with that context */ + iommu_psdtable_flush_tlb(tbl, smmu_domain, asid); + + xa_erase(&arm_smmu_asid_xa, asid); + return NULL; +} + +struct arm_smmu_ctx_desc * +arm_smmu_alloc_shared_cd(struct iommu_pasid_table *tbl, struct mm_struct *mm) +{ + u16 asid; + int err = 0; + u64 tcr, par, reg; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_ctx_desc *ret = NULL; + + asid = arm64_mm_context_get(mm); + if (!asid) + return ERR_PTR(-ESRCH); + + cd = kzalloc(sizeof(*cd), GFP_KERNEL); + if (!cd) { + err = -ENOMEM; + goto out_put_context; + } + + refcount_set(&cd->refs, 1); + + mutex_lock(&arm_smmu_asid_lock); + ret = arm_smmu_share_asid(mm, asid); + if (ret) { + mutex_unlock(&arm_smmu_asid_lock); + goto out_free_cd; + } + + err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL); + mutex_unlock(&arm_smmu_asid_lock); + + if (err) + goto out_free_asid; + + tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) | + FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | + FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | + CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; + + switch (PAGE_SIZE) { + case SZ_4K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K); + break; + case SZ_16K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K); + break; + case SZ_64K: + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K); + break; + default: + WARN_ON(1); + err = -EINVAL; + goto out_free_asid; + } + + reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); + par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT); + tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par); + + cd->ttbr = virt_to_phys(mm->pgd); + cd->tcr = tcr; + /* + * MAIR value is pretty much constant and global, so we can just get it + * from the current CPU register + */ + cd->mair = read_sysreg(mair_el1); + cd->asid = asid; + cd->mm = mm; + + return cd; + +out_free_asid: + iommu_psdtable_free_asid(tbl, &arm_smmu_asid_xa, cd); +out_free_cd: + kfree(cd); +out_put_context: + arm64_mm_context_put(mm); + return err < 0 ? ERR_PTR(err) : ret; +} + +void arm_smmu_free_shared_cd(struct iommu_pasid_table *tbl, + struct arm_smmu_ctx_desc *cd) +{ + if (iommu_psdtable_free_asid(tbl, &arm_smmu_asid_xa, cd)) { + /* Unpin ASID */ + arm64_mm_context_put(cd->mm); + kfree(cd); + } +} + struct iommu_vendor_psdtable_ops arm_cd_table_ops = { .alloc = arm_smmu_alloc_cd_tables, .free = arm_smmu_free_cd_tables, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 775c34899935..da35d4cc0c1e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -37,153 +37,6 @@ struct arm_smmu_bond { static DEFINE_MUTEX(sva_lock); -/* - * Check if the CPU ASID is available on the SMMU side. If a private context - * descriptor is using it, try to replace it. - */ -static struct arm_smmu_ctx_desc * -arm_smmu_share_asid(struct mm_struct *mm, u16 asid) -{ - int ret; - u32 new_asid; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_device *smmu; - struct arm_smmu_domain *smmu_domain; - struct iommu_pasid_table *tbl; - - cd = xa_load(&arm_smmu_asid_xa, asid); - if (!cd) - return NULL; - - if (cd->mm) { - if (WARN_ON(cd->mm != mm)) - return ERR_PTR(-EINVAL); - /* All devices bound to this mm use the same cd struct. */ - refcount_inc(&cd->refs); - return cd; - } - - smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); - smmu = smmu_domain->smmu; - tbl = smmu_domain->tbl; - - ret = xa_alloc(&arm_smmu_asid_xa, &new_asid, cd, - XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); - if (ret) - return ERR_PTR(-ENOSPC); - /* - * Race with unmap: TLB invalidations will start targeting the new ASID, - * which isn't assigned yet. We'll do an invalidate-all on the old ASID - * later, so it doesn't matter. - */ - cd->asid = new_asid; - /* - * Update ASID and invalidate CD in all associated masters. There will - * be some overlap between use of both ASIDs, until we invalidate the - * TLB. - */ - ret = iommu_psdtable_write(tbl, &tbl->cfg, 0, cd); - if (ret) - return ERR_PTR(-ENOSYS); - - /* Invalidate TLB entries previously associated with that context */ - iommu_psdtable_flush_tlb(tbl, smmu_domain, asid); - - xa_erase(&arm_smmu_asid_xa, asid); - return NULL; -} - -static struct arm_smmu_ctx_desc * -arm_smmu_alloc_shared_cd(struct iommu_pasid_table *tbl, struct mm_struct *mm) -{ - u16 asid; - int err = 0; - u64 tcr, par, reg; - struct arm_smmu_ctx_desc *cd; - struct arm_smmu_ctx_desc *ret = NULL; - - asid = arm64_mm_context_get(mm); - if (!asid) - return ERR_PTR(-ESRCH); - - cd = kzalloc(sizeof(*cd), GFP_KERNEL); - if (!cd) { - err = -ENOMEM; - goto out_put_context; - } - - refcount_set(&cd->refs, 1); - - mutex_lock(&arm_smmu_asid_lock); - ret = arm_smmu_share_asid(mm, asid); - if (ret) { - mutex_unlock(&arm_smmu_asid_lock); - goto out_free_cd; - } - - err = xa_insert(&arm_smmu_asid_xa, asid, cd, GFP_KERNEL); - mutex_unlock(&arm_smmu_asid_lock); - - if (err) - goto out_free_asid; - - tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - vabits_actual) | - FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | - FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | - CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; - - switch (PAGE_SIZE) { - case SZ_4K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_4K); - break; - case SZ_16K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_16K); - break; - case SZ_64K: - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_TG0, ARM_LPAE_TCR_TG0_64K); - break; - default: - WARN_ON(1); - err = -EINVAL; - goto out_free_asid; - } - - reg = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - par = cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR0_PARANGE_SHIFT); - tcr |= FIELD_PREP(CTXDESC_CD_0_TCR_IPS, par); - - cd->ttbr = virt_to_phys(mm->pgd); - cd->tcr = tcr; - /* - * MAIR value is pretty much constant and global, so we can just get it - * from the current CPU register - */ - cd->mair = read_sysreg(mair_el1); - cd->asid = asid; - cd->mm = mm; - - return cd; - -out_free_asid: - iommu_psdtable_free_asid(tbl, &arm_smmu_asid_xa, cd); -out_free_cd: - kfree(cd); -out_put_context: - arm64_mm_context_put(mm); - return err < 0 ? ERR_PTR(err) : ret; -} - -static void arm_smmu_free_shared_cd(struct iommu_pasid_table *tbl, - struct arm_smmu_ctx_desc *cd) -{ - if (iommu_psdtable_free_asid(tbl, &arm_smmu_asid_xa, cd)) { - /* Unpin ASID */ - arm64_mm_context_put(cd->mm); - kfree(cd); - } -} - static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index d5cf2d521c29..13ff024ab0d5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -773,6 +773,10 @@ struct iommu_sva *arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void arm_smmu_sva_unbind(struct iommu_sva *handle); u32 arm_smmu_sva_get_pasid(struct iommu_sva *handle); void arm_smmu_sva_notifier_synchronize(void); +struct arm_smmu_ctx_desc * +arm_smmu_alloc_shared_cd(struct iommu_pasid_table *tbl, struct mm_struct *mm); +void arm_smmu_free_shared_cd(struct iommu_pasid_table *tbl, + struct arm_smmu_ctx_desc *cd); #else /* CONFIG_ARM_SMMU_V3_SVA */ static inline bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) { @@ -828,5 +832,13 @@ static inline u32 arm_smmu_sva_get_pasid(struct iommu_sva *handle) } static inline void arm_smmu_sva_notifier_synchronize(void) {} +struct arm_smmu_ctx_desc * +arm_smmu_alloc_shared_cd(struct iommu_pasid_table *tbl, struct mm_struct *mm) +{ + return ERR_PTR(-EINVAL); +} + +static inline void arm_smmu_free_shared_cd(struct iommu_pasid_table *tbl, + struct arm_smmu_ctx_desc *cd) {} #endif /* CONFIG_ARM_SMMU_V3_SVA */ #endif /* _ARM_SMMU_V3_H */ diff --git a/drivers/iommu/iommu-pasid-table.h b/drivers/iommu/iommu-pasid-table.h index 4d6590e60f9b..0e774ae6682a 100644 --- a/drivers/iommu/iommu-pasid-table.h +++ b/drivers/iommu/iommu-pasid-table.h @@ -8,6 +8,7 @@ #define __IOMMU_PASID_TABLE_H #include +#include #include #include "arm/arm-smmu-v3/arm-smmu-v3.h"