From patchwork Wed Feb 21 22:59:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Crouse X-Patchwork-Id: 10234273 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 21A2E602A0 for ; Wed, 21 Feb 2018 23:00:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1535B285FC for ; Wed, 21 Feb 2018 23:00:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0585E28534; Wed, 21 Feb 2018 23:00:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3480228534 for ; Wed, 21 Feb 2018 23:00:09 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4545D6EAD0; Wed, 21 Feb 2018 22:59:46 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from smtp.codeaurora.org (smtp.codeaurora.org [198.145.29.96]) by gabe.freedesktop.org (Postfix) with ESMTPS id 217236EACA; Wed, 21 Feb 2018 22:59:44 +0000 (UTC) Received: by smtp.codeaurora.org (Postfix, from userid 1000) id C0C1060FA9; Wed, 21 Feb 2018 22:59:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1519253983; bh=NGrFl2HYLgvb2U901x5D4Xsy2sKi6E+Y/6L0rB0PJWk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TIZfPZxFOqdgPQ8yjTvkFfQQzxHdA41B1eACc3+LqcWYC9tQl7SqqODJC2Z6fGSAG +mpejnlc93ROxY8OaEmGkFaPjQ6UH+XewbAzhd+5XjM7x7LM8kmKgarsmP1YC3sw2d vKrwsfWol5ougHsOFFQTQ+n6G65dO4i3VRr/rgAQ= Received: from jcrouse-lnx.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: jcrouse@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id 9158B60F6E; Wed, 21 Feb 2018 22:59:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1519253979; bh=NGrFl2HYLgvb2U901x5D4Xsy2sKi6E+Y/6L0rB0PJWk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OtPelCT4Vh4uTA3MaIdOnkslSUeXTTKFUOtjXmDeFkLl6OadTNoTqaedIh66ospKo ToTMi3Vx7DzYMSu/IenSd0HEiCGf4BeHZtEh7p3ugTdCbxP/zyJXvKvF9zMSnayl9j RRgq75Bwn6ryvoVFpeX2y9Gd5PwClsxAIRu/SC48= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 9158B60F6E Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=jcrouse@codeaurora.org From: Jordan Crouse To: freedreno@lists.freedesktop.org Subject: [PATCH 04/14] iommu: sva: Add support for pasid allocation Date: Wed, 21 Feb 2018 15:59:14 -0700 Message-Id: <20180221225924.30737-5-jcrouse@codeaurora.org> X-Mailer: git-send-email 2.16.1 In-Reply-To: <20180221225924.30737-1-jcrouse@codeaurora.org> References: <20180221225924.30737-1-jcrouse@codeaurora.org> X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: jean-philippe.brucker@arm.com, linux-arm-msm@vger.kernel.org, dri-devel@lists.freedesktop.org, tfiga@chromium.org, iommu@lists.linux-foundation.org, vivek.gautam@codeaurora.org, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" X-Virus-Scanned: ClamAV using ClamSMTP Some older SMMU implementations that do not have a fully featured PASID model have alternate workarounds for using multiple pagetables. For example, MSM GPUs have logic to automatically switch the user pagetable from hardware by writing the context bank directly. Instead of binding and sharing CPU pagetables these implementations need to a new pagetable structure and populate it manually. Add a new set of API functions to create and populate a pagetable structure identified by a pasid. Signed-off-by: Jordan Crouse --- drivers/iommu/iommu-sva.c | 239 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/iommu/iommu.c | 3 +- include/linux/iommu.h | 56 +++++++++++ 3 files changed, 297 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 5fc689b1ef72..c48fde5b0bbd 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -809,3 +809,242 @@ struct mm_struct *iommu_sva_find(int pasid) return mm; } EXPORT_SYMBOL_GPL(iommu_sva_find); + +int iommu_sva_alloc_pasid(struct iommu_domain *domain, struct device *dev) +{ + int ret, pasid; + struct io_pasid *io_pasid; + + if (!domain->ops->pasid_alloc || !domain->ops->pasid_free) + return -ENODEV; + + io_pasid = kzalloc(sizeof(*io_pasid), GFP_KERNEL); + if (!io_pasid) + return -ENOMEM; + + io_pasid->domain = domain; + io_pasid->base.type = IO_TYPE_PASID; + + idr_preload(GFP_KERNEL); + spin_lock(&iommu_sva_lock); + pasid = idr_alloc_cyclic(&iommu_pasid_idr, &io_pasid->base, + 1, (1 << 31), GFP_ATOMIC); + io_pasid->base.pasid = pasid; + spin_unlock(&iommu_sva_lock); + idr_preload_end(); + + if (pasid < 0) { + kfree(io_pasid); + return pasid; + } + + ret = domain->ops->pasid_alloc(domain, dev, pasid); + if (!ret) + return pasid; + + spin_lock(&iommu_sva_lock); + idr_remove(&iommu_pasid_idr, io_pasid->base.pasid); + spin_unlock(&iommu_sva_lock); + + kfree(io_pasid); + + return ret; +} +EXPORT_SYMBOL_GPL(iommu_sva_alloc_pasid); + +static struct io_pasid *get_io_pasid(int pasid) +{ + struct io_base *io_base; + struct io_pasid *io_pasid = NULL; + + spin_lock(&iommu_sva_lock); + io_base = idr_find(&iommu_pasid_idr, pasid); + if (io_base && io_base->type == IO_TYPE_PASID) + io_pasid = container_of(io_base, struct io_pasid, base); + spin_unlock(&iommu_sva_lock); + + return io_pasid; +} + +int iommu_sva_map(int pasid, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + unsigned long orig_iova = iova; + unsigned int min_pagesz; + size_t orig_size = size; + struct io_pasid *io_pasid; + struct iommu_domain *domain; + int ret = 0; + + io_pasid = get_io_pasid(pasid); + if (!io_pasid) + return -ENODEV; + + domain = io_pasid->domain; + + if (unlikely(domain->ops->sva_map == NULL || + domain->pgsize_bitmap == 0UL)) + return -ENODEV; + + /* find out the minimum page size supported */ + min_pagesz = 1 << __ffs(domain->pgsize_bitmap); + + /* + * both the virtual address and the physical one, as well as + * the size of the mapping, must be aligned (at least) to the + * size of the smallest page supported by the hardware + */ + if (!IS_ALIGNED(iova | paddr | size, min_pagesz)) { + pr_err("unaligned: iova 0x%lx pa %pa size 0x%zx min_pagesz 0x%x\n", + iova, &paddr, size, min_pagesz); + return -EINVAL; + } + + while (size) { + size_t pgsize = iommu_pgsize(domain, iova | paddr, size); + + pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", + iova, &paddr, pgsize); + + ret = domain->ops->sva_map(domain, pasid, iova, paddr, pgsize, + prot); + if (ret) + break; + + iova += pgsize; + paddr += pgsize; + size -= pgsize; + } + + /* unroll mapping in case something went wrong */ + if (ret) + iommu_sva_unmap(pasid, orig_iova, orig_size - size); + + return ret; +} +EXPORT_SYMBOL_GPL(iommu_sva_map); + +size_t iommu_sva_map_sg(int pasid, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + struct io_pasid *io_pasid; + struct iommu_domain *domain; + struct scatterlist *s; + size_t mapped = 0; + unsigned int i, min_pagesz; + int ret; + + io_pasid = get_io_pasid(pasid); + if (!io_pasid) + return -ENODEV; + + domain = io_pasid->domain; + + if (unlikely(domain->pgsize_bitmap == 0UL)) + return 0; + + min_pagesz = 1 << __ffs(domain->pgsize_bitmap); + + for_each_sg(sg, s, nents, i) { + phys_addr_t phys = page_to_phys(sg_page(s)) + s->offset; + + /* + * We are mapping on IOMMU page boundaries, so offset within + * the page must be 0. However, the IOMMU may support pages + * smaller than PAGE_SIZE, so s->offset may still represent + * an offset of that boundary within the CPU page. + */ + if (!IS_ALIGNED(s->offset, min_pagesz)) + goto out_err; + + ret = iommu_sva_map(pasid, iova + mapped, phys, s->length, + prot); + if (ret) + goto out_err; + + mapped += s->length; + } + + return mapped; + +out_err: + /* undo mappings already done */ + iommu_unmap(domain, iova, mapped); + + return 0; + +} +EXPORT_SYMBOL_GPL(iommu_sva_map_sg); + +size_t iommu_sva_unmap(int pasid, unsigned long iova, size_t size) +{ + const struct iommu_ops *ops; + struct io_pasid *io_pasid; + struct iommu_domain *domain; + size_t unmapped_page, unmapped = 0; + unsigned int min_pagesz; + + io_pasid = get_io_pasid(pasid); + if (!io_pasid) + return -ENODEV; + + domain = io_pasid->domain; + ops = domain->ops; + + if (unlikely(ops->sva_unmap == NULL || + domain->pgsize_bitmap == 0UL)) + return -ENODEV; + + /* find out the minimum page size supported */ + min_pagesz = 1 << __ffs(domain->pgsize_bitmap); + + /* + * The virtual address, as well as the size of the mapping, must be + * aligned (at least) to the size of the smallest page supported + * by the hardware + */ + if (!IS_ALIGNED(iova | size, min_pagesz)) { + pr_err("unaligned: iova 0x%lx size 0x%zx min_pagesz 0x%x\n", + iova, size, min_pagesz); + return -EINVAL; + } + + /* + * Keep iterating until we either unmap 'size' bytes (or more) + * or we hit an area that isn't mapped. + */ + while (unmapped < size) { + size_t pgsize = iommu_pgsize(domain, iova, size - unmapped); + + unmapped_page = ops->sva_unmap(domain, pasid, iova, pgsize); + if (!unmapped_page) + break; + + iova += unmapped_page; + unmapped += unmapped_page; + } + + return unmapped; +} +EXPORT_SYMBOL_GPL(iommu_sva_unmap); + +void iommu_sva_free_pasid(int pasid) +{ + struct io_pasid *io_pasid; + struct iommu_domain *domain; + + io_pasid = get_io_pasid(pasid); + if (!io_pasid) + return; + + domain = io_pasid->domain; + + domain->ops->pasid_free(domain, pasid); + + spin_lock(&iommu_sva_lock); + idr_remove(&iommu_pasid_idr, io_pasid->base.pasid); + spin_unlock(&iommu_sva_lock); + + kfree(io_pasid); +} +EXPORT_SYMBOL_GPL(iommu_sva_free_pasid); diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 7f8395b620b1..da3728388069 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1627,7 +1627,7 @@ phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) } EXPORT_SYMBOL_GPL(iommu_iova_to_phys); -static size_t iommu_pgsize(struct iommu_domain *domain, +size_t iommu_pgsize(struct iommu_domain *domain, unsigned long addr_merge, size_t size) { unsigned int pgsize_idx; @@ -1658,6 +1658,7 @@ static size_t iommu_pgsize(struct iommu_domain *domain, return pgsize; } +EXPORT_SYMBOL_GPL(iommu_pgsize); int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index e998389cf195..6e34b87655a7 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -112,6 +112,7 @@ struct iommu_domain { enum iommu_io_type { IO_TYPE_MM, + IO_TYPE_PASID, }; struct io_base { @@ -134,6 +135,11 @@ struct io_mm { struct rcu_head rcu; }; +struct io_pasid { + struct io_base base; + struct iommu_domain *domain; +}; + enum iommu_cap { IOMMU_CAP_CACHE_COHERENCY, /* IOMMU can enforce cache coherent DMA transactions */ @@ -347,6 +353,15 @@ struct iommu_ops { int (*page_response)(struct iommu_domain *domain, struct device *dev, struct page_response_msg *msg); + int (*pasid_alloc)(struct iommu_domain *domain, struct device *dev, + int pasid); + int (*sva_map)(struct iommu_domain *domain, int pasid, + unsigned long iova, phys_addr_t paddr, size_t size, + int prot); + size_t (*sva_unmap)(struct iommu_domain *domain, int pasid, + unsigned long iova, size_t size); + void (*pasid_free)(struct iommu_domain *domain, int pasid); + unsigned long pgsize_bitmap; }; @@ -577,6 +592,9 @@ extern int iommu_domain_get_attr(struct iommu_domain *domain, enum iommu_attr, extern int iommu_domain_set_attr(struct iommu_domain *domain, enum iommu_attr, void *data); +size_t iommu_pgsize(struct iommu_domain *domain, + unsigned long addr_merge, size_t size); + /* Window handling function prototypes */ extern int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, phys_addr_t offset, u64 size, @@ -1002,6 +1020,16 @@ extern int iommu_register_mm_exit_handler(struct device *dev, extern int iommu_unregister_mm_exit_handler(struct device *dev); extern struct mm_struct *iommu_sva_find(int pasid); + +extern int iommu_sva_alloc_pasid(struct iommu_domain *domain, + struct device *dev); +extern int iommu_sva_map(int pasid, unsigned long iova, phys_addr_t physaddr, + size_t size, int prot); +extern size_t iommu_sva_map_sg(int pasid, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot); +extern size_t iommu_sva_unmap(int pasid, unsigned long iova, size_t size); +extern void iommu_sva_free_pasid(int pasid); + #else /* CONFIG_IOMMU_SVA */ static inline int iommu_sva_device_init(struct device *dev, unsigned long features, @@ -1046,6 +1074,34 @@ static inline struct mm_struct *iommu_sva_find(int pasid) { return NULL; } + +static inline int iommu_sva_alloc_pasid(struct iommu_domain *domain, + struct device *dev) +{ + return -EOPNOTSUPP; +} + +static inline int iommu_sva_map(int pasid, unsigned long iova, + phys_addr_t physaddr, size_t size, int prot) +{ + return -ENODEV; +} + + +size_t iommu_sva_map_sg(int pasid, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + return 0; +} + +static inline size_t iommu_sva_unmap(int pasid, unsigned long iova, size_t size) +{ + return size; +} + +extern void iommu_sva_free_pasid(int pasid) { } + + #endif /* CONFIG_IOMMU_SVA */ #ifdef CONFIG_IOMMU_FAULT