From patchwork Thu May 23 07:00:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956963 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A6A80933 for ; Thu, 23 May 2019 07:00:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D753237A5 for ; Thu, 23 May 2019 07:00:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7135328173; Thu, 23 May 2019 07:00:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D24FE237A5 for ; Thu, 23 May 2019 07:00:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pfvG2Je1Ju2qOmLwR05+rz7cS9tkINesi1gQDqj2tOY=; b=kQfLVJv3fRer+Y /xElkbDhqrQtIMbLG/MsLFWemUOk7Z5MO+BiZVyF+aPpJGyQEvCLSaqk7MSg4N0u0qqsj27xms/K4 7J2UOLS19LvuiyMjyzJvdt+C6sKX4TxmJl35Sw5U5rdknTft0Kukj/9To0C+GFniX/Kp5tUe9wZsx aEBATDQGm8vX/G/pEC5YZ8SnRlGbd6Cj7WKjkFOiRH5IcjGu3ESK1+ZzrQaN/CiTIHSRzZLfl3jlU TxgGIqJFmQnp42M8BulariTmI5NSBazwxVfK4ojoHN0hY6kuqxuvj3zLXWxbopr5NjLOeM7Vm6DOz v3KhrnuFAgWh0lhIpsOQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiX-0004tb-31; Thu, 23 May 2019 07:00:45 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiM-0004hI-Ez; Thu, 23 May 2019 07:00:34 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 01/23] iommu/dma: Cleanup dma-iommu.h Date: Thu, 23 May 2019 09:00:06 +0200 Message-Id: <20190523070028.7435-2-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP No need for a __KERNEL__ guard outside uapi and add a missing comment describing the #else cpp statement. Last but not least include instead of the asm version, which is frowned upon. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- include/linux/dma-iommu.h | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h index 476e0c54de2d..dfb83f9c24dc 100644 --- a/include/linux/dma-iommu.h +++ b/include/linux/dma-iommu.h @@ -16,9 +16,8 @@ #ifndef __DMA_IOMMU_H #define __DMA_IOMMU_H -#ifdef __KERNEL__ +#include #include -#include #ifdef CONFIG_IOMMU_DMA #include @@ -86,7 +85,7 @@ void iommu_dma_compose_msi_msg(struct msi_desc *desc, void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list); -#else +#else /* CONFIG_IOMMU_DMA */ struct iommu_domain; struct msi_desc; @@ -128,5 +127,4 @@ static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_he } #endif /* CONFIG_IOMMU_DMA */ -#endif /* __KERNEL__ */ #endif /* __DMA_IOMMU_H */ From patchwork Thu May 23 07:00:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956965 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 20E0F933 for ; Thu, 23 May 2019 07:00:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E60C327CEA for ; Thu, 23 May 2019 07:00:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D68642811E; Thu, 23 May 2019 07:00:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6ACC7281B7 for ; Thu, 23 May 2019 07:00:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RMf1yn9NeGH1J+3cUB1ekSMKM1TwuJo6Vap0NK0ibD0=; b=D2U68oTsNthSKg fGsn8bDy0yboiE72bZpSoaOd/fU7D/7mpfkxLvzmHgvDG13SsxZciZZ7ppD5h8CHjt6yyAl9wSwpV KBbYjnySUHbhdrM4YQGPL5ZCnZLIea0o9MvOfXoLeX4cK0buh1fmuqafS+vw/VN9D0EyJiwj2MsCt gwVQ+uLAau7T4APtnWWBjRUS/WFl404BhIWVcrtZl8lNoyEt4lwYrOdhRZv9HSXtHv3OX4eptJzKH QdmBwq56HttQMaEbGdqaL/b0Yf0gC2Z21wvhLUOjbdvNnyVrxym3s/cgrgvwYnOZBybDwHtZVoYVN EU8d7WzCgdSiVQyJziPg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThie-000562-VA; Thu, 23 May 2019 07:00:53 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiP-0004iC-6v; Thu, 23 May 2019 07:00:37 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 02/23] iommu/dma: Remove the flush_page callback Date: Thu, 23 May 2019 09:00:07 +0200 Message-Id: <20190523070028.7435-3-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP We now have a arch_dma_prep_coherent architecture hook that is used for the generic DMA remap allocator, and we should use the same interface for the dma-iommu code. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy Acked-by: Catalin Marinas --- arch/arm64/mm/dma-mapping.c | 8 +------- drivers/iommu/dma-iommu.c | 8 +++----- include/linux/dma-iommu.h | 3 +-- 3 files changed, 5 insertions(+), 14 deletions(-) diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index 674860e3e478..10a8852c8b6a 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -104,12 +104,6 @@ arch_initcall(arm64_dma_init); #include #include -/* Thankfully, all cache ops are by VA so we can ignore phys here */ -static void flush_page(struct device *dev, const void *virt, phys_addr_t phys) -{ - __dma_flush_area(virt, PAGE_SIZE); -} - static void *__iommu_alloc_attrs(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp, unsigned long attrs) @@ -186,7 +180,7 @@ static void *__iommu_alloc_attrs(struct device *dev, size_t size, struct page **pages; pages = iommu_dma_alloc(dev, iosize, gfp, attrs, ioprot, - handle, flush_page); + handle); if (!pages) return NULL; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 129c4badf9ae..aac12433ffef 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -560,8 +561,6 @@ void iommu_dma_free(struct device *dev, struct page **pages, size_t size, * @attrs: DMA attributes for this allocation * @prot: IOMMU mapping flags * @handle: Out argument for allocated DMA handle - * @flush_page: Arch callback which must ensure PAGE_SIZE bytes from the - * given VA/PA are visible to the given non-coherent device. * * If @size is less than PAGE_SIZE, then a full CPU page will be allocated, * but an IOMMU which supports smaller pages might not map the whole thing. @@ -570,8 +569,7 @@ void iommu_dma_free(struct device *dev, struct page **pages, size_t size, * or NULL on failure. */ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, - unsigned long attrs, int prot, dma_addr_t *handle, - void (*flush_page)(struct device *, const void *, phys_addr_t)) + unsigned long attrs, int prot, dma_addr_t *handle) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -615,7 +613,7 @@ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, */ sg_miter_start(&miter, sgt.sgl, sgt.orig_nents, SG_MITER_FROM_SG); while (sg_miter_next(&miter)) - flush_page(dev, miter.addr, page_to_phys(miter.page)); + arch_dma_prep_coherent(miter.page, PAGE_SIZE); sg_miter_stop(&miter); } diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h index dfb83f9c24dc..e1ef265b578b 100644 --- a/include/linux/dma-iommu.h +++ b/include/linux/dma-iommu.h @@ -44,8 +44,7 @@ int dma_info_to_prot(enum dma_data_direction dir, bool coherent, * the arch code to take care of attributes and cache maintenance */ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, - unsigned long attrs, int prot, dma_addr_t *handle, - void (*flush_page)(struct device *, const void *, phys_addr_t)); + unsigned long attrs, int prot, dma_addr_t *handle); void iommu_dma_free(struct device *dev, struct page **pages, size_t size, dma_addr_t *handle); From patchwork Thu May 23 07:00:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956967 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 40CF315A6 for ; Thu, 23 May 2019 07:01:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 168C427CEA for ; Thu, 23 May 2019 07:01:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0AC43281DB; Thu, 23 May 2019 07:01:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A9F1C27CEA for ; Thu, 23 May 2019 07:01:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QMdvzCrJ5a4y+rxm3OpM+WIVZGF7bYneJ/X3s1ytx6U=; b=au3+tOn/nDxJgn zLraE22JIUhYwVly2+4+GE4Zh2eP/TvQ7jo7bjK1V3a+awCs4dY5x91DTk5SuRCKUqlBcMTBoB7RF FbveAdAnrlmjciv/WKcOds9aDyIx9RAEL1mAC7USG+VnTzOLvd1Q9PFbI8h5yBljjZhfaLlcof3bP LzvC7WZVnfwOHNl3ivKhlk+sJMYBtoXIB17kz/K+gjMcaMRiJab9npJEF4LU6cE6S7fENx2cu25NH pPUBqtc0y9vO5hUM0DIsShEFBo8ForniyiYRv7m2I9U8H3teSF9BAfImMbp3AFANxJxfA9gsVELGF VTJMo8bFCc3wcZ+y1S/Q==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThis-0005Pp-4B; Thu, 23 May 2019 07:01:06 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiS-0004mA-1l; Thu, 23 May 2019 07:00:40 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 03/23] iommu/dma: Use for_each_sg in iommu_dma_alloc Date: Thu, 23 May 2019 09:00:08 +0200 Message-Id: <20190523070028.7435-4-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP arch_dma_prep_coherent can handle physically contiguous ranges larger than PAGE_SIZE just fine, which means we don't need a page-based iterator. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index aac12433ffef..9b7f120d7381 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -606,15 +606,11 @@ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, goto out_free_iova; if (!(prot & IOMMU_CACHE)) { - struct sg_mapping_iter miter; - /* - * The CPU-centric flushing implied by SG_MITER_TO_SG isn't - * sufficient here, so skip it by using the "wrong" direction. - */ - sg_miter_start(&miter, sgt.sgl, sgt.orig_nents, SG_MITER_FROM_SG); - while (sg_miter_next(&miter)) - arch_dma_prep_coherent(miter.page, PAGE_SIZE); - sg_miter_stop(&miter); + struct scatterlist *sg; + int i; + + for_each_sg(sgt.sgl, sg, sgt.orig_nents, i) + arch_dma_prep_coherent(sg_page(sg), sg->length); } if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, prot) From patchwork Thu May 23 07:00:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956969 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6223F933 for ; Thu, 23 May 2019 07:01:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31F67281B7 for ; Thu, 23 May 2019 07:01:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2656528173; Thu, 23 May 2019 07:01:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3C95F281B7 for ; Thu, 23 May 2019 07:01:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=I1SzXknw5VWuXTQUwh6WgWZoKqOHaETdbHCF3Px8IHQ=; b=kCMKjxYSlatZb4 etS0FsJj1ZauU/HH4x/ax0qUdLn19qjA9hjg3lqMyrRaVnyt7SYlfdSq5IJ26ct738WsHEkZtJqZm zBJ0bGYPV1IgsDB1MtENDS1g/5PAf7gGn6BfvHYxZzzqvI46dtDzTj/d+20UIUMTiUCtvYXqj7egy iLC71NsKoJkh/rMs7E1bw+/sC2ehVoBszBn+Xd94dkD59qRCFHcdhLIsoQKZc+KbSZkM+SKe/jg87 fOXGn8mS3WN1peXiAJ+aoET2vq5Py8+P2GcvLnqqlCXg0yBHIHJ5tZxgEgMCT33LQmd0XyPIXMSUY KRrJDtXwBfCbtdh7yXXQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjI-000630-PY; Thu, 23 May 2019 07:01:32 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiU-0004qA-Ow; Thu, 23 May 2019 07:00:43 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 04/23] iommu/dma: move the arm64 wrappers to common code Date: Thu, 23 May 2019 09:00:09 +0200 Message-Id: <20190523070028.7435-5-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP There is nothing really arm64 specific in the iommu_dma_ops implementation, so move it to dma-iommu.c and keep a lot of symbols self-contained. Note the implementation does depend on the DMA_DIRECT_REMAP infrastructure for now, so we'll have to make the DMA_IOMMU support depend on it, but this will be relaxed soon. Signed-off-by: Christoph Hellwig Acked-by: Robin Murphy --- arch/arm64/mm/dma-mapping.c | 394 +---------------------------------- drivers/iommu/Kconfig | 1 + drivers/iommu/dma-iommu.c | 398 +++++++++++++++++++++++++++++++++--- include/linux/dma-iommu.h | 42 +--- 4 files changed, 378 insertions(+), 457 deletions(-) diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index 10a8852c8b6a..d1661f78eb4d 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -58,37 +59,6 @@ void arch_dma_prep_coherent(struct page *page, size_t size) __dma_flush_area(page_address(page), size); } -#ifdef CONFIG_IOMMU_DMA -static int __swiotlb_get_sgtable_page(struct sg_table *sgt, - struct page *page, size_t size) -{ - int ret = sg_alloc_table(sgt, 1, GFP_KERNEL); - - if (!ret) - sg_set_page(sgt->sgl, page, PAGE_ALIGN(size), 0); - - return ret; -} - -static int __swiotlb_mmap_pfn(struct vm_area_struct *vma, - unsigned long pfn, size_t size) -{ - int ret = -ENXIO; - unsigned long nr_vma_pages = vma_pages(vma); - unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; - unsigned long off = vma->vm_pgoff; - - if (off < nr_pages && nr_vma_pages <= (nr_pages - off)) { - ret = remap_pfn_range(vma, vma->vm_start, - pfn + off, - vma->vm_end - vma->vm_start, - vma->vm_page_prot); - } - - return ret; -} -#endif /* CONFIG_IOMMU_DMA */ - static int __init arm64_dma_init(void) { WARN_TAINT(ARCH_DMA_MINALIGN < cache_line_size(), @@ -100,374 +70,18 @@ static int __init arm64_dma_init(void) arch_initcall(arm64_dma_init); #ifdef CONFIG_IOMMU_DMA -#include -#include -#include - -static void *__iommu_alloc_attrs(struct device *dev, size_t size, - dma_addr_t *handle, gfp_t gfp, - unsigned long attrs) -{ - bool coherent = dev_is_dma_coherent(dev); - int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); - size_t iosize = size; - void *addr; - - if (WARN(!dev, "cannot create IOMMU mapping for unknown device\n")) - return NULL; - - size = PAGE_ALIGN(size); - - /* - * Some drivers rely on this, and we probably don't want the - * possibility of stale kernel data being read by devices anyway. - */ - gfp |= __GFP_ZERO; - - if (!gfpflags_allow_blocking(gfp)) { - struct page *page; - /* - * In atomic context we can't remap anything, so we'll only - * get the virtually contiguous buffer we need by way of a - * physically contiguous allocation. - */ - if (coherent) { - page = alloc_pages(gfp, get_order(size)); - addr = page ? page_address(page) : NULL; - } else { - addr = dma_alloc_from_pool(size, &page, gfp); - } - if (!addr) - return NULL; - - *handle = iommu_dma_map_page(dev, page, 0, iosize, ioprot); - if (*handle == DMA_MAPPING_ERROR) { - if (coherent) - __free_pages(page, get_order(size)); - else - dma_free_from_pool(addr, size); - addr = NULL; - } - } else if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); - struct page *page; - - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), gfp & __GFP_NOWARN); - if (!page) - return NULL; - - *handle = iommu_dma_map_page(dev, page, 0, iosize, ioprot); - if (*handle == DMA_MAPPING_ERROR) { - dma_release_from_contiguous(dev, page, - size >> PAGE_SHIFT); - return NULL; - } - addr = dma_common_contiguous_remap(page, size, VM_USERMAP, - prot, - __builtin_return_address(0)); - if (addr) { - if (!coherent) - __dma_flush_area(page_to_virt(page), iosize); - memset(addr, 0, size); - } else { - iommu_dma_unmap_page(dev, *handle, iosize, 0, attrs); - dma_release_from_contiguous(dev, page, - size >> PAGE_SHIFT); - } - } else { - pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); - struct page **pages; - - pages = iommu_dma_alloc(dev, iosize, gfp, attrs, ioprot, - handle); - if (!pages) - return NULL; - - addr = dma_common_pages_remap(pages, size, VM_USERMAP, prot, - __builtin_return_address(0)); - if (!addr) - iommu_dma_free(dev, pages, iosize, handle); - } - return addr; -} - -static void __iommu_free_attrs(struct device *dev, size_t size, void *cpu_addr, - dma_addr_t handle, unsigned long attrs) -{ - size_t iosize = size; - - size = PAGE_ALIGN(size); - /* - * @cpu_addr will be one of 4 things depending on how it was allocated: - * - A remapped array of pages for contiguous allocations. - * - A remapped array of pages from iommu_dma_alloc(), for all - * non-atomic allocations. - * - A non-cacheable alias from the atomic pool, for atomic - * allocations by non-coherent devices. - * - A normal lowmem address, for atomic allocations by - * coherent devices. - * Hence how dodgy the below logic looks... - */ - if (dma_in_atomic_pool(cpu_addr, size)) { - iommu_dma_unmap_page(dev, handle, iosize, 0, 0); - dma_free_from_pool(cpu_addr, size); - } else if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - struct page *page = vmalloc_to_page(cpu_addr); - - iommu_dma_unmap_page(dev, handle, iosize, 0, attrs); - dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); - dma_common_free_remap(cpu_addr, size, VM_USERMAP); - } else if (is_vmalloc_addr(cpu_addr)){ - struct vm_struct *area = find_vm_area(cpu_addr); - - if (WARN_ON(!area || !area->pages)) - return; - iommu_dma_free(dev, area->pages, iosize, &handle); - dma_common_free_remap(cpu_addr, size, VM_USERMAP); - } else { - iommu_dma_unmap_page(dev, handle, iosize, 0, 0); - __free_pages(virt_to_page(cpu_addr), get_order(size)); - } -} - -static int __iommu_mmap_attrs(struct device *dev, struct vm_area_struct *vma, - void *cpu_addr, dma_addr_t dma_addr, size_t size, - unsigned long attrs) -{ - struct vm_struct *area; - int ret; - - vma->vm_page_prot = arch_dma_mmap_pgprot(dev, vma->vm_page_prot, attrs); - - if (dma_mmap_from_dev_coherent(dev, vma, cpu_addr, size, &ret)) - return ret; - - if (!is_vmalloc_addr(cpu_addr)) { - unsigned long pfn = page_to_pfn(virt_to_page(cpu_addr)); - return __swiotlb_mmap_pfn(vma, pfn, size); - } - - if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - /* - * DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, - * hence in the vmalloc space. - */ - unsigned long pfn = vmalloc_to_pfn(cpu_addr); - return __swiotlb_mmap_pfn(vma, pfn, size); - } - - area = find_vm_area(cpu_addr); - if (WARN_ON(!area || !area->pages)) - return -ENXIO; - - return iommu_dma_mmap(area->pages, size, vma); -} - -static int __iommu_get_sgtable(struct device *dev, struct sg_table *sgt, - void *cpu_addr, dma_addr_t dma_addr, - size_t size, unsigned long attrs) -{ - unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; - struct vm_struct *area = find_vm_area(cpu_addr); - - if (!is_vmalloc_addr(cpu_addr)) { - struct page *page = virt_to_page(cpu_addr); - return __swiotlb_get_sgtable_page(sgt, page, size); - } - - if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - /* - * DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, - * hence in the vmalloc space. - */ - struct page *page = vmalloc_to_page(cpu_addr); - return __swiotlb_get_sgtable_page(sgt, page, size); - } - - if (WARN_ON(!area || !area->pages)) - return -ENXIO; - - return sg_alloc_table_from_pages(sgt, area->pages, count, 0, size, - GFP_KERNEL); -} - -static void __iommu_sync_single_for_cpu(struct device *dev, - dma_addr_t dev_addr, size_t size, - enum dma_data_direction dir) -{ - phys_addr_t phys; - - if (dev_is_dma_coherent(dev)) - return; - - phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dev_addr); - arch_sync_dma_for_cpu(dev, phys, size, dir); -} - -static void __iommu_sync_single_for_device(struct device *dev, - dma_addr_t dev_addr, size_t size, - enum dma_data_direction dir) -{ - phys_addr_t phys; - - if (dev_is_dma_coherent(dev)) - return; - - phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dev_addr); - arch_sync_dma_for_device(dev, phys, size, dir); -} - -static dma_addr_t __iommu_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - bool coherent = dev_is_dma_coherent(dev); - int prot = dma_info_to_prot(dir, coherent, attrs); - dma_addr_t dev_addr = iommu_dma_map_page(dev, page, offset, size, prot); - - if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && - dev_addr != DMA_MAPPING_ERROR) - __dma_map_area(page_address(page) + offset, size, dir); - - return dev_addr; -} - -static void __iommu_unmap_page(struct device *dev, dma_addr_t dev_addr, - size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) - __iommu_sync_single_for_cpu(dev, dev_addr, size, dir); - - iommu_dma_unmap_page(dev, dev_addr, size, dir, attrs); -} - -static void __iommu_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nelems, - enum dma_data_direction dir) -{ - struct scatterlist *sg; - int i; - - if (dev_is_dma_coherent(dev)) - return; - - for_each_sg(sgl, sg, nelems, i) - arch_sync_dma_for_cpu(dev, sg_phys(sg), sg->length, dir); -} - -static void __iommu_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nelems, - enum dma_data_direction dir) -{ - struct scatterlist *sg; - int i; - - if (dev_is_dma_coherent(dev)) - return; - - for_each_sg(sgl, sg, nelems, i) - arch_sync_dma_for_device(dev, sg_phys(sg), sg->length, dir); -} - -static int __iommu_map_sg_attrs(struct device *dev, struct scatterlist *sgl, - int nelems, enum dma_data_direction dir, - unsigned long attrs) -{ - bool coherent = dev_is_dma_coherent(dev); - - if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) - __iommu_sync_sg_for_device(dev, sgl, nelems, dir); - - return iommu_dma_map_sg(dev, sgl, nelems, - dma_info_to_prot(dir, coherent, attrs)); -} - -static void __iommu_unmap_sg_attrs(struct device *dev, - struct scatterlist *sgl, int nelems, - enum dma_data_direction dir, - unsigned long attrs) -{ - if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) - __iommu_sync_sg_for_cpu(dev, sgl, nelems, dir); - - iommu_dma_unmap_sg(dev, sgl, nelems, dir, attrs); -} - -static const struct dma_map_ops iommu_dma_ops = { - .alloc = __iommu_alloc_attrs, - .free = __iommu_free_attrs, - .mmap = __iommu_mmap_attrs, - .get_sgtable = __iommu_get_sgtable, - .map_page = __iommu_map_page, - .unmap_page = __iommu_unmap_page, - .map_sg = __iommu_map_sg_attrs, - .unmap_sg = __iommu_unmap_sg_attrs, - .sync_single_for_cpu = __iommu_sync_single_for_cpu, - .sync_single_for_device = __iommu_sync_single_for_device, - .sync_sg_for_cpu = __iommu_sync_sg_for_cpu, - .sync_sg_for_device = __iommu_sync_sg_for_device, - .map_resource = iommu_dma_map_resource, - .unmap_resource = iommu_dma_unmap_resource, -}; - -static int __init __iommu_dma_init(void) -{ - return iommu_dma_init(); -} -arch_initcall(__iommu_dma_init); - -static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, - const struct iommu_ops *ops) -{ - struct iommu_domain *domain; - - if (!ops) - return; - - /* - * The IOMMU core code allocates the default DMA domain, which the - * underlying IOMMU driver needs to support via the dma-iommu layer. - */ - domain = iommu_get_domain_for_dev(dev); - - if (!domain) - goto out_err; - - if (domain->type == IOMMU_DOMAIN_DMA) { - if (iommu_dma_init_domain(domain, dma_base, size, dev)) - goto out_err; - - dev->dma_ops = &iommu_dma_ops; - } - - return; - -out_err: - pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n", - dev_name(dev)); -} - void arch_teardown_dma_ops(struct device *dev) { dev->dma_ops = NULL; } - -#else - -static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, - const struct iommu_ops *iommu) -{ } - -#endif /* CONFIG_IOMMU_DMA */ +#endif void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, const struct iommu_ops *iommu, bool coherent) { dev->dma_coherent = coherent; - __iommu_setup_dma_ops(dev, dma_base, size, iommu); + if (iommu) + iommu_setup_dma_ops(dev, dma_base, size); #ifdef CONFIG_XEN if (xen_initial_domain()) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 83664db5221d..d6d063160dd6 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -97,6 +97,7 @@ config IOMMU_DMA select IOMMU_IOVA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH + depends on DMA_DIRECT_REMAP config FSL_PAMU bool "Freescale IOMMU support" diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 9b7f120d7381..e34ba23353cb 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -21,6 +21,7 @@ #include #include +#include #include #include #include @@ -79,11 +80,6 @@ static struct iommu_dma_cookie *cookie_alloc(enum iommu_dma_cookie_type type) return cookie; } -int iommu_dma_init(void) -{ - return iova_cache_get(); -} - /** * iommu_get_dma_cookie - Acquire DMA-API resources for a domain * @domain: IOMMU domain to prepare for DMA-API usage @@ -314,7 +310,7 @@ static void iommu_dma_flush_iotlb_all(struct iova_domain *iovad) * to ensure it is an invalid IOVA. It is safe to reinitialise a domain, but * any change which could make prior IOVAs invalid will fail. */ -int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, +static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, u64 size, struct device *dev) { struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -365,7 +361,6 @@ int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, return iova_reserve_iommu_regions(dev, domain); } -EXPORT_SYMBOL(iommu_dma_init_domain); /** * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API @@ -376,7 +371,7 @@ EXPORT_SYMBOL(iommu_dma_init_domain); * * Return: corresponding IOMMU API page protection flags */ -int dma_info_to_prot(enum dma_data_direction dir, bool coherent, +static int dma_info_to_prot(enum dma_data_direction dir, bool coherent, unsigned long attrs) { int prot = coherent ? IOMMU_CACHE : 0; @@ -535,17 +530,17 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev, } /** - * iommu_dma_free - Free a buffer allocated by iommu_dma_alloc() + * iommu_dma_free - Free a buffer allocated by __iommu_dma_alloc() * @dev: Device which owns this buffer - * @pages: Array of buffer pages as returned by iommu_dma_alloc() + * @pages: Array of buffer pages as returned by __iommu_dma_alloc() * @size: Size of buffer in bytes * @handle: DMA address of buffer * * Frees both the pages associated with the buffer, and the array * describing them */ -void iommu_dma_free(struct device *dev, struct page **pages, size_t size, - dma_addr_t *handle) +static void __iommu_dma_free(struct device *dev, struct page **pages, + size_t size, dma_addr_t *handle) { __iommu_dma_unmap(iommu_get_dma_domain(dev), *handle, size); __iommu_dma_free_pages(pages, PAGE_ALIGN(size) >> PAGE_SHIFT); @@ -553,7 +548,7 @@ void iommu_dma_free(struct device *dev, struct page **pages, size_t size, } /** - * iommu_dma_alloc - Allocate and map a buffer contiguous in IOVA space + * __iommu_dma_alloc - Allocate and map a buffer contiguous in IOVA space * @dev: Device to allocate memory for. Must be a real device * attached to an iommu_dma_domain * @size: Size of buffer in bytes @@ -568,8 +563,8 @@ void iommu_dma_free(struct device *dev, struct page **pages, size_t size, * Return: Array of struct page pointers describing the buffer, * or NULL on failure. */ -struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, - unsigned long attrs, int prot, dma_addr_t *handle) +static struct page **__iommu_dma_alloc(struct device *dev, size_t size, + gfp_t gfp, unsigned long attrs, int prot, dma_addr_t *handle) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -631,20 +626,72 @@ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, } /** - * iommu_dma_mmap - Map a buffer into provided user VMA - * @pages: Array representing buffer from iommu_dma_alloc() + * __iommu_dma_mmap - Map a buffer into provided user VMA + * @pages: Array representing buffer from __iommu_dma_alloc() * @size: Size of buffer in bytes * @vma: VMA describing requested userspace mapping * * Maps the pages of the buffer in @pages into @vma. The caller is responsible * for verifying the correct size and protection of @vma beforehand. */ - -int iommu_dma_mmap(struct page **pages, size_t size, struct vm_area_struct *vma) +static int __iommu_dma_mmap(struct page **pages, size_t size, + struct vm_area_struct *vma) { return vm_map_pages(vma, pages, PAGE_ALIGN(size) >> PAGE_SHIFT); } +static void iommu_dma_sync_single_for_cpu(struct device *dev, + dma_addr_t dma_handle, size_t size, enum dma_data_direction dir) +{ + phys_addr_t phys; + + if (dev_is_dma_coherent(dev)) + return; + + phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); + arch_sync_dma_for_cpu(dev, phys, size, dir); +} + +static void iommu_dma_sync_single_for_device(struct device *dev, + dma_addr_t dma_handle, size_t size, enum dma_data_direction dir) +{ + phys_addr_t phys; + + if (dev_is_dma_coherent(dev)) + return; + + phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); + arch_sync_dma_for_device(dev, phys, size, dir); +} + +static void iommu_dma_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nelems, + enum dma_data_direction dir) +{ + struct scatterlist *sg; + int i; + + if (dev_is_dma_coherent(dev)) + return; + + for_each_sg(sgl, sg, nelems, i) + arch_sync_dma_for_cpu(dev, sg_phys(sg), sg->length, dir); +} + +static void iommu_dma_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nelems, + enum dma_data_direction dir) +{ + struct scatterlist *sg; + int i; + + if (dev_is_dma_coherent(dev)) + return; + + for_each_sg(sgl, sg, nelems, i) + arch_sync_dma_for_device(dev, sg_phys(sg), sg->length, dir); +} + static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t size, int prot, struct iommu_domain *domain) { @@ -668,19 +715,44 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, return iova + iova_off; } -dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, +static dma_addr_t __iommu_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, int prot) { return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot, iommu_get_dma_domain(dev)); } -void iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size, - enum dma_data_direction dir, unsigned long attrs) +static void __iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, + size_t size, enum dma_data_direction dir, unsigned long attrs) { __iommu_dma_unmap(iommu_get_dma_domain(dev), handle, size); } +static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + phys_addr_t phys = page_to_phys(page) + offset; + bool coherent = dev_is_dma_coherent(dev); + dma_addr_t dma_handle; + + dma_handle =__iommu_dma_map(dev, phys, size, + dma_info_to_prot(dir, coherent, attrs), + iommu_get_dma_domain(dev)); + if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && + dma_handle != DMA_MAPPING_ERROR) + arch_sync_dma_for_device(dev, phys, size, dir); + return dma_handle; +} + +static void iommu_dma_unmap_page(struct device *dev, dma_addr_t dma_handle, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + iommu_dma_sync_single_for_cpu(dev, dma_handle, size, dir); + __iommu_dma_unmap(iommu_get_dma_domain(dev), dma_handle, size); +} + /* * Prepare a successfully-mapped scatterlist to give back to the caller. * @@ -763,18 +835,22 @@ static void __invalidate_sg(struct scatterlist *sg, int nents) * impedance-matching, to be able to hand off a suitably-aligned list, * but still preserve the original offsets and sizes for the caller. */ -int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, - int nents, int prot) +static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, unsigned long attrs) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; struct scatterlist *s, *prev = NULL; + int prot = dma_info_to_prot(dir, dev_is_dma_coherent(dev), attrs); dma_addr_t iova; size_t iova_len = 0; unsigned long mask = dma_get_seg_boundary(dev); int i; + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + iommu_dma_sync_sg_for_device(dev, sg, nents, dir); + /* * Work out how much IOVA space we need, and align the segments to * IOVA granules for the IOMMU driver to handle. With some clever @@ -834,12 +910,16 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, return 0; } -void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, - enum dma_data_direction dir, unsigned long attrs) +static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, unsigned long attrs) { dma_addr_t start, end; struct scatterlist *tmp; int i; + + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0) + iommu_dma_sync_sg_for_cpu(dev, sg, nents, dir); + /* * The scatterlist segments are mapped into a single * contiguous IOVA allocation, so this is incredibly easy. @@ -854,7 +934,7 @@ void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, __iommu_dma_unmap(iommu_get_dma_domain(dev), start, end - start); } -dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, +static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { return __iommu_dma_map(dev, phys, size, @@ -862,12 +942,268 @@ dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, iommu_get_dma_domain(dev)); } -void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, +static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir, unsigned long attrs) { __iommu_dma_unmap(iommu_get_dma_domain(dev), handle, size); } +static void *iommu_dma_alloc(struct device *dev, size_t size, + dma_addr_t *handle, gfp_t gfp, unsigned long attrs) +{ + bool coherent = dev_is_dma_coherent(dev); + int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); + size_t iosize = size; + void *addr; + + size = PAGE_ALIGN(size); + gfp |= __GFP_ZERO; + + if (!gfpflags_allow_blocking(gfp)) { + struct page *page; + /* + * In atomic context we can't remap anything, so we'll only + * get the virtually contiguous buffer we need by way of a + * physically contiguous allocation. + */ + if (coherent) { + page = alloc_pages(gfp, get_order(size)); + addr = page ? page_address(page) : NULL; + } else { + addr = dma_alloc_from_pool(size, &page, gfp); + } + if (!addr) + return NULL; + + *handle = __iommu_dma_map_page(dev, page, 0, iosize, ioprot); + if (*handle == DMA_MAPPING_ERROR) { + if (coherent) + __free_pages(page, get_order(size)); + else + dma_free_from_pool(addr, size); + addr = NULL; + } + } else if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { + pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); + struct page *page; + + page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, + get_order(size), gfp & __GFP_NOWARN); + if (!page) + return NULL; + + *handle = __iommu_dma_map_page(dev, page, 0, iosize, ioprot); + if (*handle == DMA_MAPPING_ERROR) { + dma_release_from_contiguous(dev, page, + size >> PAGE_SHIFT); + return NULL; + } + addr = dma_common_contiguous_remap(page, size, VM_USERMAP, + prot, + __builtin_return_address(0)); + if (addr) { + if (!coherent) + arch_dma_prep_coherent(page, iosize); + memset(addr, 0, size); + } else { + __iommu_dma_unmap_page(dev, *handle, iosize, 0, attrs); + dma_release_from_contiguous(dev, page, + size >> PAGE_SHIFT); + } + } else { + pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); + struct page **pages; + + pages = __iommu_dma_alloc(dev, iosize, gfp, attrs, ioprot, + handle); + if (!pages) + return NULL; + + addr = dma_common_pages_remap(pages, size, VM_USERMAP, prot, + __builtin_return_address(0)); + if (!addr) + __iommu_dma_free(dev, pages, iosize, handle); + } + return addr; +} + +static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, + dma_addr_t handle, unsigned long attrs) +{ + size_t iosize = size; + + size = PAGE_ALIGN(size); + /* + * @cpu_addr will be one of 4 things depending on how it was allocated: + * - A remapped array of pages for contiguous allocations. + * - A remapped array of pages from __iommu_dma_alloc(), for all + * non-atomic allocations. + * - A non-cacheable alias from the atomic pool, for atomic + * allocations by non-coherent devices. + * - A normal lowmem address, for atomic allocations by + * coherent devices. + * Hence how dodgy the below logic looks... + */ + if (dma_in_atomic_pool(cpu_addr, size)) { + __iommu_dma_unmap_page(dev, handle, iosize, 0, 0); + dma_free_from_pool(cpu_addr, size); + } else if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { + struct page *page = vmalloc_to_page(cpu_addr); + + __iommu_dma_unmap_page(dev, handle, iosize, 0, attrs); + dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); + dma_common_free_remap(cpu_addr, size, VM_USERMAP); + } else if (is_vmalloc_addr(cpu_addr)){ + struct vm_struct *area = find_vm_area(cpu_addr); + + if (WARN_ON(!area || !area->pages)) + return; + __iommu_dma_free(dev, area->pages, iosize, &handle); + dma_common_free_remap(cpu_addr, size, VM_USERMAP); + } else { + __iommu_dma_unmap_page(dev, handle, iosize, 0, 0); + __free_pages(virt_to_page(cpu_addr), get_order(size)); + } +} + +static int __iommu_dma_mmap_pfn(struct vm_area_struct *vma, + unsigned long pfn, size_t size) +{ + int ret = -ENXIO; + unsigned long nr_vma_pages = vma_pages(vma); + unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; + unsigned long off = vma->vm_pgoff; + + if (off < nr_pages && nr_vma_pages <= (nr_pages - off)) { + ret = remap_pfn_range(vma, vma->vm_start, + pfn + off, + vma->vm_end - vma->vm_start, + vma->vm_page_prot); + } + + return ret; +} + +static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, + void *cpu_addr, dma_addr_t dma_addr, size_t size, + unsigned long attrs) +{ + unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; + unsigned long off = vma->vm_pgoff; + struct vm_struct *area; + int ret; + + vma->vm_page_prot = arch_dma_mmap_pgprot(dev, vma->vm_page_prot, attrs); + + if (dma_mmap_from_dev_coherent(dev, vma, cpu_addr, size, &ret)) + return ret; + + if (off >= nr_pages || vma_pages(vma) > nr_pages - off) + return -ENXIO; + + if (!is_vmalloc_addr(cpu_addr)) { + unsigned long pfn = page_to_pfn(virt_to_page(cpu_addr)); + return __iommu_dma_mmap_pfn(vma, pfn, size); + } + + if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { + /* + * DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, + * hence in the vmalloc space. + */ + unsigned long pfn = vmalloc_to_pfn(cpu_addr); + return __iommu_dma_mmap_pfn(vma, pfn, size); + } + + area = find_vm_area(cpu_addr); + if (WARN_ON(!area || !area->pages)) + return -ENXIO; + + return __iommu_dma_mmap(area->pages, size, vma); +} + +static int __iommu_dma_get_sgtable_page(struct sg_table *sgt, struct page *page, + size_t size) +{ + int ret = sg_alloc_table(sgt, 1, GFP_KERNEL); + + if (!ret) + sg_set_page(sgt->sgl, page, PAGE_ALIGN(size), 0); + return ret; +} + +static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, + void *cpu_addr, dma_addr_t dma_addr, size_t size, + unsigned long attrs) +{ + unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; + struct vm_struct *area = find_vm_area(cpu_addr); + + if (!is_vmalloc_addr(cpu_addr)) { + struct page *page = virt_to_page(cpu_addr); + return __iommu_dma_get_sgtable_page(sgt, page, size); + } + + if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { + /* + * DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, + * hence in the vmalloc space. + */ + struct page *page = vmalloc_to_page(cpu_addr); + return __iommu_dma_get_sgtable_page(sgt, page, size); + } + + if (WARN_ON(!area || !area->pages)) + return -ENXIO; + + return sg_alloc_table_from_pages(sgt, area->pages, count, 0, size, + GFP_KERNEL); +} + +static const struct dma_map_ops iommu_dma_ops = { + .alloc = iommu_dma_alloc, + .free = iommu_dma_free, + .mmap = iommu_dma_mmap, + .get_sgtable = iommu_dma_get_sgtable, + .map_page = iommu_dma_map_page, + .unmap_page = iommu_dma_unmap_page, + .map_sg = iommu_dma_map_sg, + .unmap_sg = iommu_dma_unmap_sg, + .sync_single_for_cpu = iommu_dma_sync_single_for_cpu, + .sync_single_for_device = iommu_dma_sync_single_for_device, + .sync_sg_for_cpu = iommu_dma_sync_sg_for_cpu, + .sync_sg_for_device = iommu_dma_sync_sg_for_device, + .map_resource = iommu_dma_map_resource, + .unmap_resource = iommu_dma_unmap_resource, +}; + +/* + * The IOMMU core code allocates the default DMA domain, which the underlying + * IOMMU driver needs to support via the dma-iommu layer. + */ +void iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size) +{ + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + + if (!domain) + goto out_err; + + /* + * The IOMMU core code allocates the default DMA domain, which the + * underlying IOMMU driver needs to support via the dma-iommu layer. + */ + if (domain->type == IOMMU_DOMAIN_DMA) { + if (iommu_dma_init_domain(domain, dma_base, size, dev)) + goto out_err; + dev->dma_ops = &iommu_dma_ops; + } + + return; +out_err: + pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n", + dev_name(dev)); +} + static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, phys_addr_t msi_addr, struct iommu_domain *domain) { @@ -948,3 +1284,9 @@ void iommu_dma_compose_msi_msg(struct msi_desc *desc, msg->address_lo &= cookie_msi_granule(domain->iova_cookie) - 1; msg->address_lo += lower_32_bits(msi_page->iova); } + +static int iommu_dma_init(void) +{ + return iova_cache_get(); +} +arch_initcall(iommu_dma_init); diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h index e1ef265b578b..b3cc3fb84079 100644 --- a/include/linux/dma-iommu.h +++ b/include/linux/dma-iommu.h @@ -24,49 +24,13 @@ #include #include -int iommu_dma_init(void); - /* Domain management interface for IOMMU drivers */ int iommu_get_dma_cookie(struct iommu_domain *domain); int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base); void iommu_put_dma_cookie(struct iommu_domain *domain); /* Setup call for arch DMA mapping code */ -int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, - u64 size, struct device *dev); - -/* General helpers for DMA-API <-> IOMMU-API interaction */ -int dma_info_to_prot(enum dma_data_direction dir, bool coherent, - unsigned long attrs); - -/* - * These implement the bulk of the relevant DMA mapping callbacks, but require - * the arch code to take care of attributes and cache maintenance - */ -struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, - unsigned long attrs, int prot, dma_addr_t *handle); -void iommu_dma_free(struct device *dev, struct page **pages, size_t size, - dma_addr_t *handle); - -int iommu_dma_mmap(struct page **pages, size_t size, struct vm_area_struct *vma); - -dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, int prot); -int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, - int nents, int prot); - -/* - * Arch code with no special attribute handling may use these - * directly as DMA mapping callbacks for simplicity - */ -void iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size, - enum dma_data_direction dir, unsigned long attrs); -void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, - enum dma_data_direction dir, unsigned long attrs); -dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, - size_t size, enum dma_data_direction dir, unsigned long attrs); -void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir, unsigned long attrs); +void iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size); /* The DMA API isn't _quite_ the whole story, though... */ /* @@ -91,9 +55,9 @@ struct msi_desc; struct msi_msg; struct device; -static inline int iommu_dma_init(void) +static inline void iommu_setup_dma_ops(struct device *dev, u64 dma_base, + u64 size) { - return 0; } static inline int iommu_get_dma_cookie(struct iommu_domain *domain) From patchwork Thu May 23 07:00:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2350933 for ; Thu, 23 May 2019 07:01:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 89D0227CEA for ; Thu, 23 May 2019 07:01:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7E655281D2; Thu, 23 May 2019 07:01:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 667EA27CEA for ; Thu, 23 May 2019 07:01:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jbz4wYEquOeHRPuOoKgCA9TWEGQuca8uef/yf8jPulM=; b=MuEPlxgCpVifs7 vqp6MHdxCH+OYfyVpMpgn6uTXiHHunsU059lK56JaVhlrOhIiF0DzJhsNgTKm+AtByW/d3KS91TR7 izglRvhY9A0Vuld/9QHFkR11f5eMwNqsnLdm2mbu4IJUZNq97chmZZhqvTwommJqC52+Wwg4BPCqB Xl6xOmnegokjG4vumSdMpEHNrVIifR8qkPQMHl2dklWu73Em37i1msNig352Z+FYrtScRXrnPkezm kwtwY751RbzFMBDybTZWldzSbhxPzj8gTVPNhb8RU+wNH1eRFyT4+2LzRrW1UNHryPGxWExbrs7m4 ef8k8iq74kVBgzlA8XWg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjQ-0006B3-FL; Thu, 23 May 2019 07:01:40 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiX-0004tP-Nh; Thu, 23 May 2019 07:00:46 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 05/23] iommu/dma: Move __iommu_dma_map Date: Thu, 23 May 2019 09:00:10 +0200 Message-Id: <20190523070028.7435-6-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Moving this function up to its unmap counterpart helps to keep related code together for the following changes. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 46 +++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index e34ba23353cb..c406abe3be01 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -464,6 +464,29 @@ static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr, iommu_dma_free_iova(cookie, dma_addr, size); } +static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, + size_t size, int prot, struct iommu_domain *domain) +{ + struct iommu_dma_cookie *cookie = domain->iova_cookie; + size_t iova_off = 0; + dma_addr_t iova; + + if (cookie->type == IOMMU_DMA_IOVA_COOKIE) { + iova_off = iova_offset(&cookie->iovad, phys); + size = iova_align(&cookie->iovad, size + iova_off); + } + + iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); + if (!iova) + return DMA_MAPPING_ERROR; + + if (iommu_map(domain, iova, phys - iova_off, size, prot)) { + iommu_dma_free_iova(cookie, iova, size); + return DMA_MAPPING_ERROR; + } + return iova + iova_off; +} + static void __iommu_dma_free_pages(struct page **pages, int count) { while (count--) @@ -692,29 +715,6 @@ static void iommu_dma_sync_sg_for_device(struct device *dev, arch_sync_dma_for_device(dev, sg_phys(sg), sg->length, dir); } -static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, - size_t size, int prot, struct iommu_domain *domain) -{ - struct iommu_dma_cookie *cookie = domain->iova_cookie; - size_t iova_off = 0; - dma_addr_t iova; - - if (cookie->type == IOMMU_DMA_IOVA_COOKIE) { - iova_off = iova_offset(&cookie->iovad, phys); - size = iova_align(&cookie->iovad, size + iova_off); - } - - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); - if (!iova) - return DMA_MAPPING_ERROR; - - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { - iommu_dma_free_iova(cookie, iova, size); - return DMA_MAPPING_ERROR; - } - return iova + iova_off; -} - static dma_addr_t __iommu_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, int prot) { From patchwork Thu May 23 07:00:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956973 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2691115A6 for ; Thu, 23 May 2019 07:01:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F048B26256 for ; Thu, 23 May 2019 07:01:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E49BB2807B; Thu, 23 May 2019 07:01:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4A93E26256 for ; Thu, 23 May 2019 07:01:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RZKpYueokxcyksmHHmfwvLH+vyD1Ggoir/Qlv4Ml0rI=; b=T2mAzfS2gSmy5E xCdHyvuZHhdfcgK6f4zW5h99SKTlGzQOkd1DvuqaHmVRDqZDlmyf7+cZN8xzy0+kW0NrFiH1YpJsJ /RIEF+6vFYvzcMbA3Zd8nOyPI+vs9YYy5IMu6fimIHIAwFi1nIzwIcCbMZ+Tv2FDxRbyJpe4iABKh iU9tlC3TmI9jTAMbKxiRSJ9is6R30MmfGcPpD0hf0mB1VX8cv6UbK3bwxWYIbaPOrCkW9nYgY2IMy q5B2QvnghwL2B2Qa8thDKLNPBdiUL0Ar9s7EbdpWrZkcdgNnL7s7bfbSj0iMIEwatP1WUT4EjMTsm CSL3bXTZo5YyOWtw1NPg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjZ-0006Q1-3J; Thu, 23 May 2019 07:01:49 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThia-0004x9-HI; Thu, 23 May 2019 07:00:49 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 06/23] iommu/dma: Move domain lookup into __iommu_dma_{map, unmap} Date: Thu, 23 May 2019 09:00:11 +0200 Message-Id: <20190523070028.7435-7-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy Most of the callers don't care, and the couple that do already have the domain to hand for other reasons are in slow paths where the (trivial) overhead of a repeated lookup will be utterly immaterial. Signed-off-by: Robin Murphy Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index c406abe3be01..c04450a4adec 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -448,9 +448,10 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, size >> iova_shift(iovad)); } -static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr, +static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, size_t size) { + struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; size_t iova_off = iova_offset(iovad, dma_addr); @@ -465,8 +466,9 @@ static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr, } static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, - size_t size, int prot, struct iommu_domain *domain) + size_t size, int prot) { + struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; size_t iova_off = 0; dma_addr_t iova; @@ -565,7 +567,7 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev, static void __iommu_dma_free(struct device *dev, struct page **pages, size_t size, dma_addr_t *handle) { - __iommu_dma_unmap(iommu_get_dma_domain(dev), *handle, size); + __iommu_dma_unmap(dev, *handle, size); __iommu_dma_free_pages(pages, PAGE_ALIGN(size) >> PAGE_SHIFT); *handle = DMA_MAPPING_ERROR; } @@ -718,14 +720,13 @@ static void iommu_dma_sync_sg_for_device(struct device *dev, static dma_addr_t __iommu_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, int prot) { - return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot, - iommu_get_dma_domain(dev)); + return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot); } static void __iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir, unsigned long attrs) { - __iommu_dma_unmap(iommu_get_dma_domain(dev), handle, size); + __iommu_dma_unmap(dev, handle, size); } static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, @@ -734,11 +735,10 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, { phys_addr_t phys = page_to_phys(page) + offset; bool coherent = dev_is_dma_coherent(dev); + int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle =__iommu_dma_map(dev, phys, size, - dma_info_to_prot(dir, coherent, attrs), - iommu_get_dma_domain(dev)); + dma_handle =__iommu_dma_map(dev, phys, size, prot); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -750,7 +750,7 @@ static void iommu_dma_unmap_page(struct device *dev, dma_addr_t dma_handle, { if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) iommu_dma_sync_single_for_cpu(dev, dma_handle, size, dir); - __iommu_dma_unmap(iommu_get_dma_domain(dev), dma_handle, size); + __iommu_dma_unmap(dev, dma_handle, size); } /* @@ -931,21 +931,20 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, sg = tmp; } end = sg_dma_address(sg) + sg_dma_len(sg); - __iommu_dma_unmap(iommu_get_dma_domain(dev), start, end - start); + __iommu_dma_unmap(dev, start, end - start); } static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { return __iommu_dma_map(dev, phys, size, - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, - iommu_get_dma_domain(dev)); + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); } static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, size_t size, enum dma_data_direction dir, unsigned long attrs) { - __iommu_dma_unmap(iommu_get_dma_domain(dev), handle, size); + __iommu_dma_unmap(dev, handle, size); } static void *iommu_dma_alloc(struct device *dev, size_t size, @@ -1205,9 +1204,8 @@ void iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size) } static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, - phys_addr_t msi_addr, struct iommu_domain *domain) + phys_addr_t msi_addr, struct iommu_dma_cookie *cookie) { - struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iommu_dma_msi_page *msi_page; dma_addr_t iova; int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO; @@ -1222,7 +1220,7 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, if (!msi_page) return NULL; - iova = __iommu_dma_map(dev, msi_addr, size, prot, domain); + iova = __iommu_dma_map(dev, msi_addr, size, prot); if (iova == DMA_MAPPING_ERROR) goto out_free_page; @@ -1258,7 +1256,7 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) * of an MSI from within an IPI handler. */ spin_lock_irqsave(&cookie->msi_lock, flags); - msi_page = iommu_dma_get_msi_page(dev, msi_addr, domain); + msi_page = iommu_dma_get_msi_page(dev, msi_addr, cookie); spin_unlock_irqrestore(&cookie->msi_lock, flags); msi_desc_set_iommu_cookie(desc, msi_page); From patchwork Thu May 23 07:00:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956975 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8476A15A6 for ; Thu, 23 May 2019 07:02:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 596A026AE3 for ; Thu, 23 May 2019 07:02:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C11C2807B; Thu, 23 May 2019 07:02:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id AB4B62807B for ; Thu, 23 May 2019 07:02:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sG3svyPTPeBxywZg5oCoBOQHapi7rK1ASzqntOuKasg=; b=ss2t7uYok/haEj 2ZPV9CJhNK6NPANJFq1GUaGKiBjSlnAtoXOLc/qKhpfAwEEf87VLgb7i916XVsDSoB1hnZz1FWnsM wqK1v3OCBGQCl7w5zALlDnZW7hM2GP8V27K0Sitv/qSg3dQ1JD4j4COZGUiNbt4jIOgjXvXXA1wc9 dTb9wxX0R0btBdfAQc+05GvHTd9gE3jMDlfpHxVtaFjFI8WZVHszIuCdJBi9Y5GZ6G4CeHxDmSEYM kdlQMMuZI8jzgqZ9J4p3Sk0QSYq8UvQiPJ2k6AkX66wJ5GIZU+y8iUqh2yfdSYy/0OiZ+51ayr/Ls Hff0eKNA2arMB4QOgbVA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjk-0006gU-BV; Thu, 23 May 2019 07:02:00 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThid-00052B-Bb; Thu, 23 May 2019 07:00:51 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 07/23] iommu/dma: Squash __iommu_dma_{map,unmap}_page helpers Date: Thu, 23 May 2019 09:00:12 +0200 Message-Id: <20190523070028.7435-8-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy The remaining internal callsites don't care about having prototypes compatible with the relevant dma_map_ops callbacks, so the extra level of indirection just wastes space and complictaes things. Signed-off-by: Robin Murphy Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index c04450a4adec..4596e4860da8 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -717,18 +717,6 @@ static void iommu_dma_sync_sg_for_device(struct device *dev, arch_sync_dma_for_device(dev, sg_phys(sg), sg->length, dir); } -static dma_addr_t __iommu_dma_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, int prot) -{ - return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot); -} - -static void __iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - __iommu_dma_unmap(dev, handle, size); -} - static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -974,7 +962,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!addr) return NULL; - *handle = __iommu_dma_map_page(dev, page, 0, iosize, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), iosize, + ioprot); if (*handle == DMA_MAPPING_ERROR) { if (coherent) __free_pages(page, get_order(size)); @@ -991,7 +980,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!page) return NULL; - *handle = __iommu_dma_map_page(dev, page, 0, iosize, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), iosize, ioprot); if (*handle == DMA_MAPPING_ERROR) { dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); @@ -1005,7 +994,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, arch_dma_prep_coherent(page, iosize); memset(addr, 0, size); } else { - __iommu_dma_unmap_page(dev, *handle, iosize, 0, attrs); + __iommu_dma_unmap(dev, *handle, iosize); dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); } @@ -1044,12 +1033,12 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, * Hence how dodgy the below logic looks... */ if (dma_in_atomic_pool(cpu_addr, size)) { - __iommu_dma_unmap_page(dev, handle, iosize, 0, 0); + __iommu_dma_unmap(dev, handle, iosize); dma_free_from_pool(cpu_addr, size); } else if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { struct page *page = vmalloc_to_page(cpu_addr); - __iommu_dma_unmap_page(dev, handle, iosize, 0, attrs); + __iommu_dma_unmap(dev, handle, iosize); dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); dma_common_free_remap(cpu_addr, size, VM_USERMAP); } else if (is_vmalloc_addr(cpu_addr)){ @@ -1060,7 +1049,7 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, __iommu_dma_free(dev, area->pages, iosize, &handle); dma_common_free_remap(cpu_addr, size, VM_USERMAP); } else { - __iommu_dma_unmap_page(dev, handle, iosize, 0, 0); + __iommu_dma_unmap(dev, handle, iosize); __free_pages(virt_to_page(cpu_addr), get_order(size)); } } From patchwork Thu May 23 07:00:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956977 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18D48933 for ; Thu, 23 May 2019 07:02:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E33B526AE3 for ; Thu, 23 May 2019 07:02:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D7BF22811E; Thu, 23 May 2019 07:02:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6819626AE3 for ; Thu, 23 May 2019 07:02:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1n+rhslkyAoZQm1xMsdPCGbb1mae/+l6lP7xNnq2nTw=; b=WmnKDr1ChsUaO7 n1wDoAIiYPGmO5wtpkr3Yu+Zs8OruEpDh3H9LZys7gYzMZPKwkuP47ioFky9BG41d70R2hWpjwuXQ 28+N5bKBUlmVWUU6XiR4C8IFsABZ0Q4f4u7hxS29wY+86P2QgzPkoZBPrS9TjmB1Ec95iBVBHnECs f60usXfQs1NYSXdKlgXtFl5RDXHC6Mf2by+3ZR+kfGfGrbmjCCCYwVL9zziOPpqK+5quqCTU4V77a f3NOlPoZJR7B75LWE5DOrJMqCYTez4nKjnpagQz2U6ls3mSZiSgtgZQLO5wi2tIhdFyNBuiCeJ4yN uV3fxwCxE3kI7gc9dueQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThju-0006uE-0k; Thu, 23 May 2019 07:02:10 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThig-00056P-31; Thu, 23 May 2019 07:00:54 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 08/23] iommu/dma: Factor out remapped pages lookup Date: Thu, 23 May 2019 09:00:13 +0200 Message-Id: <20190523070028.7435-9-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy Since we duplicate the find_vm_area() logic a few times in places where we only care aboute the pages, factor out a helper to abstract it. Signed-off-by: Robin Murphy [hch: don't warn when not finding a region, as we'll rely on that later] Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 4596e4860da8..e870ea59a34d 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -554,6 +554,15 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev, return pages; } +static struct page **__iommu_dma_get_pages(void *cpu_addr) +{ + struct vm_struct *area = find_vm_area(cpu_addr); + + if (!area || !area->pages) + return NULL; + return area->pages; +} + /** * iommu_dma_free - Free a buffer allocated by __iommu_dma_alloc() * @dev: Device which owns this buffer @@ -1042,11 +1051,11 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); dma_common_free_remap(cpu_addr, size, VM_USERMAP); } else if (is_vmalloc_addr(cpu_addr)){ - struct vm_struct *area = find_vm_area(cpu_addr); + struct page **pages = __iommu_dma_get_pages(cpu_addr); - if (WARN_ON(!area || !area->pages)) + if (!pages) return; - __iommu_dma_free(dev, area->pages, iosize, &handle); + __iommu_dma_free(dev, pages, iosize, &handle); dma_common_free_remap(cpu_addr, size, VM_USERMAP); } else { __iommu_dma_unmap(dev, handle, iosize); @@ -1078,7 +1087,7 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, { unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; unsigned long off = vma->vm_pgoff; - struct vm_struct *area; + struct page **pages; int ret; vma->vm_page_prot = arch_dma_mmap_pgprot(dev, vma->vm_page_prot, attrs); @@ -1103,11 +1112,10 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, return __iommu_dma_mmap_pfn(vma, pfn, size); } - area = find_vm_area(cpu_addr); - if (WARN_ON(!area || !area->pages)) + pages = __iommu_dma_get_pages(cpu_addr); + if (!pages) return -ENXIO; - - return __iommu_dma_mmap(area->pages, size, vma); + return __iommu_dma_mmap(pages, size, vma); } static int __iommu_dma_get_sgtable_page(struct sg_table *sgt, struct page *page, @@ -1125,7 +1133,7 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, unsigned long attrs) { unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; - struct vm_struct *area = find_vm_area(cpu_addr); + struct page **pages; if (!is_vmalloc_addr(cpu_addr)) { struct page *page = virt_to_page(cpu_addr); @@ -1141,10 +1149,10 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, return __iommu_dma_get_sgtable_page(sgt, page, size); } - if (WARN_ON(!area || !area->pages)) + pages = __iommu_dma_get_pages(cpu_addr); + if (!pages) return -ENXIO; - - return sg_alloc_table_from_pages(sgt, area->pages, count, 0, size, + return sg_alloc_table_from_pages(sgt, pages, count, 0, size, GFP_KERNEL); } From patchwork Thu May 23 07:00:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956979 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE63C15A6 for ; Thu, 23 May 2019 07:02:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A4FB6237A5 for ; Thu, 23 May 2019 07:02:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 994172811E; Thu, 23 May 2019 07:02:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F26C0281D2 for ; Thu, 23 May 2019 07:02:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=zq3o+34j1b77q+qpGhoEc/zl6RwBTrxHJC6XeTJGXbE=; b=athQPHEgANKtP+ RzlaVW1gOVilyz8uxHla+vBlj63EK47lVzL+IPexJnzK7pmvs3ns14QKXMNcSH43X26QrM1FTdxTr 8FFHyXyieCh3fMPll5xu9Vk7/0L48IuX5pEV+w2usyZq0kLMjEN7WUNYoaY1xUSQFg+YREnvvjR7t +PklX9ZwO0wmiTfLIEGiJOW6kk85fWs/QDCJbv5Bj86tRCvnYlvwOrElf47S2JXq0yuBZVrY4wXZv 9bksC3/LrhhwklOZLaGTfFHeF4iJtXd+s7wX2aUcxVienEnFPdAEtU8rUXjqwY5d1bxfa9bO0OYSF LZTl/zzuXLCWKr4yHVkA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThk3-00078j-7D; Thu, 23 May 2019 07:02:19 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThij-0005B9-0p; Thu, 23 May 2019 07:00:57 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 09/23] iommu/dma: Refactor the page array remapping allocator Date: Thu, 23 May 2019 09:00:14 +0200 Message-Id: <20190523070028.7435-10-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Move the call to dma_common_pages_remap into __iommu_dma_alloc and rename it to iommu_dma_alloc_remap. This creates a self-contained helper for remapped pages allocation and mapping. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 54 +++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index e870ea59a34d..0ccc25fd5c86 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -564,9 +564,9 @@ static struct page **__iommu_dma_get_pages(void *cpu_addr) } /** - * iommu_dma_free - Free a buffer allocated by __iommu_dma_alloc() + * iommu_dma_free - Free a buffer allocated by iommu_dma_alloc_remap() * @dev: Device which owns this buffer - * @pages: Array of buffer pages as returned by __iommu_dma_alloc() + * @pages: Array of buffer pages as returned by __iommu_dma_alloc_remap() * @size: Size of buffer in bytes * @handle: DMA address of buffer * @@ -582,33 +582,35 @@ static void __iommu_dma_free(struct device *dev, struct page **pages, } /** - * __iommu_dma_alloc - Allocate and map a buffer contiguous in IOVA space + * iommu_dma_alloc_remap - Allocate and map a buffer contiguous in IOVA space * @dev: Device to allocate memory for. Must be a real device * attached to an iommu_dma_domain * @size: Size of buffer in bytes + * @dma_handle: Out argument for allocated DMA handle * @gfp: Allocation flags * @attrs: DMA attributes for this allocation - * @prot: IOMMU mapping flags - * @handle: Out argument for allocated DMA handle * * If @size is less than PAGE_SIZE, then a full CPU page will be allocated, * but an IOMMU which supports smaller pages might not map the whole thing. * - * Return: Array of struct page pointers describing the buffer, - * or NULL on failure. + * Return: Mapped virtual address, or NULL on failure. */ -static struct page **__iommu_dma_alloc(struct device *dev, size_t size, - gfp_t gfp, unsigned long attrs, int prot, dma_addr_t *handle) +static void *iommu_dma_alloc_remap(struct device *dev, size_t size, + dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; + bool coherent = dev_is_dma_coherent(dev); + int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); + pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); + unsigned int count, min_size, alloc_sizes = domain->pgsize_bitmap; struct page **pages; struct sg_table sgt; dma_addr_t iova; - unsigned int count, min_size, alloc_sizes = domain->pgsize_bitmap; + void *vaddr; - *handle = DMA_MAPPING_ERROR; + *dma_handle = DMA_MAPPING_ERROR; min_size = alloc_sizes & -alloc_sizes; if (min_size < PAGE_SIZE) { @@ -634,7 +636,7 @@ static struct page **__iommu_dma_alloc(struct device *dev, size_t size, if (sg_alloc_table_from_pages(&sgt, pages, count, 0, size, GFP_KERNEL)) goto out_free_iova; - if (!(prot & IOMMU_CACHE)) { + if (!(ioprot & IOMMU_CACHE)) { struct scatterlist *sg; int i; @@ -642,14 +644,21 @@ static struct page **__iommu_dma_alloc(struct device *dev, size_t size, arch_dma_prep_coherent(sg_page(sg), sg->length); } - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, prot) + if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) < size) goto out_free_sg; - *handle = iova; + vaddr = dma_common_pages_remap(pages, size, VM_USERMAP, prot, + __builtin_return_address(0)); + if (!vaddr) + goto out_unmap; + + *dma_handle = iova; sg_free_table(&sgt); - return pages; + return vaddr; +out_unmap: + __iommu_dma_unmap(dev, iova, size); out_free_sg: sg_free_table(&sgt); out_free_iova: @@ -1008,18 +1017,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, size >> PAGE_SHIFT); } } else { - pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); - struct page **pages; - - pages = __iommu_dma_alloc(dev, iosize, gfp, attrs, ioprot, - handle); - if (!pages) - return NULL; - - addr = dma_common_pages_remap(pages, size, VM_USERMAP, prot, - __builtin_return_address(0)); - if (!addr) - __iommu_dma_free(dev, pages, iosize, handle); + addr = iommu_dma_alloc_remap(dev, iosize, handle, gfp, attrs); } return addr; } @@ -1033,7 +1031,7 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, /* * @cpu_addr will be one of 4 things depending on how it was allocated: * - A remapped array of pages for contiguous allocations. - * - A remapped array of pages from __iommu_dma_alloc(), for all + * - A remapped array of pages from iommu_dma_alloc_remap(), for all * non-atomic allocations. * - A non-cacheable alias from the atomic pool, for atomic * allocations by non-coherent devices. From patchwork Thu May 23 07:00:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65A7815A6 for ; Thu, 23 May 2019 07:02:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D4942811E for ; Thu, 23 May 2019 07:02:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 318D626256; Thu, 23 May 2019 07:02:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D679A2811E for ; Thu, 23 May 2019 07:02:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TZk2FmL7FOtzvW5efViqX01Wb47hqURmSm3fqiDE72Q=; b=CrdTOP8hVVYRqB qcEa65HaERmtYICy12/rOLwQ5B0VKcKLaf6rFIwoQ8AiZ383KT3Pf3UuJJ3fIwqhPGFCgzcMG0VIX QCQZ4yef4HUejzaafQmyAzGcZUle3OkIpb+UkM9+xya3MlgqS8eoZ5OmxHOiAuSgZnTS5fX6VAC0P CHn3MFXZc8j+f0fOqLnBh5dQhwwe1SAB2hhaGZtizN4gLrRAnMOj5xLaxi9XrnXs1j6mk8l7o7KBe L9o2KkQLkWwuf2B67GJ/bNHh8ykg+qyfqnBxQCZP5mofMwd/2VwsBgXfMd83cA3eyLPc5eLd1xZ1b w6j2Bt3Ezbx3ui49EoCw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThkE-0007QZ-QU; Thu, 23 May 2019 07:02:30 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThil-0005G0-Qw; Thu, 23 May 2019 07:01:00 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 10/23] iommu/dma: Remove __iommu_dma_free Date: Thu, 23 May 2019 09:00:15 +0200 Message-Id: <20190523070028.7435-11-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP We only have a single caller of this function left, so open code it there. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 0ccc25fd5c86..191c0a4c8e31 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -563,24 +563,6 @@ static struct page **__iommu_dma_get_pages(void *cpu_addr) return area->pages; } -/** - * iommu_dma_free - Free a buffer allocated by iommu_dma_alloc_remap() - * @dev: Device which owns this buffer - * @pages: Array of buffer pages as returned by __iommu_dma_alloc_remap() - * @size: Size of buffer in bytes - * @handle: DMA address of buffer - * - * Frees both the pages associated with the buffer, and the array - * describing them - */ -static void __iommu_dma_free(struct device *dev, struct page **pages, - size_t size, dma_addr_t *handle) -{ - __iommu_dma_unmap(dev, *handle, size); - __iommu_dma_free_pages(pages, PAGE_ALIGN(size) >> PAGE_SHIFT); - *handle = DMA_MAPPING_ERROR; -} - /** * iommu_dma_alloc_remap - Allocate and map a buffer contiguous in IOVA space * @dev: Device to allocate memory for. Must be a real device @@ -1053,7 +1035,8 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, if (!pages) return; - __iommu_dma_free(dev, pages, iosize, &handle); + __iommu_dma_unmap(dev, handle, iosize); + __iommu_dma_free_pages(pages, size >> PAGE_SHIFT); dma_common_free_remap(cpu_addr, size, VM_USERMAP); } else { __iommu_dma_unmap(dev, handle, iosize); From patchwork Thu May 23 07:00:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956983 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95228933 for ; Thu, 23 May 2019 07:02:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6921A2811E for ; Thu, 23 May 2019 07:02:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5D8E6281DB; Thu, 23 May 2019 07:02:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D9B522811E for ; Thu, 23 May 2019 07:02:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kgZgaBdWDXF+AE3oXBBSefIMjdGk3N3uh0SDb2k91+w=; b=Wq/OsmZVMUPxIr U6e3A/gtI1GgVHfBSqhTRBHZzuh7XaYJTOoE+eXeiSYkdvlpTvwKZAnR3/mqbwcLrui6mNZ+qkaEf SzU9oKo/aNhqoXLVnlbEKASca7Q2PnizILydZqv3R2wIe0w990EeEZiEHpFmymlMt8AJRItdix/YV ftS4EGiQaCXLfKD0u9XMQoGHD4wfXkJ0/nYW6bxNlqXYLvcCJ3er/KPpBu6Db3rmD3Zw5XFWoDPzM pfpsWVAAK5tEXcvWApLjVWGm+FMKQqtX0B72JzyNiWHOQf9NuHFcur9q2MEv61hpTUh+tHVXhPb6b SU9y1NNWnPDMdR/nF/+A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThkO-0007eP-EI; Thu, 23 May 2019 07:02:40 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThio-0005Kw-Lw; Thu, 23 May 2019 07:01:03 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 11/23] iommu/dma: Refactor iommu_dma_free Date: Thu, 23 May 2019 09:00:16 +0200 Message-Id: <20190523070028.7435-12-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy The freeing logic was made particularly horrible by part of it being opaque to the arch wrapper, which led to a lot of convoluted repetition to ensure each path did everything in the right order. Now that it's all private, we can pick apart and consolidate the logically-distinct steps of freeing the IOMMU mapping, the underlying pages, and the CPU remap (if necessary) into something much more manageable. Signed-off-by: Robin Murphy [various cosmetic changes to the code flow] Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 73 ++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 40 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 191c0a4c8e31..f61e3f8861a8 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -935,6 +935,39 @@ static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, __iommu_dma_unmap(dev, handle, size); } +static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, + dma_addr_t handle, unsigned long attrs) +{ + size_t alloc_size = PAGE_ALIGN(size); + int count = alloc_size >> PAGE_SHIFT; + struct page *page = NULL, **pages = NULL; + + __iommu_dma_unmap(dev, handle, size); + + /* Non-coherent atomic allocation? Easy */ + if (dma_free_from_pool(cpu_addr, alloc_size)) + return; + + if (is_vmalloc_addr(cpu_addr)) { + /* + * If it the address is remapped, then it's either non-coherent + * or highmem CMA, or an iommu_dma_alloc_remap() construction. + */ + pages = __iommu_dma_get_pages(cpu_addr); + if (!pages) + page = vmalloc_to_page(cpu_addr); + dma_common_free_remap(cpu_addr, alloc_size, VM_USERMAP); + } else { + /* Lowmem means a coherent atomic or CMA allocation */ + page = virt_to_page(cpu_addr); + } + + if (pages) + __iommu_dma_free_pages(pages, count); + if (page && !dma_release_from_contiguous(dev, page, count)) + __free_pages(page, get_order(alloc_size)); +} + static void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp, unsigned long attrs) { @@ -1004,46 +1037,6 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, return addr; } -static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, - dma_addr_t handle, unsigned long attrs) -{ - size_t iosize = size; - - size = PAGE_ALIGN(size); - /* - * @cpu_addr will be one of 4 things depending on how it was allocated: - * - A remapped array of pages for contiguous allocations. - * - A remapped array of pages from iommu_dma_alloc_remap(), for all - * non-atomic allocations. - * - A non-cacheable alias from the atomic pool, for atomic - * allocations by non-coherent devices. - * - A normal lowmem address, for atomic allocations by - * coherent devices. - * Hence how dodgy the below logic looks... - */ - if (dma_in_atomic_pool(cpu_addr, size)) { - __iommu_dma_unmap(dev, handle, iosize); - dma_free_from_pool(cpu_addr, size); - } else if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - struct page *page = vmalloc_to_page(cpu_addr); - - __iommu_dma_unmap(dev, handle, iosize); - dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); - dma_common_free_remap(cpu_addr, size, VM_USERMAP); - } else if (is_vmalloc_addr(cpu_addr)){ - struct page **pages = __iommu_dma_get_pages(cpu_addr); - - if (!pages) - return; - __iommu_dma_unmap(dev, handle, iosize); - __iommu_dma_free_pages(pages, size >> PAGE_SHIFT); - dma_common_free_remap(cpu_addr, size, VM_USERMAP); - } else { - __iommu_dma_unmap(dev, handle, iosize); - __free_pages(virt_to_page(cpu_addr), get_order(size)); - } -} - static int __iommu_dma_mmap_pfn(struct vm_area_struct *vma, unsigned long pfn, size_t size) { From patchwork Thu May 23 07:00:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956985 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C546933 for ; Thu, 23 May 2019 07:02:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E638F27CEA for ; Thu, 23 May 2019 07:02:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DA280281B7; Thu, 23 May 2019 07:02:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7337D27CEA for ; Thu, 23 May 2019 07:02:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9IEqCBG760AALy1VhLUgGAhMMXPTncH0xzKxU4ZIEGs=; b=A+AO5ncU5G4L74 BRse77/IiTG46WMOvkm86FK9Tyn9FPdcFcDBtx37paes1AY6HjgsAB9bkbCirnUqacQPBFkF4zO9y 0vqRxu+BgnobTieFLBR6c1F4RRAHOAbtQzspaqIK0aIfzvTSTtglpXl7o5tlrTm13HI1DmeLl/uLl 0gKY+vJdAf1f+cttN3/5AVMLKFU6Zs6Onaf9OE0z+ifel2AxS1npI7gFiXh/09F2S4DgTGL9QgPlW DGG+C38FQq3l2IeOrcSQi5ehlDGe9Rx2xWZShs9l7ZC7u18SHWD6UXmfQ42Ez0CeSwJSCmWiE6Br/ lPQvw1ppBFmwmIGvttrA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThkZ-0007sA-IR; Thu, 23 May 2019 07:02:51 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThir-0005Pg-Ea; Thu, 23 May 2019 07:01:05 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 12/23] iommu/dma: Refactor iommu_dma_alloc Date: Thu, 23 May 2019 09:00:17 +0200 Message-Id: <20190523070028.7435-13-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy Shuffle around the self-contained atomic and non-contiguous cases to return early and get out of the way of the CMA case that we're about to work on next. Signed-off-by: Robin Murphy [hch: slight changes to the code flow] Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 60 +++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index f61e3f8861a8..41e87756c076 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -973,14 +973,19 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, { bool coherent = dev_is_dma_coherent(dev); int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); + pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); size_t iosize = size; + struct page *page; void *addr; size = PAGE_ALIGN(size); gfp |= __GFP_ZERO; + if (gfpflags_allow_blocking(gfp) && + !(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) + return iommu_dma_alloc_remap(dev, iosize, handle, gfp, attrs); + if (!gfpflags_allow_blocking(gfp)) { - struct page *page; /* * In atomic context we can't remap anything, so we'll only * get the virtually contiguous buffer we need by way of a @@ -1002,39 +1007,34 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, __free_pages(page, get_order(size)); else dma_free_from_pool(addr, size); - addr = NULL; - } - } else if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); - struct page *page; - - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), gfp & __GFP_NOWARN); - if (!page) return NULL; - - *handle = __iommu_dma_map(dev, page_to_phys(page), iosize, ioprot); - if (*handle == DMA_MAPPING_ERROR) { - dma_release_from_contiguous(dev, page, - size >> PAGE_SHIFT); - return NULL; - } - addr = dma_common_contiguous_remap(page, size, VM_USERMAP, - prot, - __builtin_return_address(0)); - if (addr) { - if (!coherent) - arch_dma_prep_coherent(page, iosize); - memset(addr, 0, size); - } else { - __iommu_dma_unmap(dev, *handle, iosize); - dma_release_from_contiguous(dev, page, - size >> PAGE_SHIFT); } - } else { - addr = iommu_dma_alloc_remap(dev, iosize, handle, gfp, attrs); + return addr; } + + page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, + get_order(size), gfp & __GFP_NOWARN); + if (!page) + return NULL; + + *handle = __iommu_dma_map(dev, page_to_phys(page), iosize, ioprot); + if (*handle == DMA_MAPPING_ERROR) + goto out_free_pages; + + addr = dma_common_contiguous_remap(page, size, VM_USERMAP, prot, + __builtin_return_address(0)); + if (!addr) + goto out_unmap; + + if (!coherent) + arch_dma_prep_coherent(page, iosize); + memset(addr, 0, size); return addr; +out_unmap: + __iommu_dma_unmap(dev, *handle, iosize); +out_free_pages: + dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); + return NULL; } static int __iommu_dma_mmap_pfn(struct vm_area_struct *vma, From patchwork Thu May 23 07:00:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956987 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C26E0933 for ; Thu, 23 May 2019 07:03:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 989882807B for ; Thu, 23 May 2019 07:03:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8D4762811E; Thu, 23 May 2019 07:03:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3E1A928173 for ; Thu, 23 May 2019 07:03:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nI1nQnrRKypgsPPF1Bbs8sazGJtiCTBeoR+aPlnqv1k=; b=sjQt3sGEfAwWR6 xPyb4od4vRzQ3HGzbU27s1EQTywVlXtxgrNSiM3ffrfB1MH+5UFQMxv8ZSILHOoKYHX2gm5wgjA8q qSL2AA+HPDm7AxKpTl6vkSfGeV6s1ZVeewXy/rd86PRujHlIw2NAIZfDLKfNTyEKkQKcNMBSoU4qW 3EE0Q9s1fmxTwqvnkAtBw/GhCmsXC59NB0prk52xZgjyvxNKMOXeRNWXGvbk13kkixlMY1T+eLEw2 dtmHTpmn8pkIoBUW95bSrLsoXiN4C03q+oujS3CrnWMnin3xUPn0wU2Zfg0t2grPEXy4lYuxI6hLr VB9DSgEwm2uRntnW49vw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThkj-00086v-9h; Thu, 23 May 2019 07:03:01 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiu-0005UL-7s; Thu, 23 May 2019 07:01:08 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 13/23] iommu/dma: Don't remap CMA unnecessarily Date: Thu, 23 May 2019 09:00:18 +0200 Message-Id: <20190523070028.7435-14-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy Always remapping CMA allocations was largely a bodge to keep the freeing logic manageable when it was split between here and an arch wrapper. Now that it's all together and streamlined, we can relax that limitation. Signed-off-by: Robin Murphy Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 41e87756c076..3629bc2f59ee 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -973,7 +973,6 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, { bool coherent = dev_is_dma_coherent(dev); int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); - pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); size_t iosize = size; struct page *page; void *addr; @@ -1021,13 +1020,19 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (*handle == DMA_MAPPING_ERROR) goto out_free_pages; - addr = dma_common_contiguous_remap(page, size, VM_USERMAP, prot, - __builtin_return_address(0)); - if (!addr) - goto out_unmap; + if (!coherent || PageHighMem(page)) { + pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); - if (!coherent) - arch_dma_prep_coherent(page, iosize); + addr = dma_common_contiguous_remap(page, size, VM_USERMAP, prot, + __builtin_return_address(0)); + if (!addr) + goto out_unmap; + + if (!coherent) + arch_dma_prep_coherent(page, iosize); + } else { + addr = page_address(page); + } memset(addr, 0, size); return addr; out_unmap: From patchwork Thu May 23 07:00:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956989 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E48DC933 for ; Thu, 23 May 2019 07:03:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BB7602807B for ; Thu, 23 May 2019 07:03:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AFA102811E; Thu, 23 May 2019 07:03:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5AAA626256 for ; Thu, 23 May 2019 07:03:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wv79XlPzNKQcXZJQcsaZP6zfrqTxYpF/g5+drwo7KLo=; b=GJC/350txTAKn2 QaVFmjHe4Ld1661kWoS8AW/WeKsm1YYMpNcok0WCNUIsq+ymlovR8bwYS+CTb3CRUDzgWovBNMUPN MrEoxLynLIlPGPgbTErm97lVg1HKkHUbcxh0S4+R4KvWQbfR1QCPZeZ6oO+hbaiAsiwGg0iOK/hFF zlXStC6Y++NlY+Gzi5W0fBvRRkMz8G359qXVKw5gX3vy15Q0RX1azDgpAH4sZWEI+mHFiA+0JUJU4 mHZ4+0RNjTxIz+U9/4eqmviTqifVNj+XCcL0U9jceCy4Hb0FOshJtUiP5Jx2l3w6jTKQcQWOw5X7e szwEYTlQO7dOYw3kHu+g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThku-0008Ki-JT; Thu, 23 May 2019 07:03:12 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThix-0005ZM-17; Thu, 23 May 2019 07:01:11 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 14/23] iommu/dma: Merge the CMA and alloc_pages allocation paths Date: Thu, 23 May 2019 09:00:19 +0200 Message-Id: <20190523070028.7435-15-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of having a separate code path for the non-blocking alloc_pages and CMA allocations paths merge them into one. There is a slight behavior change here in that we try the page allocator if CMA fails. This matches what dma-direct and other iommu drivers do and will be needed to use the dma-iommu code on architectures without DMA remapping later on. Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 3629bc2f59ee..6b8cedae7cff 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -974,7 +974,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, bool coherent = dev_is_dma_coherent(dev); int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); size_t iosize = size; - struct page *page; + struct page *page = NULL; void *addr; size = PAGE_ALIGN(size); @@ -984,35 +984,26 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, !(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) return iommu_dma_alloc_remap(dev, iosize, handle, gfp, attrs); - if (!gfpflags_allow_blocking(gfp)) { - /* - * In atomic context we can't remap anything, so we'll only - * get the virtually contiguous buffer we need by way of a - * physically contiguous allocation. - */ - if (coherent) { - page = alloc_pages(gfp, get_order(size)); - addr = page ? page_address(page) : NULL; - } else { - addr = dma_alloc_from_pool(size, &page, gfp); - } + if (!gfpflags_allow_blocking(gfp) && !coherent) { + addr = dma_alloc_from_pool(size, &page, gfp); if (!addr) return NULL; *handle = __iommu_dma_map(dev, page_to_phys(page), iosize, ioprot); if (*handle == DMA_MAPPING_ERROR) { - if (coherent) - __free_pages(page, get_order(size)); - else - dma_free_from_pool(addr, size); + dma_free_from_pool(addr, size); return NULL; } return addr; } - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), gfp & __GFP_NOWARN); + if (gfpflags_allow_blocking(gfp)) + page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, + get_order(size), + gfp & __GFP_NOWARN); + if (!page) + page = alloc_pages(gfp, get_order(size)); if (!page) return NULL; @@ -1038,7 +1029,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, out_unmap: __iommu_dma_unmap(dev, *handle, iosize); out_free_pages: - dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT); + if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) + __free_pages(page, get_order(size)); return NULL; } From patchwork Thu May 23 07:00:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956991 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95F19933 for ; Thu, 23 May 2019 07:03:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C36226256 for ; Thu, 23 May 2019 07:03:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 608ED26E49; Thu, 23 May 2019 07:03:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 037E128173 for ; Thu, 23 May 2019 07:03:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AmYlqkq35q3QJGrsi+8oSmx3fhX8F7D8WWr/krdetLk=; b=a/XGDlSzpc0FnM rkqTJZXoz0M3nBOafG0i2l2Tfk1FMjAQ3gFy3xxtWEUMLqwUhGoN575dTpif911dOk4D+43UM2oQj GDxA+EYwPOwGtGMOOqaTsOUnspPmaS9RNyogUSH2zBqVLFqdNqFuDWUIyYQrDNDAAlp4UQ0ye8Ntn La4YLoBytT5iwpiOs2AQ2GshbyFzTfvuL5EPprDNwz0nSWlOxguoLLLKayBD/wWAZQZ2BybbLZW/Q E7XAco9MLkxW12/qAUe2DtAFRCfuOTbUcwih4l3OG90aQg15OgYxNU0zfuEtpMoBYqWwiCh3MEoPy S5W8asuKvQ7j2RTivTZw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThl3-00005n-0Q; Thu, 23 May 2019 07:03:21 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThiz-0005e1-Pi; Thu, 23 May 2019 07:01:14 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 15/23] iommu/dma: Split iommu_dma_free Date: Thu, 23 May 2019 09:00:20 +0200 Message-Id: <20190523070028.7435-16-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy Most of it can double up to serve the failure cleanup path for iommu_dma_alloc(). Signed-off-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 6b8cedae7cff..33d1ce8cc640 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -935,15 +935,12 @@ static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, __iommu_dma_unmap(dev, handle, size); } -static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, - dma_addr_t handle, unsigned long attrs) +static void __iommu_dma_free(struct device *dev, size_t size, void *cpu_addr) { size_t alloc_size = PAGE_ALIGN(size); int count = alloc_size >> PAGE_SHIFT; struct page *page = NULL, **pages = NULL; - __iommu_dma_unmap(dev, handle, size); - /* Non-coherent atomic allocation? Easy */ if (dma_free_from_pool(cpu_addr, alloc_size)) return; @@ -968,6 +965,13 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, __free_pages(page, get_order(alloc_size)); } +static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, + dma_addr_t handle, unsigned long attrs) +{ + __iommu_dma_unmap(dev, handle, size); + __iommu_dma_free(dev, size, cpu_addr); +} + static void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, gfp_t gfp, unsigned long attrs) { From patchwork Thu May 23 07:00:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956993 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6EA6015A6 for ; Thu, 23 May 2019 07:03:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 442A7281B7 for ; Thu, 23 May 2019 07:03:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 388FD27CEA; Thu, 23 May 2019 07:03:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C89AB28173 for ; Thu, 23 May 2019 07:03:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Bw9rG+68D8nisr1dxkAtC19LTO3NdKNkbXfwhtV4krA=; b=NxS2+kbKzrOg9Q FDBYoK2GQLJw8CSv7AVY2jGeyvNa6nBlax4JG3d0EDI5HhmCWwDh2B4IVdxcr1w9JIzG6QS0rXpO4 5uJrclclDUi1wGOBQMkEK5jULk0AL1swnVq4hfVa7E+TMwUuiQrmqi2x+mr0vyDywgYda309CK6nR DDnUy+aTaaOMCLnOtUBNdOvFDmBcc4hN1a/SiOqv94u6tLarBBFqg7+0iCKcpBXprP/bo3+RfW74D ba559NIsBcm/YllGECXaPX44R1JIbVTQp14rTEbkU+rUZI34yJhy6n8gKgqrCM5hfWdEBnjqzaagL QK1Ry3zrv1EGiYA2E8Tw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThlE-0000Kv-Ga; Thu, 23 May 2019 07:03:32 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThj2-0005ij-N4; Thu, 23 May 2019 07:01:17 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 16/23] iommu/dma: Cleanup variable naming in iommu_dma_alloc Date: Thu, 23 May 2019 09:00:21 +0200 Message-Id: <20190523070028.7435-17-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Robin Murphy Most importantly clear up the size / iosize confusion. Also rename addr to cpu_addr to match the surrounding code and make the intention a little more clear. Signed-off-by: Robin Murphy [hch: split from a larger patch] Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 45 +++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 33d1ce8cc640..9ac76d286df1 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -977,64 +977,63 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, { bool coherent = dev_is_dma_coherent(dev); int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); - size_t iosize = size; + size_t alloc_size = PAGE_ALIGN(size); struct page *page = NULL; - void *addr; + void *cpu_addr; - size = PAGE_ALIGN(size); gfp |= __GFP_ZERO; if (gfpflags_allow_blocking(gfp) && !(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) - return iommu_dma_alloc_remap(dev, iosize, handle, gfp, attrs); + return iommu_dma_alloc_remap(dev, size, handle, gfp, attrs); if (!gfpflags_allow_blocking(gfp) && !coherent) { - addr = dma_alloc_from_pool(size, &page, gfp); - if (!addr) + cpu_addr = dma_alloc_from_pool(alloc_size, &page, gfp); + if (!cpu_addr) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), iosize, + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); if (*handle == DMA_MAPPING_ERROR) { - dma_free_from_pool(addr, size); + dma_free_from_pool(cpu_addr, alloc_size); return NULL; } - return addr; + return cpu_addr; } if (gfpflags_allow_blocking(gfp)) - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), + page = dma_alloc_from_contiguous(dev, alloc_size >> PAGE_SHIFT, + get_order(alloc_size), gfp & __GFP_NOWARN); if (!page) - page = alloc_pages(gfp, get_order(size)); + page = alloc_pages(gfp, get_order(alloc_size)); if (!page) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), iosize, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); if (*handle == DMA_MAPPING_ERROR) goto out_free_pages; if (!coherent || PageHighMem(page)) { pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); - addr = dma_common_contiguous_remap(page, size, VM_USERMAP, prot, - __builtin_return_address(0)); - if (!addr) + cpu_addr = dma_common_contiguous_remap(page, alloc_size, + VM_USERMAP, prot, __builtin_return_address(0)); + if (!cpu_addr) goto out_unmap; if (!coherent) - arch_dma_prep_coherent(page, iosize); + arch_dma_prep_coherent(page, size); } else { - addr = page_address(page); + cpu_addr = page_address(page); } - memset(addr, 0, size); - return addr; + memset(cpu_addr, 0, alloc_size); + return cpu_addr; out_unmap: - __iommu_dma_unmap(dev, *handle, iosize); + __iommu_dma_unmap(dev, *handle, size); out_free_pages: - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); + if (!dma_release_from_contiguous(dev, page, alloc_size >> PAGE_SHIFT)) + __free_pages(page, get_order(alloc_size)); return NULL; } From patchwork Thu May 23 07:00:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956995 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1C0615A6 for ; Thu, 23 May 2019 07:03:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87C5C26AE3 for ; Thu, 23 May 2019 07:03:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7BF0E26E49; Thu, 23 May 2019 07:03:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 11AEA27CEA for ; Thu, 23 May 2019 07:03:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bwIdEH87zvp1bd4tdoJfxgDmQHhok5IXhjVAuI4Q1/M=; b=EHoeiKDgezlsTG TliWCErmIyPHS0YAZNu5E3AAk3j4a4jNDoyRlvDoxY3yll5CllwC0poMRbrO4ndYVTt/f6JmaxLZN yoRTiRBUf7HRNyp887vgdYDSixl0uuwcnVP8bnRtT2n+jIkkQvOMIyVts7QjWXbjdIGZyW4ixcojp eMk+s/hWwLvjcVZfR2gyjMAJwTyhsnywDs2jKNXyS6gcdl6F5/5+3HB1P2xZCdmfpuBpYvat4T7l8 N7yjGGuqQrd69rSl+oXh9CdAqLYnAHEy5X1q4rjaGzAuhtdMjdryB1x0/6kEZw0VhO6uccNkceItF mdKPTi1WyyJEkGcy75PQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThlN-0000ZK-UW; Thu, 23 May 2019 07:03:41 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThj5-0005nU-GH; Thu, 23 May 2019 07:01:20 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 17/23] iommu/dma: Refactor iommu_dma_alloc, part 2 Date: Thu, 23 May 2019 09:00:22 +0200 Message-Id: <20190523070028.7435-18-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP All the logic in iommu_dma_alloc that deals with page allocation from the CMA or page allocators can be split into a self-contained helper, and we can than map the result of that or the atomic pool allocation with the iommu later. This also allows reusing __iommu_dma_free to tear down the allocations and MMU mappings when the IOMMU mapping fails. Based on a patch from Robin Murphy. Signed-off-by: Christoph Hellwig --- drivers/iommu/dma-iommu.c | 65 +++++++++++++++++++++------------------ 1 file changed, 35 insertions(+), 30 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 9ac76d286df1..9f0aa80f2bdd 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -972,35 +972,14 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr, __iommu_dma_free(dev, size, cpu_addr); } -static void *iommu_dma_alloc(struct device *dev, size_t size, - dma_addr_t *handle, gfp_t gfp, unsigned long attrs) +static void *iommu_dma_alloc_pages(struct device *dev, size_t size, + struct page **pagep, gfp_t gfp, unsigned long attrs) { bool coherent = dev_is_dma_coherent(dev); - int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); size_t alloc_size = PAGE_ALIGN(size); struct page *page = NULL; void *cpu_addr; - gfp |= __GFP_ZERO; - - if (gfpflags_allow_blocking(gfp) && - !(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) - return iommu_dma_alloc_remap(dev, size, handle, gfp, attrs); - - if (!gfpflags_allow_blocking(gfp) && !coherent) { - cpu_addr = dma_alloc_from_pool(alloc_size, &page, gfp); - if (!cpu_addr) - return NULL; - - *handle = __iommu_dma_map(dev, page_to_phys(page), size, - ioprot); - if (*handle == DMA_MAPPING_ERROR) { - dma_free_from_pool(cpu_addr, alloc_size); - return NULL; - } - return cpu_addr; - } - if (gfpflags_allow_blocking(gfp)) page = dma_alloc_from_contiguous(dev, alloc_size >> PAGE_SHIFT, get_order(alloc_size), @@ -1010,33 +989,59 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!page) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); - if (*handle == DMA_MAPPING_ERROR) - goto out_free_pages; - if (!coherent || PageHighMem(page)) { pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); cpu_addr = dma_common_contiguous_remap(page, alloc_size, VM_USERMAP, prot, __builtin_return_address(0)); if (!cpu_addr) - goto out_unmap; + goto out_free_pages; if (!coherent) arch_dma_prep_coherent(page, size); } else { cpu_addr = page_address(page); } + + *pagep = page; memset(cpu_addr, 0, alloc_size); return cpu_addr; -out_unmap: - __iommu_dma_unmap(dev, *handle, size); out_free_pages: if (!dma_release_from_contiguous(dev, page, alloc_size >> PAGE_SHIFT)) __free_pages(page, get_order(alloc_size)); return NULL; } +static void *iommu_dma_alloc(struct device *dev, size_t size, + dma_addr_t *handle, gfp_t gfp, unsigned long attrs) +{ + bool coherent = dev_is_dma_coherent(dev); + int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); + struct page *page = NULL; + void *cpu_addr; + + gfp |= __GFP_ZERO; + + if (gfpflags_allow_blocking(gfp) && + !(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) + return iommu_dma_alloc_remap(dev, size, handle, gfp, attrs); + + if (!gfpflags_allow_blocking(gfp) && !coherent) + cpu_addr = dma_alloc_from_pool(PAGE_ALIGN(size), &page, gfp); + else + cpu_addr = iommu_dma_alloc_pages(dev, size, &page, gfp, attrs); + if (!cpu_addr) + return NULL; + + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); + if (*handle == DMA_MAPPING_ERROR) { + __iommu_dma_free(dev, size, cpu_addr); + return NULL; + } + + return cpu_addr; +} + static int __iommu_dma_mmap_pfn(struct vm_area_struct *vma, unsigned long pfn, size_t size) { From patchwork Thu May 23 07:00:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956997 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B260A933 for ; Thu, 23 May 2019 07:04:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8975427CEA for ; Thu, 23 May 2019 07:04:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D528281B7; Thu, 23 May 2019 07:04:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 22C1627CEA for ; Thu, 23 May 2019 07:04:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=h40Y723hUhWz9QH0Emz7Z//OikZBKWCNdnAXzwhlbeQ=; b=VOEV0gcqG+p8zr 12+iI4WRVGNTVVaWJZ+GmZ7OMQ8MK3WPkXPj6bAKbc3+e+8L9DUkc1pibKqokUcTl2rDAPrALXZnl gFHBnNf73Oq0OZuaP8rjVRuR+1jBqV1L4cpsIsz8TcdrvDS/nkKl0RZ2OrO4J2jBcuMYE7TxiS7/d Vv9t8OIBVBweZHuMByqEC/VUYTZZUKgtVUHpwAHxlEK04Iyw697IX1YZUG6UFY6SFqFmmKjPAUuH0 qe7rZytikX0f2cAUgcQ04EbjLFmduCPIfhd7b8OXM+HFjv+TLnnYafFFEbZfgbh/bKERmyNf0/DBc i+ast+ygevNJIpTr+Vxw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThla-0000na-JY; Thu, 23 May 2019 07:03:54 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThj8-0005s9-C6; Thu, 23 May 2019 07:01:22 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 18/23] iommu/dma: Refactor iommu_dma_get_sgtable Date: Thu, 23 May 2019 09:00:23 +0200 Message-Id: <20190523070028.7435-19-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Inline __iommu_dma_get_sgtable_page into the main function, and use the fact that __iommu_dma_get_pages return NULL for remapped contigous allocations to simplify the code flow a bit. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 45 +++++++++++++++------------------------ 1 file changed, 17 insertions(+), 28 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 9f0aa80f2bdd..b56bd8e7d5f9 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1097,42 +1097,31 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, return __iommu_dma_mmap(pages, size, vma); } -static int __iommu_dma_get_sgtable_page(struct sg_table *sgt, struct page *page, - size_t size) -{ - int ret = sg_alloc_table(sgt, 1, GFP_KERNEL); - - if (!ret) - sg_set_page(sgt->sgl, page, PAGE_ALIGN(size), 0); - return ret; -} - static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs) { - unsigned int count = PAGE_ALIGN(size) >> PAGE_SHIFT; - struct page **pages; + struct page *page; + int ret; - if (!is_vmalloc_addr(cpu_addr)) { - struct page *page = virt_to_page(cpu_addr); - return __iommu_dma_get_sgtable_page(sgt, page, size); - } + if (is_vmalloc_addr(cpu_addr)) { + struct page **pages = __iommu_dma_get_pages(cpu_addr); - if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - /* - * DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, - * hence in the vmalloc space. - */ - struct page *page = vmalloc_to_page(cpu_addr); - return __iommu_dma_get_sgtable_page(sgt, page, size); + if (pages) { + return sg_alloc_table_from_pages(sgt, pages, + PAGE_ALIGN(size) >> PAGE_SHIFT, + 0, size, GFP_KERNEL); + } + + page = vmalloc_to_page(cpu_addr); + } else { + page = virt_to_page(cpu_addr); } - pages = __iommu_dma_get_pages(cpu_addr); - if (!pages) - return -ENXIO; - return sg_alloc_table_from_pages(sgt, pages, count, 0, size, - GFP_KERNEL); + ret = sg_alloc_table(sgt, 1, GFP_KERNEL); + if (!ret) + sg_set_page(sgt->sgl, page, PAGE_ALIGN(size), 0); + return ret; } static const struct dma_map_ops iommu_dma_ops = { From patchwork Thu May 23 07:00:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10956999 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 744AC933 for ; Thu, 23 May 2019 07:04:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 482C326256 for ; Thu, 23 May 2019 07:04:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 38445281D2; Thu, 23 May 2019 07:04:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7733826256 for ; Thu, 23 May 2019 07:04:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Lea3NVoGNG+oqwt9g1S/EPJRGe2Rsnpbqno4YGXi9AE=; b=cJO6YfKLvvkNL/ /y/t5eFlyneNmjxZyiVBGvgH34XLtB1w3PlJkdSKcThXXr9Vg1rjuYbGjIh44yv+WIsOdlpjw7pW5 1jKfu+rr1vs5a121zgr8a6kZ0WFH0YrboLOUJqAzRsPSdXv2Eca4ZUWjxU077MNeBP/DcEp58ZzX9 1pEQ9AnJ5a5KAdZTr4nxLo27L2VxtIUqd6aMVopc4+6A1+Qz93CaaaNt8UwfrhHoMMgSbPY2XzN1Y 50Qg9PqyHEBYbWzyr5+iH9+HXwxUl9egLNcLyVa6fEHkgZ1wJCC15si6f/1URwESmnQRAYxwQCjC1 fFYK615M+1qB5iANxHQQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThli-000115-K4; Thu, 23 May 2019 07:04:02 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjB-0005vy-41; Thu, 23 May 2019 07:01:25 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 19/23] iommu/dma: Refactor iommu_dma_mmap Date: Thu, 23 May 2019 09:00:24 +0200 Message-Id: <20190523070028.7435-20-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Inline __iommu_dma_mmap and __iommu_dma_mmap_pfn into the main function, and use the fact that __iommu_dma_get_pages return NULL for remapped contigous allocations to simplify the code flow a bit. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 60 +++++++-------------------------------- 1 file changed, 11 insertions(+), 49 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index b56bd8e7d5f9..ea2797d10070 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -650,21 +650,6 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, return NULL; } -/** - * __iommu_dma_mmap - Map a buffer into provided user VMA - * @pages: Array representing buffer from __iommu_dma_alloc() - * @size: Size of buffer in bytes - * @vma: VMA describing requested userspace mapping - * - * Maps the pages of the buffer in @pages into @vma. The caller is responsible - * for verifying the correct size and protection of @vma beforehand. - */ -static int __iommu_dma_mmap(struct page **pages, size_t size, - struct vm_area_struct *vma) -{ - return vm_map_pages(vma, pages, PAGE_ALIGN(size) >> PAGE_SHIFT); -} - static void iommu_dma_sync_single_for_cpu(struct device *dev, dma_addr_t dma_handle, size_t size, enum dma_data_direction dir) { @@ -1042,31 +1027,13 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, return cpu_addr; } -static int __iommu_dma_mmap_pfn(struct vm_area_struct *vma, - unsigned long pfn, size_t size) -{ - int ret = -ENXIO; - unsigned long nr_vma_pages = vma_pages(vma); - unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; - unsigned long off = vma->vm_pgoff; - - if (off < nr_pages && nr_vma_pages <= (nr_pages - off)) { - ret = remap_pfn_range(vma, vma->vm_start, - pfn + off, - vma->vm_end - vma->vm_start, - vma->vm_page_prot); - } - - return ret; -} - static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs) { unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT; unsigned long off = vma->vm_pgoff; - struct page **pages; + unsigned long pfn; int ret; vma->vm_page_prot = arch_dma_mmap_pgprot(dev, vma->vm_page_prot, attrs); @@ -1077,24 +1044,19 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, if (off >= nr_pages || vma_pages(vma) > nr_pages - off) return -ENXIO; - if (!is_vmalloc_addr(cpu_addr)) { - unsigned long pfn = page_to_pfn(virt_to_page(cpu_addr)); - return __iommu_dma_mmap_pfn(vma, pfn, size); - } + if (is_vmalloc_addr(cpu_addr)) { + struct page **pages = __iommu_dma_get_pages(cpu_addr); - if (attrs & DMA_ATTR_FORCE_CONTIGUOUS) { - /* - * DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, - * hence in the vmalloc space. - */ - unsigned long pfn = vmalloc_to_pfn(cpu_addr); - return __iommu_dma_mmap_pfn(vma, pfn, size); + if (pages) + return vm_map_pages(vma, pages, nr_pages); + pfn = vmalloc_to_pfn(cpu_addr); + } else { + pfn = page_to_pfn(virt_to_page(cpu_addr)); } - pages = __iommu_dma_get_pages(cpu_addr); - if (!pages) - return -ENXIO; - return __iommu_dma_mmap(pages, size, vma); + return remap_pfn_range(vma, vma->vm_start, pfn + off, + vma->vm_end - vma->vm_start, + vma->vm_page_prot); } static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, From patchwork Thu May 23 07:00:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10957001 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D3D815A6 for ; Thu, 23 May 2019 07:04:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 649AB27CEA for ; Thu, 23 May 2019 07:04:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5967D281E1; Thu, 23 May 2019 07:04:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E1B4E27CEA for ; Thu, 23 May 2019 07:04:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UbP6z3lJ90QteqVB85vebDOEllFMVVACLF4B7G84t68=; b=ZLBe8S+loA6bD1 VQGTrNoEDTTcTbfoeCosbr2kFWaKfZ0OBsUWTv5EyXFi+E+D0oVnYbs6Q8KHcdfleBAegjh03H34c 1gptv1QJxTHeQG/wR1d6jPNX1I1G7gmWWHamaxbFpzzG3akeM6hu7nvHwfgpepZYmrSYh7WNt/fy1 GMXeIvqwEP1HiR5QuGlLn4nFbPaswfzQzv3wVKk3v/mE0zed1XT7gi+KzA2RNyI3H1wzkr2x0LqKy SLaKBVcwoH2KiGAoTjGUQ1yvzi2wm4Iaqb3i4E1IUbBIRSKDNtmy51HA9Tl8PFzcPtZ3TaPsUSCxE uO8q9n59CfNjRqcZqwnA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThlu-0001GA-Ks; Thu, 23 May 2019 07:04:14 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjE-00060F-4z; Thu, 23 May 2019 07:01:28 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 20/23] iommu/dma: Don't depend on CONFIG_DMA_DIRECT_REMAP Date: Thu, 23 May 2019 09:00:25 +0200 Message-Id: <20190523070028.7435-21-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP For entirely dma coherent architectures there is no requirement to ever remap dma coherent allocation. Move all the remap and pool code under IS_ENABLED() checks and drop the Kconfig dependency. Signed-off-by: Christoph Hellwig Reviewed-by: Robin Murphy --- drivers/iommu/Kconfig | 1 - drivers/iommu/dma-iommu.c | 16 +++++++++------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index d6d063160dd6..83664db5221d 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -97,7 +97,6 @@ config IOMMU_DMA select IOMMU_IOVA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH - depends on DMA_DIRECT_REMAP config FSL_PAMU bool "Freescale IOMMU support" diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index ea2797d10070..567c300d1926 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -927,10 +927,11 @@ static void __iommu_dma_free(struct device *dev, size_t size, void *cpu_addr) struct page *page = NULL, **pages = NULL; /* Non-coherent atomic allocation? Easy */ - if (dma_free_from_pool(cpu_addr, alloc_size)) + if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && + dma_free_from_pool(cpu_addr, alloc_size)) return; - if (is_vmalloc_addr(cpu_addr)) { + if (IS_ENABLED(CONFIG_DMA_REMAP) && is_vmalloc_addr(cpu_addr)) { /* * If it the address is remapped, then it's either non-coherent * or highmem CMA, or an iommu_dma_alloc_remap() construction. @@ -974,7 +975,7 @@ static void *iommu_dma_alloc_pages(struct device *dev, size_t size, if (!page) return NULL; - if (!coherent || PageHighMem(page)) { + if (IS_ENABLED(CONFIG_DMA_REMAP) && (!coherent || PageHighMem(page))) { pgprot_t prot = arch_dma_mmap_pgprot(dev, PAGE_KERNEL, attrs); cpu_addr = dma_common_contiguous_remap(page, alloc_size, @@ -1007,11 +1008,12 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, gfp |= __GFP_ZERO; - if (gfpflags_allow_blocking(gfp) && + if (IS_ENABLED(CONFIG_DMA_REMAP) && gfpflags_allow_blocking(gfp) && !(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) return iommu_dma_alloc_remap(dev, size, handle, gfp, attrs); - if (!gfpflags_allow_blocking(gfp) && !coherent) + if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && + !gfpflags_allow_blocking(gfp) && !coherent) cpu_addr = dma_alloc_from_pool(PAGE_ALIGN(size), &page, gfp); else cpu_addr = iommu_dma_alloc_pages(dev, size, &page, gfp, attrs); @@ -1044,7 +1046,7 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, if (off >= nr_pages || vma_pages(vma) > nr_pages - off) return -ENXIO; - if (is_vmalloc_addr(cpu_addr)) { + if (IS_ENABLED(CONFIG_DMA_REMAP) && is_vmalloc_addr(cpu_addr)) { struct page **pages = __iommu_dma_get_pages(cpu_addr); if (pages) @@ -1066,7 +1068,7 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, struct page *page; int ret; - if (is_vmalloc_addr(cpu_addr)) { + if (IS_ENABLED(CONFIG_DMA_REMAP) && is_vmalloc_addr(cpu_addr)) { struct page **pages = __iommu_dma_get_pages(cpu_addr); if (pages) { From patchwork Thu May 23 07:00:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10957003 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 567781708 for ; Thu, 23 May 2019 07:04:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D12E28173 for ; Thu, 23 May 2019 07:04:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 21CCC281F9; Thu, 23 May 2019 07:04:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B3BE728173 for ; Thu, 23 May 2019 07:04:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=f2ccugCxMWHJAEFHJ+UqfPvJyyP3UYpBzKDPb6hqOuA=; b=KE1yqtwl+DP/nl I0xyu1kpDJj9Q2Ad1r0NKFI06E9DMXa8Om2/AZULx1haNEq1ACXNOczX1e3TI0VocwDRRFDzyc1lu BMSTUZAaq4k0IlcsUH3UIjRHZewmhcqsJ1coYZuh5AvIMxMiduQsntmXW1Fh+Y/19mbsd9VLuY6YD uq7GxIEtbafegPD7u+j/UMDy0Q6xfwDIqv/t86kZeKY5ruFVqIQSoxlqNPqcwbj+uGqLjddgD867X kXMON1HTbNrkeFV7ZKP6LcqG38G7+sEL3rB39amWV4IoVopwk3yFwd+ThQGFqV6noB44UfuPaZffq 0eQDp8L3P1Vxe9GCvJmg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThm2-0001Uy-Pj; Thu, 23 May 2019 07:04:22 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjG-00062C-Rw; Thu, 23 May 2019 07:01:31 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 21/23] iommu/dma: Switch copyright boilerplace to SPDX Date: Thu, 23 May 2019 09:00:26 +0200 Message-Id: <20190523070028.7435-22-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Christoph Hellwig Acked-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 13 +------------ include/linux/dma-iommu.h | 13 +------------ 2 files changed, 2 insertions(+), 24 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 567c300d1926..0233195dd196 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * A fairly generic DMA-API to IOMMU-API glue layer. * @@ -5,18 +6,6 @@ * * based in part on arch/arm/mm/dma-mapping.c: * Copyright (C) 2000-2004 Russell King - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see . */ #include diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h index b3cc3fb84079..05556f4d9cce 100644 --- a/include/linux/dma-iommu.h +++ b/include/linux/dma-iommu.h @@ -1,17 +1,6 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (C) 2014-2015 ARM Ltd. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see . */ #ifndef __DMA_IOMMU_H #define __DMA_IOMMU_H From patchwork Thu May 23 07:00:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10957005 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3FB82933 for ; Thu, 23 May 2019 07:05:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 162B828249 for ; Thu, 23 May 2019 07:05:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0A198282E8; Thu, 23 May 2019 07:05:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A05FE282EC for ; Thu, 23 May 2019 07:05:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eukRVLGwHPU+CMYDKjAh32+OE+SV7k5YDe5unlqercA=; b=LLN90bMugK8exU llAcAb3agmjRMjaKWG60TfPZ1jqVfq7h08/QjZ/RG1/chVKjppaB4k9cRnp1trq2NrtOlrjOmYcC7 G48w30V7xyDMcwes3wAdzIifBW2LwQbSqB8KPY7UhpS23aO63LGn7CpApFoqIYNJX0NUtKB1rIK0L Z4WHpwEi0IjSRuzibRUjeh9WC+Ekk4ORccT3OXRrjXR3uv8L0Hw4ipfMBuTLrTzXYDe2h97QMd7Fz Zwtcgz2DOTAYUoEphkoUa884rzQYfP14dZOqc2aWjj+dQ3sJc6w+nl0o9YPJFgPI2IC9EuP4hLr0K i/rcK/VU8M0yvYbjHfaw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThnQ-0003JW-AI; Thu, 23 May 2019 07:05:48 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjJ-00062m-MF; Thu, 23 May 2019 07:01:34 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 22/23] arm64: switch copyright boilerplace to SPDX in dma-mapping.c Date: Thu, 23 May 2019 09:00:27 +0200 Message-Id: <20190523070028.7435-23-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Mukesh Ojha , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Christoph Hellwig Acked-by: Robin Murphy Reviewed-by: Mukesh Ojha Acked-by: Catalin Marinas --- arch/arm64/mm/dma-mapping.c | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index d1661f78eb4d..184ef9ccd69d 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -1,20 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0 /* - * SWIOTLB-based DMA API implementation - * * Copyright (C) 2012 ARM Ltd. * Author: Catalin Marinas - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see . */ #include From patchwork Thu May 23 07:00:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 10957007 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E462933 for ; Thu, 23 May 2019 07:05:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E878227F94 for ; Thu, 23 May 2019 07:05:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DCD0C28249; Thu, 23 May 2019 07:05:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 97530282EC for ; Thu, 23 May 2019 07:05:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4uWrkX9R64iHrg6+6obZ+1TeTHa2/McDIRHo0DFNXj8=; b=p8EaT9R22QyBGu 31aVFKbClRrHkbmbqDoLW4SGKnFigya1EpH4owM8JPpvZMKL11qtfqx9tlkvz4QUp6cbhw3ze7eky yQqxEP6OHLmQD0yElopaoPuIuzfdiqBN2GGthfLOF9UNkKZvfNcedMFHrthb/h2G5y1Z86hCiPzl5 ShHjNxKoBauXNnvpQ/NLVpr4YWvBS1of9PtecB23aFL8r20hczHqZaupJI0GN+jYZWHn9LKoAx1Gv QRhoJZ68ZDKzt+ED3He1/t4xKxC34on9zy5+q3MhO9oRqxLUszNQ1eCGublWiYTYjVmJBDrTgVwzv KBMwCHJds2MojcP6H5ng==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThnU-0003PU-UM; Thu, 23 May 2019 07:05:52 +0000 Received: from 213-225-10-46.nat.highway.a1.net ([213.225.10.46] helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.90_1 #2 (Red Hat Linux)) id 1hThjM-00065b-BR; Thu, 23 May 2019 07:01:37 +0000 From: Christoph Hellwig To: Robin Murphy Subject: [PATCH 23/23] arm64: trim includes in dma-mapping.c Date: Thu, 23 May 2019 09:00:28 +0200 Message-Id: <20190523070028.7435-24-hch@lst.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190523070028.7435-1-hch@lst.de> References: <20190523070028.7435-1-hch@lst.de> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Murphy , Catalin Marinas , Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP With most of the previous functionality now elsewhere a lot of the headers included in this file are not needed. Signed-off-by: Christoph Hellwig Acked-by: Catalin Marinas --- arch/arm64/mm/dma-mapping.c | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index 184ef9ccd69d..1669618db08a 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -5,19 +5,9 @@ */ #include -#include -#include #include -#include -#include -#include -#include #include -#include #include -#include -#include -#include #include