From patchwork Thu Apr 1 16:47:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12178409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7407DC433ED for ; Thu, 1 Apr 2021 16:50:25 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C4A061389 for ; Thu, 1 Apr 2021 16:50:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C4A061389 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BC5hBRWzkR+Kd5t3qoj9oprRikRm0a3VVISAQlFUixg=; b=Ojnf/0/RGxz+GryccQZL/8rHd ovKTbsc/2yqqyD4pyORZJq2A7lAAaTjdoGvfBdfNMJmRp9zS61+kEi5lGRKq3puCkrXHPcfqpaa0r 5U4Z1VLOgD3e4vXNJfnXBl2BDzKW2Ol/6p3L4+shq/xSlvbuePoRB0NALKHojF6SK4iV3oH21jm4a 0p4BkO2XZ1NqaIP8srVBd1Q3DXtTfzgmt+5NS6ss8oCJjX7U1vM6fojH2B+k6V9ZrJohisVGjMoy+ 15kAmBDkdpYfoHWiAFW53JshihHAGfsx9YSAEoujERbyZ69fylmBk/Tg4wNoKKXokEEQBf5EHpo1x YdxqWrhMA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Us-00ANjj-BE; Thu, 01 Apr 2021 16:48:42 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Ul-00ANi7-E6 for linux-arm-kernel@lists.infradead.org; Thu, 01 Apr 2021 16:48:37 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id E527761382; Thu, 1 Apr 2021 16:48:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617295714; bh=sT6+Td6tnAXtRRYdO8fsQ55JCnLxj+SLLHhf7/QPLgo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KSb8ii2ppv2mF8vBxF0OCjLP8ZcATsDmg6EiZi4GbeXuPK86p2he/8WFSIsseUsxX 9/7AwZ5w1q9mN+552kDlDeLGTmA46rCURAupIFTdKN9ayFlSm5PRPBngmD5KcCsh+v HjtKd6tar5UZW+PZvdZkiCF3Xr+J8BmSNyBuAuiqZIQgXjIKEAU+P2ilKKEerTzYFH VzLXTp+kd6aioAWdor7fu0lmr969vRt4iXTme8NpbQxiPBFgFGqa6YoH85lK6XdSJ+ 2/TzrP0vKFE+aniahiBoZI5KAQz3s3WthCbANsV+akPxtbGQAsE1NdT1pRysw4BZ3A h2foJF1F6B+vw== From: Will Deacon To: iommu@lists.linux-foundation.org Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , "Isaac J. Manjarres" , Pratik Patel , Robin Murphy , Lu Baolu Subject: [RFC PATCH 1/6] iommu/io-pgtable: Introduce unmap_pages() as a page table op Date: Thu, 1 Apr 2021 17:47:33 +0100 Message-Id: <20210401164738.9513-2-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210401164738.9513-1-will@kernel.org> References: <20210401164738.9513-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210401_174835_713416_45812500 X-CRM114-Status: GOOD ( 11.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Isaac J. Manjarres" The io-pgtable code expects to operate on a single block or granule of memory that is supported by the IOMMU hardware when unmapping memory. This means that when a large buffer that consists of multiple such blocks is unmapped, the io-pgtable code will walk the page tables to the correct level to unmap each block, even for blocks that are virtually contiguous and at the same level, which can incur an overhead in performance. Introduce the unmap_pages() page table op to express to the io-pgtable code that it should unmap a number of blocks of the same size, instead of a single block. Doing so allows multiple blocks to be unmapped in one call to the io-pgtable code, reducing the number of page table walks, and indirect calls. Signed-off-by: Isaac J. Manjarres Suggested-by: Will Deacon Signed-off-by: Will Deacon --- include/linux/io-pgtable.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/io-pgtable.h b/include/linux/io-pgtable.h index a4c9ca2c31f1..2ed0c057d9e7 100644 --- a/include/linux/io-pgtable.h +++ b/include/linux/io-pgtable.h @@ -144,6 +144,7 @@ struct io_pgtable_cfg { * * @map: Map a physically contiguous memory region. * @unmap: Unmap a physically contiguous memory region. + * @unmap_pages: Unmap a range of virtually contiguous pages of the same size. * @iova_to_phys: Translate iova to physical address. * * These functions map directly onto the iommu_ops member functions with @@ -154,6 +155,9 @@ struct io_pgtable_ops { phys_addr_t paddr, size_t size, int prot, gfp_t gfp); size_t (*unmap)(struct io_pgtable_ops *ops, unsigned long iova, size_t size, struct iommu_iotlb_gather *gather); + size_t (*unmap_pages)(struct io_pgtable_ops *ops, unsigned long iova, + size_t pgsize, size_t pgcount, + struct iommu_iotlb_gather *gather); phys_addr_t (*iova_to_phys)(struct io_pgtable_ops *ops, unsigned long iova); }; From patchwork Thu Apr 1 16:47:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12178413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD251C43460 for ; Thu, 1 Apr 2021 16:50:29 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4BD206120D for ; Thu, 1 Apr 2021 16:50:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4BD206120D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/TGgnhZO50HGdLPjgqUJkDpyWfcRl1kYFGAE6wcF5Cw=; b=klbPL2CDk/kR8w1S41FBd5BXQ 7HcGe+gw9UgYC3U3/oDLL70dej/bQ3Ky6jUfRGceDXBclSvdOZLRSf+D3P8xKwzkvuAa5PoZAD9bP TBX1BS6BLoRIPFhQN6VshbMIQWQllWzZmO1RWy1wOyYXyyiQJpciTLbVIO3rjzl9ODvUuP/JmjUHu DjiG20Z/cR7B3GExJpnI1dGKYz9XCUatVCFsrEezegUxycfyqlSzJtlcV50916X/tuQDR8lghmCky HZxQNKcfoZ2WG9EPyXCI/YsVY6N6LBPzi53D3oUaJ49ifNtD0jjSXHTINi0ApZlGL7eof/7sv6rYU y0OULyj1g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS0V5-00ANmL-6F; Thu, 01 Apr 2021 16:48:55 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Un-00ANia-37 for linux-arm-kernel@lists.infradead.org; Thu, 01 Apr 2021 16:48:39 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id A013A61384; Thu, 1 Apr 2021 16:48:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617295715; bh=p4hoXNujhDB52s4CkDR3mvKvMLihsFk6ec9xg1tPeII=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HjXHx5Wcc0eFWhBXlm6rO7YJkyJ7ISLX/ZG6hikgYM0O4Dv6A8PZ+BJvSevosmyID BMQHqPjKuj9LWOkmQBsN2muIcjZTGAE2S9vpZ0sZyqdl7coMotpFWX4qkB9DASdAdA 8MOYbNkn1wTKrQKd9ElOL38sqBegH///SqfyTM38A9hpPK/aN5tj6UmNSjt5wr21zl BEE7c57t1BkZB+Txn4rPCbRMopaWbR81Q0MfM4/F0ybVIPYJcb5sQmEA17MS2mYWRv 6cJA2/T8yfcQnbM5oXhZ76tbrih3c8kbk+J0gmyWhpKEDFlqjqyaLgPyIOPNP5j71X YE5ETBtoFYGLA== From: Will Deacon To: iommu@lists.linux-foundation.org Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , "Isaac J. Manjarres" , Pratik Patel , Robin Murphy , Lu Baolu Subject: [RFC PATCH 2/6] iommu: Add an unmap_pages() op for IOMMU drivers Date: Thu, 1 Apr 2021 17:47:34 +0100 Message-Id: <20210401164738.9513-3-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210401164738.9513-1-will@kernel.org> References: <20210401164738.9513-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210401_174837_359563_BD180F54 X-CRM114-Status: GOOD ( 10.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: "Isaac J. Manjarres" Add a callback for IOMMU drivers to provide a path for the IOMMU framework to call into an IOMMU driver, which can call into the io-pgtable code, to unmap a virtually contiguous range of pages of the same size. For IOMMU drivers that do not specify an unmap_pages() callback, the existing logic of unmapping memory one page block at a time will be used. Signed-off-by: Isaac J. Manjarres Suggested-by: Will Deacon Signed-off-by: Will Deacon --- include/linux/iommu.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 5e7fe519430a..9cf81242581a 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -193,6 +193,7 @@ struct iommu_iotlb_gather { * @detach_dev: detach device from an iommu domain * @map: map a physically contiguous memory region to an iommu domain * @unmap: unmap a physically contiguous memory region from an iommu domain + * @unmap_pages: unmap a number of pages of the same size from an iommu domain * @flush_iotlb_all: Synchronously flush all hardware TLBs for this domain * @iotlb_sync_map: Sync mappings created recently using @map to the hardware * @iotlb_sync: Flush all queued ranges from the hardware TLBs and empty flush @@ -245,6 +246,9 @@ struct iommu_ops { phys_addr_t paddr, size_t size, int prot, gfp_t gfp); size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, size_t size, struct iommu_iotlb_gather *iotlb_gather); + size_t (*unmap_pages)(struct iommu_domain *domain, unsigned long iova, + size_t pgsize, size_t pgcount, + struct iommu_iotlb_gather *iotlb_gather); void (*flush_iotlb_all)(struct iommu_domain *domain); void (*iotlb_sync_map)(struct iommu_domain *domain, unsigned long iova, size_t size); From patchwork Thu Apr 1 16:47:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12178415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14B5CC433ED for ; Thu, 1 Apr 2021 16:50:32 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 99FF66120D for ; Thu, 1 Apr 2021 16:50:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99FF66120D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=KzYSWvaM0tbELdJo+Uhu0+kKqNPnTpvr4wgFuXPuiJY=; b=glBLpaXkgWfVH7e25ndKXFCNE FFT5BbHk1TVbKr4kd6g9CVzCAFlPz1kN+OEKx8cop9EOnzd1DodtK0XYeRUi0NtV+F8mR47J5gFL1 cBOyx1B6id2MEojDrQBByAPC+glPVkFY34spHU3e8UMW5kv+JTDi4uPS8xS+epPhhK/3E5LRtlqRV bQug/L5zMKxpPkz5XzVhN8jTlJJ7rZJtFLket+uVejnr0+UsqxgtMyHiQ46H2XUb2fQ/SPEIyqp4D a0rVIlTZNOK/RaSh67wPo+6Vw31FUnIFTJd0fjWQ/OKc71Z34KFzlx5Adk6VGTza491J7iEaRwwWM DldjSGuyA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS0VC-00ANoD-8u; Thu, 01 Apr 2021 16:49:02 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Uo-00ANj9-Qc for linux-arm-kernel@lists.infradead.org; Thu, 01 Apr 2021 16:48:40 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 53F8C6138D; Thu, 1 Apr 2021 16:48:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617295717; bh=N+JQHEXYptK0jkhRJ2ALB6gJZB1U2b6jwOELVOFQslc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dyBIUYidA0aR58n5PWMp820F/rDFXTSM8yCGLBKWONn4w4YyyWHoki6al//cvPPJZ lTL0cUXAKkdpFPNrBAugtS7PfjXac63FEpX3R/N2JXxnGgNMiWzNDObQUPvhPe6Glx Tb1ISVZvD6fu4o7lVVi1WOJrSuQ3hHCToh9IxIahdQgXDVcl1tV1V4nfU4LQ5A5DSZ BJUqTeEE66z94WutT0LUndVp65sIc5EeYSUaSdHcnOKcyFTG8L0axwPJFAndsZN/QV MXSt+6///1Yq/aEGrqu2CYE+svzFLP0DRQ4dMO0beaGS1AL+5Mk5lp2R3dP1sckHvl 2v+vFHB5TYdfg== From: Will Deacon To: iommu@lists.linux-foundation.org Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , "Isaac J. Manjarres" , Pratik Patel , Robin Murphy , Lu Baolu Subject: [RFC PATCH 3/6] iommu: Use bitmap to calculate page size in iommu_pgsize() Date: Thu, 1 Apr 2021 17:47:35 +0100 Message-Id: <20210401164738.9513-4-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210401164738.9513-1-will@kernel.org> References: <20210401164738.9513-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210401_174839_088641_7FDE1510 X-CRM114-Status: GOOD ( 13.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Avoid the potential for shifting values by amounts greater than the width of their type by using a bitmap to compute page size in iommu_pgsize(). Signed-off-by: Will Deacon --- drivers/iommu/iommu.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index d0b0a15dba84..bcd623862bf9 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -2360,30 +2361,22 @@ static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long addr_merge, size_t size) { unsigned int pgsize_idx; + unsigned long pgsizes; size_t pgsize; - /* Max page size that still fits into 'size' */ - pgsize_idx = __fls(size); + /* Page sizes supported by the hardware and small enough for @size */ + pgsizes = domain->pgsize_bitmap & GENMASK(__fls(size), 0); - /* need to consider alignment requirements ? */ - if (likely(addr_merge)) { - /* Max page size allowed by address */ - unsigned int align_pgsize_idx = __ffs(addr_merge); - pgsize_idx = min(pgsize_idx, align_pgsize_idx); - } - - /* build a mask of acceptable page sizes */ - pgsize = (1UL << (pgsize_idx + 1)) - 1; - - /* throw away page sizes not supported by the hardware */ - pgsize &= domain->pgsize_bitmap; + /* Constrain the page sizes further based on the maximum alignment */ + if (likely(addr_merge)) + pgsizes &= GENMASK(__ffs(addr_merge), 0); - /* make sure we're still sane */ - BUG_ON(!pgsize); + /* Make sure we have at least one suitable page size */ + BUG_ON(!pgsizes); - /* pick the biggest page */ - pgsize_idx = __fls(pgsize); - pgsize = 1UL << pgsize_idx; + /* Pick the biggest page size remaining */ + pgsize_idx = __fls(pgsizes); + pgsize = BIT(pgsize_idx); return pgsize; } From patchwork Thu Apr 1 16:47:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12178417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68E74C433B4 for ; Thu, 1 Apr 2021 16:50:33 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 00A2761389 for ; Thu, 1 Apr 2021 16:50:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00A2761389 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=o9ODgvUPFUgTEG9V85OCBGMBrQpHx49jOPE0rduR1rQ=; b=eA0RZGJ04HwI2w7Dchc2k9Znd 87gRcN9PnJ1ytoo7vxZBFEJ41BXfZeUDTnjhlPL8e9X8iI9AaPmWUCqmbg7YrxPvZZS+A/rh2EkcA Se0416m019IpiRK+1+UryI18lNTRUVEXytZHRhjYVvFQTfP861S3dHzNqiKBcZSoBgq+Eg9fGz1OU j1U5i9u7n4W8YO0Lp7QuYsbg1cq/1qE6q/4i0a+k1o0ElRx79V/C2So3Wjv9rfF2NKyzXQdhTvZ39 gbvI9r1n4QJoOPriCbyBoZOfp5PBi36cJshAmSTi5Xa8qODn2I3UQNxTzoSNwGx8Q0tQQuLrSZ+HM wEwmVKlMg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS0VK-00ANqE-UN; Thu, 01 Apr 2021 16:49:11 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Uq-00ANjn-Lh for linux-arm-kernel@lists.infradead.org; Thu, 01 Apr 2021 16:48:43 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 095EC61382; Thu, 1 Apr 2021 16:48:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617295719; bh=U+eQcrRlMdYPEEIk+UyCEfZu0TkCUUl69Fe662VNo/g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L0l+kRwkYGUy1DE5enMdmDL48W0/3m0sxt9BJefph5sb0+BtyjsYU9e1nP8l8OgnE Cn7VggTIAiH7PLBBA15n8Rc7WsglI17WFqQfWCZvg8jjVwddHtw6LJrsU9/Ikv6mpo jkqclznt+kFIAtXGq+R34otxYYGcbZKucmx/1Ankg2XUIB9XCYtZfCZbT64g7emC+p pLZZqgHvcUSR8n5kFcTE9L+1QhnfU1r82rdBVvHV2LVMw9KIPAZqdg5/qINZACnR1G FiFzbs3V2naTnE/6nf7ftp8HjfXPBBpjDXsBhLJtjlTT9w6suSgReCBaMNN9GKw9VV S/WLl3oAwDtTQ== From: Will Deacon To: iommu@lists.linux-foundation.org Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , "Isaac J. Manjarres" , Pratik Patel , Robin Murphy , Lu Baolu Subject: [RFC PATCH 4/6] iommu: Split 'addr_merge' argument to iommu_pgsize() into separate parts Date: Thu, 1 Apr 2021 17:47:36 +0100 Message-Id: <20210401164738.9513-5-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210401164738.9513-1-will@kernel.org> References: <20210401164738.9513-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210401_174842_170483_AE786BDA X-CRM114-Status: GOOD ( 10.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The 'addr_merge' parameter to iommu_pgsize() is a fabricated address intended to describe the alignment requirements to consider when choosing an appropriate page size. On the iommu_map() path, this address is the logical OR of the virtual and physical addresses. Subsequent improvements to iommu_pgsize() will need to check the alignment of the virtual and physical components of 'addr_merge' independently, so pass them in as separate parameters and reconstruct 'addr_merge' locally. No functional change. Signed-off-by: Will Deacon --- drivers/iommu/iommu.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index bcd623862bf9..ab689611a03b 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2357,12 +2357,13 @@ phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) } EXPORT_SYMBOL_GPL(iommu_iova_to_phys); -static size_t iommu_pgsize(struct iommu_domain *domain, - unsigned long addr_merge, size_t size) +static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size) { unsigned int pgsize_idx; unsigned long pgsizes; size_t pgsize; + phys_addr_t addr_merge = paddr | iova; /* Page sizes supported by the hardware and small enough for @size */ pgsizes = domain->pgsize_bitmap & GENMASK(__fls(size), 0); @@ -2415,7 +2416,7 @@ static int __iommu_map(struct iommu_domain *domain, unsigned long iova, pr_debug("map: iova 0x%lx pa %pa size 0x%zx\n", iova, &paddr, size); while (size) { - size_t pgsize = iommu_pgsize(domain, iova | paddr, size); + size_t pgsize = iommu_pgsize(domain, iova, paddr, size); pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", iova, &paddr, pgsize); @@ -2503,8 +2504,9 @@ static size_t __iommu_unmap(struct iommu_domain *domain, * or we hit an area that isn't mapped. */ while (unmapped < size) { - size_t pgsize = iommu_pgsize(domain, iova, size - unmapped); + size_t pgsize; + pgsize = iommu_pgsize(domain, iova, iova, size - unmapped); unmapped_page = ops->unmap(domain, iova, pgsize, iotlb_gather); if (!unmapped_page) break; From patchwork Thu Apr 1 16:47:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12178419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19B6EC433ED for ; Thu, 1 Apr 2021 16:50:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 99ADF6120D for ; Thu, 1 Apr 2021 16:50:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99ADF6120D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=C6jlV+XvOPTQmXX0iYUavtDrZA2L9fR7twBnNc7kOnA=; b=lBt1VOEwhGOcMx898Ev7Pkdtc 3CMEKoAWxB9kZTOLS0yqkFsSXvZjUAwuPSDK8C3tGDQP2FlQaIIEOnb9p8Yqql5Qg6y/Zlbm9eB/p gKnL5rXUfEgaK6SkoN2kOZ+PudrcbQV3Qb+3tFQvqo7r7Rq+UC8L4J5CIuAIHYiKGuxYQadjBsqd3 HUWxwW6Gl/TvSNyLC6RKToKg6wY+77+m3s8EaUhKg+VE9UFCeoItAM/OrrAV7gYGx5DTrTCISFFRI KwxjKrajOfh+xHB5g8qPYxC57KDrn/17bkZfR21n/snQRxI8oHaHAQIfgxqFRRiRi572XjZhM7yLN 9+Wb8JDrg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS0VS-00ANsS-Ty; Thu, 01 Apr 2021 16:49:19 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Us-00ANjw-6L for linux-arm-kernel@lists.infradead.org; Thu, 01 Apr 2021 16:48:44 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id B05F961390; Thu, 1 Apr 2021 16:48:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617295721; bh=ixFflCA3kCijj1uJNIph6hG5Yjr6CXFdBFZADdeuL4c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HWnD6Wamspc9siukU7TVrJBt2v6ocZOyiorbLmeV44TQBx0DJ9wvkOP2F+MZwQgXP ukS2b6AR2BEqGVss/XPIWjC9jsIHeweXtxWi2E+423vyvItRyq3ILSrY2femLSLJ9H LWQ7SQnTOREQzL3GCFgb4VXtBcGPHp1zrvH1523V3kNSdRgDnwvQFELKKFEEALRIsh ZfGlgbfweLQGwEjVwdjm1+Cm3G9qYYUTSDM66r11VaX0fziBJ2qth7a8eml2AZ6OIN Iw9oIL0eDyGNUC+U5gexN885crYSDlYo9iRdctga7V0t5PddsDFhGE+YLoeqIf/ljL bWmVgaVRkoOGA== From: Will Deacon To: iommu@lists.linux-foundation.org Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , "Isaac J. Manjarres" , Pratik Patel , Robin Murphy , Lu Baolu Subject: [RFC PATCH 5/6] iommu: Hook up '->unmap_pages' driver callback Date: Thu, 1 Apr 2021 17:47:37 +0100 Message-Id: <20210401164738.9513-6-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210401164738.9513-1-will@kernel.org> References: <20210401164738.9513-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210401_174842_588099_3A9ED846 X-CRM114-Status: GOOD ( 10.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Extend iommu_pgsize() to populate an optional 'count' paramater so that we can direct unmapping operation to the ->unmap_pages callback if it has been provided by the driver. Signed-off-by: Will Deacon --- drivers/iommu/iommu.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index ab689611a03b..fe186691fc21 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2358,7 +2358,7 @@ phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) EXPORT_SYMBOL_GPL(iommu_iova_to_phys); static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size) + phys_addr_t paddr, size_t size, size_t *count) { unsigned int pgsize_idx; unsigned long pgsizes; @@ -2379,6 +2379,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long iova, pgsize_idx = __fls(pgsizes); pgsize = BIT(pgsize_idx); + if (count) + *count = size >> pgsize_idx; return pgsize; } @@ -2416,7 +2418,7 @@ static int __iommu_map(struct iommu_domain *domain, unsigned long iova, pr_debug("map: iova 0x%lx pa %pa size 0x%zx\n", iova, &paddr, size); while (size) { - size_t pgsize = iommu_pgsize(domain, iova, paddr, size); + size_t pgsize = iommu_pgsize(domain, iova, paddr, size, NULL); pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", iova, &paddr, pgsize); @@ -2467,6 +2469,19 @@ int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, } EXPORT_SYMBOL_GPL(iommu_map_atomic); +static size_t __iommu_unmap_pages(struct iommu_domain *domain, + unsigned long iova, size_t size, + struct iommu_iotlb_gather *iotlb_gather) +{ + const struct iommu_ops *ops = domain->ops; + size_t pgsize, count; + + pgsize = iommu_pgsize(domain, iova, iova, size, &count); + return ops->unmap_pages ? + ops->unmap_pages(domain, iova, pgsize, count, iotlb_gather) : + ops->unmap(domain, iova, pgsize, iotlb_gather); +} + static size_t __iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size, struct iommu_iotlb_gather *iotlb_gather) @@ -2504,10 +2519,9 @@ static size_t __iommu_unmap(struct iommu_domain *domain, * or we hit an area that isn't mapped. */ while (unmapped < size) { - size_t pgsize; - - pgsize = iommu_pgsize(domain, iova, iova, size - unmapped); - unmapped_page = ops->unmap(domain, iova, pgsize, iotlb_gather); + unmapped_page = __iommu_unmap_pages(domain, iova, + size - unmapped, + iotlb_gather); if (!unmapped_page) break; From patchwork Thu Apr 1 16:47:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 12178421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE6EEC433ED for ; Thu, 1 Apr 2021 16:50:47 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 717F761389 for ; Thu, 1 Apr 2021 16:50:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 717F761389 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fao7DpY+0baWqpKJdltXDInoKht5/JmXXVoNqFc3B5o=; b=TK28YSoRfYNwTxhl9vLa3xCUt 76VAvQKLjLXEaYmnez1KY3qsFqDuA2OcAYoFZoYe6wNY8zCvvgxFGmOsZ4xkI8aHZmKfNMtGo7Jlb gFzV0Vs8K3ggIVoCXISCfIjpfAaXjjmIBhCEM+DGAd13TWRfIcFny6XMvj2gYVf0U6PgoPbAemfhT DJl/jrtJldUNuqg6wEqAqPJjFM3d0ZA9vNkqd+5YxW9hcxHKrEudKYCyvU/KSnjvamBO/4Yx0Ov8Y eCDjNWqvCP7bFj7WoGubaQx/UJIEdH94ir4zbzag9t/AufQRtgVPZcw7bb3Rl5UXpqQkMaOGxhLqv sXyxa7KTA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Vf-00ANwd-Ux; Thu, 01 Apr 2021 16:49:32 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS0Ut-00ANkc-Ra for linux-arm-kernel@lists.infradead.org; Thu, 01 Apr 2021 16:48:45 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 6738C61389; Thu, 1 Apr 2021 16:48:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617295722; bh=m+QOtK4MBnW2L0WweNAXI08Q1Wjuv41DugP348K1TW8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KS94AAUr/WbiLXrpHLRYc4VWJiMwix2aQ1o6JIJ17PhemR9TIwjtwD7/clEb8GS0x 10aQlthgJR8DPT6EQe8r+DH69DdqWx1yA1qTZLRUxrxA/Mlt8xgSWPtEm5uROVn7Pu eJ18hz8j/aN6DcgkDZt7+Jf975cRFo1Pa1TbgTCIBabCcBC9z5EdLYwgG1w1XdFxYE d39LF7XGXBa5AlqPRRbTqyLuYznr6y2niVDSSSm/DAwkCzk098q9c0+OqfZThd/P0u vrTar44pZxuoxmIEztfv6NpN/8B4ufvHuPh7Xp4f//atqCRQmvElnj47lqJ1MUjZXB u64Er8zWs32oA== From: Will Deacon To: iommu@lists.linux-foundation.org Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , "Isaac J. Manjarres" , Pratik Patel , Robin Murphy , Lu Baolu Subject: [RFC PATCH 6/6] iommu: Accomodate larger pages in iommu_pgsize() 'count' calculation Date: Thu, 1 Apr 2021 17:47:38 +0100 Message-Id: <20210401164738.9513-7-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210401164738.9513-1-will@kernel.org> References: <20210401164738.9513-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210401_174844_121820_731A3B73 X-CRM114-Status: GOOD ( 15.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Extend the calculation of 'count' in iommu_pgsize() so that it takes larger page sizes into consideration and returns a value which will allow a larger page size to be used on the next call. Signed-off-by: Will Deacon --- drivers/iommu/iommu.c | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index fe186691fc21..0572a4dcd9e2 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2360,9 +2360,9 @@ EXPORT_SYMBOL_GPL(iommu_iova_to_phys); static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, size_t *count) { - unsigned int pgsize_idx; + unsigned int pgsize_idx, pgsize_idx_next; unsigned long pgsizes; - size_t pgsize; + size_t offset, pgsize, pgsize_next; phys_addr_t addr_merge = paddr | iova; /* Page sizes supported by the hardware and small enough for @size */ @@ -2378,9 +2378,37 @@ static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long iova, /* Pick the biggest page size remaining */ pgsize_idx = __fls(pgsizes); pgsize = BIT(pgsize_idx); + if (!count) + return pgsize; - if (count) - *count = size >> pgsize_idx; + + /* Find the next biggest support page size, if it exists */ + pgsizes = domain->pgsize_bitmap & ~GENMASK(pgsize_idx, 0); + if (!pgsizes) + goto out_set_count; + + pgsize_idx_next = __ffs(pgsizes); + pgsize_next = BIT(pgsize_idx_next); + + /* + * There's no point trying a bigger page size unless the virtual + * and physical addresses are similarly offset within the larger page. + */ + if ((iova ^ paddr) & (pgsize_next - 1)) + goto out_set_count; + + /* Calculate the offset to the next page size alignment boundary */ + offset = pgsize_next - (addr_merge & (pgsize_next - 1)); + + /* + * If size is big enough to accomodate the larger page, reduce + * the number of smaller pages. + */ + if (offset + pgsize_next <= size) + size = offset; + +out_set_count: + *count = size >> pgsize_idx; return pgsize; }