From patchwork Mon Aug 8 17:49:34 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laura Abbott X-Patchwork-Id: 9269089 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 12BE760839 for ; Mon, 8 Aug 2016 17:52:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07F4228161 for ; Mon, 8 Aug 2016 17:52:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F0A30283A6; Mon, 8 Aug 2016 17:52:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 66AB928156 for ; Mon, 8 Aug 2016 17:52:17 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1bWohC-0001Wu-B3; Mon, 08 Aug 2016 17:50:38 +0000 Received: from mail-qt0-f182.google.com ([209.85.216.182]) by bombadil.infradead.org with esmtps (Exim 4.85_2 #1 (Red Hat Linux)) id 1bWogo-0000Cl-FS for linux-arm-kernel@lists.infradead.org; Mon, 08 Aug 2016 17:50:17 +0000 Received: by mail-qt0-f182.google.com with SMTP id x25so207346821qtx.2 for ; Mon, 08 Aug 2016 10:49:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xc1iMgirNhCVdImJme2LC0pIuqw2pfYGn4QMXJbGPMk=; b=IbSqcM3RtlUBzZDV7az5F6qzPjx6sOAEJ21c4+WGQ1pHBmv2/Zj30rsqK2IxBVGqfn PaAv9ol9KrwRA2XAz1LAZfLT3bpsDNbNY+LsQ0QttM1/kF98u0WL3jAQtbsG4CFIUqUQ kINaNmK9zjKYUGxF9GTQAYVLwMeu+p3701dJFZmPaIKXmjQ6MqlE9yDAUNYzfFkFg5XC FmldmGPnf/ofmszZ2oTAPk1gApg9fqqQIGi7zUi/Pz9EITnSyka7XQxlDMcyOneFb+Iv swecAZgD/vr3Xest43mgkDyD5oilnkHRZdpHwbpy3aJF0b7KfZR7pIpZtBocuxReUMNe v9qg== X-Gm-Message-State: AEkoouvIGJU4/B99Ub9YuMAYBDFCdhxcZLKI0n6rGfWQ0AbIFWqltumiXtAr+xKimI7HHz54 X-Received: by 10.237.49.5 with SMTP id 5mr29473609qtg.136.1470678593134; Mon, 08 Aug 2016 10:49:53 -0700 (PDT) Received: from labbott-redhat-machine.redhat.com ([2601:602:9800:177f::337f]) by smtp.gmail.com with ESMTPSA id v22sm18036577qtc.19.2016.08.08.10.49.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Aug 2016 10:49:52 -0700 (PDT) From: Laura Abbott To: Sumit Semwal , John Stultz , =?UTF-8?q?Arve=20Hj=C3=B8nnev=C3=A5g?= , Riley Andrews Subject: [RFCv2][PATCH 2/5] arm: Implement ARCH_HAS_FORCE_CACHE Date: Mon, 8 Aug 2016 10:49:34 -0700 Message-Id: <1470678577-14010-3-git-send-email-labbott@redhat.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1470678577-14010-1-git-send-email-labbott@redhat.com> References: <1470678577-14010-1-git-send-email-labbott@redhat.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160808_105014_766467_4C9F6FA0 X-CRM114-Status: GOOD ( 23.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devel@driverdev.osuosl.org, Jon Medhurst , Android Kernel Team , Greg Kroah-Hartman , Daniel Vetter , Will Deacon , Russell King , linux-kernel@vger.kernel.org, linaro-mm-sig@lists.linaro.org, Rohit kumar , Bryan Huntsman , Jeremy Gebben , Eun Taik Lee , Catalin Marinas , Liviu Dudau , Laura Abbott , Mitchel Humpherys , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP arm may need the kernel_force_cache APIs to guarantee data consistency. Implement versions of these APIs based on the DMA APIs. Signed-off-by: Laura Abbott --- arch/arm/include/asm/cacheflush.h | 4 ++ arch/arm/mm/dma-mapping.c | 119 -------------------------------------- arch/arm/mm/flush.c | 115 ++++++++++++++++++++++++++++++++++++ arch/arm/mm/mm.h | 8 +++ 4 files changed, 127 insertions(+), 119 deletions(-) diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h index 9156fc3..78eb011 100644 --- a/arch/arm/include/asm/cacheflush.h +++ b/arch/arm/include/asm/cacheflush.h @@ -518,4 +518,8 @@ static inline void secure_flush_area(const void *addr, size_t size) outer_flush_range(phys, phys + size); } +#define ARCH_HAS_FORCE_CACHE 1 +void kernel_force_cache_clean(struct page *page, size_t size); +void kernel_force_cache_invalidate(struct page *page, size_t size); + #endif diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index c6834c0..8c9296d 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -95,23 +95,6 @@ static struct arm_dma_buffer *arm_dma_buffer_find(void *virt) return found; } -/* - * The DMA API is built upon the notion of "buffer ownership". A buffer - * is either exclusively owned by the CPU (and therefore may be accessed - * by it) or exclusively owned by the DMA device. These helper functions - * represent the transitions between these two ownership states. - * - * Note, however, that on later ARMs, this notion does not work due to - * speculative prefetches. We model our approach on the assumption that - * the CPU does do speculative prefetches, which means we clean caches - * before transfers and delay cache invalidation until transfer completion. - * - */ -static void __dma_page_cpu_to_dev(struct page *, unsigned long, - size_t, enum dma_data_direction); -static void __dma_page_dev_to_cpu(struct page *, unsigned long, - size_t, enum dma_data_direction); - /** * arm_dma_map_page - map a portion of a page for streaming DMA * @dev: valid struct device pointer, or NULL for ISA and EISA-like devices @@ -945,108 +928,6 @@ int arm_dma_get_sgtable(struct device *dev, struct sg_table *sgt, return 0; } -static void dma_cache_maint_page(struct page *page, unsigned long offset, - size_t size, enum dma_data_direction dir, - void (*op)(const void *, size_t, int)) -{ - unsigned long pfn; - size_t left = size; - - pfn = page_to_pfn(page) + offset / PAGE_SIZE; - offset %= PAGE_SIZE; - - /* - * A single sg entry may refer to multiple physically contiguous - * pages. But we still need to process highmem pages individually. - * If highmem is not configured then the bulk of this loop gets - * optimized out. - */ - do { - size_t len = left; - void *vaddr; - - page = pfn_to_page(pfn); - - if (PageHighMem(page)) { - if (len + offset > PAGE_SIZE) - len = PAGE_SIZE - offset; - - if (cache_is_vipt_nonaliasing()) { - vaddr = kmap_atomic(page); - op(vaddr + offset, len, dir); - kunmap_atomic(vaddr); - } else { - vaddr = kmap_high_get(page); - if (vaddr) { - op(vaddr + offset, len, dir); - kunmap_high(page); - } - } - } else { - vaddr = page_address(page) + offset; - op(vaddr, len, dir); - } - offset = 0; - pfn++; - left -= len; - } while (left); -} - -/* - * Make an area consistent for devices. - * Note: Drivers should NOT use this function directly, as it will break - * platforms with CONFIG_DMABOUNCE. - * Use the driver DMA support - see dma-mapping.h (dma_sync_*) - */ -static void __dma_page_cpu_to_dev(struct page *page, unsigned long off, - size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr; - - dma_cache_maint_page(page, off, size, dir, dmac_map_area); - - paddr = page_to_phys(page) + off; - if (dir == DMA_FROM_DEVICE) { - outer_inv_range(paddr, paddr + size); - } else { - outer_clean_range(paddr, paddr + size); - } - /* FIXME: non-speculating: flush on bidirectional mappings? */ -} - -static void __dma_page_dev_to_cpu(struct page *page, unsigned long off, - size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr = page_to_phys(page) + off; - - /* FIXME: non-speculating: not required */ - /* in any case, don't bother invalidating if DMA to device */ - if (dir != DMA_TO_DEVICE) { - outer_inv_range(paddr, paddr + size); - - dma_cache_maint_page(page, off, size, dir, dmac_unmap_area); - } - - /* - * Mark the D-cache clean for these pages to avoid extra flushing. - */ - if (dir != DMA_TO_DEVICE && size >= PAGE_SIZE) { - unsigned long pfn; - size_t left = size; - - pfn = page_to_pfn(page) + off / PAGE_SIZE; - off %= PAGE_SIZE; - if (off) { - pfn++; - left -= PAGE_SIZE - off; - } - while (left >= PAGE_SIZE) { - page = pfn_to_page(pfn++); - set_bit(PG_dcache_clean, &page->flags); - left -= PAGE_SIZE; - } - } -} /** * arm_dma_map_sg - map a set of SG buffers for streaming mode DMA diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c index 3cced84..2b8b705 100644 --- a/arch/arm/mm/flush.c +++ b/arch/arm/mm/flush.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -20,6 +21,7 @@ #include #include "mm.h" +#include "dma.h" #ifdef CONFIG_ARM_HEAVY_MB void (*soc_mb)(void); @@ -415,3 +417,116 @@ void __flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned l */ __cpuc_flush_dcache_area(page_address(page), PAGE_SIZE); } + +static void dma_cache_maint_page(struct page *page, unsigned long offset, + size_t size, enum dma_data_direction dir, + void (*op)(const void *, size_t, int)) +{ + unsigned long pfn; + size_t left = size; + + pfn = page_to_pfn(page) + offset / PAGE_SIZE; + offset %= PAGE_SIZE; + + /* + * A single sg entry may refer to multiple physically contiguous + * pages. But we still need to process highmem pages individually. + * If highmem is not configured then the bulk of this loop gets + * optimized out. + */ + do { + size_t len = left; + void *vaddr; + + page = pfn_to_page(pfn); + + if (PageHighMem(page)) { + if (len + offset > PAGE_SIZE) + len = PAGE_SIZE - offset; + + if (cache_is_vipt_nonaliasing()) { + vaddr = kmap_atomic(page); + op(vaddr + offset, len, dir); + kunmap_atomic(vaddr); + } else { + vaddr = kmap_high_get(page); + if (vaddr) { + op(vaddr + offset, len, dir); + kunmap_high(page); + } + } + } else { + vaddr = page_address(page) + offset; + op(vaddr, len, dir); + } + offset = 0; + pfn++; + left -= len; + } while (left); +} + +/* + * Make an area consistent for devices. + * Note: Drivers should NOT use this function directly, as it will break + * platforms with CONFIG_DMABOUNCE. + * Use the driver DMA support - see dma-mapping.h (dma_sync_*) + */ +void __dma_page_cpu_to_dev(struct page *page, unsigned long off, + size_t size, enum dma_data_direction dir) +{ + phys_addr_t paddr; + + dma_cache_maint_page(page, off, size, dir, dmac_map_area); + + paddr = page_to_phys(page) + off; + if (dir == DMA_FROM_DEVICE) { + outer_inv_range(paddr, paddr + size); + } else { + outer_clean_range(paddr, paddr + size); + } + /* FIXME: non-speculating: flush on bidirectional mappings? */ +} + +void __dma_page_dev_to_cpu(struct page *page, unsigned long off, + size_t size, enum dma_data_direction dir) +{ + phys_addr_t paddr = page_to_phys(page) + off; + + /* FIXME: non-speculating: not required */ + /* in any case, don't bother invalidating if DMA to device */ + if (dir != DMA_TO_DEVICE) { + outer_inv_range(paddr, paddr + size); + + dma_cache_maint_page(page, off, size, dir, dmac_unmap_area); + } + + /* + * Mark the D-cache clean for these pages to avoid extra flushing. + */ + if (dir != DMA_TO_DEVICE && size >= PAGE_SIZE) { + unsigned long pfn; + size_t left = size; + + pfn = page_to_pfn(page) + off / PAGE_SIZE; + off %= PAGE_SIZE; + if (off) { + pfn++; + left -= PAGE_SIZE - off; + } + while (left >= PAGE_SIZE) { + page = pfn_to_page(pfn++); + set_bit(PG_dcache_clean, &page->flags); + left -= PAGE_SIZE; + } + } +} + +void kernel_force_cache_clean(struct page *page, size_t size) +{ + __dma_page_cpu_to_dev(page, 0, size, DMA_BIDIRECTIONAL); +} + +void kernel_force_cache_invalidate(struct page *page, size_t size) +{ + __dma_page_dev_to_cpu(page, 0, size, DMA_BIDIRECTIONAL); +} diff --git a/arch/arm/mm/mm.h b/arch/arm/mm/mm.h index ce727d4..9b853ac 100644 --- a/arch/arm/mm/mm.h +++ b/arch/arm/mm/mm.h @@ -1,9 +1,11 @@ #ifdef CONFIG_MMU #include #include +#include #include + /* the upper-most page table pointer */ extern pmd_t *top_pmd; @@ -97,3 +99,9 @@ void arm_mm_memblock_reserve(void); void dma_contiguous_remap(void); unsigned long __clear_cr(unsigned long mask); + +void __dma_page_dev_to_cpu(struct page *page, unsigned long off, + size_t size, enum dma_data_direction dir); + +void __dma_page_cpu_to_dev(struct page *page, unsigned long off, + size_t size, enum dma_data_direction dir);