From patchwork Wed Feb 15 20:09:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13142149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12F92C636CC for ; Wed, 15 Feb 2023 20:09:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6627C6B0073; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 57DDD6B007D; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DEB26B0078; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0C1D96B0074 for ; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D000AAB660 for ; Wed, 15 Feb 2023 20:09:31 +0000 (UTC) X-FDA: 80470616142.22.08238EF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 6509B12000A for ; Wed, 15 Feb 2023 20:09:29 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vGfYtOSz; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676491770; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rO9DpJQQWF0SyEucyLTZyrAzjeodDlsM2btoEWRp7WU=; b=1BMceY1bGr2yUdXOveYXxsL3eIKrVb8c8vMBfFb1RGSLuA9tua+ThQroTkUZxqosKr1LQg pe4rpOSTqWYDWwxldUQ1LSHJZtcYRBxw0WHSYqcWMPovZ1jNQ2ycY5x7f9ngZXGw2HfE9h Z1et3yYHM5Im20LDd3vXWfl8HVIqCXk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vGfYtOSz; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676491770; a=rsa-sha256; cv=none; b=oQYNmBupTE9RVuDA/i5KVJ4y5HL8L6s9YM02qIyDoUXXq+RNm6E/txh8+kwODpfCxak1r/ lrijnWhZdUzK4sZIbOMzG17FJq66DVPONBMN3CwshwB3GNCwEKytTlgnw5SztlCp51XLau Wzh0PxpFEAZSgp42qc3chcrZC9XjgFE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rO9DpJQQWF0SyEucyLTZyrAzjeodDlsM2btoEWRp7WU=; b=vGfYtOSzjDXVBKdUpt5Ko4JWG+ y6gE2yNUNg+f8lkuv0YSdhHYTNlAJnLynIAQgbO4ZtYEe7FRBuG98ubV+a2KILxUrhZMnVzStr5h6 XDspcBrrLtpGCXXBblT/MxYtEY0VCKwp2H2y3xh41PQ9UDK6mL1BroF9kFQUUO7dDPcoV5rSlwswq r3jGbzlqdnuGMpvdSkiaY95Y72lmnlUuMc6Cn25QSWhoPdtvnZGWkXMkdOxbDsQV8VP07aEmMmped 93oHezjWeBlePl6VBPBpp3UaLLCsyqmNVJuCpSKaf14YbwZXpG0Xf3hZptWg3mo0kAEaLXJzJkR0h uB5H0EYA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pSO5m-007lAL-OL; Wed, 15 Feb 2023 20:09:26 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, linux-ia64@vger.kernel.org, linux-arch@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 14/17] ia64: Implement the new page table range API Date: Wed, 15 Feb 2023 20:09:17 +0000 Message-Id: <20230215200920.1849567-1-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230215000446.1655635-1-willy@infradead.org> References: <20230215000446.1655635-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6509B12000A X-Stat-Signature: hzzz1ktjgsm3zg4ywxj43o4wk4cypgjt X-HE-Tag: 1676491769-609912 X-HE-Meta: U2FsdGVkX18qOdu9SsSb7c6Wxhvck2rkhn2mcwAJqcgRtIPMTNOL7DmyWFMHGFHqRTdGusu725+5Ayb86UJXG9tcTB+aPzxcIZAlWbr1Bwg+XAZZPzGj7LL7/cRhMM8UlzwKVs32gfpJEYJaBqYw2FqZ+A5ryUth5X48oL7uBF8gY5Yw2WI/2tMITUOyAUvlYxvVL0mGxN/mmwgSBev3t3glMEpH42QmU2EFEoxfbchjNgcNGAgoC8WOeHmKr+MvOhER72bWaKZBwCl8GanpttewPdZyCSOqZ/nnQKtIVAdeowLRBFHDYIVy8oN6/4p1mPLFMAVhbZI5OO3rshxxDIGxdssWeV91a/Uz9NPjQkuDh9eNRUXHhFOS0imcsg4T55GXUterAxqS6CSsb2NkvyYzmda9ega7mmBQIFItcX48gNubUOcbCo2foXXDYBqO4MC7tfSPnjQpLR43Ne7qtvlJkzBMsSPEATS0I2NT9uWUhTYEJ5L+EnUEgcr/e6CjwfGA02TQx3PBrFT/ScXXcxAx0eR8cSmk4j1zMUv7zitw5m4HSCWrJo58SR7U876j4HXuXaRi9ObFddsyqiHVDdNHVd724elHS9CQcozNNWxd+rGAZTzmVbpY9yu9wpBXwf8W8kxqg2xfwMx8drnR34YlGvFGYQRoOCGGmdg8KATWSRMr8mp2PfPN2FgSmfK9/SGo9cQVTxr0IzxwoKeQAtAjp5zyRthXr1VWImVRrLibPTnSmQTHuUrOPU/EYzcAQPizT00RSvqA4eMcs01fkaMWJHYeAMT/ptE/d2q6R1MFOBDaAPe5E6+SOvVwiaAeG2Q8OpTJkUuDvNsDuaay8gtcgCBkG5WtLEnfjWdpLHMDU1ErtBeLxz89q4Bzad5Gr2hASjMzPfUWnC4V9JnpMjy6Z84lN/8HFlZYAf3FQ7FRBvkzTXAMXn7aNcZTp+U9PGhItG5EAyGhb45HMJr l0EMjf/s Kjp1gu/URJ+KHK6sr4iUebbQdvc4cCbZC7lw0wqlSh49z1pgSMfrHXYy9x7WLowjYrkD4qUa4u3l3XuJretWIXz/jp5oxb2yDEawi/0AOYDLfQUxB59Jy3D0msej8UVgzL7XN7fmj7zDeGc920KzWYMh33RvFy3oIRcucUcWgSuxLX84jewhtRIN/1/LFTj4YQiRWnBTFN9I0Xhc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add set_ptes(), update_mmu_cache_range() and flush_dcache_folio(). PG_arch_1 (aka PG_dcache_clean) becomes a per-folio flag instead of per-page, which makes arch_dma_mark_clean() and mark_clean() a little more exciting. Signed-off-by: Matthew Wilcox (Oracle) --- arch/ia64/hp/common/sba_iommu.c | 26 +++++++++++++++----------- arch/ia64/include/asm/cacheflush.h | 14 ++++++++++---- arch/ia64/include/asm/pgtable.h | 14 +++++++++++++- arch/ia64/mm/init.c | 29 +++++++++++++++++++---------- 4 files changed, 57 insertions(+), 26 deletions(-) diff --git a/arch/ia64/hp/common/sba_iommu.c b/arch/ia64/hp/common/sba_iommu.c index 8ad6946521d8..48d475f10003 100644 --- a/arch/ia64/hp/common/sba_iommu.c +++ b/arch/ia64/hp/common/sba_iommu.c @@ -798,22 +798,26 @@ sba_io_pdir_entry(u64 *pdir_ptr, unsigned long vba) #endif #ifdef ENABLE_MARK_CLEAN -/** +/* * Since DMA is i-cache coherent, any (complete) pages that were written via * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't have to * flush them when they get mapped into an executable vm-area. */ -static void -mark_clean (void *addr, size_t size) +static void mark_clean(void *addr, size_t size) { - unsigned long pg_addr, end; - - pg_addr = PAGE_ALIGN((unsigned long) addr); - end = (unsigned long) addr + size; - while (pg_addr + PAGE_SIZE <= end) { - struct page *page = virt_to_page((void *)pg_addr); - set_bit(PG_arch_1, &page->flags); - pg_addr += PAGE_SIZE; + struct folio *folio = virt_to_folio(addr); + ssize_t left = size; + size_t offset = offset_in_folio(folio, addr); + + if (offset) { + left -= folio_size(folio) - offset; + folio = folio_next(folio); + } + + while (left >= folio_size(folio)) { + set_bit(PG_arch_1, &folio->flags); + left -= folio_size(folio); + folio = folio_next(folio); } } #endif diff --git a/arch/ia64/include/asm/cacheflush.h b/arch/ia64/include/asm/cacheflush.h index 708c0fa5d975..eac493fa9e0d 100644 --- a/arch/ia64/include/asm/cacheflush.h +++ b/arch/ia64/include/asm/cacheflush.h @@ -13,10 +13,16 @@ #include #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -#define flush_dcache_page(page) \ -do { \ - clear_bit(PG_arch_1, &(page)->flags); \ -} while (0) +static inline void flush_dcache_folio(struct folio *folio) +{ + clear_bit(PG_arch_1, &folio->flags); +} +#define flush_dcache_folio flush_dcache_folio + +static inline void flush_dcache_page(struct page *page) +{ + flush_dcache_folio(page_folio(page)); +} extern void flush_icache_range(unsigned long start, unsigned long end); #define flush_icache_range flush_icache_range diff --git a/arch/ia64/include/asm/pgtable.h b/arch/ia64/include/asm/pgtable.h index 21c97e31a28a..0c2be4ea664b 100644 --- a/arch/ia64/include/asm/pgtable.h +++ b/arch/ia64/include/asm/pgtable.h @@ -303,7 +303,18 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) *ptep = pteval; } -#define set_pte_at(mm,addr,ptep,pteval) set_pte(ptep,pteval) +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + for (;;) { + set_pte(ptep, pte); + if (--nr == 0) + break; + ptep++; + pte_val(pte) += PAGE_SIZE; + } +} +#define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, add, ptep, pte, 1) /* * Make page protection values cacheable, uncacheable, or write- @@ -396,6 +407,7 @@ pte_same (pte_t a, pte_t b) return pte_val(a) == pte_val(b); } +#define update_mmu_cache_range(vma, address, ptep, nr) do { } while (0) #define update_mmu_cache(vma, address, ptep) do { } while (0) extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c index 7f5353e28516..12aef25944aa 100644 --- a/arch/ia64/mm/init.c +++ b/arch/ia64/mm/init.c @@ -50,30 +50,39 @@ void __ia64_sync_icache_dcache (pte_t pte) { unsigned long addr; - struct page *page; + struct folio *folio; - page = pte_page(pte); - addr = (unsigned long) page_address(page); + folio = page_folio(pte_page(pte)); + addr = (unsigned long)folio_address(folio); - if (test_bit(PG_arch_1, &page->flags)) + if (test_bit(PG_arch_1, &folio->flags)) return; /* i-cache is already coherent with d-cache */ - flush_icache_range(addr, addr + page_size(page)); - set_bit(PG_arch_1, &page->flags); /* mark page as clean */ + flush_icache_range(addr, addr + folio_size(folio)); + set_bit(PG_arch_1, &folio->flags); /* mark page as clean */ } /* - * Since DMA is i-cache coherent, any (complete) pages that were written via + * Since DMA is i-cache coherent, any (complete) folios that were written via * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't have to * flush them when they get mapped into an executable vm-area. */ void arch_dma_mark_clean(phys_addr_t paddr, size_t size) { - unsigned long pfn = PHYS_PFN(paddr); + struct folio *folio = page_folio(phys_to_page(paddr)); + ssize_t left = size; + size_t offset = offset_in_folio(folio, paddr); - do { + if (offset) { + left -= folio_size(folio) - offset; + folio = folio_next(folio); + } + + while (left >= (ssize_t)folio_size(folio)) { set_bit(PG_arch_1, &pfn_to_page(pfn)->flags); - } while (++pfn <= PHYS_PFN(paddr + size - 1)); + left -= folio_size(folio); + folio = folio_next(folio); + } } inline void From patchwork Wed Feb 15 20:09:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13142148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA328C636D4 for ; Wed, 15 Feb 2023 20:09:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F20F6B0074; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 37B9E6B0073; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21BF86B007B; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0B33E6B0073 for ; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C17CAA0A8C for ; Wed, 15 Feb 2023 20:09:31 +0000 (UTC) X-FDA: 80470616142.21.3968A10 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 71B34140009 for ; Wed, 15 Feb 2023 20:09:29 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i50W5RLo; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676491770; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WK7WV98DmqNoEol8pKIPxUYYZ9kR5i0EUBhp91OqSig=; b=lVSnSfJoWbVbeRm2kO2G64LSDu3ddNstaYh7IDdYMu8NjZGepbUHMnP/bTkG8nCCfJpaQc +RHUwsih/hN8cTJrmittdTGRExp3OyR8ppQTIZQX8wYgM8JTGv0CpQBzmgNUVVuUc7JhXa I5jiSo+/cPN5btXCbDml7TfQh6JIh4A= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i50W5RLo; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676491770; a=rsa-sha256; cv=none; b=Yu9rwTz3QKr18ppWh/WV8xJZT8V+ZipfQxkzzoPk/v3KHFzwQb03v5AeYu1AC7PsZCeSl7 yrfTL2JBTqvG8qarj/28bZ+tP60nvRVvcMWaezTJYHmn14VH23iy7cJVVvkBa0d5Ag+KRy Lt3M1VktEo8TNmhMJfyYH8oE9xtK9Qo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=WK7WV98DmqNoEol8pKIPxUYYZ9kR5i0EUBhp91OqSig=; b=i50W5RLoKp2u9dkh3sbtVQHYCW y4vps7i5Zjb9gRIuKQkcVxkB2ZDAJibcvJZTosxgjyG3hUAKblPrrtsj9vVrGxMJYZ72zUaQQqJ4g 82FddwyDg7YViumA8bA3rznbcKchmb7M6R9D0M85AF7H1oJI2+FO5QeKbxtDFjDrDyQsxU11jnDtt 64SCigt+cFtVdBTeqE2xGyWT82tDeJXTKPIHugptpPVoCpe5spakHXlKVf8K/jEPOWv8M7bRUBA4k ZEyndIZz9oMorsB32W9pFxCKwVpaXe+b/dXV6+hykrTOgSI0z5P26m7L9DqJm4/IO/S1d/sKWIXoH /zTE9Kgw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pSO5m-007lAU-Rl; Wed, 15 Feb 2023 20:09:26 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, linux-m68k@lists.linux-m68k.org, Geert Uytterhoeven , linux-arch@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 15/17] m68k: Implement the new page table range API Date: Wed, 15 Feb 2023 20:09:18 +0000 Message-Id: <20230215200920.1849567-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230215200920.1849567-1-willy@infradead.org> References: <20230215000446.1655635-1-willy@infradead.org> <20230215200920.1849567-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 71B34140009 X-Stat-Signature: ohytor6xmt36eo9ra5on9n6abnxy8ze3 X-Rspam-User: X-HE-Tag: 1676491769-479518 X-HE-Meta: U2FsdGVkX1/rg6b2UJxt274xw68fuco88K+c246w2RLEv1IeaLHNt0Xv/LlikZqQLXpNL7D3RL2QJ/aeh9m0Si2KCqgJpR8CEcdK/cT+ZP9H0zWjhgZXnLWy4Nyf6aJ6nGFUznLW+qVbfctMkmvdisasMfaeLtwSYoH+RVGBaH5s5wxcC3w+hWVh3LPebWlYty61SFMAcxvKbmEHMcmTg4yMb5AQjFV8if1ucTW+wtBLI2NzVjaMcsOv2BXBjIH5e5mIeobDtw3b6JGFigTqh65+arevwmnlzLN7WSxaNX5ArectQc3olRaXdSE1NiUIzSajHWZHRZANTn7Ese7CVG07Yt5aOG1UiB25TTegBrkgcwKdQPJMSUijfRazhdkDTRxxOKYqmMggUAowbvIuSDiS2GMniZhL2Gja95YqaYQQ8MPE8QkYf7YplQstQyjAmCwIHd3AP+nEsN3P0k0JgXoVsotfMPXH8hl1yofsgVkxluDtqAz+ayW0LK1AGHNWd8eHGLHUXQzMkAk8OCQu7hws9VGrX6Li83koBkHH/NMP9GvBLGYkHutyahQQSrSoq+XaZTIGCD8EC/onOKC3I16v2xWgcwMHJTHvGSCbtKyXuD35lG1isIS+3jLrDgy8cPGfmXywimYcqthDAfZ0YsWgQUQbyyKoGefAGGNxA9G8DT7sOrTIEam44y5PYc1EbsKPTegGZ4fGtvEpPH0Vc7REWi0efGwb9KkwdwCkYQ72gSb1CAc6KgLhoJf+HojP6BTy2MiBNUomlV2kqi4X/KEsfP38RJYt35d7cKMJl1Yt1BUq0ae6ByMkCozp6Mh9h3/I6xLHZWFVfVaQeieA8kf9ws2JpBXs9sNDx3Eo7zTCI9cXv1/6m1w8FGMOh2z1KUmNYpB+k3irdzCVib9aR8hr+4tye8jvAymPfgomNE+1JTE6yxCEwyCu5qPKp/b+agTVLFscfwDy/3JFJyh lHxcMlyX GRiaJsEu9vjgjC7VC48snwmgonCCU3FjGIr544TsGb0VJvtx66b1vWAGWOIXWfMBvt8Tey/AdSXjc1mC+kXLGBSpAO+bhqKwxXkFXGaNnb66IB/9kG3k4TZfSOq7+tqo1+SJ/tEKFiHIhVZfVs8ckvrncWviIld4ka1i8RYuDsNlXjEnXx0Zm1Tv096A3ZUTcCeyt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add set_ptes(), update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio(). I'm not entirely certain that the 040/060 case in __flush_pages_to_ram() is correct. Signed-off-by: Matthew Wilcox (Oracle) --- arch/m68k/include/asm/cacheflush_mm.h | 12 ++++++++---- arch/m68k/include/asm/pgtable_mm.h | 21 ++++++++++++++++++--- arch/m68k/mm/motorola.c | 2 +- 3 files changed, 27 insertions(+), 8 deletions(-) diff --git a/arch/m68k/include/asm/cacheflush_mm.h b/arch/m68k/include/asm/cacheflush_mm.h index 1ac55e7b47f0..2244c35178d0 100644 --- a/arch/m68k/include/asm/cacheflush_mm.h +++ b/arch/m68k/include/asm/cacheflush_mm.h @@ -220,13 +220,13 @@ static inline void flush_cache_page(struct vm_area_struct *vma, unsigned long vm /* Push the page at kernel virtual address and clear the icache */ /* RZ: use cpush %bc instead of cpush %dc, cinv %ic */ -static inline void __flush_page_to_ram(void *vaddr) +static inline void __flush_pages_to_ram(void *vaddr, unsigned int nr) { if (CPU_IS_COLDFIRE) { unsigned long addr, start, end; addr = ((unsigned long) vaddr) & ~(PAGE_SIZE - 1); start = addr & ICACHE_SET_MASK; - end = (addr + PAGE_SIZE - 1) & ICACHE_SET_MASK; + end = (addr + nr * PAGE_SIZE - 1) & ICACHE_SET_MASK; if (start > end) { flush_cf_bcache(0, end); end = ICACHE_MAX_ADDR; @@ -249,10 +249,14 @@ static inline void __flush_page_to_ram(void *vaddr) } #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -#define flush_dcache_page(page) __flush_page_to_ram(page_address(page)) +#define flush_dcache_page(page) __flush_pages_to_ram(page_address(page), 1) +#define flush_dcache_folio(folio) \ + __flush_pages_to_ram(folio_address(folio), folio_nr_pages(folio)) #define flush_dcache_mmap_lock(mapping) do { } while (0) #define flush_dcache_mmap_unlock(mapping) do { } while (0) -#define flush_icache_page(vma, page) __flush_page_to_ram(page_address(page)) +#define flush_icache_pages(vma, page, nr) \ + __flush_pages_to_ram(page_address(page), nr) +#define flush_icache_page(vma, page) flush_icache_pages(vma, page, 1) extern void flush_icache_user_page(struct vm_area_struct *vma, struct page *page, unsigned long addr, int len); diff --git a/arch/m68k/include/asm/pgtable_mm.h b/arch/m68k/include/asm/pgtable_mm.h index b93c41fe2067..400206c17c97 100644 --- a/arch/m68k/include/asm/pgtable_mm.h +++ b/arch/m68k/include/asm/pgtable_mm.h @@ -31,8 +31,20 @@ do{ \ *(pteptr) = (pteval); \ } while(0) -#define set_pte_at(mm,addr,ptep,pteval) set_pte(ptep,pteval) +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + for (;;) { + set_pte(ptep, pte); + if (--nr == 0) + break; + ptep++; + pte_val(pte) += PAGE_SIZE; + } +} + +#define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1) /* PMD_SHIFT determines the size of the area a second-level page table can map */ #if CONFIG_PGTABLE_LEVELS == 3 @@ -138,11 +150,14 @@ extern void kernel_set_cachemode(void *addr, unsigned long size, int cmode); * tables contain all the necessary information. The Sun3 does, but * they are updated on demand. */ -static inline void update_mmu_cache(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline void update_mmu_cache_range(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, unsigned int nr) { } +#define update_mmu_cache(vma, addr, ptep) \ + update_mmu_cache_range(vma, addr, ptep, 1) + #endif /* !__ASSEMBLY__ */ /* MMU-specific headers */ diff --git a/arch/m68k/mm/motorola.c b/arch/m68k/mm/motorola.c index 2a375637e007..7784d0fcdf6e 100644 --- a/arch/m68k/mm/motorola.c +++ b/arch/m68k/mm/motorola.c @@ -81,7 +81,7 @@ static inline void cache_page(void *vaddr) void mmu_page_ctor(void *page) { - __flush_page_to_ram(page); + __flush_pages_to_ram(page, 1); flush_tlb_kernel_page(page); nocache_page(page); } From patchwork Wed Feb 15 20:09:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13142151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8AECC64ED9 for ; Wed, 15 Feb 2023 20:09:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E4C36B007B; Wed, 15 Feb 2023 15:09:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 06DBF6B007D; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E55EC6B0080; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D37CE6B007D for ; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A2A37C105E for ; Wed, 15 Feb 2023 20:09:32 +0000 (UTC) X-FDA: 80470616184.16.77D02AE Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id AF4F4140016 for ; Wed, 15 Feb 2023 20:09:29 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RLi9F9Xz; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676491770; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J8/1q6frBYGwgbqjjavZ1BISvaOhK/l8i4/4YVlRXgY=; b=R373lxgPv6NKLFROvzJoH/EBuWJ0H6MCO13WRS6EfF6lUrZPfJ4DljpJLnJxsQGBiXNm+J +lEhPAAry5GJvF95q5miX2iOpAKCngWsLWIzlwGvpvvlr+k/bQyTYiPC5otDIA5QZPNKwz hwjbPQrODlxLq9yAI8Z9/6irmi3qDB4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RLi9F9Xz; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676491770; a=rsa-sha256; cv=none; b=yVnENeKPLAfdNrWRrsmUvK3Jfy1E6Dz5TwZhHf3Q5wvODs1lrUzwyqsOb59O8VPmnjCjaA iyD3CWKNPDED54Rz6Z7RGXAMy5X2sCpfwk9sD5aOTd6IXrQFKGYSHULk1IvCxiI7BZeP0z WKV8N/v7yWKAmuMkXH+lKc7knBZX7dM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=J8/1q6frBYGwgbqjjavZ1BISvaOhK/l8i4/4YVlRXgY=; b=RLi9F9Xz0A8fF8Tmhtavo1XYMA r8Xf+3tOYAmOgF7m74JMcmt6Gx/ie4sI17SorHbeiyt8M7wpxuFdtuv1d2qb1qq/QZyPcElSaUFWu rAt8XqS7rSk4QUgE9PT9H9Fey21K9TTAs3G0xx8gxL7ONo1tCBZuctczJUAvO7pBg0mcB35Au0qa8 fg5e5qJdRkX6zpJHABfbuLcYwMW7PbVdxt2ruX6Lo1yY403QrceexS25viA0WVFbIQhp47/O12c0L QV5Xg17GQOzsyh3q1FBMKM8oRWmVLM8IwZ3zorcfVWp0zXiGRdULrrKoczXNiPqD+zf9ywi5oTV++ /kFO+MFQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pSO5m-007lAY-V4; Wed, 15 Feb 2023 20:09:26 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, Michal Simek , linux-arch@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 16/17] microblaze: Implement the new page table range API Date: Wed, 15 Feb 2023 20:09:19 +0000 Message-Id: <20230215200920.1849567-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230215200920.1849567-1-willy@infradead.org> References: <20230215000446.1655635-1-willy@infradead.org> <20230215200920.1849567-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AF4F4140016 X-Stat-Signature: b96wh8z5epkb3nyzy99cgwcmezzmsa8t X-HE-Tag: 1676491769-749738 X-HE-Meta: U2FsdGVkX1/MouNZQpuPnuMlEo9DALi5CqVAjRWxRNt5RKUKLKd59oNZM/UPRFIhNM3ehCSViND7weicTGGsw90z9I7mBkLHLtaAtIX80lGuxI56AQdzviEx6ZNyoeXPLz9A/XF5SpIQYvtIy7U/aC4QI2VEKwjM9UyeIvErWsNNiukUX/LlqKetPexYxCogMWMNdHT+Uhp57YpJDEaN9bhvRL5GfM7EiQ1i5TOkaIAuGUtGfaScGVhWgQcpCegC3CW+Tiakd+XOoiVjlFn87sB9dnwaPp1Bi0gcOi+zh1pc3TztOJTMOYycu0uQLW3RlJM1kQpnkd7KeuMd49nUU+aE1RcN+9s7jdqOgL1ZCLdGYzYHXAlp1F6ULB/rBZ3ueKFH8xHlAdVEnq8gNkHPXbVozsoZ+FqSYcBC3XaIQdczOfu1Zu4MLEnJMjeRQkyTC7sxsbq96os4KdcsHXjRHEOTmDxoat/L8oAfQjdRGnFEErT6sqw8yGceGBZ9aaP9hhUYBdtk62Lk36RTOaDgBTiNgrgyXsgSer9fnJ03wAyVfvQbnbFd/uxW9LycVXw7k1fjZmy4r1gLbGQMlo4DI5tfBf3bj/IacBTWfEowhk5CjXvkIBs7CQd4oEO86kQ6fNHs7UBpapVU2k23zyogF5MhblXWlf0wrjWl99oVZloPguqRV5m9FmVmpGoVQdtrFKYPiTJhrZ08oFvuACmKfcaHcS3X8zduzpetPCDKSgtQTeCq3Ju51HJvQofinxrRTOtvH11KUSxOg7xWHS8TVNV+ccWN0Cwd//uoTRu5H1nfsVpV71U4rT/Mt29qcdvnx2EWZBUi4xwLJkRAAfTiEeLTaR62WK6QQAForBG7+AZqsDAADwXbJbFIiet8RNCcIRJt2946MbhzbL7vgYZcB5c94YlphTk1qQ4egMEpnk0hvORYMpH+kSym5i2ONLEjVD7nOLVyNsUFxCVHbO7 Dm/VVLsL Z4Lc0d4SO/hJ5m8Rz9cTOLjCk6oUzSfnjm43LueGDK2LEYeFdl6vzCwc2gkJRPncaRTTIEBSjihJVSN5ETgYWGGL20XYxoYiqtdc99zA6yox01RefW9lef6XcbgiBzR0N2OeoczmB+DKVgHV4fQtQKH653rItF1JtAql5gE3opKm67iVZqri7kpcaiHBR2uHEpgUYqLJR7t/uwoA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add set_ptes(), update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio(). Also change the calling convention for set_pte() to be the same as other architectures. Signed-off-by: Matthew Wilcox (Oracle) --- arch/microblaze/include/asm/cacheflush.h | 8 ++++++++ arch/microblaze/include/asm/pgtable.h | 17 ++++++++++++----- arch/microblaze/include/asm/tlbflush.h | 4 +++- 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/arch/microblaze/include/asm/cacheflush.h b/arch/microblaze/include/asm/cacheflush.h index 39f8fb6768d8..e6641ff98cb3 100644 --- a/arch/microblaze/include/asm/cacheflush.h +++ b/arch/microblaze/include/asm/cacheflush.h @@ -74,6 +74,14 @@ do { \ flush_dcache_range((unsigned) (addr), (unsigned) (addr) + PAGE_SIZE); \ } while (0); +static void flush_dcache_folio(struct folio *folio) +{ + unsigned long addr = folio_pfn(folio) << PAGE_SHIFT; + + flush_dcache_range(addr, addr + folio_size(folio)); +} +#define flush_dcache_folio flush_dcache_folio + #define flush_cache_page(vma, vmaddr, pfn) \ flush_dcache_range(pfn << PAGE_SHIFT, (pfn << PAGE_SHIFT) + PAGE_SIZE); diff --git a/arch/microblaze/include/asm/pgtable.h b/arch/microblaze/include/asm/pgtable.h index d1b8272abcd9..a01e1369b486 100644 --- a/arch/microblaze/include/asm/pgtable.h +++ b/arch/microblaze/include/asm/pgtable.h @@ -330,18 +330,25 @@ static inline unsigned long pte_update(pte_t *p, unsigned long clr, /* * set_pte stores a linux PTE into the linux page table. */ -static inline void set_pte(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) +static inline void set_pte(pte_t *ptep, pte_t pte) { *ptep = pte; } -static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) { - *ptep = pte; + for (;;) { + set_pte(ptep, pte); + if (--nr == 0) + break; + ptep++; + pte_val(pte) += 1 << PFN_SHIFT_OFFSET; + } } +#define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1) + #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) diff --git a/arch/microblaze/include/asm/tlbflush.h b/arch/microblaze/include/asm/tlbflush.h index 2038168ed128..1b179e5e9062 100644 --- a/arch/microblaze/include/asm/tlbflush.h +++ b/arch/microblaze/include/asm/tlbflush.h @@ -33,7 +33,9 @@ static inline void local_flush_tlb_range(struct vm_area_struct *vma, #define flush_tlb_kernel_range(start, end) do { } while (0) -#define update_mmu_cache(vma, addr, ptep) do { } while (0) +#define update_mmu_cache_range(vma, addr, ptep, nr) do { } while (0) +#define update_mmu_cache(vma, addr, pte) \ + update_mmu_cache_range(vma, addr, ptep, 1) #define flush_tlb_all local_flush_tlb_all #define flush_tlb_mm local_flush_tlb_mm From patchwork Wed Feb 15 20:09:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13142150 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBBDAC64EC4 for ; Wed, 15 Feb 2023 20:09:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA4D26B0078; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C2CDE6B007B; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA61F6B007D; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 957576B0078 for ; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 53CDF12064E for ; Wed, 15 Feb 2023 20:09:32 +0000 (UTC) X-FDA: 80470616184.15.7371C1E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 8F07C1C000E for ; Wed, 15 Feb 2023 20:09:30 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IAA2niVM; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676491770; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HL+fzotLRQPwweMMbF3JhsBS1ohDo5FxCNgNopahTIo=; b=V+7QteGS6vKaCmRTZAoxd3kZD5Lp8XCwPYlNkAH4kfxlLBC04gHGew0Y0xayRqrvUYFvee On7cZr/G5XPzxHCV+46fR6KSW9LAweLW2UP3nABcrR7JWIxBBXF6sW+KHVeHEtbmBVsKSz mYe+V1dfXbSjEoWJ5Z76Va61Ug0tIIA= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IAA2niVM; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676491770; a=rsa-sha256; cv=none; b=hsdKYvNU6OSlzD7Oow6vTIvLAHotTL4vOR/WEARav48hhsOT9Icgwc5I45CJwSN9z+EKvn Kb6rvPjHXSKmw1lKYEO3ao+sceY8xPHszCl8zHQVw2SRm3WPQVA+U8XsmBRnr1ke9PHq7Z VrPWF9bluShHa+bHdE3NVsYwRP/nOgA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=HL+fzotLRQPwweMMbF3JhsBS1ohDo5FxCNgNopahTIo=; b=IAA2niVM5trdK75ZXymjCHccsn Sm5GLbnK5ojQScpqWcF+pp0XxEbugFAf4bX6i30k7h08K3eQfrvbw6WYm65CP+yM5BXxaLlbG2UY4 0DaLQaIxTWly3H8I7QkKKZ2NvOQ1rMjrNR4QQQDxfrn0xeI/943JgCae2FNLoc2uTHpS22YLEZfXI NcM9Z/eJ6/E5HQGL3RFyEmOWKVri4lGGkh0FAfrIhvruzI7Pv67LGe+RGPWjoDfgo6HB5m8MZmKvt jNh9tSTR7RNfGRYse6pdH2AF1UjoHMWVTUg5v5FO+8kKFTJp3Jm0SoYV/4hm2tUpfRtSumBq9PoRo BJHBp6xA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pSO5n-007lAt-1I; Wed, 15 Feb 2023 20:09:27 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, linux-mips@vger.kernel.org, Thomas Bogendoerfer , linux-arch@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 17/17] mips: Implement the new page table range API Date: Wed, 15 Feb 2023 20:09:20 +0000 Message-Id: <20230215200920.1849567-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230215200920.1849567-1-willy@infradead.org> References: <20230215000446.1655635-1-willy@infradead.org> <20230215200920.1849567-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8F07C1C000E X-Stat-Signature: g3mndtg687t443bxidh8c7gbjze7tjt7 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676491770-728576 X-HE-Meta: U2FsdGVkX1/DHtgey0OlgQsH6B4QPLq+F5eISCnDwIN2gf5AUUds1OvmUxzAk7xVO/dVeo274aGumJigMTK9f2em3dETzmVkvFPVoqspB/OBNIaTaHj0IUctTjvMaH89hax2XMUwHvLIolMH0LtF5Rk175QUc0CNjxixy9QzEM3PKP6cs+HC8Rl0zV4VbYF7AYMpCWTdscAA9DvF/InKETUdVSet9ImzHOWrtiUug3ll/5wWdmarEDQ7ioHAagy96SHqqNGImq7i8jlZpBRLI1vRDDIjAEiTs+x/kZAr8FqQUrtLfWiDDaENGLtS7/RVERMZ6xfJPrR18h28ZC1dRZDaA3MdiuJcUo1sQOHSHD8f1wAm06yjpP5zHNmiPmcdxK7UAtmWzTvEn0yK80ulRnBf0IPBQvXqpiq1DUSurBqaFPbAOrMbBAYVkFxtTz4piX4ePC5M8D6zwmsivbLpTWkGonXSQDbIcXIHGkYL5FqUfExm0/VYkqL196j9Be0H7YvkkPXv6jW4ua9NurNxsc25n5Gcvc94Z2WwlodCto94pJFrQfvxqWH9gmh7nkl+5yAzPPvXA3XxtfMBj95x8yAa2btZSp5cMkXRdRJCKl2pGm3HLIx9GbrWL35tTv9Pvat/k/ZwTSvnKZEicL9g3pqFJMqKnJCsZ7QmnBeq6Rda1NFXcIwGGORwDZIFPbCQqPhKH/S7xBygUZ5jx0p8eYjlGap1zTvOhvL2azuSsdhAwzGScDPa4+HS5t8Qdk35DcRExMw+PSqRE9UzYIzm7UezFWGM9/5F3qunUnXBwd36EQhTWUbGl1Y8TV5XqCglMO+bQ6EvvrmE0u9mTGFidDcpU4xBTIx06n384dK16y3zMnwVWmdBNKMgJGa+7NWGjE4eNhQ1sYJcb5f+xsnImwrV+dfvk0oEA9aSpUZL++AytH1ey/igUqEe0kPy2j+pWpUiBQaA+QkpjFj04gX iyhemdSy 8DBdo2iE6qXNL/qHGxrAzawIP3lne3uiyI6su+Mm5z2g8x8pwYF/Tz2yfBd6ieJoxA4w7suHA7bAVasSruEOCSwaQGbEg+ogacYzqqrZjtO+DtXyTR728fH4eImhXYW5alJK0aTvDmbdGZA8Sgbu8ViQzfQ5eKwux840ESwReac1ay3XJfUTbhUR/E6Ya4wI0C6kF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add set_ptes(), update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio(). PG_arch_1 (aka PG_dcache_dirty) becomes a per-folio flag instead of per-page. Signed-off-by: Matthew Wilcox (Oracle) --- arch/mips/include/asm/cacheflush.h | 32 +++++++++++------ arch/mips/mm/c-r4k.c | 5 +-- arch/mips/mm/cache.c | 56 +++++++++++++++--------------- arch/mips/mm/init.c | 17 +++++---- 4 files changed, 63 insertions(+), 47 deletions(-) diff --git a/arch/mips/include/asm/cacheflush.h b/arch/mips/include/asm/cacheflush.h index b3dc9c589442..2683cade42ef 100644 --- a/arch/mips/include/asm/cacheflush.h +++ b/arch/mips/include/asm/cacheflush.h @@ -36,12 +36,12 @@ */ #define PG_dcache_dirty PG_arch_1 -#define Page_dcache_dirty(page) \ - test_bit(PG_dcache_dirty, &(page)->flags) -#define SetPageDcacheDirty(page) \ - set_bit(PG_dcache_dirty, &(page)->flags) -#define ClearPageDcacheDirty(page) \ - clear_bit(PG_dcache_dirty, &(page)->flags) +#define folio_test_dcache_dirty(folio) \ + test_bit(PG_dcache_dirty, &(folio)->flags) +#define folio_set_dcache_dirty(folio) \ + set_bit(PG_dcache_dirty, &(folio)->flags) +#define folio_clear_dcache_dirty(folio) \ + clear_bit(PG_dcache_dirty, &(folio)->flags) extern void (*flush_cache_all)(void); extern void (*__flush_cache_all)(void); @@ -50,15 +50,24 @@ extern void (*flush_cache_mm)(struct mm_struct *mm); extern void (*flush_cache_range)(struct vm_area_struct *vma, unsigned long start, unsigned long end); extern void (*flush_cache_page)(struct vm_area_struct *vma, unsigned long page, unsigned long pfn); -extern void __flush_dcache_page(struct page *page); +extern void __flush_dcache_pages(struct page *page, unsigned int nr); #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +static inline void flush_dcache_folio(struct folio *folio) +{ + if (cpu_has_dc_aliases) + __flush_dcache_pages(&folio->page, folio_nr_pages(folio)); + else if (!cpu_has_ic_fills_f_dc) + folio_set_dcache_dirty(folio); +} +#define flush_dcache_folio flush_dcache_folio + static inline void flush_dcache_page(struct page *page) { if (cpu_has_dc_aliases) - __flush_dcache_page(page); + __flush_dcache_pages(page, 1); else if (!cpu_has_ic_fills_f_dc) - SetPageDcacheDirty(page); + folio_set_dcache_dirty(page_folio(page)); } #define flush_dcache_mmap_lock(mapping) do { } while (0) @@ -73,10 +82,11 @@ static inline void flush_anon_page(struct vm_area_struct *vma, __flush_anon_page(page, vmaddr); } -static inline void flush_icache_page(struct vm_area_struct *vma, - struct page *page) +static inline void flush_icache_pages(struct vm_area_struct *vma, + struct page *page, unsigned int nr) { } +#define flush_icache_page(vma, page) flush_icache_pages(vma, page, 1) extern void (*flush_icache_range)(unsigned long start, unsigned long end); extern void (*local_flush_icache_range)(unsigned long start, unsigned long end); diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c index a549fa98c2f4..7d2a42f0cffd 100644 --- a/arch/mips/mm/c-r4k.c +++ b/arch/mips/mm/c-r4k.c @@ -679,13 +679,14 @@ static inline void local_r4k_flush_cache_page(void *args) if ((mm == current->active_mm) && (pte_val(*ptep) & _PAGE_VALID)) vaddr = NULL; else { + struct folio *folio = page_folio(page); /* * Use kmap_coherent or kmap_atomic to do flushes for * another ASID than the current one. */ map_coherent = (cpu_has_dc_aliases && - page_mapcount(page) && - !Page_dcache_dirty(page)); + folio_mapped(folio) && + !folio_test_dcache_dirty(folio)); if (map_coherent) vaddr = kmap_coherent(page, addr); else diff --git a/arch/mips/mm/cache.c b/arch/mips/mm/cache.c index 11b3e7ddafd5..0668435521fc 100644 --- a/arch/mips/mm/cache.c +++ b/arch/mips/mm/cache.c @@ -82,13 +82,15 @@ SYSCALL_DEFINE3(cacheflush, unsigned long, addr, unsigned long, bytes, return 0; } -void __flush_dcache_page(struct page *page) +void __flush_dcache_pages(struct page *page, unsigned int nr) { - struct address_space *mapping = page_mapping_file(page); + struct folio *folio = page_folio(page); + struct address_space *mapping = folio_flush_mapping(folio); unsigned long addr; + unsigned int i; if (mapping && !mapping_mapped(mapping)) { - SetPageDcacheDirty(page); + folio_set_dcache_dirty(folio); return; } @@ -97,25 +99,21 @@ void __flush_dcache_page(struct page *page) * case is for exec env/arg pages and those are %99 certainly going to * get faulted into the tlb (and thus flushed) anyways. */ - if (PageHighMem(page)) - addr = (unsigned long)kmap_atomic(page); - else - addr = (unsigned long)page_address(page); - - flush_data_cache_page(addr); - - if (PageHighMem(page)) - kunmap_atomic((void *)addr); + for (i = 0; i < nr; i++) { + addr = (unsigned long)kmap_local_page(page + i); + flush_data_cache_page(addr); + kunmap_local((void *)addr); + } } - -EXPORT_SYMBOL(__flush_dcache_page); +EXPORT_SYMBOL(__flush_dcache_pages); void __flush_anon_page(struct page *page, unsigned long vmaddr) { unsigned long addr = (unsigned long) page_address(page); + struct folio *folio = page_folio(page); if (pages_do_alias(addr, vmaddr)) { - if (page_mapcount(page) && !Page_dcache_dirty(page)) { + if (folio_mapped(folio) && !folio_test_dcache_dirty(folio)) { void *kaddr; kaddr = kmap_coherent(page, vmaddr); @@ -130,27 +128,29 @@ EXPORT_SYMBOL(__flush_anon_page); void __update_cache(unsigned long address, pte_t pte) { - struct page *page; + struct folio *folio; unsigned long pfn, addr; int exec = !pte_no_exec(pte) && !cpu_has_ic_fills_f_dc; + unsigned int i; pfn = pte_pfn(pte); if (unlikely(!pfn_valid(pfn))) return; - page = pfn_to_page(pfn); - if (Page_dcache_dirty(page)) { - if (PageHighMem(page)) - addr = (unsigned long)kmap_atomic(page); - else - addr = (unsigned long)page_address(page); - - if (exec || pages_do_alias(addr, address & PAGE_MASK)) - flush_data_cache_page(addr); - if (PageHighMem(page)) - kunmap_atomic((void *)addr); + folio = page_folio(pfn_to_page(pfn)); + address &= PAGE_MASK; + address -= offset_in_folio(folio, pfn << PAGE_SHIFT); + + if (folio_test_dcache_dirty(folio)) { + for (i = 0; i < folio_nr_pages(folio); i++) { + addr = (unsigned long)kmap_local_folio(folio, i); - ClearPageDcacheDirty(page); + if (exec || pages_do_alias(addr, address)) + flush_data_cache_page(addr); + kunmap_local((void *)addr); + address += PAGE_SIZE; + } + folio_clear_dcache_dirty(folio); } } diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c index 5a8002839550..19d4ca3b3fbd 100644 --- a/arch/mips/mm/init.c +++ b/arch/mips/mm/init.c @@ -88,7 +88,7 @@ static void *__kmap_pgprot(struct page *page, unsigned long addr, pgprot_t prot) pte_t pte; int tlbidx; - BUG_ON(Page_dcache_dirty(page)); + BUG_ON(folio_test_dcache_dirty(page_folio(page))); preempt_disable(); pagefault_disable(); @@ -169,11 +169,12 @@ void kunmap_coherent(void) void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr, struct vm_area_struct *vma) { + struct folio *src = page_folio(from); void *vfrom, *vto; vto = kmap_atomic(to); if (cpu_has_dc_aliases && - page_mapcount(from) && !Page_dcache_dirty(from)) { + folio_mapped(src) && !folio_test_dcache_dirty(src)) { vfrom = kmap_coherent(from, vaddr); copy_page(vto, vfrom); kunmap_coherent(); @@ -194,15 +195,17 @@ void copy_to_user_page(struct vm_area_struct *vma, struct page *page, unsigned long vaddr, void *dst, const void *src, unsigned long len) { + struct folio *folio = page_folio(page); + if (cpu_has_dc_aliases && - page_mapcount(page) && !Page_dcache_dirty(page)) { + folio_mapped(folio) && !folio_test_dcache_dirty(folio)) { void *vto = kmap_coherent(page, vaddr) + (vaddr & ~PAGE_MASK); memcpy(vto, src, len); kunmap_coherent(); } else { memcpy(dst, src, len); if (cpu_has_dc_aliases) - SetPageDcacheDirty(page); + folio_set_dcache_dirty(folio); } if (vma->vm_flags & VM_EXEC) flush_cache_page(vma, vaddr, page_to_pfn(page)); @@ -212,15 +215,17 @@ void copy_from_user_page(struct vm_area_struct *vma, struct page *page, unsigned long vaddr, void *dst, const void *src, unsigned long len) { + struct folio *folio = page_folio(page); + if (cpu_has_dc_aliases && - page_mapcount(page) && !Page_dcache_dirty(page)) { + folio_mapped(folio) && !folio_test_dcache_dirty(folio)) { void *vfrom = kmap_coherent(page, vaddr) + (vaddr & ~PAGE_MASK); memcpy(dst, vfrom, len); kunmap_coherent(); } else { memcpy(dst, src, len); if (cpu_has_dc_aliases) - SetPageDcacheDirty(page); + folio_set_dcache_dirty(folio); } } EXPORT_SYMBOL_GPL(copy_from_user_page);