From patchwork Wed Feb 15 20:09:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13142149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12F92C636CC for ; Wed, 15 Feb 2023 20:09:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6627C6B0073; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 57DDD6B007D; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DEB26B0078; Wed, 15 Feb 2023 15:09:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0C1D96B0074 for ; Wed, 15 Feb 2023 15:09:32 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D000AAB660 for ; Wed, 15 Feb 2023 20:09:31 +0000 (UTC) X-FDA: 80470616142.22.08238EF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 6509B12000A for ; Wed, 15 Feb 2023 20:09:29 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vGfYtOSz; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676491770; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rO9DpJQQWF0SyEucyLTZyrAzjeodDlsM2btoEWRp7WU=; b=1BMceY1bGr2yUdXOveYXxsL3eIKrVb8c8vMBfFb1RGSLuA9tua+ThQroTkUZxqosKr1LQg pe4rpOSTqWYDWwxldUQ1LSHJZtcYRBxw0WHSYqcWMPovZ1jNQ2ycY5x7f9ngZXGw2HfE9h Z1et3yYHM5Im20LDd3vXWfl8HVIqCXk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vGfYtOSz; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676491770; a=rsa-sha256; cv=none; b=oQYNmBupTE9RVuDA/i5KVJ4y5HL8L6s9YM02qIyDoUXXq+RNm6E/txh8+kwODpfCxak1r/ lrijnWhZdUzK4sZIbOMzG17FJq66DVPONBMN3CwshwB3GNCwEKytTlgnw5SztlCp51XLau Wzh0PxpFEAZSgp42qc3chcrZC9XjgFE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rO9DpJQQWF0SyEucyLTZyrAzjeodDlsM2btoEWRp7WU=; b=vGfYtOSzjDXVBKdUpt5Ko4JWG+ y6gE2yNUNg+f8lkuv0YSdhHYTNlAJnLynIAQgbO4ZtYEe7FRBuG98ubV+a2KILxUrhZMnVzStr5h6 XDspcBrrLtpGCXXBblT/MxYtEY0VCKwp2H2y3xh41PQ9UDK6mL1BroF9kFQUUO7dDPcoV5rSlwswq r3jGbzlqdnuGMpvdSkiaY95Y72lmnlUuMc6Cn25QSWhoPdtvnZGWkXMkdOxbDsQV8VP07aEmMmped 93oHezjWeBlePl6VBPBpp3UaLLCsyqmNVJuCpSKaf14YbwZXpG0Xf3hZptWg3mo0kAEaLXJzJkR0h uB5H0EYA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pSO5m-007lAL-OL; Wed, 15 Feb 2023 20:09:26 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, linux-ia64@vger.kernel.org, linux-arch@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 14/17] ia64: Implement the new page table range API Date: Wed, 15 Feb 2023 20:09:17 +0000 Message-Id: <20230215200920.1849567-1-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230215000446.1655635-1-willy@infradead.org> References: <20230215000446.1655635-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6509B12000A X-Stat-Signature: hzzz1ktjgsm3zg4ywxj43o4wk4cypgjt X-HE-Tag: 1676491769-609912 X-HE-Meta: U2FsdGVkX18qOdu9SsSb7c6Wxhvck2rkhn2mcwAJqcgRtIPMTNOL7DmyWFMHGFHqRTdGusu725+5Ayb86UJXG9tcTB+aPzxcIZAlWbr1Bwg+XAZZPzGj7LL7/cRhMM8UlzwKVs32gfpJEYJaBqYw2FqZ+A5ryUth5X48oL7uBF8gY5Yw2WI/2tMITUOyAUvlYxvVL0mGxN/mmwgSBev3t3glMEpH42QmU2EFEoxfbchjNgcNGAgoC8WOeHmKr+MvOhER72bWaKZBwCl8GanpttewPdZyCSOqZ/nnQKtIVAdeowLRBFHDYIVy8oN6/4p1mPLFMAVhbZI5OO3rshxxDIGxdssWeV91a/Uz9NPjQkuDh9eNRUXHhFOS0imcsg4T55GXUterAxqS6CSsb2NkvyYzmda9ega7mmBQIFItcX48gNubUOcbCo2foXXDYBqO4MC7tfSPnjQpLR43Ne7qtvlJkzBMsSPEATS0I2NT9uWUhTYEJ5L+EnUEgcr/e6CjwfGA02TQx3PBrFT/ScXXcxAx0eR8cSmk4j1zMUv7zitw5m4HSCWrJo58SR7U876j4HXuXaRi9ObFddsyqiHVDdNHVd724elHS9CQcozNNWxd+rGAZTzmVbpY9yu9wpBXwf8W8kxqg2xfwMx8drnR34YlGvFGYQRoOCGGmdg8KATWSRMr8mp2PfPN2FgSmfK9/SGo9cQVTxr0IzxwoKeQAtAjp5zyRthXr1VWImVRrLibPTnSmQTHuUrOPU/EYzcAQPizT00RSvqA4eMcs01fkaMWJHYeAMT/ptE/d2q6R1MFOBDaAPe5E6+SOvVwiaAeG2Q8OpTJkUuDvNsDuaay8gtcgCBkG5WtLEnfjWdpLHMDU1ErtBeLxz89q4Bzad5Gr2hASjMzPfUWnC4V9JnpMjy6Z84lN/8HFlZYAf3FQ7FRBvkzTXAMXn7aNcZTp+U9PGhItG5EAyGhb45HMJr l0EMjf/s Kjp1gu/URJ+KHK6sr4iUebbQdvc4cCbZC7lw0wqlSh49z1pgSMfrHXYy9x7WLowjYrkD4qUa4u3l3XuJretWIXz/jp5oxb2yDEawi/0AOYDLfQUxB59Jy3D0msej8UVgzL7XN7fmj7zDeGc920KzWYMh33RvFy3oIRcucUcWgSuxLX84jewhtRIN/1/LFTj4YQiRWnBTFN9I0Xhc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add set_ptes(), update_mmu_cache_range() and flush_dcache_folio(). PG_arch_1 (aka PG_dcache_clean) becomes a per-folio flag instead of per-page, which makes arch_dma_mark_clean() and mark_clean() a little more exciting. Signed-off-by: Matthew Wilcox (Oracle) --- arch/ia64/hp/common/sba_iommu.c | 26 +++++++++++++++----------- arch/ia64/include/asm/cacheflush.h | 14 ++++++++++---- arch/ia64/include/asm/pgtable.h | 14 +++++++++++++- arch/ia64/mm/init.c | 29 +++++++++++++++++++---------- 4 files changed, 57 insertions(+), 26 deletions(-) diff --git a/arch/ia64/hp/common/sba_iommu.c b/arch/ia64/hp/common/sba_iommu.c index 8ad6946521d8..48d475f10003 100644 --- a/arch/ia64/hp/common/sba_iommu.c +++ b/arch/ia64/hp/common/sba_iommu.c @@ -798,22 +798,26 @@ sba_io_pdir_entry(u64 *pdir_ptr, unsigned long vba) #endif #ifdef ENABLE_MARK_CLEAN -/** +/* * Since DMA is i-cache coherent, any (complete) pages that were written via * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't have to * flush them when they get mapped into an executable vm-area. */ -static void -mark_clean (void *addr, size_t size) +static void mark_clean(void *addr, size_t size) { - unsigned long pg_addr, end; - - pg_addr = PAGE_ALIGN((unsigned long) addr); - end = (unsigned long) addr + size; - while (pg_addr + PAGE_SIZE <= end) { - struct page *page = virt_to_page((void *)pg_addr); - set_bit(PG_arch_1, &page->flags); - pg_addr += PAGE_SIZE; + struct folio *folio = virt_to_folio(addr); + ssize_t left = size; + size_t offset = offset_in_folio(folio, addr); + + if (offset) { + left -= folio_size(folio) - offset; + folio = folio_next(folio); + } + + while (left >= folio_size(folio)) { + set_bit(PG_arch_1, &folio->flags); + left -= folio_size(folio); + folio = folio_next(folio); } } #endif diff --git a/arch/ia64/include/asm/cacheflush.h b/arch/ia64/include/asm/cacheflush.h index 708c0fa5d975..eac493fa9e0d 100644 --- a/arch/ia64/include/asm/cacheflush.h +++ b/arch/ia64/include/asm/cacheflush.h @@ -13,10 +13,16 @@ #include #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 -#define flush_dcache_page(page) \ -do { \ - clear_bit(PG_arch_1, &(page)->flags); \ -} while (0) +static inline void flush_dcache_folio(struct folio *folio) +{ + clear_bit(PG_arch_1, &folio->flags); +} +#define flush_dcache_folio flush_dcache_folio + +static inline void flush_dcache_page(struct page *page) +{ + flush_dcache_folio(page_folio(page)); +} extern void flush_icache_range(unsigned long start, unsigned long end); #define flush_icache_range flush_icache_range diff --git a/arch/ia64/include/asm/pgtable.h b/arch/ia64/include/asm/pgtable.h index 21c97e31a28a..0c2be4ea664b 100644 --- a/arch/ia64/include/asm/pgtable.h +++ b/arch/ia64/include/asm/pgtable.h @@ -303,7 +303,18 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) *ptep = pteval; } -#define set_pte_at(mm,addr,ptep,pteval) set_pte(ptep,pteval) +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + for (;;) { + set_pte(ptep, pte); + if (--nr == 0) + break; + ptep++; + pte_val(pte) += PAGE_SIZE; + } +} +#define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, add, ptep, pte, 1) /* * Make page protection values cacheable, uncacheable, or write- @@ -396,6 +407,7 @@ pte_same (pte_t a, pte_t b) return pte_val(a) == pte_val(b); } +#define update_mmu_cache_range(vma, address, ptep, nr) do { } while (0) #define update_mmu_cache(vma, address, ptep) do { } while (0) extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c index 7f5353e28516..12aef25944aa 100644 --- a/arch/ia64/mm/init.c +++ b/arch/ia64/mm/init.c @@ -50,30 +50,39 @@ void __ia64_sync_icache_dcache (pte_t pte) { unsigned long addr; - struct page *page; + struct folio *folio; - page = pte_page(pte); - addr = (unsigned long) page_address(page); + folio = page_folio(pte_page(pte)); + addr = (unsigned long)folio_address(folio); - if (test_bit(PG_arch_1, &page->flags)) + if (test_bit(PG_arch_1, &folio->flags)) return; /* i-cache is already coherent with d-cache */ - flush_icache_range(addr, addr + page_size(page)); - set_bit(PG_arch_1, &page->flags); /* mark page as clean */ + flush_icache_range(addr, addr + folio_size(folio)); + set_bit(PG_arch_1, &folio->flags); /* mark page as clean */ } /* - * Since DMA is i-cache coherent, any (complete) pages that were written via + * Since DMA is i-cache coherent, any (complete) folios that were written via * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't have to * flush them when they get mapped into an executable vm-area. */ void arch_dma_mark_clean(phys_addr_t paddr, size_t size) { - unsigned long pfn = PHYS_PFN(paddr); + struct folio *folio = page_folio(phys_to_page(paddr)); + ssize_t left = size; + size_t offset = offset_in_folio(folio, paddr); - do { + if (offset) { + left -= folio_size(folio) - offset; + folio = folio_next(folio); + } + + while (left >= (ssize_t)folio_size(folio)) { set_bit(PG_arch_1, &pfn_to_page(pfn)->flags); - } while (++pfn <= PHYS_PFN(paddr + size - 1)); + left -= folio_size(folio); + folio = folio_next(folio); + } } inline void