From patchwork Sun Apr 25 14:12:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F089EC43460 for ; Sun, 25 Apr 2021 14:15:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F55B61369 for ; Sun, 25 Apr 2021 14:15:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F55B61369 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GvB5m35kEXsgveAtV/LCKDoDTK0IQ7QByh7LQIdrjvY=; b=qMWeYGPhipWZyKrVFQ6UKB9+3 7oQ9AZt3WqAKtOyJqJ4gnU8EXr4MJNitN6XJd5ZYvHjmK5jDYq/bI7ZBgOycNuCuM3zzWSjQ8HSlz BI/PRZxJ6tbIJwQ1kpxlPYLGpH9FstxH8Hz2GCnVaprc6UZlOV0ORib51yxlT4N58rd7WOr5IYRNO xawMhAxvmhdbjjnzRQjB2JzCNB+xHihCw+OVQU9yoOG8zx09gsqtzsA5omokzWvZGHNuMYwrB8KxE 7hnIPDlPEpUdCydvfOF17rLO04F3U5uvDX6/uT0YiEp2QQy6KdVz1NDFg5tSA+lw7A2TnMpIptU0J NWYBG08vQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafW2-005iFq-Cd; Sun, 25 Apr 2021 14:13:42 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafVz-005iFf-AD for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:13:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=KW55YbQnlz7SIqfPLToGbEidmfkJ1g+UVynIfCIMsRE=; b=mLV15XgjswALmkS4TMyxUq/WuZ AzamfVyTw2DLCQTHNlO4mOrsqJuSggEy6FogophL7uhnhjfkYQvytbU3sktbVdjhv5L9CXa0K1njc Hgw2sEiwjMUW+4VvEbECZgDI9TvSNCkNS2+Jgr1dU09SRWEs/caAu0aspiVDkXWBqREt6ZQJvKpMJ La09S+ObZTsSdO/UvUZqkuDo5Mo4ES4B46XYeXJ5h2qOpN2Y1R0PvvHfqXZptFsgVEBtsWkNKPrsl nGaDwF1L4AZF6a6UiC/+7l+QU+92BnPRUwoREVNgj1G5UBE5ICfJOEpYEE7YX+bRi/LRxDJsA2oYC RJb0QYLQ==; Received: from mail-pj1-x1035.google.com ([2607:f8b0:4864:20::1035]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafVv-00FT3x-K1 for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:13:38 +0000 Received: by mail-pj1-x1035.google.com with SMTP id y22-20020a17090a8b16b0290150ae1a6d2bso3808897pjn.0 for ; Sun, 25 Apr 2021 07:13:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KW55YbQnlz7SIqfPLToGbEidmfkJ1g+UVynIfCIMsRE=; b=J/I88VtUm4TabcGsDPLY+FoRRZxAcj6Fz4AFVQdPVM5KydnQdB5oEf13HPojCWkm5w jpBhVkYG2IUpYEaJmvl7TFU7IY6qG4mOqZ3XjTv/fcyckV7SQcWKQIYHvtpIiXJmOxv0 g2UdNGw29770Tod6y8WNeD5KrTq6MermXcBsHJE0+WhT5CSWyDTPA8kCkMP03a3zdOm7 7E1t+iSHkLtq0HNiUBnNhnX5Bjr7MBOsY35xbpHvW4Sd/xCg9H0PCqaVxwOD3P8RJb7x JMBQCQrjxl1pSM6BRPkIxYOEDkd4BuwK1pTb0yZoZjwQv/IvE8CT/mLD9VMyrLT+C9ss DpTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KW55YbQnlz7SIqfPLToGbEidmfkJ1g+UVynIfCIMsRE=; b=hI5t+kNrWYik07T6qK+KkziVuKxlHoZNITI4zCT2d4/wyXECrgljFf3re+CL57VlkN 7237JzD46gU+HBXGvtAXij5DVGbo6WyhD5h5rtyKCeDlxqaUlnQZaCkhK+ilpYm2mkxk rry+keRD5IloMB27qfK6TZsKvr4gBKPGVB9a99HTZG065DyTa4jFss9dGdRGWOfG77YF mosggh/zX8AoHhdFWWeZVn8dbDdA2AsDQBNQK8+lzVg+IRU0qwdBoy/K9l1iuTTHk51x KkKQ8OQLGOVHMxOwotEySmw1dCYYegBCmccI3rRV1d/WlIVFO6221zqHRmQvlN5UAxtw u8jA== X-Gm-Message-State: AOAM532XxppVo4AlIVnx1VVQxBckjCOHOZ55eTRwZY67gT6nmV1TkvNn gbKijHAW0yQxd9b08X3l6dgNU4+qww== X-Google-Smtp-Source: ABdhPJxJPERun+Wu2dMUnxngmCzDCmfk83f7A+WB0VlC1b4ZrBvr2pAD5NF0bjzhpOn+HaqkJiNYzQ== X-Received: by 2002:a17:90a:f298:: with SMTP id fs24mr15768349pjb.129.1619360013457; Sun, 25 Apr 2021 07:13:33 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:13:26 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 01/10] arm64/mm: split out __create_pgd_mapping() routines Date: Sun, 25 Apr 2021 22:12:55 +0800 Message-Id: <20210425141304.32721-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071335_693705_33D892A1 X-CRM114-Status: GOOD ( 30.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If VA_BITS < 48, an additional table level should be used to allow an ID mapping to cover system RAM. At present, it is done by macro create_table_entry in head.S. But with second thoughts, this expansion can be easily achieved by redefinition of CONFIG_PGTABLE_LEVELS. Split out the routines for __create_pgd_mapping(), so that two sets of code will be generated to manipulate pgtable under two different CONFIG_PGTABLE_LEVELS. Later the one generated under 'CONFIG_PGTABLE_LEVELS + 1' can be used for idmap. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- RFC -> v2: consider the config ARM64_16K_PAGES && ARM64_VA_BITS_36 and move redefinition of CONFIG_PGTABLE_LEVELS to the head of idmap_mmu.c --- arch/arm64/Kconfig | 4 + arch/arm64/mm/Makefile | 2 + arch/arm64/mm/idmap_mmu.c | 34 +++++ arch/arm64/mm/mmu.c | 263 +---------------------------------- arch/arm64/mm/mmu_include.c | 270 ++++++++++++++++++++++++++++++++++++ 5 files changed, 312 insertions(+), 261 deletions(-) create mode 100644 arch/arm64/mm/idmap_mmu.c create mode 100644 arch/arm64/mm/mmu_include.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index e4e1b6550115..79755ade5d27 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -327,6 +327,10 @@ config PGTABLE_LEVELS default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47 default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48 +config IDMAP_PGTABLE_EXPAND + def_bool y + depends on (ARM64_16K_PAGES && ARM64_VA_BITS_36) || (ARM64_64K_PAGES && ARM64_VA_BITS_42) || (ARM64_4K_PAGES && ARM64_VA_BITS_39) || (ARM64_16K_PAGES && ARM64_VA_BITS_47) + config ARCH_SUPPORTS_UPROBES def_bool y diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index f188c9092696..f9283cb9a201 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,6 +3,8 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o + +obj-$(CONFIG_IDMAP_PGTABLE_EXPAND) += idmap_mmu.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/idmap_mmu.c b/arch/arm64/mm/idmap_mmu.c new file mode 100644 index 000000000000..42a27dd5cc9f --- /dev/null +++ b/arch/arm64/mm/idmap_mmu.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#ifdef CONFIG_IDMAP_PGTABLE_EXPAND + +#if CONFIG_PGTABLE_LEVELS == 2 +#define EXTEND_LEVEL 3 +#elif CONFIG_PGTABLE_LEVELS == 3 +#define EXTEND_LEVEL 4 +#endif + +#undef CONFIG_PGTABLE_LEVELS +#define CONFIG_PGTABLE_LEVELS EXTEND_LEVEL + +#include + +#include +#include +#include +#include +#include + +#include "./mmu_include.c" + +void __create_pgd_mapping_extend(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + __create_pgd_mapping(pgdir, phys, virt, size, prot, pgtable_alloc, flags); +} +#endif + + diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 5d9550fdb9cf..56e4f25e8d6d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -37,9 +37,6 @@ #include #include -#define NO_BLOCK_MAPPINGS BIT(0) -#define NO_CONT_MAPPINGS BIT(1) - u64 idmap_t0sz = TCR_T0SZ(VA_BITS_MIN); u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; @@ -116,264 +113,6 @@ static phys_addr_t __init early_pgtable_alloc(int shift) return phys; } -static bool pgattr_change_is_safe(u64 old, u64 new) -{ - /* - * The following mapping attributes may be updated in live - * kernel mappings without the need for break-before-make. - */ - pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; - - /* creating or taking down mappings is always safe */ - if (old == 0 || new == 0) - return true; - - /* live contiguous mappings may not be manipulated at all */ - if ((old | new) & PTE_CONT) - return false; - - /* Transitioning from Non-Global to Global is unsafe */ - if (old & ~new & PTE_NG) - return false; - - /* - * Changing the memory type between Normal and Normal-Tagged is safe - * since Tagged is considered a permission attribute from the - * mismatched attribute aliases perspective. - */ - if (((old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || - (old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED)) && - ((new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || - (new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED))) - mask |= PTE_ATTRINDX_MASK; - - return ((old ^ new) & ~mask) == 0; -} - -static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) -{ - pte_t *ptep; - - ptep = pte_set_fixmap_offset(pmdp, addr); - do { - pte_t old_pte = READ_ONCE(*ptep); - - set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); - - /* - * After the PTE entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - READ_ONCE(pte_val(*ptep)))); - - phys += PAGE_SIZE; - } while (ptep++, addr += PAGE_SIZE, addr != end); - - pte_clear_fixmap(); -} - -static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long next; - pmd_t pmd = READ_ONCE(*pmdp); - - BUG_ON(pmd_sect(pmd)); - if (pmd_none(pmd)) { - phys_addr_t pte_phys; - BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(PAGE_SHIFT); - __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); - pmd = READ_ONCE(*pmdp); - } - BUG_ON(pmd_bad(pmd)); - - do { - pgprot_t __prot = prot; - - next = pte_cont_addr_end(addr, end); - - /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && - (flags & NO_CONT_MAPPINGS) == 0) - __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - - init_pte(pmdp, addr, next, phys, __prot); - - phys += next - addr; - } while (addr = next, addr != end); -} - -static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) -{ - unsigned long next; - pmd_t *pmdp; - - pmdp = pmd_set_fixmap_offset(pudp, addr); - do { - pmd_t old_pmd = READ_ONCE(*pmdp); - - next = pmd_addr_end(addr, end); - - /* try section mapping first */ - if (((addr | next | phys) & ~SECTION_MASK) == 0 && - (flags & NO_BLOCK_MAPPINGS) == 0) { - pmd_set_huge(pmdp, phys, prot); - - /* - * After the PMD entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), - READ_ONCE(pmd_val(*pmdp)))); - } else { - alloc_init_cont_pte(pmdp, addr, next, phys, prot, - pgtable_alloc, flags); - - BUG_ON(pmd_val(old_pmd) != 0 && - pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); - } - phys += next - addr; - } while (pmdp++, addr = next, addr != end); - - pmd_clear_fixmap(); -} - -static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) -{ - unsigned long next; - pud_t pud = READ_ONCE(*pudp); - - /* - * Check for initial section mappings in the pgd/pud. - */ - BUG_ON(pud_sect(pud)); - if (pud_none(pud)) { - phys_addr_t pmd_phys; - BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(PMD_SHIFT); - __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); - pud = READ_ONCE(*pudp); - } - BUG_ON(pud_bad(pud)); - - do { - pgprot_t __prot = prot; - - next = pmd_cont_addr_end(addr, end); - - /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PMD_MASK) == 0) && - (flags & NO_CONT_MAPPINGS) == 0) - __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - - init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); - - phys += next - addr; - } while (addr = next, addr != end); -} - -static inline bool use_1G_block(unsigned long addr, unsigned long next, - unsigned long phys) -{ - if (PAGE_SHIFT != 12) - return false; - - if (((addr | next | phys) & ~PUD_MASK) != 0) - return false; - - return true; -} - -static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long next; - pud_t *pudp; - p4d_t *p4dp = p4d_offset(pgdp, addr); - p4d_t p4d = READ_ONCE(*p4dp); - - if (p4d_none(p4d)) { - phys_addr_t pud_phys; - BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(PUD_SHIFT); - __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); - p4d = READ_ONCE(*p4dp); - } - BUG_ON(p4d_bad(p4d)); - - pudp = pud_set_fixmap_offset(p4dp, addr); - do { - pud_t old_pud = READ_ONCE(*pudp); - - next = pud_addr_end(addr, end); - - /* - * For 4K granule only, attempt to put down a 1GB block - */ - if (use_1G_block(addr, next, phys) && - (flags & NO_BLOCK_MAPPINGS) == 0) { - pud_set_huge(pudp, phys, prot); - - /* - * After the PUD entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), - READ_ONCE(pud_val(*pudp)))); - } else { - alloc_init_cont_pmd(pudp, addr, next, phys, prot, - pgtable_alloc, flags); - - BUG_ON(pud_val(old_pud) != 0 && - pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); - } - phys += next - addr; - } while (pudp++, addr = next, addr != end); - - pud_clear_fixmap(); -} - -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long addr, end, next; - pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); - - /* - * If the virtual and physical address don't have the same offset - * within a page, we cannot map the region as the caller expects. - */ - if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) - return; - - phys &= PAGE_MASK; - addr = virt & PAGE_MASK; - end = PAGE_ALIGN(virt + size); - - do { - next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, - flags); - phys += next - addr; - } while (pgdp++, addr = next, addr != end); -} - static phys_addr_t __pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL); @@ -404,6 +143,8 @@ static phys_addr_t pgd_pgtable_alloc(int shift) return pa; } +#include "./mmu_include.c" + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c new file mode 100644 index 000000000000..95ff35a3c6cb --- /dev/null +++ b/arch/arm64/mm/mmu_include.c @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * This file is shared + * And these functions should be PIC because they are called under both MMU-disable + * and MMU-enable + */ + +#define NO_BLOCK_MAPPINGS BIT(0) +#define NO_CONT_MAPPINGS BIT(1) + +static bool pgattr_change_is_safe(u64 old, u64 new) +{ + /* + * The following mapping attributes may be updated in live + * kernel mappings without the need for break-before-make. + */ + pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; + + /* creating or taking down mappings is always safe */ + if (old == 0 || new == 0) + return true; + + /* live contiguous mappings may not be manipulated at all */ + if ((old | new) & PTE_CONT) + return false; + + /* Transitioning from Non-Global to Global is unsafe */ + if (old & ~new & PTE_NG) + return false; + + /* + * Changing the memory type between Normal and Normal-Tagged is safe + * since Tagged is considered a permission attribute from the + * mismatched attribute aliases perspective. + */ + if (((old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || + (old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED)) && + ((new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || + (new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED))) + mask |= PTE_ATTRINDX_MASK; + + return ((old ^ new) & ~mask) == 0; +} + +static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot) +{ + pte_t *ptep; + + ptep = pte_set_fixmap_offset(pmdp, addr); + do { + pte_t old_pte = READ_ONCE(*ptep); + + set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + + /* + * After the PTE entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), + READ_ONCE(pte_val(*ptep)))); + + phys += PAGE_SIZE; + } while (ptep++, addr += PAGE_SIZE, addr != end); + + pte_clear_fixmap(); +} + +static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pmd_t pmd = READ_ONCE(*pmdp); + + BUG_ON(pmd_sect(pmd)); + if (pmd_none(pmd)) { + phys_addr_t pte_phys; + + BUG_ON(!pgtable_alloc); + pte_phys = pgtable_alloc(PAGE_SHIFT); + __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); + pmd = READ_ONCE(*pmdp); + } + BUG_ON(pmd_bad(pmd)); + + do { + pgprot_t __prot = prot; + + next = pte_cont_addr_end(addr, end); + + /* use a contiguous mapping if the range is suitably aligned */ + if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && + (flags & NO_CONT_MAPPINGS) == 0) + __prot = __pgprot(pgprot_val(prot) | PTE_CONT); + + init_pte(pmdp, addr, next, phys, __prot); + + phys += next - addr; + } while (addr = next, addr != end); +} + +static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) +{ + unsigned long next; + pmd_t *pmdp; + + pmdp = pmd_set_fixmap_offset(pudp, addr); + do { + pmd_t old_pmd = READ_ONCE(*pmdp); + + next = pmd_addr_end(addr, end); + + /* try section mapping first */ + if (((addr | next | phys) & ~SECTION_MASK) == 0 && + (flags & NO_BLOCK_MAPPINGS) == 0) { + pmd_set_huge(pmdp, phys, prot); + + /* + * After the PMD entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), + READ_ONCE(pmd_val(*pmdp)))); + } else { + alloc_init_cont_pte(pmdp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(pmd_val(old_pmd) != 0 && + pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); + } + phys += next - addr; + } while (pmdp++, addr = next, addr != end); + + pmd_clear_fixmap(); +} + +static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) +{ + unsigned long next; + pud_t pud = READ_ONCE(*pudp); + + /* + * Check for initial section mappings in the pgd/pud. + */ + BUG_ON(pud_sect(pud)); + if (pud_none(pud)) { + phys_addr_t pmd_phys; + + BUG_ON(!pgtable_alloc); + pmd_phys = pgtable_alloc(PMD_SHIFT); + __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); + pud = READ_ONCE(*pudp); + } + BUG_ON(pud_bad(pud)); + + do { + pgprot_t __prot = prot; + + next = pmd_cont_addr_end(addr, end); + + /* use a contiguous mapping if the range is suitably aligned */ + if ((((addr | next | phys) & ~CONT_PMD_MASK) == 0) && + (flags & NO_CONT_MAPPINGS) == 0) + __prot = __pgprot(pgprot_val(prot) | PTE_CONT); + + init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); + + phys += next - addr; + } while (addr = next, addr != end); +} + +static inline bool use_1G_block(unsigned long addr, unsigned long next, + unsigned long phys) +{ + if (PAGE_SHIFT != 12) + return false; + + if (((addr | next | phys) & ~PUD_MASK) != 0) + return false; + + return true; +} + +static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pud_t *pudp; + p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t p4d = READ_ONCE(*p4dp); + + if (p4d_none(p4d)) { + phys_addr_t pud_phys; + + BUG_ON(!pgtable_alloc); + pud_phys = pgtable_alloc(PUD_SHIFT); + __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); + p4d = READ_ONCE(*p4dp); + } + BUG_ON(p4d_bad(p4d)); + + pudp = pud_set_fixmap_offset(p4dp, addr); + do { + pud_t old_pud = READ_ONCE(*pudp); + + next = pud_addr_end(addr, end); + + /* + * For 4K granule only, attempt to put down a 1GB block + */ + if (use_1G_block(addr, next, phys) && + (flags & NO_BLOCK_MAPPINGS) == 0) { + pud_set_huge(pudp, phys, prot); + + /* + * After the PUD entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), + READ_ONCE(pud_val(*pudp)))); + } else { + alloc_init_cont_pmd(pudp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(pud_val(old_pud) != 0 && + pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); + } + phys += next - addr; + } while (pudp++, addr = next, addr != end); + + pud_clear_fixmap(); +} + +static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long addr, end, next; + pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + + /* + * If the virtual and physical address don't have the same offset + * within a page, we cannot map the region as the caller expects. + */ + if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) + return; + + phys &= PAGE_MASK; + addr = virt & PAGE_MASK; + end = PAGE_ALIGN(virt + size); + + do { + next = pgd_addr_end(addr, end); + alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, + flags); + phys += next - addr; + } while (pgdp++, addr = next, addr != end); +} From patchwork Sun Apr 25 14:12:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 880A9C433B4 for ; Sun, 25 Apr 2021 14:15:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07DB66128B for ; Sun, 25 Apr 2021 14:15:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07DB66128B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=G3iBv8BkNgSMIXYzE9Bovpr3+kjcVh6P8pRrgIxT4T8=; b=gDWeZuAg+i11wfdn7XXtGbM9o 90MXAeOLd3eoLbViwS8VhByJw3xT0/tECeYdsDZFpj8EcyIe9oPmh9C7+brhEk/UgeujAgZCBj8HE CqQUmyTA68jHVYcVmpKJFjDqhqjDZx7Eas8aKgzbJZqeBNl0tRjmsNze+JKjb4i3QlmEIxnLKlMLm v+qD49DyM0Qg7VdO7hxHXvp+rpJkTfp6+muD4CKjoms1SVkJzjT5GL2DmlUHO6f0cu6gp0e+jb30D 6/l4BBkmdAgeVPo6C3hrCrDLTAcN8kuJ5ebld1NSdeE+wmU3LH2YYGg6ZlyoSLeOK+VPBP9/2euY3 yyt5YIwAw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafWE-005iH4-Gi; Sun, 25 Apr 2021 14:13:54 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafW4-005iGH-Io for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:13:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=zxK5dXUW170Ydn4SsP2MT19YDrd+YAyUvlsbJP4btws=; b=DZmIVjwYUFLXF6axX1aOCn8OEo fNNiJWAZgPYOlf6G8ntkCmLb9nJ3N2Ipu10ZKdJc5Be8+AbL2BPiLKaprnqDZtnHyX6y3m4dPdkrX 4/+hEIYiMYtUDWQB+T5F47nXDt5yQ/ByPhmLwi4bB2ZETnNMVIbH7tmN/AbM1NoKk/PYHtRkPB4Jo V07WRlEp20P1ch1SH/ErH8UvkMRVxAIwm/ibCXNbz72gCu3fAimSqAtPYbWQjQlaQinB4Yf6orCoa SMIlQo3QSm8T9Wug6dDi/Y9R2B+1P/VvFBtxJd8N/0SxNbSHrgcy+haQ6A7SuHLU0kp+8a1O+wueI BgFiBbvg==; Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafW1-00FT4G-P0 for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:13:43 +0000 Received: by mail-pl1-x635.google.com with SMTP id o16so13949227plg.5 for ; Sun, 25 Apr 2021 07:13:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zxK5dXUW170Ydn4SsP2MT19YDrd+YAyUvlsbJP4btws=; b=SJos2+7BsnKGyh+6pJlCE6LNA3pxYOmlUPtpuMr1qprUk7oKS7HwJ1wikKILj+Ao6R LrgDg0SRKlI2h9276di6n77Fr9ZDhluVLcPU9x0TicUjMqysNfG2kZhQnskbHChBKWlQ jFhKzTBvWclTkyZx2LyArbnBtz6Kh0ciQLnspbkdvDlvYEdeKvsJ8JUfvhTC4VQfQ+gD JZseYKy4J7lGi7ZSiKeIBO4t0QZu32J+z1BW+79HcRwODPgn3q3ZkHNAoHf0ov+4Yyuy jrsTYidhW19HOmnpPCdU1QnunTITk2zUVMDcuJSmEkvbH/wtrS87N6+7OlobrpO6X9rW eEmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zxK5dXUW170Ydn4SsP2MT19YDrd+YAyUvlsbJP4btws=; b=JXPcuMliXmK2bBNNXHYvmLPHauoiY1Aw5mz5M7tNxNBVjA6CrA09m1PocMX19J9X12 2e+m1wGqOxyGOiuNrA7QCnFP/aKdAwuN+NbcrBtZ/xnJE7n+BgGyPTM4lbkmf2Hic7sZ JKenTu98NNPJ8xeaUVaKOqfxOHU9TpeYHVxAlKKtUWNxjyK7CeYjj52UgNSWbBCk/zyy RiDUsuyyf8K6PHGbs57dnCTK+oV7HnPRWzFJ2XixNI9GfHKoQO9hu28brL9DFUAcxIlt Vot1DbUv5oAETVYaivHbHO4MsNos3AFuN3kARatqMcKvi9QpM03e2Zp01enoziWerh8Z QCCQ== X-Gm-Message-State: AOAM532A57SQIS3YwBKQtugoLt3ompSaQL3q9DA/FtJnktSdBNYzFNpe uXZPc/GtGftXzqiBPpBN7oo974/dNw== X-Google-Smtp-Source: ABdhPJw85jvQV2QAesRB0jodAI6dX0mLONuKCslJoJHG+5KMJ+2D80LNcfdIbpctyB8O6nSlBIiTSA== X-Received: by 2002:a17:902:8697:b029:ea:a5dc:925f with SMTP id g23-20020a1709028697b02900eaa5dc925fmr13857390plo.56.1619360019048; Sun, 25 Apr 2021 07:13:39 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:13:38 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 02/10] arm64/mm: change __create_pgd_mapping() to accept nr_entries param and introduce create_idmap() Date: Sun, 25 Apr 2021 22:12:56 +0800 Message-Id: <20210425141304.32721-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071341_838095_77DDE82E X-CRM114-Status: GOOD ( 18.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As idmap_ptrs_per_pgd may have pgd entries greater than PTRS_PER_PGD, the prototype of __create_pgd_mapping() needs change to cope with that to create idmap. Now this adaption, create_idmap() API can be introduced to create idmap handly for all kinds of CONFIG_PGTABLE_LEVEL Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 7 ++++++ arch/arm64/kernel/head.S | 3 +++ arch/arm64/mm/idmap_mmu.c | 16 ++++++++----- arch/arm64/mm/mmu.c | 41 ++++++++++++++++++++++++++------ arch/arm64/mm/mmu_include.c | 9 +++++-- 5 files changed, 61 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 3c6a7f5988b1..555792921af0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -83,4 +83,11 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) } #define pmd_pgtable(pmd) pmd_page(pmd) +extern void __create_pgd_mapping_extend(pgd_t *pgdir, + unsigned int entries_cnt, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags); + #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 840bda1869e9..e19649dbbafb 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -341,6 +341,9 @@ SYM_FUNC_START_LOCAL(__create_page_tables) #if VA_BITS != EXTRA_SHIFT #error "Mismatch between VA_BITS and page size/number of translation levels" #endif + adr_l x4, idmap_extend_pgtable + mov x5, #1 + str x5, [x4] //require expanded pagetable mov x4, EXTRA_PTRS create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 diff --git a/arch/arm64/mm/idmap_mmu.c b/arch/arm64/mm/idmap_mmu.c index 42a27dd5cc9f..bff1bffee321 100644 --- a/arch/arm64/mm/idmap_mmu.c +++ b/arch/arm64/mm/idmap_mmu.c @@ -21,13 +21,17 @@ #include "./mmu_include.c" -void __create_pgd_mapping_extend(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +void __create_pgd_mapping_extend(pgd_t *pgdir, + unsigned int entries_cnt, + phys_addr_t phys, + unsigned long virt, + phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { - __create_pgd_mapping(pgdir, phys, virt, size, prot, pgtable_alloc, flags); + __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, prot, + pgtable_alloc, flags); } #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 56e4f25e8d6d..70a5a7b032dc 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -145,6 +145,33 @@ static phys_addr_t pgd_pgtable_alloc(int shift) #include "./mmu_include.c" +int idmap_extend_pgtable; + +/* + * lock: no lock protection alongside this function call + * todo: tear down idmap. (no requirement at present) + */ +void create_idmap(pgd_t *pgdir, phys_addr_t phys, + phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + u64 ptrs_per_pgd = idmap_ptrs_per_pgd; + +#ifdef CONFIG_IDMAP_PGTABLE_EXPAND + if (idmap_extend_pgtable) + __create_pgd_mapping_extend(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); + else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#endif +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -158,7 +185,7 @@ static void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, &phys, virt); return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); } @@ -173,7 +200,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, if (page_mappings_only) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(mm->pgd, phys, virt, size, prot, + __create_pgd_mapping(mm->pgd, PTRS_PER_PGD, phys, virt, size, prot, pgd_pgtable_alloc, flags); } @@ -186,7 +213,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); /* flush the TLBs after updating live kernel mappings */ @@ -196,7 +223,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, phys_addr_t end, pgprot_t prot, int flags) { - __create_pgd_mapping(pgdp, start, __phys_to_virt(start), end - start, + __create_pgd_mapping(pgdp, PTRS_PER_PGD, start, __phys_to_virt(start), end - start, prot, early_pgtable_alloc, flags); } @@ -297,7 +324,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, + __create_pgd_mapping(pgdp, PTRS_PER_PGD, pa_start, (unsigned long)va_start, size, prot, early_pgtable_alloc, flags); if (!(vm_flags & VM_NO_GUARD)) @@ -341,7 +368,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); - __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, + __create_pgd_mapping(tramp_pg_dir, PTRS_PER_PGD, pa_start, TRAMP_VALIAS, PAGE_SIZE, prot, __pgd_pgtable_alloc, 0); /* Map both the text and data into the kernel page table */ @@ -1233,7 +1260,7 @@ int arch_add_memory(int nid, u64 start, u64 size, IS_ENABLED(CONFIG_KFENCE)) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), + __create_pgd_mapping(swapper_pg_dir, PTRS_PER_PGD, start, __phys_to_virt(start), size, params->pgprot, __pgd_pgtable_alloc, flags); diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index 95ff35a3c6cb..be51689d1133 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -241,14 +241,19 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, +static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long addr, end, next; - pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + pgd_t *pgdp; + + if (likely(entries_cnt == PTRS_PER_PGD)) + pgdp = pgd_offset_pgd(pgdir, virt); + else + pgdp = pgdir + ((virt >> PGDIR_SHIFT) & (entries_cnt - 1)); /* * If the virtual and physical address don't have the same offset From patchwork Sun Apr 25 14:12:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E352C43461 for ; Sun, 25 Apr 2021 14:15:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B34E061369 for ; Sun, 25 Apr 2021 14:15:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B34E061369 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XHDrNnuCcW+ke01fcTy0kdtG8Cm63Gb0eKlRpsK1kyA=; b=dpIMjVDaGjW2+D0o9/9FBLlwA jJiLQJ1Fux89x28yyVMXv2LLf+vv17TtXrH2sSqcL21uDpUgC3OhXW1wULQR0quGpwSziowLqX1n0 nY2MXOawldIaath9YrfQGxTvD7PlW5CP8y+lKZ5ucgVZ5B15JlweYOQfJwEXbRtLaINHrJL/NQ41K Q8qYWzRCEjAGyeYAPeqZ7nS6nwJjqTNQT27jzSylIbcwkmE5JkpPeTTeAE1zmMHYj2fglscth2tFX UfepggYMHexyNUtpTmyCC/+uFaoKwwbOZxtrozE15eeqh9MuTQNVAHa4L7e0ks/aM3xi+WMXdnAOS ZfI8gt3Wg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafWP-005iI1-0o; Sun, 25 Apr 2021 14:14:05 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafW9-005iGS-10 for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:13:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=U5gbanVTJ9Aw8b1zBkgEbWv2enAd86V77nRaPs+4B70=; b=YCohjPtbEam9W1wfo9y24PNcz8 zjgBKhcoK/Q9Ntmcr7yoDZyDrb57vH+9h+5aHK2gY63BoJ7IihAv1fRG/crLnh6AFnOnl2iJxY23D HPocDvpgxy5itJbqh6dV2zj2JcoaZMa8UU2shv1DgIpZAsPRV56neaTBGrWNymkxEdcxmMXt4MZG0 02RYinAlTC1bio59lwDXm5k3sCvXsyhurx0Yr812lYFmvKzN0RJgCIDT5u7dPPSXQUI5fKFi7+NA1 zhBDcwCU6/YUHUx9cRnYK0w1YhebsR9b+ffxO4Numgb7hhwTLZvnsvp7Ba3D5JBEm797Hl/FLPeya jT1avw/w==; Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafW5-00FT4f-Te for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:13:47 +0000 Received: by mail-pf1-x42a.google.com with SMTP id e15so453642pfv.10 for ; Sun, 25 Apr 2021 07:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U5gbanVTJ9Aw8b1zBkgEbWv2enAd86V77nRaPs+4B70=; b=APYcP23C3nQSUnPnF2xSxmB84yPSKF4Rpg2Pjdru057jxnglVILgvEAw5mdyXG6N9y jvZis4aW6tSs3SYrHu6B27wcdrGKBNuA1RjyfdlTIxAmLwpgYOqffg5510+XL7V9JGqA SI/pPUdyTHSle32HaVtx/5CC2MZxAvDu709HmSuBcvjrf/c1vWCfJMW7xNuq7ifgeSjd tTfLqCxjZwcd68+CaoX+IRFoheyyhhr9bMhX1BA7Kp02+nKW2Ch3Vxjw1OBxZ49NlUGc oifZK+V2Hb0Gz+/o2PwzPHiylMmwO0spkYLxQSxsKOmr5ENEhsE+JsmVEtiig48UA8k6 aonw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U5gbanVTJ9Aw8b1zBkgEbWv2enAd86V77nRaPs+4B70=; b=PWpklxREIvotKsuKspwvtGC1xHxKIoqgtfJxyzu78bcGul84hNsi/rIjLsNCzvLD62 eg7RtHEjtZtJvgwr5v7a4j5azj7cUT72pVxFrRzwbjatt88IN1cmTZ4cSxl9HJZPaKJR fLE4Ks5dnu05xMc1vT34kSrgTh3wUZVpze+LnlSpjE41NZZXvM2epqV5FP1Rrv0LQ7vD ypiuz2oCn/IQGCYYE5LouBlxH9Wr07Me5hFMbcnoyF6ec0VAxSWi74A+Zwg0gZxW4mfX 8Z7YdjCN7xRapDKZcufuNHvEy085kCJUgiA8+ZfgyhQoSHJa2aG6XX0ZDw7sSv9TXzl1 u/Vw== X-Gm-Message-State: AOAM531jIYe1TkJ1j1AmZ2XYkYW6AKzcRQIkHWZi7xUYOLDRrmTzlAEA A+oF79OrbiUGnmHDbLvLmY6WOFo1Fw== X-Google-Smtp-Source: ABdhPJybdQ2fg+d64YE4YVdQ3LK2jNNIUM4hFnaOZ0Z9NuxhMvz01Ns1jl5Ffp+Zgq5W7JKguzr3JA== X-Received: by 2002:a63:91c9:: with SMTP id l192mr12692090pge.339.1619360024841; Sun, 25 Apr 2021 07:13:44 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:13:44 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 03/10] arm64/mm: change __create_pgd_mapping() to accept extra parameter for allocator Date: Sun, 25 Apr 2021 22:12:57 +0800 Message-Id: <20210425141304.32721-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071345_989754_EBA2DA6A X-CRM114-Status: GOOD ( 16.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current pgtable allocator only accepts the depth of pgtable as parameter. And the allocator function itself determines the memory pool info. But incoming pgtable allocator needs an extra param to get local pool info, which directs the allocation. Here preparing the prototype for the incoming change. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 5 +++- arch/arm64/mm/idmap_mmu.c | 5 ++-- arch/arm64/mm/mmu.c | 31 +++++++++++++------------ arch/arm64/mm/mmu_include.c | 39 +++++++++++++++++++------------- 4 files changed, 46 insertions(+), 34 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 555792921af0..42f602528b90 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -83,11 +83,14 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) } #define pmd_pgtable(pmd) pmd_page(pmd) +typedef phys_addr_t (*pgtable_alloc)(unsigned long shift, void *data); + extern void __create_pgd_mapping_extend(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags); #endif diff --git a/arch/arm64/mm/idmap_mmu.c b/arch/arm64/mm/idmap_mmu.c index bff1bffee321..4477cc2704a7 100644 --- a/arch/arm64/mm/idmap_mmu.c +++ b/arch/arm64/mm/idmap_mmu.c @@ -27,11 +27,12 @@ void __create_pgd_mapping_extend(pgd_t *pgdir, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, prot, - pgtable_alloc, flags); + allocator, info, flags); } #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 70a5a7b032dc..520738c43874 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -86,7 +86,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); -static phys_addr_t __init early_pgtable_alloc(int shift) +static phys_addr_t __init early_pgtable_alloc(unsigned long unused_a, void *unused_b) { phys_addr_t phys; void *ptr; @@ -113,7 +113,7 @@ static phys_addr_t __init early_pgtable_alloc(int shift) return phys; } -static phys_addr_t __pgd_pgtable_alloc(int shift) +static phys_addr_t __pgd_pgtable_alloc(unsigned long unused_a, void *unused_b) { void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL); BUG_ON(!ptr); @@ -123,9 +123,9 @@ static phys_addr_t __pgd_pgtable_alloc(int shift) return __pa(ptr); } -static phys_addr_t pgd_pgtable_alloc(int shift) +static phys_addr_t pgd_pgtable_alloc(unsigned long shift, void *unused) { - phys_addr_t pa = __pgd_pgtable_alloc(shift); + phys_addr_t pa = __pgd_pgtable_alloc(shift, unused); /* * Call proper page table ctor in case later we need to @@ -154,7 +154,8 @@ int idmap_extend_pgtable; void create_idmap(pgd_t *pgdir, phys_addr_t phys, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { u64 ptrs_per_pgd = idmap_ptrs_per_pgd; @@ -162,13 +163,13 @@ void create_idmap(pgd_t *pgdir, phys_addr_t phys, #ifdef CONFIG_IDMAP_PGTABLE_EXPAND if (idmap_extend_pgtable) __create_pgd_mapping_extend(pgdir, ptrs_per_pgd, - phys, phys, size, prot, pgtable_alloc, flags); + phys, phys, size, prot, allocator, info, flags); else __create_pgd_mapping(pgdir, ptrs_per_pgd, - phys, phys, size, prot, pgtable_alloc, flags); + phys, phys, size, prot, allocator, info, flags); #else __create_pgd_mapping(pgdir, ptrs_per_pgd, - phys, phys, size, prot, pgtable_alloc, flags); + phys, phys, size, prot, allocator, info, flags); #endif } @@ -186,7 +187,7 @@ static void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, return; } __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, - NO_CONT_MAPPINGS); + NULL, NO_CONT_MAPPINGS); } void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, @@ -201,7 +202,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(mm->pgd, PTRS_PER_PGD, phys, virt, size, prot, - pgd_pgtable_alloc, flags); + pgd_pgtable_alloc, NULL, flags); } static void update_mapping_prot(phys_addr_t phys, unsigned long virt, @@ -214,7 +215,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, } __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, - NO_CONT_MAPPINGS); + NULL, NO_CONT_MAPPINGS); /* flush the TLBs after updating live kernel mappings */ flush_tlb_kernel_range(virt, virt + size); @@ -224,7 +225,7 @@ static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, phys_addr_t end, pgprot_t prot, int flags) { __create_pgd_mapping(pgdp, PTRS_PER_PGD, start, __phys_to_virt(start), end - start, - prot, early_pgtable_alloc, flags); + prot, early_pgtable_alloc, NULL, flags); } void __init mark_linear_text_alias_ro(void) @@ -325,7 +326,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(size)); __create_pgd_mapping(pgdp, PTRS_PER_PGD, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); + early_pgtable_alloc, NULL, flags); if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -369,7 +370,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); __create_pgd_mapping(tramp_pg_dir, PTRS_PER_PGD, pa_start, TRAMP_VALIAS, PAGE_SIZE, - prot, __pgd_pgtable_alloc, 0); + prot, __pgd_pgtable_alloc, NULL, 0); /* Map both the text and data into the kernel page table */ __set_fixmap(FIX_ENTRY_TRAMP_TEXT, pa_start, prot); @@ -1261,7 +1262,7 @@ int arch_add_memory(int nid, u64 start, u64 size, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(swapper_pg_dir, PTRS_PER_PGD, start, __phys_to_virt(start), - size, params->pgprot, __pgd_pgtable_alloc, + size, params->pgprot, __pgd_pgtable_alloc, NULL, flags); memblock_clear_nomap(start, size); diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index be51689d1133..732e603fe3fc 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -69,7 +69,8 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { unsigned long next; @@ -79,8 +80,8 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, if (pmd_none(pmd)) { phys_addr_t pte_phys; - BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(PAGE_SHIFT); + BUG_ON(!allocator); + pte_phys = allocator(PAGE_SHIFT, info); __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); pmd = READ_ONCE(*pmdp); } @@ -104,7 +105,9 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) + pgtable_alloc allocator, + void *info, + int flags) { unsigned long next; pmd_t *pmdp; @@ -128,7 +131,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, READ_ONCE(pmd_val(*pmdp)))); } else { alloc_init_cont_pte(pmdp, addr, next, phys, prot, - pgtable_alloc, flags); + allocator, info, flags); BUG_ON(pmd_val(old_pmd) != 0 && pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); @@ -142,7 +145,9 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) + pgtable_alloc allocator, + void *info, + int flags) { unsigned long next; pud_t pud = READ_ONCE(*pudp); @@ -154,8 +159,8 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, if (pud_none(pud)) { phys_addr_t pmd_phys; - BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(PMD_SHIFT); + BUG_ON(!allocator); + pmd_phys = allocator(PMD_SHIFT, info); __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); pud = READ_ONCE(*pudp); } @@ -171,7 +176,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); + init_pmd(pudp, addr, next, phys, __prot, allocator, info, flags); phys += next - addr; } while (addr = next, addr != end); @@ -191,7 +196,8 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { unsigned long next; @@ -202,8 +208,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, if (p4d_none(p4d)) { phys_addr_t pud_phys; - BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(PUD_SHIFT); + BUG_ON(!allocator); + pud_phys = allocator(PUD_SHIFT, info); __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); p4d = READ_ONCE(*p4dp); } @@ -230,7 +236,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, READ_ONCE(pud_val(*pudp)))); } else { alloc_init_cont_pmd(pudp, addr, next, phys, prot, - pgtable_alloc, flags); + allocator, info, flags); BUG_ON(pud_val(old_pud) != 0 && pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); @@ -244,7 +250,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { unsigned long addr, end, next; @@ -268,8 +275,8 @@ static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_ad do { next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, - flags); + alloc_init_pud(pgdp, addr, next, phys, prot, allocator, + info, flags); phys += next - addr; } while (pgdp++, addr = next, addr != end); } From patchwork Sun Apr 25 14:12:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56953C433B4 for ; Sun, 25 Apr 2021 14:16:03 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DAA5560232 for ; Sun, 25 Apr 2021 14:16:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DAA5560232 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2PgQEJRroA0A0/Pqo+XTB+Spqx/1mPyompvNcUBOHhk=; b=nchC3lPdVG3a/eR0byP2MbVv1 bNZiaf5XW7LhecWZZ5mkoUPdgWqlVqwocl+mni+CvwObhNeAdfFQHCrfanKuUXeT/9wJaH4V2ptqP WgkItrTQxCePa/QvIyq90UeXgcHFHB7oDVePLuNiQEYXApfgEhNtMsz10qLIgHlkg/Bw53f/9hK00 R0WO69N4r1Fy8wdXe7gz0YyfTjPmAhMBAAVyMUXufCGJNTyXTaorjurSKZmeFsuWjraGUmuT34B03 oTBXcELgE3Wacjf1XqLIE49A8bAdmxF4XOKtOAraJ2ZL8mJfcBbeMud77tunZtM39xKtQZQbl95IX Ud+ho69iw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafWc-005iKq-Dv; Sun, 25 Apr 2021 14:14:20 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWF-005iHG-JN for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:13:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=Rq5MdgbFwq5YnoQQmOm0iH5gEV1BaGEp7qhwvG8Ibyo=; b=aOVPcQEuWNh0y4zyDVkI8md18v OvniJbXmFMO7SPlgDOWcwp8P5W7X1wIv7DVKjjZQNjRhLbOC7+YNPvyflAK3gAhgnUan/mZzQAFsF hZCqx65RECjIEXx0ECwvtqjvvVCpawjGA2bqJ5GPcJo7JThusbA9VDU25bhuDh5ZGnAoV/RyZz29U F3IosrBNunpSbXYGxiWurU6Dk6NiBYmQsNGDWklfgDl0yGa1cfSi1e32mnn4NUcdgLUCuITTCrxSO YNslbkrPKqlApUVlAyS8dIyBLYy+vg+Pmk1KH+xe3g3AR9z5Sdfqpy1H+CMwZSP0c2GFrsENrIUz2 joJxwA7A==; Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWD-00FT5M-05 for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:13:54 +0000 Received: by mail-pl1-x62f.google.com with SMTP id y1so11907399plg.11 for ; Sun, 25 Apr 2021 07:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Rq5MdgbFwq5YnoQQmOm0iH5gEV1BaGEp7qhwvG8Ibyo=; b=mV0xuKIeEI6j+1MXq1J6oreOpwP7pc5dWy2qVaG2QHH9RuVFsbKit3jHuna5FP7Vsa /hgxzFC/e/cQIzpjwYuKWG5HEaNbLoQWR+ttUNCokZlELF7D1fnBBmkyB77xJ8qtzOGA UOMOOLejM3eRGvrRaNjNQuQnWOxacsHgw8+jQD1zXhC3ANUkRcfOe3ji2hdw3O4cOZ0o EqAgrM4I227LvIeR1Ybb4Hw40Xzgo1d+uyh+YEHClIaKHeu2XM62ZDwuhCvo6P6PmhOS td3tKOu17vDHeLDsZHtTejK5sZQVhTqkOhKHu+EfsBAFgsMdEMFXCMBmNPkKZlsI2OZu ntSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Rq5MdgbFwq5YnoQQmOm0iH5gEV1BaGEp7qhwvG8Ibyo=; b=WEYlmTFNr4ZqJgp3IT57rKE2BcbmVnwAenWajY8WuMAH/EhGpqKZg/0Yc9sYarWbWJ Ch0FNF7BbJa22iuju24g6RBNQv18lYLbzVwCTHPC4Z+XTxf7HTzR43zgPzmi74N76SN8 oV61zS6LEmCilmIudCSGuYzsFdZS3gMo5Ebqfn17QxT95DlvoYy38zQxqldmZZIpEHda aR1MHSIlNz7EvQCjFbbO0DKx8vdE4gZCx6Jzm3NbfeSnvHKoESPRWDoiNXAmiUnxb1vX qek8NVntAc1Wb6wZ/Y6HUcYk5vA/QxpSYaMhZtaW20wLwVitPzAfsS0IT8bzxObPZSKe Okjw== X-Gm-Message-State: AOAM530vTEzEyWbSMJXGzeimj8nQdhJwhV7TT5dx5FcsHZj6DyPlmCEb +yeoJjV2+4Gn8uUNoZq/WbhVZl1obw== X-Google-Smtp-Source: ABdhPJwTeWXmjB8ZBQi5cLL8QK5euu4MxeHQtTtWnaYJ1mc2au6QXlfgc7+DDNNORNyTxlLbrBPeBA== X-Received: by 2002:a17:902:9008:b029:e6:f37a:2183 with SMTP id a8-20020a1709029008b02900e6f37a2183mr13662623plp.49.1619360030341; Sun, 25 Apr 2021 07:13:50 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:13:50 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 04/10] arm64/mm: enable __create_pgd_mapping() to run across different pgtable Date: Sun, 25 Apr 2021 22:12:58 +0800 Message-Id: <20210425141304.32721-5-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071353_062712_5E8B3F86 X-CRM114-Status: GOOD ( 12.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org __create_pgd_mapping() is planned not only to be used in boot process, but also after fully bootup. The latter means several callers can run __create_pgd_mapping() concurrenlty. In this case, PUP/PMD/PTE fixmap should be not used, instead, the virtual addresss owned by each pgtable is used. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- rfc -> v2: change allocator return type back to phys_addr_t --- arch/arm64/mm/mmu_include.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index 732e603fe3fc..ac8850fe6ce2 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -7,6 +7,7 @@ #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) +#define NO_FIXMAP BIT(2) static bool pgattr_change_is_safe(u64 old, u64 new) { @@ -43,11 +44,14 @@ static bool pgattr_change_is_safe(u64 old, u64 new) } static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) + phys_addr_t phys, pgprot_t prot, int flags) { pte_t *ptep; - ptep = pte_set_fixmap_offset(pmdp, addr); + if (likely(!(flags & NO_FIXMAP))) + ptep = pte_set_fixmap_offset(pmdp, addr); + else + ptep = pte_offset_kernel(pmdp, addr); do { pte_t old_pte = READ_ONCE(*ptep); @@ -63,7 +67,8 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); - pte_clear_fixmap(); + if (likely(!(flags & NO_FIXMAP))) + pte_clear_fixmap(); } static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, @@ -97,7 +102,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pte(pmdp, addr, next, phys, __prot); + init_pte(pmdp, addr, next, phys, __prot, flags); phys += next - addr; } while (addr = next, addr != end); @@ -112,7 +117,10 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, unsigned long next; pmd_t *pmdp; - pmdp = pmd_set_fixmap_offset(pudp, addr); + if (likely(!(flags & NO_FIXMAP))) + pmdp = pmd_set_fixmap_offset(pudp, addr); + else + pmdp = pmd_offset(pudp, addr); do { pmd_t old_pmd = READ_ONCE(*pmdp); @@ -139,7 +147,8 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys += next - addr; } while (pmdp++, addr = next, addr != end); - pmd_clear_fixmap(); + if (likely(!(flags & NO_FIXMAP))) + pmd_clear_fixmap(); } static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, @@ -215,7 +224,10 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, } BUG_ON(p4d_bad(p4d)); - pudp = pud_set_fixmap_offset(p4dp, addr); + if (likely(!(flags & NO_FIXMAP))) + pudp = pud_set_fixmap_offset(p4dp, addr); + else + pudp = pud_offset(p4dp, addr); do { pud_t old_pud = READ_ONCE(*pudp); @@ -244,7 +256,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys += next - addr; } while (pudp++, addr = next, addr != end); - pud_clear_fixmap(); + if (likely(!(flags & NO_FIXMAP))) + pud_clear_fixmap(); } static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, From patchwork Sun Apr 25 14:12:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96193C433ED for ; Sun, 25 Apr 2021 14:16:04 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1685960232 for ; Sun, 25 Apr 2021 14:16:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1685960232 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QRT2k5QSypmam4y6c/viDSX9KACvA0wP4Zn8Hy5wGsM=; b=GVq3Elc/BzxKC0N4m623o/msA a4vQzcIsZaIHmgwA1oiyRtIM6FOTFBVxoC6zwjxXFbnhfkyqhF57JzM89a+qabV/tV5XnDUaS/+Kx rdDmqEvUQHLzZvzyFbuNn82amAFhETS/rlODxXJx/D+1zflJtr8YTMLQ3u2zad4qGaVvTXBY2NSGf K/Pe6ByQmGCA8tTMpcNJ8O7LssWehabN0wT1AfVsmTnsd2IdQVYb1Q1qX6ro5LqyCWodF3XVu5JEX bKoXjvG5+Pl81tzcus0BeZUYjd0lt8B3gl+6hUkno+7PniDMbEPgoE8f1xuj1wKDlTRO3lOhvDz/u eMZEEfOmA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafWq-005iNy-TA; Sun, 25 Apr 2021 14:14:33 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWJ-005iHV-4A for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:13:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=omzWB5edwu3XFhxPye5tZAjJqjhDewo7D+4eMltP9rM=; b=Oe6+7gCtgniDkd+WbjMolLIAVB WcbjU/J3got+eDIQUD9fo2lS/ha5To3nZDhYz0GJe96gIhM+aqb1LFlkEmP7XXoSZG2EBZ+1p8RbS fSRPvZFTVrsqLWeNEGe41ddRNgLdQh2ta3dKZpjzfx36q9XMI4JvQiGw0oiDDly0BY6gF+IUq1+jI pQq+0VifS15qkr6YCzeXL5KOa/9IDDEdeaVS42p+k3ycF/oVVO/3j1F2EzrB3pvlKS0iUEM1tVzqH J6SnRi4vrhL+2MH3y3xxYkCR0BKTo/LRKS+/vBdIszL5JCLwOLRU+bDsCYqpG0jQ0h7QC6xUrjLkz kt2VhzNA==; Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWG-00FT5e-Gh for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:13:57 +0000 Received: by mail-pj1-x102b.google.com with SMTP id g1-20020a17090adac1b0290150d07f9402so3662454pjx.5 for ; Sun, 25 Apr 2021 07:13:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=omzWB5edwu3XFhxPye5tZAjJqjhDewo7D+4eMltP9rM=; b=CJklDUJwaUQ7grE8IHQQgCnJ7NgyN7cE5etDu4FIovegx3xSDBgylvo9902NVWewkz EwFfpRRj3gjO2/rL8uVYo4y2E/TkNhlg2ayhNj1C6DcJkxBCRlfjw7Lzmph64adGdSuq cZMELOkYC+T2dhNl5UtOGXiaR6c2c0xySNvwk5fK+Lbk0WbDnPJOqWBgNsBjcHAdxj0g 9BghqdtfcRalZGix1O0gcUmrErlvMmEVa3jLIzlyLLx4cm04oBO0qIIeeX0rBw1ueyRm hZXenBMism1F+JjSQutxpZS3bS80l7SQiivsd8jx1vmVDtcDyIAt7XsmMQXHUkGWJaLc KGog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=omzWB5edwu3XFhxPye5tZAjJqjhDewo7D+4eMltP9rM=; b=F3m+jjNEnZ8JQqKTvLex7cJzl6bjNNT17Csa/LXUIWPhVIC0b8P7sIsRmGSKzTY2lm Gr1RFiK/xqeCwJ76GYWCKfPheyRdOr3m84LapTDErUgmAIA6tBRKSNXkTOBXq1p2OFbQ Ihn/zTI+B0yidQFTBXZzcmXIxIaVKTTWvGZ3wi/2vnAyCsCbIv274XbuczI9nLt5C2YH uN3vI/7rLIDEEmLcPcsmJEgnYv1EdwMqmqOLKorODfC+t+qAS4trGJa5eUBeCz3+vPc/ ISx2/hhX2HB0nwG+k8LUPhj+wCwK+GHoW1E1hTLmDoEKBIqyz71gprHQFLuNbnU5xq/3 G7lw== X-Gm-Message-State: AOAM532WPAVkuzQlcpUOIEe2deY/4qtsdkG4SEPdh2pvGVqv/0tcXCdV h0MVKrirDtoIQ7e1jafb5NSpQi++Fw== X-Google-Smtp-Source: ABdhPJxf5WBDGc28WmG8ltqUyM57M/mK/sPYEJBwwIjNnkWSRdAlpgePytU83lUYCJMFcNFxsMNm8Q== X-Received: by 2002:a17:90a:bb0c:: with SMTP id u12mr15772703pjr.234.1619360035618; Sun, 25 Apr 2021 07:13:55 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:13:55 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 05/10] arm64/mm: port trans_pgd_idmap_page() onto create_idmap() Date: Sun, 25 Apr 2021 22:12:59 +0800 Message-Id: <20210425141304.32721-6-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071356_576449_6EA4D185 X-CRM114-Status: GOOD ( 14.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org At present, trans_pgd_idmap_page() has its own logic to set up idmap. To share the common code, porting it onto create_idmap(). Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 13 +++++++ arch/arm64/mm/mmu_include.c | 4 --- arch/arm64/mm/trans_pgd.c | 62 ++++++++++++++++---------------- 3 files changed, 43 insertions(+), 36 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 42f602528b90..8e6638b4d1dd 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -93,4 +93,17 @@ extern void __create_pgd_mapping_extend(pgd_t *pgdir, void *info, int flags); +extern int idmap_extend_pgtable; + +extern void create_idmap(pgd_t *pgdir, phys_addr_t phys, + phys_addr_t size, + pgprot_t prot, + pgtable_alloc allocator, + void *info, + int flags); + +#define NO_BLOCK_MAPPINGS BIT(0) +#define NO_CONT_MAPPINGS BIT(1) +#define NO_FIXMAP BIT(2) + #endif diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index ac8850fe6ce2..98c560197ea9 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -5,10 +5,6 @@ * and MMU-enable */ -#define NO_BLOCK_MAPPINGS BIT(0) -#define NO_CONT_MAPPINGS BIT(1) -#define NO_FIXMAP BIT(2) - static bool pgattr_change_is_safe(u64 old, u64 new) { /* diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 527f0a39c3da..9f4512ab8659 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -274,6 +274,14 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, return 0; } +static phys_addr_t allocator_trans_alloc(unsigned long unused, void *info) +{ + unsigned long *p; + + p = trans_alloc(info); + return virt_to_phys(p); +} + /* * The page we want to idmap may be outside the range covered by VA_BITS that * can be built using the kernel's p?d_populate() helpers. As a one off, for a @@ -287,38 +295,28 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, unsigned long *t0sz, void *page) { - phys_addr_t dst_addr = virt_to_phys(page); - unsigned long pfn = __phys_to_pfn(dst_addr); - int max_msb = (dst_addr & GENMASK(52, 48)) ? 51 : 47; - int bits_mapped = PAGE_SHIFT - 4; - unsigned long level_mask, prev_level_entry, *levels[4]; - int this_level, index, level_lsb, level_msb; - - dst_addr &= PAGE_MASK; - prev_level_entry = pte_val(pfn_pte(pfn, PAGE_KERNEL_EXEC)); - - for (this_level = 3; this_level >= 0; this_level--) { - levels[this_level] = trans_alloc(info); - if (!levels[this_level]) - return -ENOMEM; - - level_lsb = ARM64_HW_PGTABLE_LEVEL_SHIFT(this_level); - level_msb = min(level_lsb + bits_mapped, max_msb); - level_mask = GENMASK_ULL(level_msb, level_lsb); - - index = (dst_addr & level_mask) >> level_lsb; - *(levels[this_level] + index) = prev_level_entry; - - pfn = virt_to_pfn(levels[this_level]); - prev_level_entry = pte_val(pfn_pte(pfn, - __pgprot(PMD_TYPE_TABLE))); - - if (level_msb == max_msb) - break; - } - - *trans_ttbr0 = phys_to_ttbr(__pfn_to_phys(pfn)); - *t0sz = TCR_T0SZ(max_msb + 1); + pgd_t *pgdir = trans_alloc(info); + unsigned long base, step, level, va_bits; + int flags = NO_FIXMAP; + +#ifdef CONFIG_ARM64_64K_PAGES + base = 16; + step = 13; +#elif defined(CONFIG_ARM64_4K_PAGES) + base = 12; + step = 9; +#elif defined(CONFIG_ARM64_16K_PAGES) + base = 14; + step = 11; +#endif + create_idmap(pgdir, virt_to_phys(page), PAGE_SIZE, PAGE_KERNEL_EXEC, + allocator_trans_alloc, info, flags); + + *trans_ttbr0 = phys_to_ttbr(__virt_to_phys(pgdir)); + level = CONFIG_PGTABLE_LEVELS + idmap_extend_pgtable ? 1 : 0; + va_bits = base + step * level; + va_bits = min(va_bits, vabits_actual); + *t0sz = 64 - va_bits; return 0; } From patchwork Sun Apr 25 14:13:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3E48C433B4 for ; Sun, 25 Apr 2021 14:16:14 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2F55961364 for ; Sun, 25 Apr 2021 14:16:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2F55961364 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=k5lAntggxDOIIP0GeOMhWdLIQDPIRGhgHM5ElL123XQ=; b=CqS0Lh0UxT3E+AtuyGQRyT/Tz YzVJnamisPfx2QeTXHl0wQwqVGSNPcmhQkmLAN0042qQLBPtUexJQBiQkkSMjSDKs5fuU+7wkbPFV 0SQlvSaaHImsZZnfXQ8S8i07KlyQS1gLJmKkRMoY5iXAGqWQ4v8C/NfWNhd0SXLlSDLZemfZjV3Dt 6+QvBjra45mtLEFoymj+MlMqRfwTpR0WxKqvCsYvuEYdnBNVMBi0wdDj8u6SX6aEKENUTaVo/iicU 4DoLdLimf9IDJ57Z9yCJDn4RKPP1z/0LUJsxOxep8a8D+mvgU/NoYztuCfFVu+eZloeFvtZ2TSC5X /kbywDl2A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafX5-005iRe-6s; Sun, 25 Apr 2021 14:14:47 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWO-005iHx-96 for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:14:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=sfT3LasW4fyUJsDzudYiBtZnqE1vuRDuVen9QAUkGmM=; b=gm0q9VvCsTfDGCd86+cCn2kSLR U+4BwL7njVJlx4nM7lhjGU6kHxSW4Gitd3d+YFmaRnI8lgTuGY4ig2ikujg65Y0157z9EfwphMC7Q sKKcVIkfR27/YFBASuwKm8en7tfRlj3qWc4Tye2HLhS41gAc3+moX8Wx31YfOysOokcDYk6T3cGlz Eya8Y53mjIF794ztpORdgIZU8LaL4rK/bumy7CvCJ92Fat5+rzWJTPgwxFJVR/On0njmbIADyrrxA I65USYsbw5Ex3+DbpTOwjofF5uP+KPLHQX53dVkdb4vL+oumgIlT3tYSFYb2BICicfqKApBS4e8jq f+Fz5oTQ==; Received: from mail-pj1-x102f.google.com ([2607:f8b0:4864:20::102f]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWL-00FT5y-Qr for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:14:03 +0000 Received: by mail-pj1-x102f.google.com with SMTP id f2-20020a17090a4a82b02900c67bf8dc69so3692074pjh.1 for ; Sun, 25 Apr 2021 07:14:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sfT3LasW4fyUJsDzudYiBtZnqE1vuRDuVen9QAUkGmM=; b=D/PCsLDdtSO2/FiUVgUXklRTWGMRztDo8W7TMjQrb9yUH74aKvmSD2Cskp6xfiFZYt vFUGPNssOZWTd37uXBK3jlieE8o6dqx9Ev3qEIqHY97E9vApas7VFUlyoBa8OeUSi9RX /2UhDrkCwNSbpx6+drctTOMjUt4OTYRH/OsqM+TPm/F/5GL5HIleKyhl03oZdVG1ZC1t Ff1p3XuwLJ/7PeEZPI9SRZL8BI/yGdtT0nBJmsmzgPc6qJVvwu88vRMLrhDEzbXcqS+3 ZXNmxYOMeEUeDCyi/hj/pJpJ0/b85aVzu/Fgwp7qTkJPsuYXlr9MzZVWAqxoX0VeuVFu SvZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sfT3LasW4fyUJsDzudYiBtZnqE1vuRDuVen9QAUkGmM=; b=JiZaAFvucVZsv82sc8YMp12lt9DC0OunKgk8yS04b0uToBbrmLymPDec9tBuIntEsx XtLq5JbcZ9+GqSiylvJM3rAemcuLpFNeYoM7xJ892P5m/pVpGTe3nX/vgSQ5IPixchQW DKmMn1Os58IIiMzeZCSKCmy7oOApHKjlCHo2xtuo/JneKOQ9PPBmH5TDgleaHEw3LQxH fzCLtDWp9LyC0gd1rAfiIznFxdlZ4bagFs1KCW4p9vL/tU7fdcZ8PCuyezltkjkPs4Nt HMWIY8qZJaBdfSreh5mdM9qolKxJ4XIIGlHZsiVzVhD4is/J42LvAKLEzIkjoJQXYGg+ l3TQ== X-Gm-Message-State: AOAM5328ua/1GQWWNrBIblcOTH8IneQw0ZzDKMRcUIAyRSBqmHHg41pP L9BdPaCZbbRH1/uYY7g8q3um9SS+iA== X-Google-Smtp-Source: ABdhPJzsYM+D3ue5me27kamNDaIB2/1Y3qvltf7ayBkGPuJ3EHvDMPvullKl+En8mbatCMXN52TXyg== X-Received: by 2002:a17:90a:fb97:: with SMTP id cp23mr7605004pjb.169.1619360040906; Sun, 25 Apr 2021 07:14:00 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:14:00 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 06/10] arm64/mm: introduce pgtable allocator for idmap_pg_dir and init_pg_dir Date: Sun, 25 Apr 2021 22:13:00 +0800 Message-Id: <20210425141304.32721-7-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071401_890687_C330BA09 X-CRM114-Status: GOOD ( 10.77 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From the view of __create_pgd_mapping(), both idmap_pg_dir and init_pg_dir can be treated as a memory pool. Introduce an allocator working on them, so __create_pgd_mapping() can create mapping. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/mm/mmu.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 520738c43874..fa1d1d4fee8f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -86,6 +86,28 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); +struct mempool { + phys_addr_t start; + unsigned long size; + unsigned long next_idx; +}; + +struct mempool cur_pool; + +void set_cur_mempool(phys_addr_t start, unsigned long size) +{ + cur_pool.start = start; + cur_pool.size = size; + cur_pool.next_idx = 0; +} + +phys_addr_t __init head_pgtable_alloc(unsigned long unused_a, void *unused_b) +{ + unsigned long idx = cur_pool.next_idx++; + + return cur_pool.start + (idx << PAGE_SHIFT); +} + static phys_addr_t __init early_pgtable_alloc(unsigned long unused_a, void *unused_b) { phys_addr_t phys; From patchwork Sun Apr 25 14:13:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73A71C43462 for ; Sun, 25 Apr 2021 14:16:45 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F083B61370 for ; Sun, 25 Apr 2021 14:16:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F083B61370 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BYIBxQvSwKAOI3UsDbct72k+UoNiyvuT5RktXrK+iCk=; b=BBCcpi6jn9B4PjESYUgiwZ3je 3j0zYTdCsawLUHulaZ2ia4yyjEWV3tXRKk66gWrvUC5GaXT6yWwIx5mvBAjIgPTcNi8BzgvRyX//m iL/3vP4T26JjX51t/c+sFhBLRdO7brdmhgqG8PNyQrNk7LnAkIfyH+ogeQ7cxCr2suO0dpY17kOEp wc3JX1didasq1UBUhkqLQ90i+plDHZccsqd/QwAYMrQB4R/MdqPCvsE1zjPOlHwktFTsIdkp0EoDV qJAURHhYHZmJ9Q2UdapQeVW3ojnqO0RbrFh7nmnfbXTji+TN5MNs/eJjojbfp5eXUlcxZ/PDQ3rwX 708UjcdqA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafXJ-005iWZ-Nk; Sun, 25 Apr 2021 14:15:01 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWU-005iIg-5l for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:14:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=vMtiBmNRwxDsN9zQ6Ko5/YP7kk0/VhFH+hMl0hhVbXs=; b=FEF4clG6y/wiogciBie5HAivoz xWVRosQV4EIJluMdjzHruS64Dq5URX8pCh0sGb7DodUbPcbD7gr0ueRh50MhDYKuxCRbnFsQr+3Dp vmRL3svATD4SiY/c4StZVvs5kEJYOnMu4dsUSQZBLDTbXZXrxPhSUmvW5/gEpbUoGNIOjyaOUrUeW vwefo+RxwGXomZ63KZ7TJ00qbGgz1X3fBfIuDF8knxWhm2iRV944Jo+7A+eyldcD3heSfr5i4eM1e u+u2LhF51dGPXc0blJKYC624RvE589iUmq31eNiqXwOOhqck8Q2dC+9PKbPyMnv0IimWIAqe/Ense y2d2ROWg==; Received: from mail-pj1-x1029.google.com ([2607:f8b0:4864:20::1029]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWR-00FT6F-HE for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:14:08 +0000 Received: by mail-pj1-x1029.google.com with SMTP id e8-20020a17090a7288b029014e51f5a6baso3681544pjg.2 for ; Sun, 25 Apr 2021 07:14:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vMtiBmNRwxDsN9zQ6Ko5/YP7kk0/VhFH+hMl0hhVbXs=; b=R6aSTesBGB/c+sh6ed4WJev1QNjxLBFSoitglbNZaGTNpqJJO/IEqgAAZlTpwXWrV8 47DESjNl5BVSRzAVRheh0ZcUcgY7HgyRphck85BptDpwF2LtffWv1IBleBGBuzi9k7ZX l/aVU7drnuwF5M0Kc36HrNPgBvRyzNAwTZc6kKMK40fOScfWPckHwDdJ0ywnejbr6g/M jRJmFtg0nOHjdKkcL19GWRg7icKtJ35qfceZMroYrmYvUVV4aGGC0eV7AWFEkfq0cZht dLCMMX5a9dpNc82HysGYDfJ69zFZC+RFxNSTs2dF+jxQUmr1vHk2V0YzFjuKgKTM67mu CFOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vMtiBmNRwxDsN9zQ6Ko5/YP7kk0/VhFH+hMl0hhVbXs=; b=KOAl5n+NxCaFziBRqe1ylcAmPn9HjaYbEvZM2l6CT7j0zsIF7q4LyzqNp0x9D7R6vF Wl6h5n/ybis+cksA4oUpjXzRIiYeHpiUIuBTdDvr+HFMjj8bIuFJz7QBkxROzK/llViz tk4WMXGy+syZVKJjhWgTID//hG36e6tQEQsdc/LP6GezAedrpsEbhkd6Q0giKM3HjPbc spfoNL5wDaj+UbPmVJWsb1I46P8ZqBgrGtGYAObbdaSLxLVxRPj4y3cTnJJTTIXWIDPF XNHEED96ci64uKDF7NDjTz+AjGF8nkK1vngEe5kTxr9puDtpusvhEUPnoxTTkUgjSQui oXdA== X-Gm-Message-State: AOAM5326SnMCTeu+6WYqksfgytzKdEkGzhwS5E03zZ1nOUwskeGHE8/w l3lPW20z0wXGE/l691vYPk7T3bfgUQ== X-Google-Smtp-Source: ABdhPJwts5aPseEn7Q7O/fv7A0LnRWzZjfzP1agw4neo37y6HApFtb7tKr+qDhfLrMneFpRcyscZFA== X-Received: by 2002:a17:902:d913:b029:ec:c4bb:57cf with SMTP id c19-20020a170902d913b02900ecc4bb57cfmr13808491plz.12.1619360046622; Sun, 25 Apr 2021 07:14:06 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.14.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:14:06 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 07/10] arm64/pgtable-prot.h: reorganize to cope with asm Date: Sun, 25 Apr 2021 22:13:01 +0800 Message-Id: <20210425141304.32721-8-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071407_591346_9272003E X-CRM114-Status: GOOD ( 13.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch is one of the preparation for calling __create_pgd_mapping() from head.S. In order to refer PAGE_KERNEL_EXEC in head.S, reorganize this file and move the needed part under #ifdef __ASSEMBLY__ Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgtable-prot.h | 34 +++++++++++++++++---------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 9a65fb528110..424fc5e6fd69 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -33,9 +33,6 @@ extern bool arm64_use_ng_mappings; -#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) - #define PTE_MAYBE_NG (arm64_use_ng_mappings ? PTE_NG : 0) #define PMD_MAYBE_NG (arm64_use_ng_mappings ? PMD_SECT_NG : 0) @@ -49,6 +46,26 @@ extern bool arm64_use_ng_mappings; #define PTE_MAYBE_GP 0 #endif +#define PAGE_S2_MEMATTR(attr) \ + ({ \ + u64 __val; \ + if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) \ + __val = PTE_S2_MEMATTR(MT_S2_FWB_ ## attr); \ + else \ + __val = PTE_S2_MEMATTR(MT_S2_ ## attr); \ + __val; \ + }) + +#endif /* __ASSEMBLY__ */ + +#ifdef __ASSEMBLY__ +#define PTE_MAYBE_NG 0 +#define __pgprot(x) (x) +#endif + +#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) +#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) + #define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) #define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG) @@ -71,15 +88,7 @@ extern bool arm64_use_ng_mappings; #define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN) #define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT) -#define PAGE_S2_MEMATTR(attr) \ - ({ \ - u64 __val; \ - if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) \ - __val = PTE_S2_MEMATTR(MT_S2_FWB_ ## attr); \ - else \ - __val = PTE_S2_MEMATTR(MT_S2_ ## attr); \ - __val; \ - }) + #define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) /* shared+writable pages are clean by default, hence PTE_RDONLY|PTE_WRITE */ @@ -106,6 +115,5 @@ extern bool arm64_use_ng_mappings; #define __S110 PAGE_SHARED_EXEC #define __S111 PAGE_SHARED_EXEC -#endif /* __ASSEMBLY__ */ #endif /* __ASM_PGTABLE_PROT_H */ From patchwork Sun Apr 25 14:13:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84747C433ED for ; Sun, 25 Apr 2021 14:16:56 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 18F46600EF for ; Sun, 25 Apr 2021 14:16:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 18F46600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fhcXQX9ApKBmSGaWYPQVSs5GGdRH5Y0txJxFeqpxm0s=; b=OaXbxOLpOU/u5rUGtt1RXqXNE 2GPOpqdyzjFjGWfCmnXuxYG0chV8EzuuG3tmGr5NqPICpAUBujBkna7cKDsopFU0OIqQwpyeWTy/c vBic0PVMek5K/dS89kPvQMHSggl0+LltZWVeUuQskY+G1jXzFqmmSuKuGZMvgUfwlE2etzH6boXD5 +5pRVorofRZSDmpwX0Or3ZLJ8C9CSP5kVnSSa1EkOm88G9QONzeVMH21RaVEHF+UQ7rRDJAjwDxeN HM4XK/pl3zchK38EWSA9fJkPbzKeO90Wi/YHAlZARQFJEeheH0fYCA0gJM0niHr0iQY2Cs7O0Eywj yDRyVztag==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafXc-005ieU-MW; Sun, 25 Apr 2021 14:15:20 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWZ-005iKA-V0 for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:14:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=vt5vkYYlE+SDTKBQiglJcgWctT1ZgrBGHis8MwF7xGs=; b=RyPDmGArpnjklTi6Gsf+U4OwcT bfUIDmY8pvxL/u/aj/z2GD2Y5UUUpPs5YoKQWmixra2F0rifp3szdclVI0xiFhJIgAKnTxJa+JS70 JqaoAB+URZtn3Tbs+MKWz6XAFjbPyjxAI9xHcOyukKWwI+7xvDytp6Z3/Ykv5q7CeLN4n+H0+7Fad syywP6VsugLXkSjOPMDtfhgaCtswbtW9fZCxE+mBHZJGHH1HWwKQtIYoiJinN9u1E/xFa6ejjcVIS aRb7PuIwqHSw65NZb9DSFlOLsZP8EOjC+O5ffd/YoQrTE9zGEFRnzneJXk0kd7Y2WW0D5BvpSnPD0 Q00dGbwg==; Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWX-00FT6V-9r for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:14:14 +0000 Received: by mail-pg1-x529.google.com with SMTP id p12so38387019pgj.10 for ; Sun, 25 Apr 2021 07:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vt5vkYYlE+SDTKBQiglJcgWctT1ZgrBGHis8MwF7xGs=; b=ZGW4dxTLR1ywkEZMm3pGUTPepv3RRxKUm+iGtdx+JtB1Glm58Ig1bLt+OKi2PtAz+C SPzje4pptubUnH+AJc3tMvypN/B0Z9gegfUy1vMJddR7KN/AA4wiHfOl45O91c6eXK4q 8ViY5qkwjnFRczFxokV88W2lhcor/LXKRy46KqX8WBPI3/vVzLSrsM8IlD//79zEptca 7+ZD6R/C/bxL3JvkLkgN+e7k1cluupo16SS9fyp5boVGm91g9bnRblsOetRb24nOvZTg KdRnsUWmFssVEVcg89L6en6STdGBSQ0/IOxRlB3fkoTL0U8SWWhnDleyV/U/7e3H1zDy uUUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vt5vkYYlE+SDTKBQiglJcgWctT1ZgrBGHis8MwF7xGs=; b=CS65F6MxL20sybBxsKHjyu8wod0GXh+tez9sUOfNeeBEOrDtFdaCo6jBA2I82+GnhV ZUJLxmT5FnulP4R6Mzbez25bJpUVPlUFWWR3KJebsR6VEwRhCU6wZWnRYcDJpL7XIrD4 eNMNfIObnTj60CJo0VYe3r/YZAE4Mrh2RKsNkNzRNSV0+RRd59RP3Brn5NG7rhqT6A+L peGWqHrBZziLANdbFXx01ZhlgnefkNwTgcVHHtX79ZOB9fhQVS5PG/0kAQM11+Nm5+IB EZRkHwvw2Qvon6fk6Ww6EpGtOUDTP2sGQRUkDQxw0XItFNE3TLhtNUbAnDj4oGZYeOHD ldgg== X-Gm-Message-State: AOAM5322MdG6gQ4tOnnvkq4liENRXgKRqcxdy/3pPD+oIQGtpnFRwXES t6glGNrQ62Iv9bkWDYW+mofTIiyruA== X-Google-Smtp-Source: ABdhPJyRPFhgVJrpUb1hCADgLT8SRG6NynC0DHa2xotzRW73zU1/q5ugApElxtmKttYEOCX69dkADw== X-Received: by 2002:a63:b515:: with SMTP id y21mr12788026pge.253.1619360052320; Sun, 25 Apr 2021 07:14:12 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.14.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:14:11 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 08/10] arm64/mmu_include.c: disable WARN_ON() and BUG_ON() when booting. Date: Sun, 25 Apr 2021 22:13:02 +0800 Message-Id: <20210425141304.32721-9-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071413_365018_C94EA7BE X-CRM114-Status: GOOD ( 16.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch is still one of the preparation for calling __create_pgd_mapping() from head.S When calling from head.S, printk is not ready to work. Hence define SAFE_BUG_ON()/SAFE_WARN_ON(), wrapping around BUG_ON()/WARN_ON() to protect against early calling. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 1 + arch/arm64/mm/mmu_include.c | 36 +++++++++++++++++++------------- 2 files changed, 23 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 8e6638b4d1dd..c3875af99432 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -105,5 +105,6 @@ extern void create_idmap(pgd_t *pgdir, phys_addr_t phys, #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_FIXMAP BIT(2) +#define BOOT_HEAD BIT(3) #endif diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index 98c560197ea9..746cb2b502a3 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -5,6 +5,14 @@ * and MMU-enable */ +/* + * printk is not ready in the very early stage. And this pair macro should be used + * instead + */ +#define SAFE_BUG_ON(x, y) if (likely(!(x & BOOT_HEAD))) { BUG_ON(y); } +#define SAFE_WARN_ON(x, y) \ + ({ int _ret; _ret = (x & BOOT_HEAD) ? false : WARN_ON(y); }) + static bool pgattr_change_is_safe(u64 old, u64 new) { /* @@ -57,7 +65,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, * After the PTE entry has been populated once, we * only allow updates to the permission attributes. */ - BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), + SAFE_BUG_ON(flags, !pgattr_change_is_safe(pte_val(old_pte), READ_ONCE(pte_val(*ptep)))); phys += PAGE_SIZE; @@ -77,16 +85,16 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long next; pmd_t pmd = READ_ONCE(*pmdp); - BUG_ON(pmd_sect(pmd)); + SAFE_BUG_ON(flags, pmd_sect(pmd)); if (pmd_none(pmd)) { phys_addr_t pte_phys; - BUG_ON(!allocator); + SAFE_BUG_ON(flags, !allocator); pte_phys = allocator(PAGE_SHIFT, info); __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); pmd = READ_ONCE(*pmdp); } - BUG_ON(pmd_bad(pmd)); + SAFE_BUG_ON(flags, pmd_bad(pmd)); do { pgprot_t __prot = prot; @@ -131,13 +139,13 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, * After the PMD entry has been populated once, we * only allow updates to the permission attributes. */ - BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), + SAFE_BUG_ON(flags, !pgattr_change_is_safe(pmd_val(old_pmd), READ_ONCE(pmd_val(*pmdp)))); } else { alloc_init_cont_pte(pmdp, addr, next, phys, prot, allocator, info, flags); - BUG_ON(pmd_val(old_pmd) != 0 && + SAFE_BUG_ON(flags, pmd_val(old_pmd) != 0 && pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); } phys += next - addr; @@ -160,16 +168,16 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, /* * Check for initial section mappings in the pgd/pud. */ - BUG_ON(pud_sect(pud)); + SAFE_BUG_ON(flags, pud_sect(pud)); if (pud_none(pud)) { phys_addr_t pmd_phys; - BUG_ON(!allocator); + SAFE_BUG_ON(flags, !allocator); pmd_phys = allocator(PMD_SHIFT, info); __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); pud = READ_ONCE(*pudp); } - BUG_ON(pud_bad(pud)); + SAFE_BUG_ON(flags, pud_bad(pud)); do { pgprot_t __prot = prot; @@ -213,12 +221,12 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, if (p4d_none(p4d)) { phys_addr_t pud_phys; - BUG_ON(!allocator); + SAFE_BUG_ON(flags, !allocator); pud_phys = allocator(PUD_SHIFT, info); __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); p4d = READ_ONCE(*p4dp); } - BUG_ON(p4d_bad(p4d)); + SAFE_BUG_ON(flags, p4d_bad(p4d)); if (likely(!(flags & NO_FIXMAP))) pudp = pud_set_fixmap_offset(p4dp, addr); @@ -240,13 +248,13 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, * After the PUD entry has been populated once, we * only allow updates to the permission attributes. */ - BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), + SAFE_BUG_ON(flags, !pgattr_change_is_safe(pud_val(old_pud), READ_ONCE(pud_val(*pudp)))); } else { alloc_init_cont_pmd(pudp, addr, next, phys, prot, allocator, info, flags); - BUG_ON(pud_val(old_pud) != 0 && + SAFE_BUG_ON(flags, pud_val(old_pud) != 0 && pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); } phys += next - addr; @@ -275,7 +283,7 @@ static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_ad * If the virtual and physical address don't have the same offset * within a page, we cannot map the region as the caller expects. */ - if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) + if (SAFE_WARN_ON(flags, (phys ^ virt) & ~PAGE_MASK)) return; phys &= PAGE_MASK; From patchwork Sun Apr 25 14:13:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5585CC433B4 for ; Sun, 25 Apr 2021 14:17:32 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D54C1600EF for ; Sun, 25 Apr 2021 14:17:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D54C1600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BfpHguH35gKY6grGGGOdPF5boh53d0u3X8ov4BYFOZ4=; b=QIppWyHTdyUm1odGmxBFEFL5M L40Fdc34rm7tCUpqSDFYlHSEfTa89JOvuwiOFVJOo/oyByCzk0G/GF7mWPRUrBuIciOWJYQeDS8RY Ch+p6rVqFKqH+l40pKUtdt7huAxDyREW7+IknPAW8Wk8BMdERT6rKK4fafdwYhbRHy0PTtreF94ce MCTUf/5hGspyvM4YMIkNi/EvyXtMis0/PWx2ADnmgwKKauLP9r88SsJrxwoSrvfoS7Bx2iuwFvDtv Cg64aHVFmjKReqiWA0KyL6NhHUt15qDfvwRvyWnbt+ljZ5faANGn9fBfa5TWp+4hrzrmfhw/vRvz8 VWS39mP+w==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafXx-005ilO-96; Sun, 25 Apr 2021 14:15:41 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWf-005iLJ-7u for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:14:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=0DMdoS9Veuec0V12Q3JRfqDfDGvqnbvmLeiD7/pgGow=; b=wTUYKSILyjaA/gUhQ7FsOeqR45 FHypD/lqqOvzpjMGe29PLYKl/45qvvjU8U9I6z2ZZbVKRA0pblV3uJ51sf1+OYUtcluM7z+l5cCzV nbwXikmuPiAXtwofy7GH7Jjr+egf096A/KGb2ABWa4E4Fp28sctyZSFQIYDhFbAWLZHCn8/67+PK1 H6Kqaab3qTrQOPQD5gmg6mSug329fz79PV6C8xxXZ+OAIvNnmv4xuiyD7KEMXQMH2Z6FPC+U1ZHTf c2Y7SH3X05491IaPAVoeRGOrOBqOfzNDS5rtCn0mSbDVFl1ogak1z2ODGJHNjhnySJpupjPyMqIcp IcvfORgg==; Received: from mail-pj1-x1029.google.com ([2607:f8b0:4864:20::1029]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWc-00FT78-O7 for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:14:20 +0000 Received: by mail-pj1-x1029.google.com with SMTP id g1-20020a17090adac1b0290150d07f9402so3662687pjx.5 for ; Sun, 25 Apr 2021 07:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0DMdoS9Veuec0V12Q3JRfqDfDGvqnbvmLeiD7/pgGow=; b=tK0DIC5R9Leye4jivEJeBNQln26p6351TkUiOPwsWHq/Qz9AYlnk1rA3dBSDBVHPU+ SmvsQgkdAs50MeaO6jVn11Hr3evXxd+cGLzLg9zfrLlW0kMYOrp90qrS1y0ROOfdHJdP 6OP0XxYJyQbJ1qDBpnw9CstpYhTRDJ3dYdqCh1aa8fjYstjuBD6sH+7ZQpTcAWgK6O9f hQdQcwU/88CEw/GV/FR/8HtQ4HGQQik92JtYTCKMktekmd5eGNp05h0OvYGTkyp+S4eS 3HAZ29ebKFf2mU90YbOeiY0mc349Z6Zr6vjfTQBYmKo0kE/zV0UAO+0lrNGjRKTx64cj zfMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0DMdoS9Veuec0V12Q3JRfqDfDGvqnbvmLeiD7/pgGow=; b=G6RoP8w1wBAN8y3LeSACmVfzDKDXy498lbXlRruudRf1aiwKxrZU4hukiu2hC1uXzR SidhME6F+G7I1IQpL3rIVqdTA8ZhY1PqI3sMVVm6ED9NDCIYze9dBK4wLYXjgL4ddtjM hO7h6Vw7KOBkbcU3rG56WMh3XUpKHLeXUo5nRbrPbM2ho2ckVjLh27IZDfJGCUDDWcvs vHB1DsSF6Axb14WrhWCsKdwzCJvn3rDvGAqdc5rhQl9DSZFbOP1kZr2na7LgTr84DUWj h6j74o7SWfdWnE411NA+3ibirWzXn30gs2SsjXl9ayM9/lf4qaL1NkCXoS68FnJ1Hthf Idmg== X-Gm-Message-State: AOAM532mpV71bDzgJorIgT+oJsennhBRK+FgAj3eViuq203gDKtgQWpp 5erv078JNAAFkctsbg2H2AFN2TPoJA== X-Google-Smtp-Source: ABdhPJw1vKXRaq3WoSn9xF1VwgnaG0Tx42JBNMT/wuDQOxBFrSY+31y58PiWMHZ5DjfHVwkFJ8Wwnw== X-Received: by 2002:a17:902:6b8c:b029:ea:f54f:c330 with SMTP id p12-20020a1709026b8cb02900eaf54fc330mr13778469plk.10.1619360058195; Sun, 25 Apr 2021 07:14:18 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.14.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:14:17 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 09/10] arm64/mm: make __create_pgd_mapping() coped with pgtable's paddr Date: Sun, 25 Apr 2021 22:13:03 +0800 Message-Id: <20210425141304.32721-10-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071418_805576_D422C46D X-CRM114-Status: GOOD ( 14.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch is the last preparation for calling __create_pgd_mapping() from head.S. Under mmu-offset situation, pud_t */pmd_t */pte_t * points to paddr. During the building of pgtable, they should be carefully handled to avoid the involvement of __va(). Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/mm/mmu_include.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index 746cb2b502a3..c4ea00bae4df 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -54,6 +54,9 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, if (likely(!(flags & NO_FIXMAP))) ptep = pte_set_fixmap_offset(pmdp, addr); + else if (flags & BOOT_HEAD) + /* for head.S, there is no __va() */ + ptep = (pte_t *)__pmd_to_phys(*pmdp) + pte_index(addr); else ptep = pte_offset_kernel(pmdp, addr); do { @@ -121,10 +124,19 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, unsigned long next; pmd_t *pmdp; - if (likely(!(flags & NO_FIXMAP))) + if (likely(!(flags & NO_FIXMAP))) { pmdp = pmd_set_fixmap_offset(pudp, addr); - else + } else if (flags & BOOT_HEAD) { +#if CONFIG_PGTABLE_LEVELS > 2 + /* for head.S, there is no __va() */ + pmdp = (pmd_t *)__pud_to_phys(*pudp) + pmd_index(addr); +#else + pmdp = (pmd_t *)pudp; +#endif + } else { pmdp = pmd_offset(pudp, addr); + } + do { pmd_t old_pmd = READ_ONCE(*pmdp); @@ -228,10 +240,19 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, } SAFE_BUG_ON(flags, p4d_bad(p4d)); - if (likely(!(flags & NO_FIXMAP))) + if (likely(!(flags & NO_FIXMAP))) { pudp = pud_set_fixmap_offset(p4dp, addr); - else + } else if (flags & BOOT_HEAD) { +#if CONFIG_PGTABLE_LEVELS > 3 + /* for head.S, there is no __va() */ + pudp = (pud_t *)__p4d_to_phys(*p4dp) + pud_index(addr); +#else + pudp = (pud_t *)p4dp; +#endif + } else { pudp = pud_offset(p4dp, addr); + } + do { pud_t old_pud = READ_ONCE(*pudp); From patchwork Sun Apr 25 14:13:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD1EC433ED for ; Sun, 25 Apr 2021 14:17:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F38161364 for ; Sun, 25 Apr 2021 14:17:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F38161364 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fGLCDeJgHL44BNKJHiUStPlaZ4BbIhQswXIrHEJY41s=; b=Zxv4g4MsLP+xVD2OKP2sswfDf apTS/zXi9JBg/CYyT8/HbxqOT4VFZppLxk6shbptMKQ7zxk9ly9saGkYFgP7YtLWv0XAorsdE+sPc z2dGiICM8HUg2LqvhPagGhUENcOkVoC+IOXWxYeBnHAw2s3YR2raZQhLHSHPg8OwoL423+kUZM5Sm 0MlUdg0f+zJuxqwiK5iHesQob7r5a4yX4uZZ5wx9hkaIZjmUb2vubum4sif2KlvpLvHJ22ffRP/Db SmJ1EEBGUJ6dKt/w8yqscmkU9ZEMDSYOU45SZ0s/2sq1Gc1+eHdpK9qkmhQItZ/Ggnda1xNv9KS5U TyiKJDe2g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafYI-005iuY-4X; Sun, 25 Apr 2021 14:16:02 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWm-005iMy-6O for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:14:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=uyGxi/1m6tix6sJesbcYLWdFFEKB6t5K3AVKL00MdLE=; b=RPKYHEMOVCrlLR0q2z+AggLeUS 08SjW9egMZzz+Tx8edz854a481c/ckyy0L7DS+G153vTPau1M21ybcboCeZ8jzwQZwhG4aWsLmxnA jppUff7YvP73BsMe6ATFpsRutGGD0PypFioxg4uwoXW48o5UT5IWi2Jv+q+3HhhEnM459/aE7Pbxr QPk3YcHgen1LYlMI1PhNOAOdzU0MDS9wuNqtHpd+cdrnF6ugfGjz/jpv0MEsMcXUGwwQBlgBpUp77 pR+mNI/OxcRVGsqgjeenb2SekE4wfp9yMV2iaxVhdvVqkRo4kTob15J6to/eKnDu+Ey4URuJ9UeCf g+fO6bmQ==; Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWi-00FT7P-TN for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:14:27 +0000 Received: by mail-pj1-x102c.google.com with SMTP id h14-20020a17090aea8eb02901553e1cc649so1562785pjz.0 for ; Sun, 25 Apr 2021 07:14:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uyGxi/1m6tix6sJesbcYLWdFFEKB6t5K3AVKL00MdLE=; b=kGVBbb4apUaVRFfvAIzJAUdaLw9CSdzF8TntMk112imdYRvcNJix7rbn07yVCCBrD8 LZNOLLv0/WvY1aeRVJF44RqgctCyKw1YDJ3P+4YYX97gbncH3roTqz40FRAkxIIcOFL4 LkkyU4W6TUHPEmTGSujPf2VQl6HsuPNEfGOxkyG6+Nw/eHLn1nRu4L8pW3jNzahTGdkP rTdbMYHQYVjOwrFTe0qCyQ77Xj/wg9iL2MVd5v3TEJBqXQo71Jn4I1TsNOxWWKDWbR9X p72c9NqPLSM3jRoBz00L5XzFkE95VjaevdtAw7cS8lsek+oARMgmNKBI2SucIkMQuJ89 Cy4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uyGxi/1m6tix6sJesbcYLWdFFEKB6t5K3AVKL00MdLE=; b=hVdGYO0LqUy5wB6ne6AjGd7WqoKaKFGl5tbjbwk748aYpX8/GtN3Ob2AUktmlSmCzM aS0Al/8ZaIGolch8tLEx73DmhS7E6j4e5e+UdSxJ/nps8XeGHTpzXFeapeUVZlDXWh+N 3cgsnAv6JRM8pfm+QwQhww60ujd9YE//fic2xf11AFP82ZmWlO0NILZcue0QAgOjzXtS 3pIvD7ZWbPMQ3P37CmPLCTuNCPbnHcdRyySAjLTOI+2sdK+5Y325vrC2sQsRkTQyys+n m1S93eD0rcG/9J7qO6xtaBwpbddx/91PLlMaFskp0riSl+RkFb2B++mYeAERUg0knEFI k7NA== X-Gm-Message-State: AOAM531RAYDhphoECd8JJ7b+4GldGbgFw3E8/XnqglQzwTE0g6cJ3A7+ LYF1uIPvbqx+qa43hR4IxDErbb0r1w== X-Google-Smtp-Source: ABdhPJzjtHAry5eUX/C26kEznE4BwfFOovyrvxINXyNMhIlS/FXq1ZvFGapyQ+u+ZIPcH2uaWgQ38w== X-Received: by 2002:a17:90a:352:: with SMTP id 18mr15028980pjf.223.1619360063889; Sun, 25 Apr 2021 07:14:23 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.14.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:14:23 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 10/10] arm64/head: convert idmap_pg_dir and init_pg_dir to __create_pgd_mapping() Date: Sun, 25 Apr 2021 22:13:04 +0800 Message-Id: <20210425141304.32721-11-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071425_142034_967F84E6 X-CRM114-Status: GOOD ( 22.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now, everything is ready for calling __create_pgd_mapping() from head.S. Switching to these C routine and remove the asm counterpart. This patch has been successfully tested with the following config value: PAGE_SIZE VA PA PGTABLE_LEVEL 4K 48 48 4 4K 39 48 3 16K 48 48 4 16K 47 48 3 64K 52 52 3 64K 42 52 2 Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- RFC -> V2: correct the asm calling convention. --- arch/arm64/include/asm/pgalloc.h | 5 + arch/arm64/kernel/head.S | 193 ++++++++----------------------- arch/arm64/mm/mmu.c | 13 +++ 3 files changed, 66 insertions(+), 145 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index c3875af99432..b1182b656b00 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -8,6 +8,9 @@ #ifndef __ASM_PGALLOC_H #define __ASM_PGALLOC_H +#include + +#ifndef __ASSEMBLY__ #include #include #include @@ -102,6 +105,8 @@ extern void create_idmap(pgd_t *pgdir, phys_addr_t phys, void *info, int flags); +#endif + #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_FIXMAP BIT(2) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e19649dbbafb..ddb9601d61c2 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -28,6 +28,8 @@ #include #include #include +#include +#include #include #include #include @@ -92,6 +94,8 @@ SYM_CODE_START(primary_entry) bl init_kernel_el // w0=cpu_boot_mode adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 + adrp x4, init_thread_union + add sp, x4, #THREAD_SIZE bl set_cpu_boot_mode_flag bl __create_page_tables /* @@ -121,135 +125,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b __inval_dcache_area // tail call SYM_CODE_END(preserve_boot_args) -/* - * Macro to create a table entry to the next page. - * - * tbl: page table address - * virt: virtual address - * shift: #imm page table shift - * ptrs: #imm pointers per table page - * - * Preserves: virt - * Corrupts: ptrs, tmp1, tmp2 - * Returns: tbl -> next level table page address - */ - .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2 - add \tmp1, \tbl, #PAGE_SIZE - phys_to_pte \tmp2, \tmp1 - orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type - lsr \tmp1, \virt, #\shift - sub \ptrs, \ptrs, #1 - and \tmp1, \tmp1, \ptrs // table index - str \tmp2, [\tbl, \tmp1, lsl #3] - add \tbl, \tbl, #PAGE_SIZE // next level table page - .endm - -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - * shift: shift used to transform virtual address into index - * ptrs: number of entries in page table - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend, shift, ptrs - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, ptrs, istart, iend, count - lsr \iend, \vend, \shift - mov \istart, \ptrs - sub \istart, \istart, #1 - and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1) - mov \istart, \ptrs - mul \istart, \istart, \count - add \iend, \iend, \istart // iend += (count - 1) * ptrs - // our entries span multiple tables - - lsr \istart, \vstart, \shift - mov \count, \ptrs - sub \count, \count, #1 - and \istart, \istart, \count - - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: start address to map - * vend: end address to map - we map [vstart, vend] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * pgds: the number of pgd entries - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, vend, flags - * Corrupts: tbl, rtbl, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, pgds, istart, iend, tmp, count, sv - add \rtbl, \tbl, #PAGE_SIZE - mov \sv, \rtbl - mov \count, #0 - compute_indices \vstart, \vend, #PGDIR_SHIFT, \pgds, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #PTRS_PER_PUD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #PTRS_PER_PMD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #PTRS_PER_PTE, \istart, \iend, \count - bic \count, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \count, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - /* * Setup the initial page tables. We only setup the barest amount which is * required to get the kernel running. The following sections are required: @@ -345,8 +220,6 @@ SYM_FUNC_START_LOCAL(__create_page_tables) mov x5, #1 str x5, [x4] //require expanded pagetable - mov x4, EXTRA_PTRS - create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 #else /* * If VA_BITS == 48, we don't have to configure an additional @@ -356,25 +229,55 @@ SYM_FUNC_START_LOCAL(__create_page_tables) str_l x4, idmap_ptrs_per_pgd, x5 #endif 1: - ldr_l x4, idmap_ptrs_per_pgd - mov x5, x3 // __pa(__idmap_text_start) - adr_l x6, __idmap_text_end // __pa(__idmap_text_end) - - map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 + stp x0, x1, [sp, #-64]! + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #32] + stp x6, x7, [sp, #16] + + adrp x0, idmap_pg_dir + adrp x1, idmap_pg_end + sub x1, x1, x0 + bl set_cur_mempool + mov x0, #0 + mov x0, #0 + bl head_pgtable_alloc // x0 contains idmap_pg_dir + + adrp x1, __idmap_text_start + adr_l x2, __idmap_text_end + sub x2, x2, x1 + ldr x3, =PAGE_KERNEL_EXEC + adr_l x4, head_pgtable_alloc + mov x5, #0 + mov x6, #(NO_FIXMAP | BOOT_HEAD) + bl create_idmap /* * Map the kernel image (starting with PHYS_OFFSET). */ adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) - add x5, x5, x23 // add KASLR displacement - mov x4, PTRS_PER_PGD - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - - map_memory x0, x1, x5, x6, x7, x3, x4, x10, x11, x12, x13, x14 + adrp x1, init_pg_end + sub x1, x1, x0 + bl set_cur_mempool + mov x0, #0 + mov x0, #0 + bl head_pgtable_alloc // x0 is init_pg_dir + + adrp x1, _text // runtime __pa(_text) + mov_q x2, KIMAGE_VADDR // compile time __va(_text) + add x2, x2, x23 // add KASLR displacement + adrp x3, _end // runtime __pa(_end) + sub x3, x3, x1 // _end - _text + + ldr x4, =PAGE_KERNEL_EXEC + adr_l x5, head_pgtable_alloc + mov x6, #0 + mov x7, #(NO_FIXMAP | BOOT_HEAD) + + bl create_init_pgd_mapping + ldp x6, x7, [sp, #16] + ldp x4, x5, [sp, #32] + ldp x2, x3, [sp, #48] + ldp x0, x1, [sp], #64 /* * Since the page tables have been populated with non-cacheable diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index fa1d1d4fee8f..1ae72a3f2d27 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -167,6 +167,19 @@ static phys_addr_t pgd_pgtable_alloc(unsigned long shift, void *unused) #include "./mmu_include.c" +void create_init_pgd_mapping(pgd_t *pgdir, + phys_addr_t phys, + unsigned long virt, + phys_addr_t size, + pgprot_t prot, + pgtable_alloc allocator, + void* info, + int flags) +{ + __create_pgd_mapping(pgdir, PTRS_PER_PGD, phys, virt, size, + prot, allocator, info, flags); +} + int idmap_extend_pgtable; /*