From patchwork Sun Apr 25 14:12:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 880A9C433B4 for ; Sun, 25 Apr 2021 14:15:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07DB66128B for ; Sun, 25 Apr 2021 14:15:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07DB66128B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=G3iBv8BkNgSMIXYzE9Bovpr3+kjcVh6P8pRrgIxT4T8=; b=gDWeZuAg+i11wfdn7XXtGbM9o 90MXAeOLd3eoLbViwS8VhByJw3xT0/tECeYdsDZFpj8EcyIe9oPmh9C7+brhEk/UgeujAgZCBj8HE CqQUmyTA68jHVYcVmpKJFjDqhqjDZx7Eas8aKgzbJZqeBNl0tRjmsNze+JKjb4i3QlmEIxnLKlMLm v+qD49DyM0Qg7VdO7hxHXvp+rpJkTfp6+muD4CKjoms1SVkJzjT5GL2DmlUHO6f0cu6gp0e+jb30D 6/l4BBkmdAgeVPo6C3hrCrDLTAcN8kuJ5ebld1NSdeE+wmU3LH2YYGg6ZlyoSLeOK+VPBP9/2euY3 yyt5YIwAw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafWE-005iH4-Gi; Sun, 25 Apr 2021 14:13:54 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafW4-005iGH-Io for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:13:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=zxK5dXUW170Ydn4SsP2MT19YDrd+YAyUvlsbJP4btws=; b=DZmIVjwYUFLXF6axX1aOCn8OEo fNNiJWAZgPYOlf6G8ntkCmLb9nJ3N2Ipu10ZKdJc5Be8+AbL2BPiLKaprnqDZtnHyX6y3m4dPdkrX 4/+hEIYiMYtUDWQB+T5F47nXDt5yQ/ByPhmLwi4bB2ZETnNMVIbH7tmN/AbM1NoKk/PYHtRkPB4Jo V07WRlEp20P1ch1SH/ErH8UvkMRVxAIwm/ibCXNbz72gCu3fAimSqAtPYbWQjQlaQinB4Yf6orCoa SMIlQo3QSm8T9Wug6dDi/Y9R2B+1P/VvFBtxJd8N/0SxNbSHrgcy+haQ6A7SuHLU0kp+8a1O+wueI BgFiBbvg==; Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafW1-00FT4G-P0 for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:13:43 +0000 Received: by mail-pl1-x635.google.com with SMTP id o16so13949227plg.5 for ; Sun, 25 Apr 2021 07:13:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zxK5dXUW170Ydn4SsP2MT19YDrd+YAyUvlsbJP4btws=; b=SJos2+7BsnKGyh+6pJlCE6LNA3pxYOmlUPtpuMr1qprUk7oKS7HwJ1wikKILj+Ao6R LrgDg0SRKlI2h9276di6n77Fr9ZDhluVLcPU9x0TicUjMqysNfG2kZhQnskbHChBKWlQ jFhKzTBvWclTkyZx2LyArbnBtz6Kh0ciQLnspbkdvDlvYEdeKvsJ8JUfvhTC4VQfQ+gD JZseYKy4J7lGi7ZSiKeIBO4t0QZu32J+z1BW+79HcRwODPgn3q3ZkHNAoHf0ov+4Yyuy jrsTYidhW19HOmnpPCdU1QnunTITk2zUVMDcuJSmEkvbH/wtrS87N6+7OlobrpO6X9rW eEmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zxK5dXUW170Ydn4SsP2MT19YDrd+YAyUvlsbJP4btws=; b=JXPcuMliXmK2bBNNXHYvmLPHauoiY1Aw5mz5M7tNxNBVjA6CrA09m1PocMX19J9X12 2e+m1wGqOxyGOiuNrA7QCnFP/aKdAwuN+NbcrBtZ/xnJE7n+BgGyPTM4lbkmf2Hic7sZ JKenTu98NNPJ8xeaUVaKOqfxOHU9TpeYHVxAlKKtUWNxjyK7CeYjj52UgNSWbBCk/zyy RiDUsuyyf8K6PHGbs57dnCTK+oV7HnPRWzFJ2XixNI9GfHKoQO9hu28brL9DFUAcxIlt Vot1DbUv5oAETVYaivHbHO4MsNos3AFuN3kARatqMcKvi9QpM03e2Zp01enoziWerh8Z QCCQ== X-Gm-Message-State: AOAM532A57SQIS3YwBKQtugoLt3ompSaQL3q9DA/FtJnktSdBNYzFNpe uXZPc/GtGftXzqiBPpBN7oo974/dNw== X-Google-Smtp-Source: ABdhPJw85jvQV2QAesRB0jodAI6dX0mLONuKCslJoJHG+5KMJ+2D80LNcfdIbpctyB8O6nSlBIiTSA== X-Received: by 2002:a17:902:8697:b029:ea:a5dc:925f with SMTP id g23-20020a1709028697b02900eaa5dc925fmr13857390plo.56.1619360019048; Sun, 25 Apr 2021 07:13:39 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:13:38 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 02/10] arm64/mm: change __create_pgd_mapping() to accept nr_entries param and introduce create_idmap() Date: Sun, 25 Apr 2021 22:12:56 +0800 Message-Id: <20210425141304.32721-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071341_838095_77DDE82E X-CRM114-Status: GOOD ( 18.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As idmap_ptrs_per_pgd may have pgd entries greater than PTRS_PER_PGD, the prototype of __create_pgd_mapping() needs change to cope with that to create idmap. Now this adaption, create_idmap() API can be introduced to create idmap handly for all kinds of CONFIG_PGTABLE_LEVEL Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 7 ++++++ arch/arm64/kernel/head.S | 3 +++ arch/arm64/mm/idmap_mmu.c | 16 ++++++++----- arch/arm64/mm/mmu.c | 41 ++++++++++++++++++++++++++------ arch/arm64/mm/mmu_include.c | 9 +++++-- 5 files changed, 61 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 3c6a7f5988b1..555792921af0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -83,4 +83,11 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) } #define pmd_pgtable(pmd) pmd_page(pmd) +extern void __create_pgd_mapping_extend(pgd_t *pgdir, + unsigned int entries_cnt, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags); + #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 840bda1869e9..e19649dbbafb 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -341,6 +341,9 @@ SYM_FUNC_START_LOCAL(__create_page_tables) #if VA_BITS != EXTRA_SHIFT #error "Mismatch between VA_BITS and page size/number of translation levels" #endif + adr_l x4, idmap_extend_pgtable + mov x5, #1 + str x5, [x4] //require expanded pagetable mov x4, EXTRA_PTRS create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 diff --git a/arch/arm64/mm/idmap_mmu.c b/arch/arm64/mm/idmap_mmu.c index 42a27dd5cc9f..bff1bffee321 100644 --- a/arch/arm64/mm/idmap_mmu.c +++ b/arch/arm64/mm/idmap_mmu.c @@ -21,13 +21,17 @@ #include "./mmu_include.c" -void __create_pgd_mapping_extend(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +void __create_pgd_mapping_extend(pgd_t *pgdir, + unsigned int entries_cnt, + phys_addr_t phys, + unsigned long virt, + phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { - __create_pgd_mapping(pgdir, phys, virt, size, prot, pgtable_alloc, flags); + __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, prot, + pgtable_alloc, flags); } #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 56e4f25e8d6d..70a5a7b032dc 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -145,6 +145,33 @@ static phys_addr_t pgd_pgtable_alloc(int shift) #include "./mmu_include.c" +int idmap_extend_pgtable; + +/* + * lock: no lock protection alongside this function call + * todo: tear down idmap. (no requirement at present) + */ +void create_idmap(pgd_t *pgdir, phys_addr_t phys, + phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + u64 ptrs_per_pgd = idmap_ptrs_per_pgd; + +#ifdef CONFIG_IDMAP_PGTABLE_EXPAND + if (idmap_extend_pgtable) + __create_pgd_mapping_extend(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); + else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#endif +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -158,7 +185,7 @@ static void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, &phys, virt); return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); } @@ -173,7 +200,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, if (page_mappings_only) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(mm->pgd, phys, virt, size, prot, + __create_pgd_mapping(mm->pgd, PTRS_PER_PGD, phys, virt, size, prot, pgd_pgtable_alloc, flags); } @@ -186,7 +213,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); /* flush the TLBs after updating live kernel mappings */ @@ -196,7 +223,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, phys_addr_t end, pgprot_t prot, int flags) { - __create_pgd_mapping(pgdp, start, __phys_to_virt(start), end - start, + __create_pgd_mapping(pgdp, PTRS_PER_PGD, start, __phys_to_virt(start), end - start, prot, early_pgtable_alloc, flags); } @@ -297,7 +324,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, + __create_pgd_mapping(pgdp, PTRS_PER_PGD, pa_start, (unsigned long)va_start, size, prot, early_pgtable_alloc, flags); if (!(vm_flags & VM_NO_GUARD)) @@ -341,7 +368,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); - __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, + __create_pgd_mapping(tramp_pg_dir, PTRS_PER_PGD, pa_start, TRAMP_VALIAS, PAGE_SIZE, prot, __pgd_pgtable_alloc, 0); /* Map both the text and data into the kernel page table */ @@ -1233,7 +1260,7 @@ int arch_add_memory(int nid, u64 start, u64 size, IS_ENABLED(CONFIG_KFENCE)) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), + __create_pgd_mapping(swapper_pg_dir, PTRS_PER_PGD, start, __phys_to_virt(start), size, params->pgprot, __pgd_pgtable_alloc, flags); diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index 95ff35a3c6cb..be51689d1133 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -241,14 +241,19 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, +static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long addr, end, next; - pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + pgd_t *pgdp; + + if (likely(entries_cnt == PTRS_PER_PGD)) + pgdp = pgd_offset_pgd(pgdir, virt); + else + pgdp = pgdir + ((virt >> PGDIR_SHIFT) & (entries_cnt - 1)); /* * If the virtual and physical address don't have the same offset