From patchwork Sun Apr 25 14:13:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD1EC433ED for ; Sun, 25 Apr 2021 14:17:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F38161364 for ; Sun, 25 Apr 2021 14:17:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F38161364 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fGLCDeJgHL44BNKJHiUStPlaZ4BbIhQswXIrHEJY41s=; b=Zxv4g4MsLP+xVD2OKP2sswfDf apTS/zXi9JBg/CYyT8/HbxqOT4VFZppLxk6shbptMKQ7zxk9ly9saGkYFgP7YtLWv0XAorsdE+sPc z2dGiICM8HUg2LqvhPagGhUENcOkVoC+IOXWxYeBnHAw2s3YR2raZQhLHSHPg8OwoL423+kUZM5Sm 0MlUdg0f+zJuxqwiK5iHesQob7r5a4yX4uZZ5wx9hkaIZjmUb2vubum4sif2KlvpLvHJ22ffRP/Db SmJ1EEBGUJ6dKt/w8yqscmkU9ZEMDSYOU45SZ0s/2sq1Gc1+eHdpK9qkmhQItZ/Ggnda1xNv9KS5U TyiKJDe2g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafYI-005iuY-4X; Sun, 25 Apr 2021 14:16:02 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWm-005iMy-6O for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:14:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=uyGxi/1m6tix6sJesbcYLWdFFEKB6t5K3AVKL00MdLE=; b=RPKYHEMOVCrlLR0q2z+AggLeUS 08SjW9egMZzz+Tx8edz854a481c/ckyy0L7DS+G153vTPau1M21ybcboCeZ8jzwQZwhG4aWsLmxnA jppUff7YvP73BsMe6ATFpsRutGGD0PypFioxg4uwoXW48o5UT5IWi2Jv+q+3HhhEnM459/aE7Pbxr QPk3YcHgen1LYlMI1PhNOAOdzU0MDS9wuNqtHpd+cdrnF6ugfGjz/jpv0MEsMcXUGwwQBlgBpUp77 pR+mNI/OxcRVGsqgjeenb2SekE4wfp9yMV2iaxVhdvVqkRo4kTob15J6to/eKnDu+Ey4URuJ9UeCf g+fO6bmQ==; Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafWi-00FT7P-TN for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:14:27 +0000 Received: by mail-pj1-x102c.google.com with SMTP id h14-20020a17090aea8eb02901553e1cc649so1562785pjz.0 for ; Sun, 25 Apr 2021 07:14:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uyGxi/1m6tix6sJesbcYLWdFFEKB6t5K3AVKL00MdLE=; b=kGVBbb4apUaVRFfvAIzJAUdaLw9CSdzF8TntMk112imdYRvcNJix7rbn07yVCCBrD8 LZNOLLv0/WvY1aeRVJF44RqgctCyKw1YDJ3P+4YYX97gbncH3roTqz40FRAkxIIcOFL4 LkkyU4W6TUHPEmTGSujPf2VQl6HsuPNEfGOxkyG6+Nw/eHLn1nRu4L8pW3jNzahTGdkP rTdbMYHQYVjOwrFTe0qCyQ77Xj/wg9iL2MVd5v3TEJBqXQo71Jn4I1TsNOxWWKDWbR9X p72c9NqPLSM3jRoBz00L5XzFkE95VjaevdtAw7cS8lsek+oARMgmNKBI2SucIkMQuJ89 Cy4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uyGxi/1m6tix6sJesbcYLWdFFEKB6t5K3AVKL00MdLE=; b=hVdGYO0LqUy5wB6ne6AjGd7WqoKaKFGl5tbjbwk748aYpX8/GtN3Ob2AUktmlSmCzM aS0Al/8ZaIGolch8tLEx73DmhS7E6j4e5e+UdSxJ/nps8XeGHTpzXFeapeUVZlDXWh+N 3cgsnAv6JRM8pfm+QwQhww60ujd9YE//fic2xf11AFP82ZmWlO0NILZcue0QAgOjzXtS 3pIvD7ZWbPMQ3P37CmPLCTuNCPbnHcdRyySAjLTOI+2sdK+5Y325vrC2sQsRkTQyys+n m1S93eD0rcG/9J7qO6xtaBwpbddx/91PLlMaFskp0riSl+RkFb2B++mYeAERUg0knEFI k7NA== X-Gm-Message-State: AOAM531RAYDhphoECd8JJ7b+4GldGbgFw3E8/XnqglQzwTE0g6cJ3A7+ LYF1uIPvbqx+qa43hR4IxDErbb0r1w== X-Google-Smtp-Source: ABdhPJzjtHAry5eUX/C26kEznE4BwfFOovyrvxINXyNMhIlS/FXq1ZvFGapyQ+u+ZIPcH2uaWgQ38w== X-Received: by 2002:a17:90a:352:: with SMTP id 18mr15028980pjf.223.1619360063889; Sun, 25 Apr 2021 07:14:23 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.14.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:14:23 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 10/10] arm64/head: convert idmap_pg_dir and init_pg_dir to __create_pgd_mapping() Date: Sun, 25 Apr 2021 22:13:04 +0800 Message-Id: <20210425141304.32721-11-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071425_142034_967F84E6 X-CRM114-Status: GOOD ( 22.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now, everything is ready for calling __create_pgd_mapping() from head.S. Switching to these C routine and remove the asm counterpart. This patch has been successfully tested with the following config value: PAGE_SIZE VA PA PGTABLE_LEVEL 4K 48 48 4 4K 39 48 3 16K 48 48 4 16K 47 48 3 64K 52 52 3 64K 42 52 2 Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- RFC -> V2: correct the asm calling convention. --- arch/arm64/include/asm/pgalloc.h | 5 + arch/arm64/kernel/head.S | 193 ++++++++----------------------- arch/arm64/mm/mmu.c | 13 +++ 3 files changed, 66 insertions(+), 145 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index c3875af99432..b1182b656b00 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -8,6 +8,9 @@ #ifndef __ASM_PGALLOC_H #define __ASM_PGALLOC_H +#include + +#ifndef __ASSEMBLY__ #include #include #include @@ -102,6 +105,8 @@ extern void create_idmap(pgd_t *pgdir, phys_addr_t phys, void *info, int flags); +#endif + #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_FIXMAP BIT(2) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e19649dbbafb..ddb9601d61c2 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -28,6 +28,8 @@ #include #include #include +#include +#include #include #include #include @@ -92,6 +94,8 @@ SYM_CODE_START(primary_entry) bl init_kernel_el // w0=cpu_boot_mode adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 + adrp x4, init_thread_union + add sp, x4, #THREAD_SIZE bl set_cpu_boot_mode_flag bl __create_page_tables /* @@ -121,135 +125,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b __inval_dcache_area // tail call SYM_CODE_END(preserve_boot_args) -/* - * Macro to create a table entry to the next page. - * - * tbl: page table address - * virt: virtual address - * shift: #imm page table shift - * ptrs: #imm pointers per table page - * - * Preserves: virt - * Corrupts: ptrs, tmp1, tmp2 - * Returns: tbl -> next level table page address - */ - .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2 - add \tmp1, \tbl, #PAGE_SIZE - phys_to_pte \tmp2, \tmp1 - orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type - lsr \tmp1, \virt, #\shift - sub \ptrs, \ptrs, #1 - and \tmp1, \tmp1, \ptrs // table index - str \tmp2, [\tbl, \tmp1, lsl #3] - add \tbl, \tbl, #PAGE_SIZE // next level table page - .endm - -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - * shift: shift used to transform virtual address into index - * ptrs: number of entries in page table - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend, shift, ptrs - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, ptrs, istart, iend, count - lsr \iend, \vend, \shift - mov \istart, \ptrs - sub \istart, \istart, #1 - and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1) - mov \istart, \ptrs - mul \istart, \istart, \count - add \iend, \iend, \istart // iend += (count - 1) * ptrs - // our entries span multiple tables - - lsr \istart, \vstart, \shift - mov \count, \ptrs - sub \count, \count, #1 - and \istart, \istart, \count - - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: start address to map - * vend: end address to map - we map [vstart, vend] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * pgds: the number of pgd entries - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, vend, flags - * Corrupts: tbl, rtbl, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, pgds, istart, iend, tmp, count, sv - add \rtbl, \tbl, #PAGE_SIZE - mov \sv, \rtbl - mov \count, #0 - compute_indices \vstart, \vend, #PGDIR_SHIFT, \pgds, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #PTRS_PER_PUD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #PTRS_PER_PMD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #PTRS_PER_PTE, \istart, \iend, \count - bic \count, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \count, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - /* * Setup the initial page tables. We only setup the barest amount which is * required to get the kernel running. The following sections are required: @@ -345,8 +220,6 @@ SYM_FUNC_START_LOCAL(__create_page_tables) mov x5, #1 str x5, [x4] //require expanded pagetable - mov x4, EXTRA_PTRS - create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 #else /* * If VA_BITS == 48, we don't have to configure an additional @@ -356,25 +229,55 @@ SYM_FUNC_START_LOCAL(__create_page_tables) str_l x4, idmap_ptrs_per_pgd, x5 #endif 1: - ldr_l x4, idmap_ptrs_per_pgd - mov x5, x3 // __pa(__idmap_text_start) - adr_l x6, __idmap_text_end // __pa(__idmap_text_end) - - map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 + stp x0, x1, [sp, #-64]! + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #32] + stp x6, x7, [sp, #16] + + adrp x0, idmap_pg_dir + adrp x1, idmap_pg_end + sub x1, x1, x0 + bl set_cur_mempool + mov x0, #0 + mov x0, #0 + bl head_pgtable_alloc // x0 contains idmap_pg_dir + + adrp x1, __idmap_text_start + adr_l x2, __idmap_text_end + sub x2, x2, x1 + ldr x3, =PAGE_KERNEL_EXEC + adr_l x4, head_pgtable_alloc + mov x5, #0 + mov x6, #(NO_FIXMAP | BOOT_HEAD) + bl create_idmap /* * Map the kernel image (starting with PHYS_OFFSET). */ adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) - add x5, x5, x23 // add KASLR displacement - mov x4, PTRS_PER_PGD - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - - map_memory x0, x1, x5, x6, x7, x3, x4, x10, x11, x12, x13, x14 + adrp x1, init_pg_end + sub x1, x1, x0 + bl set_cur_mempool + mov x0, #0 + mov x0, #0 + bl head_pgtable_alloc // x0 is init_pg_dir + + adrp x1, _text // runtime __pa(_text) + mov_q x2, KIMAGE_VADDR // compile time __va(_text) + add x2, x2, x23 // add KASLR displacement + adrp x3, _end // runtime __pa(_end) + sub x3, x3, x1 // _end - _text + + ldr x4, =PAGE_KERNEL_EXEC + adr_l x5, head_pgtable_alloc + mov x6, #0 + mov x7, #(NO_FIXMAP | BOOT_HEAD) + + bl create_init_pgd_mapping + ldp x6, x7, [sp, #16] + ldp x4, x5, [sp, #32] + ldp x2, x3, [sp, #48] + ldp x0, x1, [sp], #64 /* * Since the page tables have been populated with non-cacheable diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index fa1d1d4fee8f..1ae72a3f2d27 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -167,6 +167,19 @@ static phys_addr_t pgd_pgtable_alloc(unsigned long shift, void *unused) #include "./mmu_include.c" +void create_init_pgd_mapping(pgd_t *pgdir, + phys_addr_t phys, + unsigned long virt, + phys_addr_t size, + pgprot_t prot, + pgtable_alloc allocator, + void* info, + int flags) +{ + __create_pgd_mapping(pgdir, PTRS_PER_PGD, phys, virt, size, + prot, allocator, info, flags); +} + int idmap_extend_pgtable; /*