From patchwork Tue Jul 2 13:51:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45526C3064D for ; Tue, 2 Jul 2024 13:51:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDDF66B00A5; Tue, 2 Jul 2024 09:51:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8DC66B00A6; Tue, 2 Jul 2024 09:51:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2EE06B00A7; Tue, 2 Jul 2024 09:51:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 947696B00A5 for ; Tue, 2 Jul 2024 09:51:35 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id ED4C6A016A for ; Tue, 2 Jul 2024 13:51:34 +0000 (UTC) X-FDA: 82294950108.15.EAA4D85 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf21.hostedemail.com (Postfix) with ESMTP id B001B1C000C for ; Tue, 2 Jul 2024 13:51:31 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928281; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VYym39zGcpp+dJgJU9GQXNYKflCQed3On6+cuBcHd04=; b=iakvO8X0NE8QJ5IISncGTlp8LjEvgLtsVXaqJrFIPwgZZjPhcFFkOHlrnko8g50OHDJRok lpWCMVH152K77oyczw5N96sFfLzsqX09z31+6u4BJbZwl3Dx9w1CChKBnaY6gH8VJPPIdO puMw4bbomeho4UjSBlaz1NIhFdwOCbY= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928281; a=rsa-sha256; cv=none; b=qhalJ7rE6ovUfh2A4Lr9ddmr/jxyx5Zhx7zc7XCt3/fP8WmzqtoSwzDzQ4cHnNoVhZNjng GRT8dX0+l2wgXB2mhNWLqFOxoOIrZ9CF6GjRoLpo6axyQJ6Ao20rupR+f1tX3IJQS17Sim vU6IUiOL05PdK3Tdo9pG9289m6yXdSg= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD47y26csz9v2V; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dTRvj7dpzd3O; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD47x3sRpz9tZl; Tue, 2 Jul 2024 15:51:25 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 800D98B775; Tue, 2 Jul 2024 15:51:25 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id ziD_a5MpHMl8; Tue, 2 Jul 2024 15:51:25 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E9E128B774; Tue, 2 Jul 2024 15:51:24 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 01/23] powerpc/64e: Remove unused IBM HTW code Date: Tue, 2 Jul 2024 15:51:13 +0200 Message-ID: <820dd1385ecc931f07b0d7a0fa827b1613917ab6.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928273; l=10427; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=2h8pbz81hz7A9kWlbZomb+s/bHgtRVC1UTR2w333/4M=; b=izE7XFJ47Sk/D0H0FwiBZtTuWj0XJrh2eDhX68TR/tBT1FcX4RLBeECK2siy5iJKIhtsN96ue gZlbvbMdVZhAGupVPaJtXuD6dZ8u8eK0JhZSFiISZgWGp0cDSFG0p66 X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: B001B1C000C X-Stat-Signature: nddos4q9nyxh5zmeq5knzw9kqd9ndjzu X-HE-Tag: 1719928291-636664 X-HE-Meta: U2FsdGVkX18ZBSP3mMGNNHyR55vONgnscLiJSvH43eD0iPnKTwLAdy4EFbfkudqfYzwEQcLWXKSHLng2iz73kdLZCtAScJmHfxSGwZd4pVL5I7qeRtgTqQCZza/jqaABLXK+tQqQhwxVCEhJdvdSK0swT/ie/fYpiqfOe2IFHYksA3dYiE4uyIcWVgMQQXRanY6TBe1V0Mtjbo7S9idtTl3TlLspyiomVnS+PJcBqD38NSrO9IO2Iv5JwLsp5J73HTaD2vYdlocjozfPza1hwFpRvfo2m0UOLzBuGhO425xmHt95826AwP9VCsb7x1o9umz54l7g68TnVX1JACZcjg9wvjrkwnKg9Ror/T9sLMiCnOpJOLy/cYmqnzQXEfP0/8VE/83aFOZMo1RlfiCSv/CbTk0vWnoeQIwYcYwyE2ymQcH4j/VsFw4AOHIyPOWV9Ku1l6yo+aPK39dfBAa4Kyuekdh11uuzMchcg2kezqbj6bnmk/2hXsgtEwtwxtxhUR0WKHA4415Aj6WSIBkJ6+1gSB0x6uVfAzH/UyVPvLAS2PLl60pYhPFryWmV3lwpcusKL1+BV45+GcWwPQVg2hjcrWJHbenYiF7JMYe7jYKYYz8h0rwC85ZbLbkLc9k2JTM13JGD27GvEJtGZ+UaJKhk1KUJGKTThxzmIOkwbOlfvDf5uKimMUhSm0Msyp3YSAX0OaUEVxfvDnJ7x8OMWrkWtu5TvK/aTu3gJ/pb4S5nkhnKv7LZhoGuXb+3YWTTrZPkdytvQjlX5nTTA+Mx2HAnlqKcXO0Y6rGyW0fQwxFzcyuPa2eIlAo1EQY9fGAp1noLWmWRJXPOztyyOfKduchNLYSpyDW25oyRYfbLj+vo3gNTeIxuQczlqF2QbM2Ugr5KocxxuXhjaZW20Ar6Hnt5lzPTgplHOEzF9GGRsrsCdLorxyNJ0n8txP8P2DiG4H5d2HugyX4IxfZsqOJ RgVs2TJo r9X32j9KT/t9c2hrzItCPkvfY5j3T1zobss+qWcUlkfff96NMZfg+bimH6W8PIq0ICyBGtp5mjJ+ctg/qDWnn7fMIuKyuWvBJg/0FY97gMtX05DwdJz5pyMMY9fSrBaxih40K X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Michael Ellerman The nohash HTW_IBM (Hardware Table Walk) code is unused since support for A2 was removed in commit fb5a515704d7 ("powerpc: Remove platforms/ wsp and associated pieces") (2014). The remaining supported CPUs use either no HTW (data_tlb_miss_bolted), or the e6500 HTW (data_tlb_miss_e6500). Signed-off-by: Michael Ellerman Signed-off-by: Christophe Leroy --- arch/powerpc/include/asm/nohash/mmu-e500.h | 3 +- arch/powerpc/mm/nohash/tlb.c | 57 +----- arch/powerpc/mm/nohash/tlb_low_64e.S | 195 --------------------- 3 files changed, 2 insertions(+), 253 deletions(-) diff --git a/arch/powerpc/include/asm/nohash/mmu-e500.h b/arch/powerpc/include/asm/nohash/mmu-e500.h index 6ddced0415cb..7dc24b8632d7 100644 --- a/arch/powerpc/include/asm/nohash/mmu-e500.h +++ b/arch/powerpc/include/asm/nohash/mmu-e500.h @@ -303,8 +303,7 @@ extern unsigned long linear_map_top; extern int book3e_htw_mode; #define PPC_HTW_NONE 0 -#define PPC_HTW_IBM 1 -#define PPC_HTW_E6500 2 +#define PPC_HTW_E6500 1 /* * 64-bit booke platforms don't load the tlb in the tlb miss handler code. diff --git a/arch/powerpc/mm/nohash/tlb.c b/arch/powerpc/mm/nohash/tlb.c index 5ffa0af4328a..a5bb87ec8578 100644 --- a/arch/powerpc/mm/nohash/tlb.c +++ b/arch/powerpc/mm/nohash/tlb.c @@ -400,9 +400,8 @@ void tlb_flush_pgtable(struct mmu_gather *tlb, unsigned long address) static void __init setup_page_sizes(void) { unsigned int tlb0cfg; - unsigned int tlb0ps; unsigned int eptcfg; - int i, psize; + int psize; #ifdef CONFIG_PPC_E500 unsigned int mmucfg = mfspr(SPRN_MMUCFG); @@ -471,50 +470,6 @@ static void __init setup_page_sizes(void) goto out; } #endif - - tlb0cfg = mfspr(SPRN_TLB0CFG); - tlb0ps = mfspr(SPRN_TLB0PS); - eptcfg = mfspr(SPRN_EPTCFG); - - /* Look for supported direct sizes */ - for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { - struct mmu_psize_def *def = &mmu_psize_defs[psize]; - - if (tlb0ps & (1U << (def->shift - 10))) - def->flags |= MMU_PAGE_SIZE_DIRECT; - } - - /* Indirect page sizes supported ? */ - if ((tlb0cfg & TLBnCFG_IND) == 0 || - (tlb0cfg & TLBnCFG_PT) == 0) - goto out; - - book3e_htw_mode = PPC_HTW_IBM; - - /* Now, we only deal with one IND page size for each - * direct size. Hopefully all implementations today are - * unambiguous, but we might want to be careful in the - * future. - */ - for (i = 0; i < 3; i++) { - unsigned int ps, sps; - - sps = eptcfg & 0x1f; - eptcfg >>= 5; - ps = eptcfg & 0x1f; - eptcfg >>= 5; - if (!ps || !sps) - continue; - for (psize = 0; psize < MMU_PAGE_COUNT; psize++) { - struct mmu_psize_def *def = &mmu_psize_defs[psize]; - - if (ps == (def->shift - 10)) - def->flags |= MMU_PAGE_SIZE_INDIRECT; - if (sps == (def->shift - 10)) - def->ind = ps + 10; - } - } - out: /* Cleanup array and print summary */ pr_info("MMU: Supported page sizes\n"); @@ -543,10 +498,6 @@ static void __init setup_mmu_htw(void) */ switch (book3e_htw_mode) { - case PPC_HTW_IBM: - patch_exception(0x1c0, exc_data_tlb_miss_htw_book3e); - patch_exception(0x1e0, exc_instruction_tlb_miss_htw_book3e); - break; #ifdef CONFIG_PPC_E500 case PPC_HTW_E6500: extlb_level_exc = EX_TLB_SIZE; @@ -577,12 +528,6 @@ static void early_init_this_mmu(void) mmu_pte_psize = MMU_PAGE_2M; break; - case PPC_HTW_IBM: - mas4 |= MAS4_INDD; - mas4 |= BOOK3E_PAGESZ_1M << MAS4_TSIZED_SHIFT; - mmu_pte_psize = MMU_PAGE_1M; - break; - case PPC_HTW_NONE: mas4 |= BOOK3E_PAGESZ_4K << MAS4_TSIZED_SHIFT; mmu_pte_psize = mmu_virtual_psize; diff --git a/arch/powerpc/mm/nohash/tlb_low_64e.S b/arch/powerpc/mm/nohash/tlb_low_64e.S index 7e0b8fe1c279..b0eb3f7eaed1 100644 --- a/arch/powerpc/mm/nohash/tlb_low_64e.S +++ b/arch/powerpc/mm/nohash/tlb_low_64e.S @@ -893,201 +893,6 @@ virt_page_table_tlb_miss_whacko_fault: TLB_MISS_EPILOG_ERROR b exc_data_storage_book3e - -/************************************************************** - * * - * TLB miss handling for Book3E with hw page table support * - * * - **************************************************************/ - - -/* Data TLB miss */ - START_EXCEPTION(data_tlb_miss_htw) - TLB_MISS_PROLOG - - /* Now we handle the fault proper. We only save DEAR in normal - * fault case since that's the only interesting values here. - * We could probably also optimize by not saving SRR0/1 in the - * linear mapping case but I'll leave that for later - */ - mfspr r14,SPRN_ESR - mfspr r16,SPRN_DEAR /* get faulting address */ - srdi r11,r16,44 /* get region */ - xoris r11,r11,0xc - cmpldi cr0,r11,0 /* linear mapping ? */ - beq tlb_load_linear /* yes -> go to linear map load */ - cmpldi cr1,r11,1 /* vmalloc mapping ? */ - - /* We do the user/kernel test for the PID here along with the RW test - */ - srdi. r11,r16,60 /* Check for user region */ - ld r15,PACAPGD(r13) /* Load user pgdir */ - beq htw_tlb_miss - - /* XXX replace the RMW cycles with immediate loads + writes */ -1: mfspr r10,SPRN_MAS1 - rlwinm r10,r10,0,16,1 /* Clear TID */ - mtspr SPRN_MAS1,r10 - ld r15,PACA_KERNELPGD(r13) /* Load kernel pgdir */ - beq+ cr1,htw_tlb_miss - - /* We got a crappy address, just fault with whatever DEAR and ESR - * are here - */ - TLB_MISS_EPILOG_ERROR - b exc_data_storage_book3e - -/* Instruction TLB miss */ - START_EXCEPTION(instruction_tlb_miss_htw) - TLB_MISS_PROLOG - - /* If we take a recursive fault, the second level handler may need - * to know whether we are handling a data or instruction fault in - * order to get to the right store fault handler. We provide that - * info by keeping a crazy value for ESR in r14 - */ - li r14,-1 /* store to exception frame is done later */ - - /* Now we handle the fault proper. We only save DEAR in the non - * linear mapping case since we know the linear mapping case will - * not re-enter. We could indeed optimize and also not save SRR0/1 - * in the linear mapping case but I'll leave that for later - * - * Faulting address is SRR0 which is already in r16 - */ - srdi r11,r16,44 /* get region */ - xoris r11,r11,0xc - cmpldi cr0,r11,0 /* linear mapping ? */ - beq tlb_load_linear /* yes -> go to linear map load */ - cmpldi cr1,r11,1 /* vmalloc mapping ? */ - - /* We do the user/kernel test for the PID here along with the RW test - */ - srdi. r11,r16,60 /* Check for user region */ - ld r15,PACAPGD(r13) /* Load user pgdir */ - beq htw_tlb_miss - - /* XXX replace the RMW cycles with immediate loads + writes */ -1: mfspr r10,SPRN_MAS1 - rlwinm r10,r10,0,16,1 /* Clear TID */ - mtspr SPRN_MAS1,r10 - ld r15,PACA_KERNELPGD(r13) /* Load kernel pgdir */ - beq+ htw_tlb_miss - - /* We got a crappy address, just fault */ - TLB_MISS_EPILOG_ERROR - b exc_instruction_storage_book3e - - -/* - * This is the guts of the second-level TLB miss handler for direct - * misses. We are entered with: - * - * r16 = virtual page table faulting address - * r15 = PGD pointer - * r14 = ESR - * r13 = PACA - * r12 = TLB exception frame in PACA - * r11 = crap (free to use) - * r10 = crap (free to use) - * - * It can be re-entered by the linear mapping miss handler. However, to - * avoid too much complication, it will save/restore things for us - */ -htw_tlb_miss: -#ifdef CONFIG_PPC_KUAP - mfspr r10,SPRN_MAS1 - rlwinm. r10,r10,0,0x3fff0000 - beq- htw_tlb_miss_fault /* KUAP fault */ -#endif - /* Search if we already have a TLB entry for that virtual address, and - * if we do, bail out. - * - * MAS1:IND should be already set based on MAS4 - */ - PPC_TLBSRX_DOT(0,R16) - beq htw_tlb_miss_done - - /* Now, we need to walk the page tables. First check if we are in - * range. - */ - rldicl. r10,r16,64-PGTABLE_EADDR_SIZE,PGTABLE_EADDR_SIZE+4 - bne- htw_tlb_miss_fault - - /* Get the PGD pointer */ - cmpldi cr0,r15,0 - beq- htw_tlb_miss_fault - - /* Get to PGD entry */ - rldicl r11,r16,64-(PGDIR_SHIFT-3),64-PGD_INDEX_SIZE-3 - clrrdi r10,r11,3 - ldx r15,r10,r15 - cmpdi cr0,r15,0 - bge htw_tlb_miss_fault - - /* Get to PUD entry */ - rldicl r11,r16,64-(PUD_SHIFT-3),64-PUD_INDEX_SIZE-3 - clrrdi r10,r11,3 - ldx r15,r10,r15 - cmpdi cr0,r15,0 - bge htw_tlb_miss_fault - - /* Get to PMD entry */ - rldicl r11,r16,64-(PMD_SHIFT-3),64-PMD_INDEX_SIZE-3 - clrrdi r10,r11,3 - ldx r15,r10,r15 - cmpdi cr0,r15,0 - bge htw_tlb_miss_fault - - /* Ok, we're all right, we can now create an indirect entry for - * a 1M or 256M page. - * - * The last trick is now that because we use "half" pages for - * the HTW (1M IND is 2K and 256M IND is 32K) we need to account - * for an added LSB bit to the RPN. For 64K pages, there is no - * problem as we already use 32K arrays (half PTE pages), but for - * 4K page we need to extract a bit from the virtual address and - * insert it into the "PA52" bit of the RPN. - */ - rlwimi r15,r16,32-9,20,20 - /* Now we build the MAS: - * - * MAS 0 : Fully setup with defaults in MAS4 and TLBnCFG - * MAS 1 : Almost fully setup - * - PID already updated by caller if necessary - * - TSIZE for now is base ind page size always - * MAS 2 : Use defaults - * MAS 3+7 : Needs to be done - */ - ori r10,r15,(BOOK3E_PAGESZ_4K << MAS3_SPSIZE_SHIFT) - - srdi r16,r10,32 - mtspr SPRN_MAS3,r10 - mtspr SPRN_MAS7,r16 - - tlbwe - -htw_tlb_miss_done: - /* We don't bother with restoring DEAR or ESR since we know we are - * level 0 and just going back to userland. They are only needed - * if you are going to take an access fault - */ - TLB_MISS_EPILOG_SUCCESS - rfi - -htw_tlb_miss_fault: - /* We need to check if it was an instruction miss. We know this - * though because r14 would contain -1 - */ - cmpdi cr0,r14,-1 - beq 1f - mtspr SPRN_DEAR,r16 - mtspr SPRN_ESR,r14 - TLB_MISS_EPILOG_ERROR - b exc_data_storage_book3e -1: TLB_MISS_EPILOG_ERROR - b exc_instruction_storage_book3e - /* * This is the guts of "any" level TLB miss handler for kernel linear * mapping misses. We are entered with: From patchwork Tue Jul 2 13:51:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4E94C3064D for ; Tue, 2 Jul 2024 13:51:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D61B6B0099; Tue, 2 Jul 2024 09:51:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 784D76B00A8; Tue, 2 Jul 2024 09:51:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B1916B00A9; Tue, 2 Jul 2024 09:51:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 356F26B00A8 for ; Tue, 2 Jul 2024 09:51:39 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E035140188 for ; Tue, 2 Jul 2024 13:51:38 +0000 (UTC) X-FDA: 82294950276.05.8CF58E1 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf30.hostedemail.com (Postfix) with ESMTP id 7E15B80019 for ; Tue, 2 Jul 2024 13:51:36 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf30.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928278; a=rsa-sha256; cv=none; b=DzjmuVm9OiymUm68jYNyhFve2PhPjvFqRbe2GEHs9Pyx36bGmBwnv3zKteBkMXcisv8D1J FIskcwxW5WYdYwPJKB6cfVJtzGEooQ7y4uMfXTODXt5wM4PfBmMeVwnYehgkGB2m4Cmk8m XMxRkAqqX1AuzUx6FhoquCQae4sO2LQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf30.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928278; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R0PVifahn1qmJ9z/bNcdf65girMvZHbArLxf8YHuaqA=; b=kEBH3nN9CUZFRuiJlwmSnzaoTl9PMSoUecbXI91RAMqBcQed/fbctAEcxdAaVGmHr4wxg1 NWr/OZOTSQ8Xzxvw7ROGoKSL8A0QudRhMX6XC6Siz5p5lfl3I+0GsgxT02hYyGFfBXQb+T 1Bw/tKbrUq9hpuiCtbXFQ/b3hCQC/NU= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD47z12CVz9v3s; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P7m0rlq-mMbZ; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD47y1JHFz9v2F; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 282B48B774; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id yeIanGXYgJun; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 7B8268B764; Tue, 2 Jul 2024 15:51:25 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 02/23] powerpc/64e: Split out nohash Book3E 64-bit code Date: Tue, 2 Jul 2024 15:51:14 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=21540; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=iy3UzMyQc8EZ4jE+FBlk+KBxyBsDFMEVWvR6uRnFe60=; b=yjlJTfT4ejFWPRZLtg+i+6CjQyOfMNCC222IY8qq0JyV0IS4J/ilIebVPxO1FHxF04S0Vkduj 2Yw0aCaVpbHBVF/mtYBzL0ostxTe0CknJ6hhvvocgPW/hWILbKqiHA0 X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7E15B80019 X-Stat-Signature: 8kw1bd3ew6o1qhspge1gdz3hqmimkhsr X-Rspam-User: X-HE-Tag: 1719928296-147014 X-HE-Meta: U2FsdGVkX19nUhbiD//I4vt1JxSWVnx5UgV61qmDwaiHdtAlXZc+S7j3MBiLjIEJTzhXYTAB/Ef0GxKFh4x8w6DA+X8olDon9l8DtBTBU/G3sKq+1vL8IZKu1gUJZveNyXHhz78AeY6OfV1dxR8B+A799AyKv+ysu/F95I237q5T+ikwjIMCDcOD7ODV2sKRJWRuXwt2oEh74K8WQtImMKTkUxTbtm6Y2YdpoR7m+ocMQqwl5+1/Wxi9ifHOGc8DnPGwWHG9TPpBsrjfmWHSmJ/OSbECLrESUHClrxMsy93AYJOXIIqxXhHEXAl1+cD3argIIgRi7YyohJOxifCBL6O2+AJkb4SUGewdmel+bvwZmtkRIveK7cDmNHB/020/Y0hBwCvp/K7vDWhOv3E0wedl2VFx+Ggm5/4Ku81gUhX5Ntt16/ZIQiPxGXPI6mHuC266vCxvk+1Lqfc9ZzPRZs9l49R/ggemAZKQ137YmMMcI9CVQhDc9GYq3I5xYIMleKh8zkos/IyXd2ObLa15yoVTzhmnl1z/tEsRl+sm0oNUphB2e5vT8O6fQ3zsmC6fkYVWSTg1khxdj1GCMS/cwCvN/z5cb5KZDx0CFAptUGEwY5dtU8Uy7ox36GN3F/rfZDiHzUPGMm++yvZarNaoayk8yyZhOswsqWCjiI3IDDFYs0dEPjqzSuI2e+H/oP64WFWea0vN3aNixQ4tA3KJxFd/w+pbwklqY6O5cc/BML3xrfOTT7dq8K+yThiqjWmpE8SvPw9m+3j2FcsVwWAIn9tADgRla+poxAgJjMTBQl5sKk3q/mhMw+2AH+tT+Z4B/5ok5PQYxaAfVEkSloYePx/be1LuXWJyrm8ym+wdaq3Q4KrgtR5tpn6WsgegioYzgYfV7s31euKDcEAVVG5YIErb3z0nZzbT2wVDjMkgOtebtqNtcOQjebU+C9qPXT8g5gleOmEAFVrOG/mNH5t uKpxTPVg lEXrqwvszmzW+a1/2WujFtTQCfdR0m6XCw0zWpnKHdMWCJqS9/Cf8xOi9V/dj6kghwtx0ERI1bPMoXIRfeOUdmi+T5yTJYdqQX66+aE1ePCUvgvoMzYzVlkyIgO1/X698D+69ujFeSLl1IY7RurIgSwjemdBn/5NOonQURLSrYeBP1cC4g87gkKNd5+5Pfckh3L0M X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Michael Ellerman A reasonable chunk of nohash/tlb.c is 64-bit only code, split it out into a separate file. Signed-off-by: Michael Ellerman Signed-off-by: Christophe Leroy --- arch/powerpc/mm/nohash/Makefile | 2 +- arch/powerpc/mm/nohash/tlb.c | 343 +------------------- arch/powerpc/mm/nohash/{tlb.c => tlb_64e.c} | 328 ------------------- 3 files changed, 2 insertions(+), 671 deletions(-) copy arch/powerpc/mm/nohash/{tlb.c => tlb_64e.c} (56%) diff --git a/arch/powerpc/mm/nohash/Makefile b/arch/powerpc/mm/nohash/Makefile index b3f0498dd42f..90e846f0c46c 100644 --- a/arch/powerpc/mm/nohash/Makefile +++ b/arch/powerpc/mm/nohash/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-y += mmu_context.o tlb.o tlb_low.o kup.o -obj-$(CONFIG_PPC_BOOK3E_64) += tlb_low_64e.o book3e_pgtable.o +obj-$(CONFIG_PPC_BOOK3E_64) += tlb_64e.o tlb_low_64e.o book3e_pgtable.o obj-$(CONFIG_40x) += 40x.o obj-$(CONFIG_44x) += 44x.o obj-$(CONFIG_PPC_8xx) += 8xx.o diff --git a/arch/powerpc/mm/nohash/tlb.c b/arch/powerpc/mm/nohash/tlb.c index a5bb87ec8578..f57dc721d063 100644 --- a/arch/powerpc/mm/nohash/tlb.c +++ b/arch/powerpc/mm/nohash/tlb.c @@ -110,28 +110,6 @@ struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { }; #endif -/* The variables below are currently only used on 64-bit Book3E - * though this will probably be made common with other nohash - * implementations at some point - */ -#ifdef CONFIG_PPC64 - -int mmu_pte_psize; /* Page size used for PTE pages */ -int mmu_vmemmap_psize; /* Page size used for the virtual mem map */ -int book3e_htw_mode; /* HW tablewalk? Value is PPC_HTW_* */ -unsigned long linear_map_top; /* Top of linear mapping */ - - -/* - * Number of bytes to add to SPRN_SPRG_TLB_EXFRAME on crit/mcheck/debug - * exceptions. This is used for bolted and e6500 TLB miss handlers which - * do not modify this SPRG in the TLB miss code; for other TLB miss handlers, - * this is set to zero. - */ -int extlb_level_exc; - -#endif /* CONFIG_PPC64 */ - #ifdef CONFIG_PPC_E500 /* next_tlbcam_idx is used to round-robin tlbcam entry assignment */ DEFINE_PER_CPU(int, next_tlbcam_idx); @@ -358,326 +336,7 @@ void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm(tlb->mm); } -/* - * Below are functions specific to the 64-bit variant of Book3E though that - * may change in the future - */ - -#ifdef CONFIG_PPC64 - -/* - * Handling of virtual linear page tables or indirect TLB entries - * flushing when PTE pages are freed - */ -void tlb_flush_pgtable(struct mmu_gather *tlb, unsigned long address) -{ - int tsize = mmu_psize_defs[mmu_pte_psize].enc; - - if (book3e_htw_mode != PPC_HTW_NONE) { - unsigned long start = address & PMD_MASK; - unsigned long end = address + PMD_SIZE; - unsigned long size = 1UL << mmu_psize_defs[mmu_pte_psize].shift; - - /* This isn't the most optimal, ideally we would factor out the - * while preempt & CPU mask mucking around, or even the IPI but - * it will do for now - */ - while (start < end) { - __flush_tlb_page(tlb->mm, start, tsize, 1); - start += size; - } - } else { - unsigned long rmask = 0xf000000000000000ul; - unsigned long rid = (address & rmask) | 0x1000000000000000ul; - unsigned long vpte = address & ~rmask; - - vpte = (vpte >> (PAGE_SHIFT - 3)) & ~0xffful; - vpte |= rid; - __flush_tlb_page(tlb->mm, vpte, tsize, 0); - } -} - -static void __init setup_page_sizes(void) -{ - unsigned int tlb0cfg; - unsigned int eptcfg; - int psize; - -#ifdef CONFIG_PPC_E500 - unsigned int mmucfg = mfspr(SPRN_MMUCFG); - int fsl_mmu = mmu_has_feature(MMU_FTR_TYPE_FSL_E); - - if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V1) { - unsigned int tlb1cfg = mfspr(SPRN_TLB1CFG); - unsigned int min_pg, max_pg; - - min_pg = (tlb1cfg & TLBnCFG_MINSIZE) >> TLBnCFG_MINSIZE_SHIFT; - max_pg = (tlb1cfg & TLBnCFG_MAXSIZE) >> TLBnCFG_MAXSIZE_SHIFT; - - for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { - struct mmu_psize_def *def; - unsigned int shift; - - def = &mmu_psize_defs[psize]; - shift = def->shift; - - if (shift == 0 || shift & 1) - continue; - - /* adjust to be in terms of 4^shift Kb */ - shift = (shift - 10) >> 1; - - if ((shift >= min_pg) && (shift <= max_pg)) - def->flags |= MMU_PAGE_SIZE_DIRECT; - } - - goto out; - } - - if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V2) { - u32 tlb1cfg, tlb1ps; - - tlb0cfg = mfspr(SPRN_TLB0CFG); - tlb1cfg = mfspr(SPRN_TLB1CFG); - tlb1ps = mfspr(SPRN_TLB1PS); - eptcfg = mfspr(SPRN_EPTCFG); - - if ((tlb1cfg & TLBnCFG_IND) && (tlb0cfg & TLBnCFG_PT)) - book3e_htw_mode = PPC_HTW_E6500; - - /* - * We expect 4K subpage size and unrestricted indirect size. - * The lack of a restriction on indirect size is a Freescale - * extension, indicated by PSn = 0 but SPSn != 0. - */ - if (eptcfg != 2) - book3e_htw_mode = PPC_HTW_NONE; - - for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { - struct mmu_psize_def *def = &mmu_psize_defs[psize]; - - if (!def->shift) - continue; - - if (tlb1ps & (1U << (def->shift - 10))) { - def->flags |= MMU_PAGE_SIZE_DIRECT; - - if (book3e_htw_mode && psize == MMU_PAGE_2M) - def->flags |= MMU_PAGE_SIZE_INDIRECT; - } - } - - goto out; - } -#endif -out: - /* Cleanup array and print summary */ - pr_info("MMU: Supported page sizes\n"); - for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { - struct mmu_psize_def *def = &mmu_psize_defs[psize]; - const char *__page_type_names[] = { - "unsupported", - "direct", - "indirect", - "direct & indirect" - }; - if (def->flags == 0) { - def->shift = 0; - continue; - } - pr_info(" %8ld KB as %s\n", 1ul << (def->shift - 10), - __page_type_names[def->flags & 0x3]); - } -} - -static void __init setup_mmu_htw(void) -{ - /* - * If we want to use HW tablewalk, enable it by patching the TLB miss - * handlers to branch to the one dedicated to it. - */ - - switch (book3e_htw_mode) { -#ifdef CONFIG_PPC_E500 - case PPC_HTW_E6500: - extlb_level_exc = EX_TLB_SIZE; - patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e); - patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e); - break; -#endif - } - pr_info("MMU: Book3E HW tablewalk %s\n", - book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported"); -} - -/* - * Early initialization of the MMU TLB code - */ -static void early_init_this_mmu(void) -{ - unsigned int mas4; - - /* Set MAS4 based on page table setting */ - - mas4 = 0x4 << MAS4_WIMGED_SHIFT; - switch (book3e_htw_mode) { - case PPC_HTW_E6500: - mas4 |= MAS4_INDD; - mas4 |= BOOK3E_PAGESZ_2M << MAS4_TSIZED_SHIFT; - mas4 |= MAS4_TLBSELD(1); - mmu_pte_psize = MMU_PAGE_2M; - break; - - case PPC_HTW_NONE: - mas4 |= BOOK3E_PAGESZ_4K << MAS4_TSIZED_SHIFT; - mmu_pte_psize = mmu_virtual_psize; - break; - } - mtspr(SPRN_MAS4, mas4); - -#ifdef CONFIG_PPC_E500 - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - unsigned int num_cams; - bool map = true; - - /* use a quarter of the TLBCAM for bolted linear map */ - num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; - - /* - * Only do the mapping once per core, or else the - * transient mapping would cause problems. - */ -#ifdef CONFIG_SMP - if (hweight32(get_tensr()) > 1) - map = false; -#endif - - if (map) - linear_map_top = map_mem_in_cams(linear_map_top, - num_cams, false, true); - } -#endif - - /* A sync won't hurt us after mucking around with - * the MMU configuration - */ - mb(); -} - -static void __init early_init_mmu_global(void) -{ - /* XXX This should be decided at runtime based on supported - * page sizes in the TLB, but for now let's assume 16M is - * always there and a good fit (which it probably is) - * - * Freescale booke only supports 4K pages in TLB0, so use that. - */ - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) - mmu_vmemmap_psize = MMU_PAGE_4K; - else - mmu_vmemmap_psize = MMU_PAGE_16M; - - /* XXX This code only checks for TLB 0 capabilities and doesn't - * check what page size combos are supported by the HW. It - * also doesn't handle the case where a separate array holds - * the IND entries from the array loaded by the PT. - */ - /* Look for supported page sizes */ - setup_page_sizes(); - - /* Look for HW tablewalk support */ - setup_mmu_htw(); - -#ifdef CONFIG_PPC_E500 - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - if (book3e_htw_mode == PPC_HTW_NONE) { - extlb_level_exc = EX_TLB_SIZE; - patch_exception(0x1c0, exc_data_tlb_miss_bolted_book3e); - patch_exception(0x1e0, - exc_instruction_tlb_miss_bolted_book3e); - } - } -#endif - - /* Set the global containing the top of the linear mapping - * for use by the TLB miss code - */ - linear_map_top = memblock_end_of_DRAM(); - - ioremap_bot = IOREMAP_BASE; -} - -static void __init early_mmu_set_memory_limit(void) -{ -#ifdef CONFIG_PPC_E500 - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - /* - * Limit memory so we dont have linear faults. - * Unlike memblock_set_current_limit, which limits - * memory available during early boot, this permanently - * reduces the memory available to Linux. We need to - * do this because highmem is not supported on 64-bit. - */ - memblock_enforce_memory_limit(linear_map_top); - } -#endif - - memblock_set_current_limit(linear_map_top); -} - -/* boot cpu only */ -void __init early_init_mmu(void) -{ - early_init_mmu_global(); - early_init_this_mmu(); - early_mmu_set_memory_limit(); -} - -void early_init_mmu_secondary(void) -{ - early_init_this_mmu(); -} - -void setup_initial_memory_limit(phys_addr_t first_memblock_base, - phys_addr_t first_memblock_size) -{ - /* On non-FSL Embedded 64-bit, we adjust the RMA size to match - * the bolted TLB entry. We know for now that only 1G - * entries are supported though that may eventually - * change. - * - * on FSL Embedded 64-bit, usually all RAM is bolted, but with - * unusual memory sizes it's possible for some RAM to not be mapped - * (such RAM is not used at all by Linux, since we don't support - * highmem on 64-bit). We limit ppc64_rma_size to what would be - * mappable if this memblock is the only one. Additional memblocks - * can only increase, not decrease, the amount that ends up getting - * mapped. We still limit max to 1G even if we'll eventually map - * more. This is due to what the early init code is set up to do. - * - * We crop it to the size of the first MEMBLOCK to - * avoid going over total available memory just in case... - */ -#ifdef CONFIG_PPC_E500 - if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - unsigned long linear_sz; - unsigned int num_cams; - - /* use a quarter of the TLBCAM for bolted linear map */ - num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; - - linear_sz = map_mem_in_cams(first_memblock_size, num_cams, - true, true); - - ppc64_rma_size = min_t(u64, linear_sz, 0x40000000); - } else -#endif - ppc64_rma_size = min_t(u64, first_memblock_size, 0x40000000); - - /* Finally limit subsequent allocations */ - memblock_set_current_limit(first_memblock_base + ppc64_rma_size); -} -#else /* ! CONFIG_PPC64 */ +#ifndef CONFIG_PPC64 void __init early_init_mmu(void) { unsigned long root = of_get_flat_dt_root(); diff --git a/arch/powerpc/mm/nohash/tlb.c b/arch/powerpc/mm/nohash/tlb_64e.c similarity index 56% copy from arch/powerpc/mm/nohash/tlb.c copy to arch/powerpc/mm/nohash/tlb_64e.c index a5bb87ec8578..7d5506d23eab 100644 --- a/arch/powerpc/mm/nohash/tlb.c +++ b/arch/powerpc/mm/nohash/tlb_64e.c @@ -1,13 +1,5 @@ // SPDX-License-Identifier: GPL-2.0-or-later /* - * This file contains the routines for TLB flushing. - * On machines where the MMU does not use a hash table to store virtual to - * physical translations (ie, SW loaded TLBs or Book3E compilant processors, - * this does -not- include 603 however which shares the implementation with - * hash based processors) - * - * -- BenH - * * Copyright 2008,2009 Ben Herrenschmidt * IBM Corp. * @@ -26,96 +18,21 @@ #include #include #include -#include #include -#include -#include #include -#include -#include #include #include #include #include #include -#include -#include #include -/* - * This struct lists the sw-supported page sizes. The hardawre MMU may support - * other sizes not listed here. The .ind field is only used on MMUs that have - * indirect page table entries. - */ -#ifdef CONFIG_PPC_E500 -struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { - [MMU_PAGE_4K] = { - .shift = 12, - .enc = BOOK3E_PAGESZ_4K, - }, - [MMU_PAGE_2M] = { - .shift = 21, - .enc = BOOK3E_PAGESZ_2M, - }, - [MMU_PAGE_4M] = { - .shift = 22, - .enc = BOOK3E_PAGESZ_4M, - }, - [MMU_PAGE_16M] = { - .shift = 24, - .enc = BOOK3E_PAGESZ_16M, - }, - [MMU_PAGE_64M] = { - .shift = 26, - .enc = BOOK3E_PAGESZ_64M, - }, - [MMU_PAGE_256M] = { - .shift = 28, - .enc = BOOK3E_PAGESZ_256M, - }, - [MMU_PAGE_1G] = { - .shift = 30, - .enc = BOOK3E_PAGESZ_1GB, - }, -}; - -static inline int mmu_get_tsize(int psize) -{ - return mmu_psize_defs[psize].enc; -} -#else -static inline int mmu_get_tsize(int psize) -{ - /* This isn't used on !Book3E for now */ - return 0; -} -#endif - -#ifdef CONFIG_PPC_8xx -struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { - [MMU_PAGE_4K] = { - .shift = 12, - }, - [MMU_PAGE_16K] = { - .shift = 14, - }, - [MMU_PAGE_512K] = { - .shift = 19, - }, - [MMU_PAGE_8M] = { - .shift = 23, - }, -}; -#endif - /* The variables below are currently only used on 64-bit Book3E * though this will probably be made common with other nohash * implementations at some point */ -#ifdef CONFIG_PPC64 - int mmu_pte_psize; /* Page size used for PTE pages */ int mmu_vmemmap_psize; /* Page size used for the virtual mem map */ int book3e_htw_mode; /* HW tablewalk? Value is PPC_HTW_* */ @@ -130,241 +47,6 @@ unsigned long linear_map_top; /* Top of linear mapping */ */ int extlb_level_exc; -#endif /* CONFIG_PPC64 */ - -#ifdef CONFIG_PPC_E500 -/* next_tlbcam_idx is used to round-robin tlbcam entry assignment */ -DEFINE_PER_CPU(int, next_tlbcam_idx); -EXPORT_PER_CPU_SYMBOL(next_tlbcam_idx); -#endif - -/* - * Base TLB flushing operations: - * - * - flush_tlb_mm(mm) flushes the specified mm context TLB's - * - flush_tlb_page(vma, vmaddr) flushes one page - * - flush_tlb_range(vma, start, end) flushes a range of pages - * - flush_tlb_kernel_range(start, end) flushes kernel pages - * - * - local_* variants of page and mm only apply to the current - * processor - */ - -#ifndef CONFIG_PPC_8xx -/* - * These are the base non-SMP variants of page and mm flushing - */ -void local_flush_tlb_mm(struct mm_struct *mm) -{ - unsigned int pid; - - preempt_disable(); - pid = mm->context.id; - if (pid != MMU_NO_CONTEXT) - _tlbil_pid(pid); - preempt_enable(); -} -EXPORT_SYMBOL(local_flush_tlb_mm); - -void __local_flush_tlb_page(struct mm_struct *mm, unsigned long vmaddr, - int tsize, int ind) -{ - unsigned int pid; - - preempt_disable(); - pid = mm ? mm->context.id : 0; - if (pid != MMU_NO_CONTEXT) - _tlbil_va(vmaddr, pid, tsize, ind); - preempt_enable(); -} - -void local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr) -{ - __local_flush_tlb_page(vma ? vma->vm_mm : NULL, vmaddr, - mmu_get_tsize(mmu_virtual_psize), 0); -} -EXPORT_SYMBOL(local_flush_tlb_page); - -void local_flush_tlb_page_psize(struct mm_struct *mm, - unsigned long vmaddr, int psize) -{ - __local_flush_tlb_page(mm, vmaddr, mmu_get_tsize(psize), 0); -} -EXPORT_SYMBOL(local_flush_tlb_page_psize); - -#endif - -/* - * And here are the SMP non-local implementations - */ -#ifdef CONFIG_SMP - -static DEFINE_RAW_SPINLOCK(tlbivax_lock); - -struct tlb_flush_param { - unsigned long addr; - unsigned int pid; - unsigned int tsize; - unsigned int ind; -}; - -static void do_flush_tlb_mm_ipi(void *param) -{ - struct tlb_flush_param *p = param; - - _tlbil_pid(p ? p->pid : 0); -} - -static void do_flush_tlb_page_ipi(void *param) -{ - struct tlb_flush_param *p = param; - - _tlbil_va(p->addr, p->pid, p->tsize, p->ind); -} - - -/* Note on invalidations and PID: - * - * We snapshot the PID with preempt disabled. At this point, it can still - * change either because: - * - our context is being stolen (PID -> NO_CONTEXT) on another CPU - * - we are invaliating some target that isn't currently running here - * and is concurrently acquiring a new PID on another CPU - * - some other CPU is re-acquiring a lost PID for this mm - * etc... - * - * However, this shouldn't be a problem as we only guarantee - * invalidation of TLB entries present prior to this call, so we - * don't care about the PID changing, and invalidating a stale PID - * is generally harmless. - */ - -void flush_tlb_mm(struct mm_struct *mm) -{ - unsigned int pid; - - preempt_disable(); - pid = mm->context.id; - if (unlikely(pid == MMU_NO_CONTEXT)) - goto no_context; - if (!mm_is_core_local(mm)) { - struct tlb_flush_param p = { .pid = pid }; - /* Ignores smp_processor_id() even if set. */ - smp_call_function_many(mm_cpumask(mm), - do_flush_tlb_mm_ipi, &p, 1); - } - _tlbil_pid(pid); - no_context: - preempt_enable(); -} -EXPORT_SYMBOL(flush_tlb_mm); - -void __flush_tlb_page(struct mm_struct *mm, unsigned long vmaddr, - int tsize, int ind) -{ - struct cpumask *cpu_mask; - unsigned int pid; - - /* - * This function as well as __local_flush_tlb_page() must only be called - * for user contexts. - */ - if (WARN_ON(!mm)) - return; - - preempt_disable(); - pid = mm->context.id; - if (unlikely(pid == MMU_NO_CONTEXT)) - goto bail; - cpu_mask = mm_cpumask(mm); - if (!mm_is_core_local(mm)) { - /* If broadcast tlbivax is supported, use it */ - if (mmu_has_feature(MMU_FTR_USE_TLBIVAX_BCAST)) { - int lock = mmu_has_feature(MMU_FTR_LOCK_BCAST_INVAL); - if (lock) - raw_spin_lock(&tlbivax_lock); - _tlbivax_bcast(vmaddr, pid, tsize, ind); - if (lock) - raw_spin_unlock(&tlbivax_lock); - goto bail; - } else { - struct tlb_flush_param p = { - .pid = pid, - .addr = vmaddr, - .tsize = tsize, - .ind = ind, - }; - /* Ignores smp_processor_id() even if set in cpu_mask */ - smp_call_function_many(cpu_mask, - do_flush_tlb_page_ipi, &p, 1); - } - } - _tlbil_va(vmaddr, pid, tsize, ind); - bail: - preempt_enable(); -} - -void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr) -{ -#ifdef CONFIG_HUGETLB_PAGE - if (vma && is_vm_hugetlb_page(vma)) - flush_hugetlb_page(vma, vmaddr); -#endif - - __flush_tlb_page(vma ? vma->vm_mm : NULL, vmaddr, - mmu_get_tsize(mmu_virtual_psize), 0); -} -EXPORT_SYMBOL(flush_tlb_page); - -#endif /* CONFIG_SMP */ - -/* - * Flush kernel TLB entries in the given range - */ -#ifndef CONFIG_PPC_8xx -void flush_tlb_kernel_range(unsigned long start, unsigned long end) -{ -#ifdef CONFIG_SMP - preempt_disable(); - smp_call_function(do_flush_tlb_mm_ipi, NULL, 1); - _tlbil_pid(0); - preempt_enable(); -#else - _tlbil_pid(0); -#endif -} -EXPORT_SYMBOL(flush_tlb_kernel_range); -#endif - -/* - * Currently, for range flushing, we just do a full mm flush. This should - * be optimized based on a threshold on the size of the range, since - * some implementation can stack multiple tlbivax before a tlbsync but - * for now, we keep it that way - */ -void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end) - -{ - if (end - start == PAGE_SIZE && !(start & ~PAGE_MASK)) - flush_tlb_page(vma, start); - else - flush_tlb_mm(vma->vm_mm); -} -EXPORT_SYMBOL(flush_tlb_range); - -void tlb_flush(struct mmu_gather *tlb) -{ - flush_tlb_mm(tlb->mm); -} - -/* - * Below are functions specific to the 64-bit variant of Book3E though that - * may change in the future - */ - -#ifdef CONFIG_PPC64 - /* * Handling of virtual linear page tables or indirect TLB entries * flushing when PTE pages are freed @@ -677,13 +359,3 @@ void setup_initial_memory_limit(phys_addr_t first_memblock_base, /* Finally limit subsequent allocations */ memblock_set_current_limit(first_memblock_base + ppc64_rma_size); } -#else /* ! CONFIG_PPC64 */ -void __init early_init_mmu(void) -{ - unsigned long root = of_get_flat_dt_root(); - - if (IS_ENABLED(CONFIG_PPC_47x) && IS_ENABLED(CONFIG_SMP) && - of_get_flat_dt_prop(root, "cooperative-partition", NULL)) - mmu_clear_feature(MMU_FTR_USE_TLBIVAX_BCAST); -} -#endif /* CONFIG_PPC64 */ From patchwork Tue Jul 2 13:51:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BF55C3064D for ; Tue, 2 Jul 2024 13:51:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A20876B00AB; Tue, 2 Jul 2024 09:51:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A88D6B00AC; Tue, 2 Jul 2024 09:51:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 849116B00AD; Tue, 2 Jul 2024 09:51:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 606186B00AB for ; Tue, 2 Jul 2024 09:51:43 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C73CBA00AE for ; Tue, 2 Jul 2024 13:51:42 +0000 (UTC) X-FDA: 82294950444.18.30A6356 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf03.hostedemail.com (Postfix) with ESMTP id 8ECBF2001C for ; Tue, 2 Jul 2024 13:51:40 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ysfc+qqhBaHzes/1dOaOQS2KSsFUBeZujVhXDSK2WXI=; b=SZNN9UWK5nwbD43Ml+u6aFhsQxPRmrL8I07Y+Pk9XAxfOGx9NnKzatS15G0U4SyO7J5b6O zeAPgIx6MVxmSQCFsDldEkfMDNiY7SKoQT124n1rZJJ0WX60eCxA9Mt78Nip9mjTin//jO 9K0v3xE/MIRoe2W9rS/wBfukVRkoNAk= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928290; a=rsa-sha256; cv=none; b=YVY7UIEEzBoe1W3laKnDGiopbFu/ftHFLZAy8qiwa1K/2l5eqLSH2ImQYABh3du50thHva zUKiq03abqYeYclBzwpfSeFTlMR1Z72JPm3lh74J+f6/lGMUOcgW3yhIcUzcgDQYqd70aZ XLiYCpBgk7GYPl7auNVQyjgKIH58++g= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4801J1Zz9v4H; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FQWF2oZxyNDX; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD47y5PTYz9tZl; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id B4B878B774; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id Fl5X96vf0VcP; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 2C3A78B775; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 03/23] powerpc/64e: Drop E500 ifdefs in 64-bit code Date: Tue, 2 Jul 2024 15:51:15 +0200 Message-ID: <7fb88809c88a1b774063eda602a9333079403f83.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=3405; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=A/9Ib78BLDVf4ZZ6WCRXnSh8JY4Xq2JsVQD062iffmM=; b=g9oCLvuqYm4sq/y4JjtDVo3V9hIsS7weLayXFdlnj+TrEWskwNYdes3hLWc80t0ePYH+p9ExO LGaYGNYQoajApTZiUVIq8OcpmyIAgR+cOjb67IdwEgpXqmtVIndKL3t X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8ECBF2001C X-Stat-Signature: wbu81cab5qmoeqcuwpbsi8s3fmu716pw X-HE-Tag: 1719928300-345742 X-HE-Meta: U2FsdGVkX1+1EV99kQJDJ+Dg0WYvT4HeDs+KvoP4LdI9AaRRq8lUzsmRjhV4gGx1vBTUqGbBUIL6Y/W7pdVJOepP0a0F+ZUvvjf9LrQLJbTOj4VT14e6apB3Uz+z7yDCEiruN8QoUSg8xQyGFmu9prhKYn3oERflyzF6DbglmJlEILX4ClZOmuDtobU4VjXYTibk1WDTOTCvS+SAw6VugyBFpENugqHLq43sqjGLwiayZ9XZjBWdM/+n32y5K/q51Gpa2yYZEJRBNYqKz8KNFxmo1OLj/T1aqsgDwCgMAn9Hh20aw5NQWYoT0by8wdyWwnGHh94AiYG2rKRZjNoRTnANi6yPH/9hUIfbh3CJf0pkv9JU6WChcWVIclA2reqH4/FjjtYn4q7DoR4ulr7Q81WDVserG9eDqOJUQ2qTJF3r2VYmAig+oz4MHb2a/p6AVnxRV2hs8xkeiY4N133VqDSLFwNkjk2JlhlKB6hEBKLhG/gQdaNsIOzvQfK05IWb9RZm78D78ejpu+YYsGjcXpDtBxXrYaGv2e8ufcGPDM/HaYcAhDaW5mZaB4gMXHqhv20suCyRrkDTEgygnyU6qC8DRXyaaaBJxtpOaPwM0GbHfilix8b3gwHWiSszybS/JdB1hiwQJK3PgtgzWMysV+h61X+aLL246/Uk8ww6X87d8ahatnZ1Z3wRSGj85hKq94nJLgXVcYYnqDsgQpaS+k+0LYv9nwTjUZZw0VJ/ukzUSmBE8vjs0A4SwHOzmxBUz8efXMOacd8Xblum7e8gqckj8ljQxu8YUgNDbbziZBR+94bQzX+v/cGjsI+gX4+kff/1uMYCDchEUsyLNi7ERjfq1vbM04kiHiRvaWM0KIVFmC0w5CYouusT0O19z6/BUk3FBh8VHUdatpSk49nqKCopOV6s7f2d9spNbsKLAOURqIVCnY/SSgXI7MvfovPd5QNdWiJCakybrLUPuqy xnlTMxPF gEHOSaB3naV3wlxP2rWzZbm/tj1Mh/I2Eaz7BWnKNJESnfaX9HCSjP3hcVorOZEQ51avElR17z6695wFIYqvF/aged9khuo7u8lo81RGkeBzMn65xISuL2ik364xaredNEOfr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Michael Ellerman All 64-bit Book3E have E500=y, so drop the unneeded ifdefs. Signed-off-by: Michael Ellerman Signed-off-by: Christophe Leroy --- arch/powerpc/mm/nohash/tlb_64e.c | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/arch/powerpc/mm/nohash/tlb_64e.c b/arch/powerpc/mm/nohash/tlb_64e.c index 7d5506d23eab..9db85ee9ba5b 100644 --- a/arch/powerpc/mm/nohash/tlb_64e.c +++ b/arch/powerpc/mm/nohash/tlb_64e.c @@ -85,7 +85,6 @@ static void __init setup_page_sizes(void) unsigned int eptcfg; int psize; -#ifdef CONFIG_PPC_E500 unsigned int mmucfg = mfspr(SPRN_MMUCFG); int fsl_mmu = mmu_has_feature(MMU_FTR_TYPE_FSL_E); @@ -151,7 +150,6 @@ static void __init setup_page_sizes(void) goto out; } -#endif out: /* Cleanup array and print summary */ pr_info("MMU: Supported page sizes\n"); @@ -180,13 +178,11 @@ static void __init setup_mmu_htw(void) */ switch (book3e_htw_mode) { -#ifdef CONFIG_PPC_E500 case PPC_HTW_E6500: extlb_level_exc = EX_TLB_SIZE; patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e); patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e); break; -#endif } pr_info("MMU: Book3E HW tablewalk %s\n", book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported"); @@ -217,7 +213,6 @@ static void early_init_this_mmu(void) } mtspr(SPRN_MAS4, mas4); -#ifdef CONFIG_PPC_E500 if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { unsigned int num_cams; bool map = true; @@ -238,7 +233,6 @@ static void early_init_this_mmu(void) linear_map_top = map_mem_in_cams(linear_map_top, num_cams, false, true); } -#endif /* A sync won't hurt us after mucking around with * the MMU configuration @@ -270,7 +264,6 @@ static void __init early_init_mmu_global(void) /* Look for HW tablewalk support */ setup_mmu_htw(); -#ifdef CONFIG_PPC_E500 if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { if (book3e_htw_mode == PPC_HTW_NONE) { extlb_level_exc = EX_TLB_SIZE; @@ -279,7 +272,6 @@ static void __init early_init_mmu_global(void) exc_instruction_tlb_miss_bolted_book3e); } } -#endif /* Set the global containing the top of the linear mapping * for use by the TLB miss code @@ -291,7 +283,6 @@ static void __init early_init_mmu_global(void) static void __init early_mmu_set_memory_limit(void) { -#ifdef CONFIG_PPC_E500 if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { /* * Limit memory so we dont have linear faults. @@ -302,7 +293,6 @@ static void __init early_mmu_set_memory_limit(void) */ memblock_enforce_memory_limit(linear_map_top); } -#endif memblock_set_current_limit(linear_map_top); } @@ -340,7 +330,6 @@ void setup_initial_memory_limit(phys_addr_t first_memblock_base, * We crop it to the size of the first MEMBLOCK to * avoid going over total available memory just in case... */ -#ifdef CONFIG_PPC_E500 if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { unsigned long linear_sz; unsigned int num_cams; @@ -353,7 +342,6 @@ void setup_initial_memory_limit(phys_addr_t first_memblock_base, ppc64_rma_size = min_t(u64, linear_sz, 0x40000000); } else -#endif ppc64_rma_size = min_t(u64, first_memblock_size, 0x40000000); /* Finally limit subsequent allocations */ From patchwork Tue Jul 2 13:51:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E580CC3064D for ; Tue, 2 Jul 2024 13:51:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D5FB6B00AE; Tue, 2 Jul 2024 09:51:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 784F66B00AF; Tue, 2 Jul 2024 09:51:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 626076B00B0; Tue, 2 Jul 2024 09:51:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4292A6B00AE for ; Tue, 2 Jul 2024 09:51:48 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E7A1E80115 for ; Tue, 2 Jul 2024 13:51:47 +0000 (UTC) X-FDA: 82294950654.17.718300F Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf01.hostedemail.com (Postfix) with ESMTP id B3B2640027 for ; Tue, 2 Jul 2024 13:51:45 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928288; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7/48v/a77YwotJqYJrVKRmh2P0K5CzoKbtbXkuxJb0k=; b=Po3uxjzOG1NZTsUoYKpXP000Rr1SAwUbExCYK3cmVKxJ6usbiNQAWJVSOt03o9IYjGfzJ9 WNASubqU0t+panzQAaKwlBXiEkcNlQa0vnakKroiy2ZBGrUU8coc/qRyRCiJ3Hvhs0bFcW Yax2taWiswYdMd5yYx40p2X7AIwvNKw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928288; a=rsa-sha256; cv=none; b=Eb6oGXnirXNMNafzILzk6SOuG4xdrSYYjBnZPmQp76CrxpW9B1nyqr8IbMv7gFD0BpDTI2 bphutoNzuN67CX3HpU3z8opMn6pRjqydgVtBENworgGPrFUfMxNlhuJtdd6pyi/dgMNlgp 7sQ88D2OUZ6W1NcligF4ePoRsq3Gxt0= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4814BXPz9v53; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 78TtuBPh8kIy; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD47z1qyLz9v2F; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 3B2288B775; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id dm-SyRIO-rOw; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id AE97B8B764; Tue, 2 Jul 2024 15:51:26 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 04/23] powerpc/64e: Drop MMU_FTR_TYPE_FSL_E checks in 64-bit code Date: Tue, 2 Jul 2024 15:51:16 +0200 Message-ID: <2b0b0bc9752e6cece222e4e2050358da70bb631d.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=7160; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=AHd2YOhqPsn0mmn3bUFrvBRmVpZgpV/7pvx5aQBWlRU=; b=tKXb2Ri9MHAabB06bOogkDY8PwgXqrNoDZ6kt50SucSZZARu44dcAwNms05x+kSodfbaW2D4G t7lEYaww4UWDUzsc8I8rnfzUghQ8IVRw8mRZXYLuxMQlBRIYFdUaP8c X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Stat-Signature: r6kar7fh3o6rxi14sgjxbzuz5sh4w67i X-Rspam-User: X-Rspamd-Queue-Id: B3B2640027 X-Rspamd-Server: rspam02 X-HE-Tag: 1719928305-26164 X-HE-Meta: U2FsdGVkX19Zf6u8XLq/auOKlwibZZQtF9Zo98gMbsYHMPsRZIu0pU0U1FlkDvemW/XzZEWrLfAxf2hi69h0EpARSyN4DPg0qiKibDFwGvLRM5zpUT846UvzUU4gUNMIC+Z4qMgF+h6GTcKZoFfuqfz6/9TMVhBxAbycloLXU5/FjqN5X3QS+K/Mar/vggApN012hLtpA0GkPGM9wh3tlYYkqU8Jt/J46zsh6eLnE3gU3x3mXu9O05AQjPOpzi45yb07nGUxmRBeCdaCqe46v8rLdE8/oCEOJqecu8GG6GxNCe9CFBH2pTwAZJPnhgEQ8+Xqu4L9Cbgqs1oZgCSaP2cgQ5oXQ0XKAWyGuixoJELO5VKTHnkl7Ck66jK21M1VnUr0xFTEm6lJeq5kyRn7EKgyLw/Dp0R+FnZ2zrjfrWA60cdJ15BN+sDdIPHFMHGQwNJX5VUL8q1VKBeOYVLkg6Sz2cIM0MMZfbEO9U/SS6wVg+22xjsBdUVl8Nv25MwmDFPx/CKEhfuwGx2R8Lrb/M284wRbL7YOQPjl4SSHd2BNkj6oLDPLaKtTDk0aIJP2VIDVF5ymNtN+RhtvdmcMJpRQRIL5/rNCsGeTj9/+8X1pKokq7Xext008RG1hPOcGrLSutvRKpvUp/AJLaf97WjC8g8zwFUvU+Mg8cNK1Gg4bYUzJ54ThxxXePHa4nNy1rPLhLmCtxBT54Z+X3MG1pFWbY7kPb4MM/w77alvzbpK/vbNjN9ocgj7UJ9TE+RJaDzYMH/eiyhRk90Wub+xj70/nDm7brrROttkoBRfv177Uvdx4mAw5Co4AgQjKi2RGV+CjxiSz1G3eLePcGGLMi8xSV1WhMprP0j4uux5JReaBXAMYMnQVWd35x9OuSWyKSKpsSf6p1joSCy764Dzx95O71v6u8Nhwug5ImDCJocspeNdMKg/txfRu7rpNik7RDmNqrkZZ+wcyalEar3p rJiKDRO4 Mwjcz4vcWXVF9prJk1kfA/iSZW8Eq+Y+iRnNZ5KgbCpyqbRVDxFBxsaBg4bb2AUbUOYex5BGjoFqQVx2Tkxe0tDaProz9zSeqtSJAF2tB/i7T2jgC/nLtJwS84RMMgORk5sj8+lV9wZ7Cgh6fVBo0Eh2iCZy1C4sKPTcwdFB0zZvu2+k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Michael Ellerman All 64-bit Book3E have MMU_FTR_TYPE_FSL_E, since A2 was removed, so remove checks for it in 64-bit only code. Signed-off-by: Michael Ellerman Signed-off-by: Christophe Leroy --- arch/powerpc/kernel/setup_64.c | 6 +- arch/powerpc/mm/nohash/tlb_64e.c | 97 ++++++++++++-------------------- 2 files changed, 38 insertions(+), 65 deletions(-) diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c index ae36a129789f..22f83fbbc762 100644 --- a/arch/powerpc/kernel/setup_64.c +++ b/arch/powerpc/kernel/setup_64.c @@ -696,11 +696,7 @@ __init u64 ppc64_bolted_size(void) { #ifdef CONFIG_PPC_BOOK3E_64 /* Freescale BookE bolts the entire linear mapping */ - /* XXX: BookE ppc64_rma_limit setup seems to disagree? */ - if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E)) - return linear_map_top; - /* Other BookE, we assume the first GB is bolted */ - return 1ul << 30; + return linear_map_top; #else /* BookS radix, does not take faults on linear mapping */ if (early_radix_enabled()) diff --git a/arch/powerpc/mm/nohash/tlb_64e.c b/arch/powerpc/mm/nohash/tlb_64e.c index 9db85ee9ba5b..21c4b2442fcf 100644 --- a/arch/powerpc/mm/nohash/tlb_64e.c +++ b/arch/powerpc/mm/nohash/tlb_64e.c @@ -86,9 +86,8 @@ static void __init setup_page_sizes(void) int psize; unsigned int mmucfg = mfspr(SPRN_MMUCFG); - int fsl_mmu = mmu_has_feature(MMU_FTR_TYPE_FSL_E); - if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V1) { + if ((mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V1) { unsigned int tlb1cfg = mfspr(SPRN_TLB1CFG); unsigned int min_pg, max_pg; @@ -115,7 +114,7 @@ static void __init setup_page_sizes(void) goto out; } - if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V2) { + if ((mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V2) { u32 tlb1cfg, tlb1ps; tlb0cfg = mfspr(SPRN_TLB0CFG); @@ -213,26 +212,24 @@ static void early_init_this_mmu(void) } mtspr(SPRN_MAS4, mas4); - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - unsigned int num_cams; - bool map = true; + unsigned int num_cams; + bool map = true; - /* use a quarter of the TLBCAM for bolted linear map */ - num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; + /* use a quarter of the TLBCAM for bolted linear map */ + num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; - /* - * Only do the mapping once per core, or else the - * transient mapping would cause problems. - */ + /* + * Only do the mapping once per core, or else the + * transient mapping would cause problems. + */ #ifdef CONFIG_SMP - if (hweight32(get_tensr()) > 1) - map = false; + if (hweight32(get_tensr()) > 1) + map = false; #endif - if (map) - linear_map_top = map_mem_in_cams(linear_map_top, - num_cams, false, true); - } + if (map) + linear_map_top = map_mem_in_cams(linear_map_top, + num_cams, false, true); /* A sync won't hurt us after mucking around with * the MMU configuration @@ -242,16 +239,10 @@ static void early_init_this_mmu(void) static void __init early_init_mmu_global(void) { - /* XXX This should be decided at runtime based on supported - * page sizes in the TLB, but for now let's assume 16M is - * always there and a good fit (which it probably is) - * + /* * Freescale booke only supports 4K pages in TLB0, so use that. */ - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) - mmu_vmemmap_psize = MMU_PAGE_4K; - else - mmu_vmemmap_psize = MMU_PAGE_16M; + mmu_vmemmap_psize = MMU_PAGE_4K; /* XXX This code only checks for TLB 0 capabilities and doesn't * check what page size combos are supported by the HW. It @@ -264,13 +255,10 @@ static void __init early_init_mmu_global(void) /* Look for HW tablewalk support */ setup_mmu_htw(); - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - if (book3e_htw_mode == PPC_HTW_NONE) { - extlb_level_exc = EX_TLB_SIZE; - patch_exception(0x1c0, exc_data_tlb_miss_bolted_book3e); - patch_exception(0x1e0, - exc_instruction_tlb_miss_bolted_book3e); - } + if (book3e_htw_mode == PPC_HTW_NONE) { + extlb_level_exc = EX_TLB_SIZE; + patch_exception(0x1c0, exc_data_tlb_miss_bolted_book3e); + patch_exception(0x1e0, exc_instruction_tlb_miss_bolted_book3e); } /* Set the global containing the top of the linear mapping @@ -283,16 +271,14 @@ static void __init early_init_mmu_global(void) static void __init early_mmu_set_memory_limit(void) { - if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - /* - * Limit memory so we dont have linear faults. - * Unlike memblock_set_current_limit, which limits - * memory available during early boot, this permanently - * reduces the memory available to Linux. We need to - * do this because highmem is not supported on 64-bit. - */ - memblock_enforce_memory_limit(linear_map_top); - } + /* + * Limit memory so we dont have linear faults. + * Unlike memblock_set_current_limit, which limits + * memory available during early boot, this permanently + * reduces the memory available to Linux. We need to + * do this because highmem is not supported on 64-bit. + */ + memblock_enforce_memory_limit(linear_map_top); memblock_set_current_limit(linear_map_top); } @@ -313,12 +299,8 @@ void early_init_mmu_secondary(void) void setup_initial_memory_limit(phys_addr_t first_memblock_base, phys_addr_t first_memblock_size) { - /* On non-FSL Embedded 64-bit, we adjust the RMA size to match - * the bolted TLB entry. We know for now that only 1G - * entries are supported though that may eventually - * change. - * - * on FSL Embedded 64-bit, usually all RAM is bolted, but with + /* + * On FSL Embedded 64-bit, usually all RAM is bolted, but with * unusual memory sizes it's possible for some RAM to not be mapped * (such RAM is not used at all by Linux, since we don't support * highmem on 64-bit). We limit ppc64_rma_size to what would be @@ -330,19 +312,14 @@ void setup_initial_memory_limit(phys_addr_t first_memblock_base, * We crop it to the size of the first MEMBLOCK to * avoid going over total available memory just in case... */ - if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { - unsigned long linear_sz; - unsigned int num_cams; - - /* use a quarter of the TLBCAM for bolted linear map */ - num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; + unsigned long linear_sz; + unsigned int num_cams; - linear_sz = map_mem_in_cams(first_memblock_size, num_cams, - true, true); + /* use a quarter of the TLBCAM for bolted linear map */ + num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; - ppc64_rma_size = min_t(u64, linear_sz, 0x40000000); - } else - ppc64_rma_size = min_t(u64, first_memblock_size, 0x40000000); + linear_sz = map_mem_in_cams(first_memblock_size, num_cams, true, true); + ppc64_rma_size = min_t(u64, linear_sz, 0x40000000); /* Finally limit subsequent allocations */ memblock_set_current_limit(first_memblock_base + ppc64_rma_size); From patchwork Tue Jul 2 13:51:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D229C3064D for ; Tue, 2 Jul 2024 13:51:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E65846B00B1; Tue, 2 Jul 2024 09:51:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E15686B00B2; Tue, 2 Jul 2024 09:51:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB5E46B00B3; Tue, 2 Jul 2024 09:51:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id ABEED6B00B1 for ; Tue, 2 Jul 2024 09:51:52 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6661DA1654 for ; Tue, 2 Jul 2024 13:51:52 +0000 (UTC) X-FDA: 82294950864.29.D6B8FA6 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf14.hostedemail.com (Postfix) with ESMTP id 4975410001B for ; Tue, 2 Jul 2024 13:51:49 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C7m8kUpZbDsAOtsrX1X8SMQPKmaS3Q/DXEHWvtxU3i8=; b=eeLcOByNQKVp6nUvs186zcW7I0GPIl+PePxYHYVZy+ZeNPx3bAg49Mwnp4eGqSlTYrQY10 y80TE49bkeYv4FazGxKN5oVi+qJlqPqQZObeZd/gZCBdn5tHp+8kPq+oQC17V9VPmwDhQu tQOy23NZ0YUrzEpUDe9H6h6cG885HFc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928299; a=rsa-sha256; cv=none; b=n+pgyKGobfu18pU6ce62P8/i2oZUJvxARED828Oq9F9+Wn2Xz3lZD8fq4AuLF7gBm9+0BO Kdcs3zZVB0YQKolxK5RtI6n+dxKOk5vdsKZo5w5/h9581p6TCIX8zzkBQ0rQg45y2hHri0 lvuVZ0/psoBcMmNh4XR3cTqMv0yZil4= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48240Jwz9v5V; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xDCoEvjglLhs; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD47z5CPhz9v49; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id AE8408B764; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id NEcuTvm6G5kf; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 38AE58B774; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 05/23] powerpc/64e: Consolidate TLB miss handler patching Date: Tue, 2 Jul 2024 15:51:17 +0200 Message-ID: <7033b37493fb48a3e5245b59d0a42afb75dabfc1.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=2333; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=yAiKCEuhEewOEpS5oDWNvn73wPUuV/itXZW+PYTcfsA=; b=TDg00DJl8BMpbdKmH0JPgZA0p/L2BdM0sYrVLzMQBcfw6y8h3qPYgMNYRUjb/jh4PmJE9dAWH QW9L/WqQjhuBNIUq6oDbJDlVXJMprhCOVas0KOxpHwHtq5wt8x/4/ao X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4975410001B X-Stat-Signature: dtccmn7id8stppu38oo31c5x3chiexuc X-HE-Tag: 1719928309-974244 X-HE-Meta: U2FsdGVkX1+tkIrd5jv4mmmJ/8OEY/mZygF51M10dpTf+qGygAYRFEX03tl/JZpKJ5Hfdu+WwitkIUwnbDGHnGJoGZ5RgER/DiWkmR0/3nx2ve0xLWYAnN+sH5g+zSyr9GC85byyDsLy82g+MAIAWIwEfAr4nyDytgippiYIZcuq83vSguuRaUc1Jcht73NibxntIbiv8FHbvpENyydjpmdVEnvNS7gWPd3anXgnX3wEBv4lYS+0WFdg1fE/QRIM8XP2AQ33zN8N19tA8olFjRhaiQiSJ4QOnVjMHZfZxGRCGA7v7zdVRbM1zrhjAbVZCfiKt+teskWh2fAftio2+1xKrFomx23nHMCrByTrBGzfm0f7VIezIGwd5sxNu7zdglLjMbiHJdmih6rXW7AzXrbl9uGRzbGo+53fIt1lYczJhDhZzMpMowRyKJ5qQE+raCApxvrsM/kjSziXM/KvlPmaNTil10XpZiEyzd+8gTLICloAClZv8zt+3SV2Jq2VOBQ3iJAJVIGW7fsVPzvkMy6FdNM4I6tb2wnigrHspm+gEDrEd9vJzF6hB+YNbgoAGO61FItb4SY5b9CtU766r2JAHtHkaOHniDYTPU2Oc9PgztIBrhQNjxFdZI4N/ItD2T5/69Dy1EuiDL9Kk8xLGYjEi5GA/Wnib7yOd8gB7KKRm+pzJVeYR6vVgCoB7SqkApQ1JFLilCy78PckqySS4vGCMSZiauzl+qwQpl8/664jptily4WVyM6s+8KKDL1qMiUDVOiubmYCGSDnU26wYPkMHCzq7PKwlECM3VNOla7fl+bOVWQrErQz0czHBuqu92GFXCJflJk1T44JOhihAQ0bUeW9ILqrd3Oo8jrvNMe2Kt6YpOf2Euu0TmKPx+qmOtnsjTgrz7M/9f3Uhd0mY8HkiBT7z07jEwW6fuIjAnSduKqv74TlEEuiljtGvVCu/sRBCEDsULZwjFy5V2K WogPyEPt PNwfKruZcmpvUy8OHTTS7ePHUQ2V9D8hCqhBb8K75sz8V+zGD0zei9uYDokFjj3WKhmHRCzb6yQDxa4aotVt77m4ZH1IV4HLVIqt6spNyqx9OkNH/TY54qqjeJsRFBeZMneFhnDK+KNUl6Dd5pX98nmRMMC1cHAmb/JBRFNRg+FSJV1A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Michael Ellerman The 64e TLB miss handler patching is done in setup_mmu_htw(), and then again immediately afterward in early_init_mmu_global(). Consolidate it into a single location. Signed-off-by: Michael Ellerman Signed-off-by: Christophe Leroy --- arch/powerpc/mm/nohash/tlb_64e.c | 38 +++++++++++++------------------- 1 file changed, 15 insertions(+), 23 deletions(-) diff --git a/arch/powerpc/mm/nohash/tlb_64e.c b/arch/powerpc/mm/nohash/tlb_64e.c index 21c4b2442fcf..d83ecf466929 100644 --- a/arch/powerpc/mm/nohash/tlb_64e.c +++ b/arch/powerpc/mm/nohash/tlb_64e.c @@ -169,24 +169,6 @@ static void __init setup_page_sizes(void) } } -static void __init setup_mmu_htw(void) -{ - /* - * If we want to use HW tablewalk, enable it by patching the TLB miss - * handlers to branch to the one dedicated to it. - */ - - switch (book3e_htw_mode) { - case PPC_HTW_E6500: - extlb_level_exc = EX_TLB_SIZE; - patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e); - patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e); - break; - } - pr_info("MMU: Book3E HW tablewalk %s\n", - book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported"); -} - /* * Early initialization of the MMU TLB code */ @@ -252,15 +234,25 @@ static void __init early_init_mmu_global(void) /* Look for supported page sizes */ setup_page_sizes(); - /* Look for HW tablewalk support */ - setup_mmu_htw(); - - if (book3e_htw_mode == PPC_HTW_NONE) { - extlb_level_exc = EX_TLB_SIZE; + /* + * If we want to use HW tablewalk, enable it by patching the TLB miss + * handlers to branch to the one dedicated to it. + */ + extlb_level_exc = EX_TLB_SIZE; + switch (book3e_htw_mode) { + case PPC_HTW_E6500: + patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e); + patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e); + break; + case PPC_HTW_NONE: patch_exception(0x1c0, exc_data_tlb_miss_bolted_book3e); patch_exception(0x1e0, exc_instruction_tlb_miss_bolted_book3e); + break; } + pr_info("MMU: Book3E HW tablewalk %s\n", + book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported"); + /* Set the global containing the top of the linear mapping * for use by the TLB miss code */ From patchwork Tue Jul 2 13:51:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFEB4C3065C for ; Tue, 2 Jul 2024 13:51:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7399D6B00B4; Tue, 2 Jul 2024 09:51:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C23B6B00B5; Tue, 2 Jul 2024 09:51:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 562906B00B6; Tue, 2 Jul 2024 09:51:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 373766B00B4 for ; Tue, 2 Jul 2024 09:51:57 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EB4DE1401E3 for ; Tue, 2 Jul 2024 13:51:56 +0000 (UTC) X-FDA: 82294951032.19.64831A9 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf09.hostedemail.com (Postfix) with ESMTP id 8B5DA14002D for ; Tue, 2 Jul 2024 13:51:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928304; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NKHZ0bImctSrg/EOw9DVdGhu/MBnNV6znsiV9hAddjE=; b=wHua0sz2AmsoKLu+ByYOSWpbcS8KjF6mUmHFjGZM+TkmxOS9yV4vrL+X2FW4MYPPS+2CI1 4d+i+8l3E68U/4Jtm9m+9532Y55GzNgvPmAN8lqQ/STPAu+Rqp9rkRt56eMNYfUk4WzTmD RXOwmgC5Lu8/+BBJhNlPj1fYfXOBp6I= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928304; a=rsa-sha256; cv=none; b=GTfhKjwlXlz/y4IOBdWH71NYmDadpkaqhcnGtlXy2MqKWhJYiDFOHO2TNzxuQ/Z5cs7ke4 u7E0M71zScG2hCv0xiQLSVwhPp3ZKq5UQXBjMST5ygvqp0wAQC8dwoIosUSYxCKBtnp5lx pjfG8kQstIG/vcFuwReP7PcAze9lUys= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4830qQ2z9v5s; Tue, 2 Jul 2024 15:51:31 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yXLzMupCXH0Z; Tue, 2 Jul 2024 15:51:31 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD48029mkz9tZl; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 46A858B775; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id 0GJp4eWPpVPM; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id B35EA8B774; Tue, 2 Jul 2024 15:51:27 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 06/23] powerpc/64e: Drop unused TLB miss handlers Date: Tue, 2 Jul 2024 15:51:18 +0200 Message-ID: <9a670adc1771fb1871fba93ace5372f7eadc286f.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=10062; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=Glt8MetGkZpHonWm+UTOTXuByxTy4QiXsMnaTdX/Gdg=; b=Xx0/uOqOjqDQzJcyRR+vH6A7qJajwJgI7vhmQT/eRG3qWPzsdmcTL1GqAZdoU3r3mpLxKtBVP 1Nr825fx0WwAlYQYQ78s/ym4rNwOk6n4doE9+JmDCjHOlKcvGqgrAW4 X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8B5DA14002D X-Stat-Signature: 7tbzzrb8joeema7a3i14yc5hf55yjgzk X-HE-Tag: 1719928314-843095 X-HE-Meta: U2FsdGVkX1+BSF0RR8dfiw3GDFVW2bhb1ZTqE+/0siMWkqXhQoTxdPWg0/yFTfix/WpHamzxLnL0ufsOyhEQZSD+m+bBVlXyPWiRmEr7jMTtyqVU4bpoZrTPEcifSFM1zR7PRietfIzJz4ybgfMbmQmxZqA5AOEoBrwzkowXq2dohlTiwcHHI4Izyj/v85DgQfCoLu0Yge5EfdIDALkUDk4ytAHToWruJYxbWrX+HAXhGPf5ix47q8IxOaUit0KQI2rRCH6a2iY/U1/j506K/r9opeZDX+VnSatZYcHcNydyJAV504gIMLSuzu7sg859/i6Vp9VFHV2kBuCyj6N65i7pAXgdAoobxqDhKR1FuIaJWTex1mAix2g5wsoit0MTA/lJr21nEuhObOKoh7hLxjYHHyOV/aJbFWyZuOoHYR4xawIDP5nzMFh8aFo5k3U54QqFLNBisu4EMdTYqygQEeX2Ew4aTye1sf+0ybleJS0nxVQ7OPe0snrfHIvU2AUlSpNRxw3ADyDFJ3GjnA0kPn7iTu2L78sl82/Mc/zEbrgu257xlTU393CiODwDLs4pi58d6QBSmacuBeNnRyVl5+DVwADv+3TINO+MVB1R1vIaQ81AJPiE0Ue13ZI4PRdZwu023GCJowm7JXy10+ogp9ncVfuSpTtFC6U516apC76rKkCJeOMtpQm3D361w17MEb/oaTXdyfMFzv0t2xnPdfkhskefmxAc0PS6Z5oC0FWJktp+OhmqJzgxlkqumsXd2OzYr3ywUJUWPAfwtLQVOm5N1HI/MU4LG+5YsSP8bS/LrupjGdLwOCEYYCSy/yRdoLi2Jah75+JXAFpPxN2Kj2lxJV39rBp3ruG1PkOJ7j+YMLwmeOVC3MJlp/BTemQo0+z8fgnJHHenxLYkVfF4QbXPdUeh+tT6QdwYGDG1ahDzWkTSuX//qi6/gsC+IExfE50RMsEllXgl64f9Dii 831TypZI Yy765k/qqcvR5YDTrwLBu5yGNT+uZr6kwQ6pGYOpIxHTa+2L8uQR1ed0MMb4nbsQdpSg2jdgoPjASsGJ216qRuM5up3BAGGT/HM7bufUsR1rN+ckOIaRcn6pG4eoC4jlCJM67sXQUC2eANMYNTS3iccfAZFGcBKwLnOC6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Michael Ellerman There are two possibilities for book3e_htw_mode, PPC_HTW_E6500 or PPC_HTW_NONE. The TLB miss handlers are patched to use, respectively: - exc_[data|indstruction]_tlb_miss_e6500_book3e - exc_[data|indstruction]_tlb_miss_bolted_book3e Which means the default handlers are never used. Remove those, and use the bolted handlers (PPC_HTW_NONE) by default. Signed-off-by: Michael Ellerman Signed-off-by: Christophe Leroy --- arch/powerpc/kernel/exceptions-64e.S | 4 +- arch/powerpc/mm/nohash/tlb_64e.c | 4 - arch/powerpc/mm/nohash/tlb_low_64e.S | 226 --------------------------- 3 files changed, 2 insertions(+), 232 deletions(-) diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S index dcf0591ad3c2..63f6b9f513a4 100644 --- a/arch/powerpc/kernel/exceptions-64e.S +++ b/arch/powerpc/kernel/exceptions-64e.S @@ -485,8 +485,8 @@ interrupt_base_book3e: /* fake trap */ EXCEPTION_STUB(0x160, decrementer) /* 0x0900 */ EXCEPTION_STUB(0x180, fixed_interval) /* 0x0980 */ EXCEPTION_STUB(0x1a0, watchdog) /* 0x09f0 */ - EXCEPTION_STUB(0x1c0, data_tlb_miss) - EXCEPTION_STUB(0x1e0, instruction_tlb_miss) + EXCEPTION_STUB(0x1c0, data_tlb_miss_bolted) + EXCEPTION_STUB(0x1e0, instruction_tlb_miss_bolted) EXCEPTION_STUB(0x200, altivec_unavailable) EXCEPTION_STUB(0x220, altivec_assist) EXCEPTION_STUB(0x260, perfmon) diff --git a/arch/powerpc/mm/nohash/tlb_64e.c b/arch/powerpc/mm/nohash/tlb_64e.c index d83ecf466929..053128a5636c 100644 --- a/arch/powerpc/mm/nohash/tlb_64e.c +++ b/arch/powerpc/mm/nohash/tlb_64e.c @@ -244,10 +244,6 @@ static void __init early_init_mmu_global(void) patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e); patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e); break; - case PPC_HTW_NONE: - patch_exception(0x1c0, exc_data_tlb_miss_bolted_book3e); - patch_exception(0x1e0, exc_instruction_tlb_miss_bolted_book3e); - break; } pr_info("MMU: Book3E HW tablewalk %s\n", diff --git a/arch/powerpc/mm/nohash/tlb_low_64e.S b/arch/powerpc/mm/nohash/tlb_low_64e.S index b0eb3f7eaed1..a54e7d6c3d0b 100644 --- a/arch/powerpc/mm/nohash/tlb_low_64e.S +++ b/arch/powerpc/mm/nohash/tlb_low_64e.S @@ -511,232 +511,6 @@ itlb_miss_fault_e6500: tlb_epilog_bolted b exc_instruction_storage_book3e -/********************************************************************** - * * - * TLB miss handling for Book3E with TLB reservation and HES support * - * * - **********************************************************************/ - - -/* Data TLB miss */ - START_EXCEPTION(data_tlb_miss) - TLB_MISS_PROLOG - - /* Now we handle the fault proper. We only save DEAR in normal - * fault case since that's the only interesting values here. - * We could probably also optimize by not saving SRR0/1 in the - * linear mapping case but I'll leave that for later - */ - mfspr r14,SPRN_ESR - mfspr r16,SPRN_DEAR /* get faulting address */ - srdi r15,r16,44 /* get region */ - xoris r15,r15,0xc - cmpldi cr0,r15,0 /* linear mapping ? */ - beq tlb_load_linear /* yes -> go to linear map load */ - cmpldi cr1,r15,1 /* vmalloc mapping ? */ - - /* The page tables are mapped virtually linear. At this point, though, - * we don't know whether we are trying to fault in a first level - * virtual address or a virtual page table address. We can get that - * from bit 0x1 of the region ID which we have set for a page table - */ - andis. r10,r15,0x1 - bne- virt_page_table_tlb_miss - - std r14,EX_TLB_ESR(r12); /* save ESR */ - std r16,EX_TLB_DEAR(r12); /* save DEAR */ - - /* We need _PAGE_PRESENT and _PAGE_ACCESSED set */ - li r11,_PAGE_PRESENT - oris r11,r11,_PAGE_ACCESSED@h - - /* We do the user/kernel test for the PID here along with the RW test - */ - srdi. r15,r16,60 /* Check for user region */ - - /* We pre-test some combination of permissions to avoid double - * faults: - * - * We move the ESR:ST bit into the position of _PAGE_BAP_SW in the PTE - * ESR_ST is 0x00800000 - * _PAGE_BAP_SW is 0x00000010 - * So the shift is >> 19. This tests for supervisor writeability. - * If the page happens to be supervisor writeable and not user - * writeable, we will take a new fault later, but that should be - * a rare enough case. - * - * We also move ESR_ST in _PAGE_DIRTY position - * _PAGE_DIRTY is 0x00001000 so the shift is >> 11 - * - * MAS1 is preset for all we need except for TID that needs to - * be cleared for kernel translations - */ - rlwimi r11,r14,32-19,27,27 - rlwimi r11,r14,32-16,19,19 - beq normal_tlb_miss_user - /* XXX replace the RMW cycles with immediate loads + writes */ -1: mfspr r10,SPRN_MAS1 - rlwinm r10,r10,0,16,1 /* Clear TID */ - mtspr SPRN_MAS1,r10 - beq+ cr1,normal_tlb_miss - - /* We got a crappy address, just fault with whatever DEAR and ESR - * are here - */ - TLB_MISS_EPILOG_ERROR - b exc_data_storage_book3e - -/* Instruction TLB miss */ - START_EXCEPTION(instruction_tlb_miss) - TLB_MISS_PROLOG - - /* If we take a recursive fault, the second level handler may need - * to know whether we are handling a data or instruction fault in - * order to get to the right store fault handler. We provide that - * info by writing a crazy value in ESR in our exception frame - */ - li r14,-1 /* store to exception frame is done later */ - - /* Now we handle the fault proper. We only save DEAR in the non - * linear mapping case since we know the linear mapping case will - * not re-enter. We could indeed optimize and also not save SRR0/1 - * in the linear mapping case but I'll leave that for later - * - * Faulting address is SRR0 which is already in r16 - */ - srdi r15,r16,44 /* get region */ - xoris r15,r15,0xc - cmpldi cr0,r15,0 /* linear mapping ? */ - beq tlb_load_linear /* yes -> go to linear map load */ - cmpldi cr1,r15,1 /* vmalloc mapping ? */ - - /* We do the user/kernel test for the PID here along with the RW test - */ - li r11,_PAGE_PRESENT|_PAGE_BAP_UX /* Base perm */ - oris r11,r11,_PAGE_ACCESSED@h - - srdi. r15,r16,60 /* Check for user region */ - std r14,EX_TLB_ESR(r12) /* write crazy -1 to frame */ - beq normal_tlb_miss_user - - li r11,_PAGE_PRESENT|_PAGE_BAP_SX /* Base perm */ - oris r11,r11,_PAGE_ACCESSED@h - /* XXX replace the RMW cycles with immediate loads + writes */ - mfspr r10,SPRN_MAS1 - rlwinm r10,r10,0,16,1 /* Clear TID */ - mtspr SPRN_MAS1,r10 - beq+ cr1,normal_tlb_miss - - /* We got a crappy address, just fault */ - TLB_MISS_EPILOG_ERROR - b exc_instruction_storage_book3e - -/* - * This is the guts of the first-level TLB miss handler for direct - * misses. We are entered with: - * - * r16 = faulting address - * r15 = region ID - * r14 = crap (free to use) - * r13 = PACA - * r12 = TLB exception frame in PACA - * r11 = PTE permission mask - * r10 = crap (free to use) - */ -normal_tlb_miss_user: -#ifdef CONFIG_PPC_KUAP - mfspr r14,SPRN_MAS1 - rlwinm. r14,r14,0,0x3fff0000 - beq- normal_tlb_miss_access_fault /* KUAP fault */ -#endif -normal_tlb_miss: - /* So we first construct the page table address. We do that by - * shifting the bottom of the address (not the region ID) by - * PAGE_SHIFT-3, clearing the bottom 3 bits (get a PTE ptr) and - * or'ing the fourth high bit. - * - * NOTE: For 64K pages, we do things slightly differently in - * order to handle the weird page table format used by linux - */ - srdi r15,r16,44 - oris r10,r15,0x1 - rldicl r14,r16,64-(PAGE_SHIFT-3),PAGE_SHIFT-3+4 - sldi r15,r10,44 - clrrdi r14,r14,19 - or r10,r15,r14 - - ld r14,0(r10) - -finish_normal_tlb_miss: - /* Check if required permissions are met */ - andc. r15,r11,r14 - bne- normal_tlb_miss_access_fault - - /* Now we build the MAS: - * - * MAS 0 : Fully setup with defaults in MAS4 and TLBnCFG - * MAS 1 : Almost fully setup - * - PID already updated by caller if necessary - * - TSIZE need change if !base page size, not - * yet implemented for now - * MAS 2 : Defaults not useful, need to be redone - * MAS 3+7 : Needs to be done - * - * TODO: mix up code below for better scheduling - */ - clrrdi r10,r16,12 /* Clear low crap in EA */ - rlwimi r10,r14,32-19,27,31 /* Insert WIMGE */ - mtspr SPRN_MAS2,r10 - - /* Check page size, if not standard, update MAS1 */ - rldicl r10,r14,64-8,64-8 - cmpldi cr0,r10,BOOK3E_PAGESZ_4K - beq- 1f - mfspr r11,SPRN_MAS1 - rlwimi r11,r14,31,21,24 - rlwinm r11,r11,0,21,19 - mtspr SPRN_MAS1,r11 -1: - /* Move RPN in position */ - rldicr r11,r14,64-(PTE_RPN_SHIFT-PAGE_SHIFT),63-PAGE_SHIFT - clrldi r15,r11,12 /* Clear crap at the top */ - rlwimi r15,r14,32-8,22,25 /* Move in U bits */ - rlwimi r15,r14,32-2,26,31 /* Move in BAP bits */ - - /* Mask out SW and UW if !DIRTY (XXX optimize this !) */ - andi. r11,r14,_PAGE_DIRTY - bne 1f - li r11,MAS3_SW|MAS3_UW - andc r15,r15,r11 -1: - srdi r16,r15,32 - mtspr SPRN_MAS3,r15 - mtspr SPRN_MAS7,r16 - - tlbwe - -normal_tlb_miss_done: - /* We don't bother with restoring DEAR or ESR since we know we are - * level 0 and just going back to userland. They are only needed - * if you are going to take an access fault - */ - TLB_MISS_EPILOG_SUCCESS - rfi - -normal_tlb_miss_access_fault: - /* We need to check if it was an instruction miss */ - andi. r10,r11,_PAGE_BAP_UX - bne 1f - ld r14,EX_TLB_DEAR(r12) - ld r15,EX_TLB_ESR(r12) - mtspr SPRN_DEAR,r14 - mtspr SPRN_ESR,r15 - TLB_MISS_EPILOG_ERROR - b exc_data_storage_book3e -1: TLB_MISS_EPILOG_ERROR - b exc_instruction_storage_book3e - - /* * This is the guts of the second-level TLB miss handler for direct * misses. We are entered with: From patchwork Tue Jul 2 13:51:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A87E6C3064D for ; Tue, 2 Jul 2024 13:52:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42DEE6B00B7; Tue, 2 Jul 2024 09:52:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DD8F6B00B8; Tue, 2 Jul 2024 09:52:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 256C06B00B9; Tue, 2 Jul 2024 09:52:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 06BDE6B00B7 for ; Tue, 2 Jul 2024 09:52:01 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A39AA1201E6 for ; Tue, 2 Jul 2024 13:52:00 +0000 (UTC) X-FDA: 82294951200.12.87618E3 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf19.hostedemail.com (Postfix) with ESMTP id 7D6D41A0008 for ; Tue, 2 Jul 2024 13:51:58 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928307; a=rsa-sha256; cv=none; b=lcmBTy8Yp52UAaiJornPh1q0ANgaIhACha2KrLoSYyGNPaMvh+o9fW+19Ql6BTlqLRXUUy rWQ071U8rQ79wqyjV4yAC2y5ht5PqXIYZlyVG1AuqwMqeLHqa55ap0Wn/g6aH0aSulWoYn Wzef1He3AAqDT0EcbzsLr0rfdF1fuYw= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928307; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XzOg91sDK1hwoFPCHuMptEqxh9SprEvaPrCX2efe22U=; b=eLo9rZir1ykRSMMffzj0LXqax/ZTqmzU+FFF4ex0Kc32LNo6I+Ekw+GqQl/ga9LpI/MZ8U 0a0k7QOIsjoDUmt3wiOvFAlCCwpjfRhDYssZsSA8mtbMS0+LbgLf4PHi1BrLXphqW2aHhq BofPW1Zjy2LsBhkSECJGA+MkxqemPHc= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4841fp7z9v6W; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C9od68sRy3VN; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4805RNyz9v4J; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id B678D8B774; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id BGNhRIrkHoLm; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 4106F8B764; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 07/23] mm: Define __pte_leaf_size() to also take a PMD entry Date: Tue, 2 Jul 2024 15:51:19 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=1803; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=NqqR+DBf4uYbt5gP7om6e3QFNLkHAAKdU42xGKHVcEo=; b=yV33hLP9H4xcUhyFxDXIt5LnxJgWqrCn1V5Xykpxq478yOcZTAZhwgAiteylu27VRD9Xbv6er iD0lsuBP5qqBVXJsr9LvPuLb4J+Nh0P8vxx5LpDXoG49QzJ3MzMfoS4 X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Stat-Signature: e5gi8eqqjh8wuoohwtuyxa4ofz6g9sk6 X-Rspamd-Queue-Id: 7D6D41A0008 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1719928318-245117 X-HE-Meta: U2FsdGVkX1+M/UyKNe9DLdo2Ndf+XbRiUahycWRR5J4lAVCoc+ojS8fTT+vtfDsy+HwKdATkW6CZyZJ+BruxlJhEM3PAd44sqN+zKZ+hly9kkG3k7zb4kb42bK5YKOz+Rxh32a3WFCrM4iiq0Jlk55h7kCBW2XUgM7W9X8aP9Wz9/LD35YZS8pBYMhPRT00ipZq3o1fjiM1S5ufYwvn8dTJvaGCOKOpuz8iM5N9+gYQTQEZ4WkT4JikShOpne0QOom8F0GyBRBP6MOjcLPsPfhhIrOLBC2G1yVMww3ydE8NoK77i6yGXIADKSBVkB+CYzsMEEZAw/JlNxVXro9itA6rs+JWNPy5Q/d7HteUE3e/1oafS6UBS+OBaCyaJyCSMywQrLou/UkUi1H4JOERmkEY18K9RpDS91dP1GAxO3ZTU8gAy0IkbcFAHJRAkknB+DjYO5bpEEVNqMlNlbrHt8HD76V4UT7bQ+fmuEEGnoKnDO8cfdutk09EB3WbbYZfgkoN4QUjKAHpm90e+vPVfwbJMgLHqf9BU23Dlu1iiqC9b6aivFCu1gXzkbpJchcD1s1YpevlBLKPPhC7/B5OKteB3ypFcInxDNv9U11S2hHLe6pEucRoClYZNuHp+fGAicrZ9KkyznIq2pT1jS0p4B8PiKtnPZ8B4XBtOyYcVWBJmSrmO1B8gLE1UrmdSc4gBH59l3RUZ0s0LlDEY0+CCFNtIR6sgQOGp0S5EDrGyFxqqhgSTirqdE/kmMZc4SHa491LzHL+xbpyBhifNyY3RraTuPd4i7kZeLovaEB8/bvBXuJ1PNJ28bxheJ0hsUG8NF4ykc6vIfl3o0+v16Q7PDb55GbeEMvpK4muiYc7ooHPxVrFiMBM2gj9UPqE87NoXX5ZQSTKvz61cbCPQglF1cNRkuxzOC8cIsiM+AF23c4U4hNI10M4e4wVS+2AJ7BhvlqVVlUHd990dGThe2Ha YTXZe502 aZhntJDTXQFJwvKBVuhxqcvRQhvcS6KyyhcmuudUQwOKJ/WY7ZPLK7p2dsz4gQNB0PAMQ4eDR+vr8G/Y8dbs4LjfM2VIoDeQbE+3LjZUnykb0URSo4pvVA8LiWuw+sre0oLuFtXuSC2XY9wqbDGvFNQ+Qihc8aKPtbgSo5nTK9z7/CgI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On powerpc 8xx, when a page is 8M size, the information is in the PMD entry. So allow architectures to provide __pte_leaf_size() instead of pte_leaf_size() and provide the PMD entry to that function. When __pte_leaf_size() is not defined, define it as a pte_leaf_size() so that architectures not interested in the PMD arguments are not impacted. Only define a default pte_leaf_size() when __pte_leaf_size() is not defined to make sure nobody adds new calls to pte_leaf_size() in the core. Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- v3: Don't change pte_leaf_size() to not impact other architectures --- include/linux/pgtable.h | 3 +++ kernel/events/core.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 2f32eaccf0b9..2a6a3cccfc36 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1907,9 +1907,12 @@ typedef unsigned int pgtbl_mod_mask; #ifndef pmd_leaf_size #define pmd_leaf_size(x) PMD_SIZE #endif +#ifndef __pte_leaf_size #ifndef pte_leaf_size #define pte_leaf_size(x) PAGE_SIZE #endif +#define __pte_leaf_size(x,y) pte_leaf_size(y) +#endif /* * We always define pmd_pfn for all archs as it's used in lots of generic diff --git a/kernel/events/core.c b/kernel/events/core.c index 8f908f077935..b2ca11bdc11e 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7609,7 +7609,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) pte = ptep_get_lockless(ptep); if (pte_present(pte)) - size = pte_leaf_size(pte); + size = __pte_leaf_size(pmd, pte); pte_unmap(ptep); #endif /* CONFIG_HAVE_GUP_FAST */ From patchwork Tue Jul 2 13:51:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0657BC3064D for ; Tue, 2 Jul 2024 13:52:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90EAD6B00BB; Tue, 2 Jul 2024 09:52:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BDD66B00BC; Tue, 2 Jul 2024 09:52:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 738036B00BD; Tue, 2 Jul 2024 09:52:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 48B746B00BB for ; Tue, 2 Jul 2024 09:52:05 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0EA2E1A01D3 for ; Tue, 2 Jul 2024 13:52:05 +0000 (UTC) X-FDA: 82294951410.03.22BD75B Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf16.hostedemail.com (Postfix) with ESMTP id D96C6180017 for ; Tue, 2 Jul 2024 13:52:02 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZRPni+mE/UBcft1hGqlqgIsvYl1G4pGJjxODv7h8tgQ=; b=XyfoNCZcg13EoZVVXoIfd3fCAt0nkPA3qBtRh7i6lQD5TC0bbKris5/0htlgGKE5oabs/K GdJs5e9tJngXI4eTKLUfVra+RghDJchSxH+XetY7nThIXOOLCFkeUaTRxfHIguVYGwQLoz OzlNDVBWoBPrhUQOHfONZL1iwPZORrQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928312; a=rsa-sha256; cv=none; b=NrvmPjpCEvAXqLUSlAffi9B/ZhC979rqT51v0m05zIlVJh2UIuCP/EClE5Qk4tW79ZGnlD NZadmOMFsXgD+805tXJAqqKPfvVvbLUmDHPBpjSwGOsKkaN7yVFF77D759QfGpDQmuM/+d rddXvV3bVJaYu0Yzi6zWxovj7Y+59Us= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4851dMfz9v6d; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RCNoT-nbWqwX; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4813SMFz9v4N; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 72A338B775; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id JkSC1z_M_3Cp; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id BF4AE8B764; Tue, 2 Jul 2024 15:51:28 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 08/23] mm: Provide mm_struct and address to huge_ptep_get() Date: Tue, 2 Jul 2024 15:51:20 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=23997; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=TVU6OPP5V7Sg5c0l3LOavpI2f8Th/LptQR0+OKvZMVE=; b=YVAw3ze7EUc7yIC481VkB0MorcLJtNBSCLbby8PBezdq40IbL6C5GDUXtL1+4bfLtzTwQp9F5 QTIMM1pdTWDCBShRgj+EWklfYgi7lJ2VxA8y+MHomE/Rg8nKmNpw1Ic X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: D96C6180017 X-Stat-Signature: szpipupeayxs3bffd1c4wmqbh7s7jdpo X-HE-Tag: 1719928322-193435 X-HE-Meta: U2FsdGVkX1+ue1CBiTgz0EbwBsASjFmgOJVJ1m+UBmqdyvq8N3DP3AkjWB4amcEac84je82MQ0thTNhvFYqDcdSRPD8gOQw6qWHmOCPWnPPDJgPWBv8AIdOHajSNTfDpzQ3NZI2VWWnPZcs7dl0qkTpf7v76cpQlm5+j+igTFhC4R+t7TTJqRTQNlLRR+0Xs6jDK8GLd3EhzNg1IFT9eF8HwJ9leKok3Gp5OcW9M68M5W3uttna59r/ewjHJWGCkiZwfkSscp2i9pDl2yopuYB3ddyffJXDoFjfHlV+vZuYEweKMH/qvcF7yONhHTDXU5d2utMfFD9Rv94IYH4t+krRphIdRS6scmwl91zHEflpVPSS/8/V8TXEnLUTgVbbn8x57DAhDBriBslfBsHnVGf8FF41Uvm4nDz26hd7J0lPyXCvt+D2lDBTeAhH2sVKt9RI+J8QtedAeFeHxiQpB3BdRiA3Sn+0xb/kxWNIr1n8EPI0Ugn9nKUUPr3hydAlFXuMFHX3F9V3uaUBEgS3/tsud8QS/zcfd8Qz1Va4e/6VUpNvePOCF/qwvvy+4Jg39tjQx8zrHKENrC9CoFhwKceFK7iRQ5XxA/KqCLshCLmDqOaLtAwVTcQg9EY65FKE+tXFn896qUQoaoYuuP27+T36i/5L7rRBoUFfz/5mWkC3fZ3Q8Z8XzRCb9K2tkGAIXoyGsKMv9nL2AzoAqE16pwEPiAAG4HTFMGEbCyi7lhso8oSP+cz2c6r2IRnj6owQqbz16wAwTKa8sQXEHLFhr3ziumYACjE9lMWFNb1QkiQ7W18E3YA143Ykz3GPagCQW8BCDVOO73H26F4bXlEgGE/3CO4HxhM4Oy9ZsxJRkA9BJLTIZpHAEAGtNumjRqXavk5vuFP2av2eUOubz3K3TbRxcjUwO/mLBjTp2XanQkO50V9gfRCfdj0i7YpA+HwTWr6gEV7WuBmIt1XbGnZQ tYOo+V/S DxsWjnUU0UsHQtBbvmYvFIG2tklgjjn4eSURGmKIAkAnLlLbqJsZPaXsSbqcR/Tga8WhhNTncEThrcKqHeP4IdflgAogvf0VfIkPg6YiBOc7AesefSUUamDv7kfeQbIYi6WMmNbxkkNVdVGS39gmUoDm7V9DUoQbhfZgRL2CVyQ7wmjU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On powerpc 8xx huge_ptep_get() will need to know whether the given ptep is a PTE entry or a PMD entry. This cannot be known with the PMD entry itself because there is no easy way to know it from the content of the entry. So huge_ptep_get() will need to know either the size of the page or get the pmd. In order to be consistent with huge_ptep_get_and_clear(), give mm and address to huge_ptep_get(). Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- v2: Add missing changes in arch implementations v3: Fixed a comment in ARM and missing changes in S390 v4: Fix missing or bad changes in mm/hugetlb.c and rebased on v6.10-rc1 v5: Fix vma->mm to vma->vm_mm --- arch/arm/include/asm/hugetlb-3level.h | 4 +-- arch/arm64/include/asm/hugetlb.h | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/s390/include/asm/hugetlb.h | 4 +-- arch/s390/mm/hugetlbpage.c | 4 +-- fs/hugetlbfs/inode.c | 2 +- fs/proc/task_mmu.c | 10 +++--- fs/userfaultfd.c | 2 +- include/asm-generic/hugetlb.h | 2 +- include/linux/swapops.h | 4 +-- mm/damon/vaddr.c | 6 ++-- mm/gup.c | 2 +- mm/hmm.c | 2 +- mm/hugetlb.c | 44 +++++++++++++-------------- mm/memory-failure.c | 2 +- mm/mempolicy.c | 2 +- mm/migrate.c | 4 +-- mm/mincore.c | 2 +- mm/userfaultfd.c | 2 +- 21 files changed, 53 insertions(+), 53 deletions(-) diff --git a/arch/arm/include/asm/hugetlb-3level.h b/arch/arm/include/asm/hugetlb-3level.h index a30be5505793..87d48e2d90ad 100644 --- a/arch/arm/include/asm/hugetlb-3level.h +++ b/arch/arm/include/asm/hugetlb-3level.h @@ -13,12 +13,12 @@ /* * If our huge pte is non-zero then mark the valid bit. - * This allows pte_present(huge_ptep_get(ptep)) to return true for non-zero + * This allows pte_present(huge_ptep_get(mm,addr,ptep)) to return true for non-zero * ptes. * (The valid bit is automatically cleared by set_pte_at for PROT_NONE ptes). */ #define __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get(pte_t *ptep) +static inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { pte_t retval = *ptep; if (pte_val(retval)) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 3954cbd2ff56..293f880865e8 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -46,7 +46,7 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); #define __HAVE_ARCH_HUGE_PTEP_GET -extern pte_t huge_ptep_get(pte_t *ptep); +extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); void __init arm64_hugetlb_cma_reserve(void); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 3f09ac73cce3..5f1e2103888b 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -127,7 +127,7 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize) return contig_ptes; } -pte_t huge_ptep_get(pte_t *ptep) +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { int ncontig, i; size_t pgsize; diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index b1ce97a9dbfc..faf3624d8057 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -44,7 +44,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, pte_t pte, int dirty); #define __HAVE_ARCH_HUGE_PTEP_GET -pte_t huge_ptep_get(pte_t *ptep); +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 0ebd968b33c9..42314f093922 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -3,7 +3,7 @@ #include #ifdef CONFIG_RISCV_ISA_SVNAPOT -pte_t huge_ptep_get(pte_t *ptep) +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { unsigned long pte_num; int i; diff --git a/arch/s390/include/asm/hugetlb.h b/arch/s390/include/asm/hugetlb.h index ce5f4fe8be4d..cf1b5d6fb1a6 100644 --- a/arch/s390/include/asm/hugetlb.h +++ b/arch/s390/include/asm/hugetlb.h @@ -19,7 +19,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz); void __set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); -pte_t huge_ptep_get(pte_t *ptep); +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); @@ -64,7 +64,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty) { - int changed = !pte_same(huge_ptep_get(ptep), pte); + int changed = !pte_same(huge_ptep_get(vma->vm_mm, addr, ptep), pte); if (changed) { huge_ptep_get_and_clear(vma->vm_mm, addr, ptep); __set_huge_pte_at(vma->vm_mm, addr, ptep, pte); diff --git a/arch/s390/mm/hugetlbpage.c b/arch/s390/mm/hugetlbpage.c index 2675aab4acc7..1be481672f4a 100644 --- a/arch/s390/mm/hugetlbpage.c +++ b/arch/s390/mm/hugetlbpage.c @@ -169,7 +169,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, __set_huge_pte_at(mm, addr, ptep, pte); } -pte_t huge_ptep_get(pte_t *ptep) +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { return __rste_to_pte(pte_val(*ptep)); } @@ -177,7 +177,7 @@ pte_t huge_ptep_get(pte_t *ptep) pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - pte_t pte = huge_ptep_get(ptep); + pte_t pte = huge_ptep_get(mm, addr, ptep); pmd_t *pmdp = (pmd_t *) ptep; pud_t *pudp = (pud_t *) ptep; diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index c34faef62daf..8d7b3d9c5a43 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -422,7 +422,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, if (!ptep) return false; - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(vma->vm_mm, addr, ptep); if (huge_pte_none(pte) || !pte_present(pte)) return false; diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index d99a390a0f41..90ad9df8ea73 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1013,7 +1013,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, { struct mem_size_stats *mss = walk->private; struct vm_area_struct *vma = walk->vma; - pte_t ptent = huge_ptep_get(pte); + pte_t ptent = huge_ptep_get(walk->mm, addr, pte); struct folio *folio = NULL; bool present = false; @@ -1882,7 +1882,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, if (vma->vm_flags & VM_SOFTDIRTY) flags |= PM_SOFT_DIRTY; - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(walk->mm, addr, ptep); if (pte_present(pte)) { struct folio *folio = page_folio(pte_page(pte)); @@ -2571,7 +2571,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, if (~p->arg.flags & PM_SCAN_WP_MATCHING) { /* Go the short route when not write-protecting pages. */ - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(walk->mm, start, ptep); categories = p->cur_vma_category | pagemap_hugetlb_category(pte); if (!pagemap_scan_is_interesting_page(categories, p)) @@ -2583,7 +2583,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, i_mmap_lock_write(vma->vm_file->f_mapping); ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep); - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(walk->mm, start, ptep); categories = p->cur_vma_category | pagemap_hugetlb_category(pte); if (!pagemap_scan_is_interesting_page(categories, p)) @@ -2979,7 +2979,7 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long end, struct mm_walk *walk) { - pte_t huge_pte = huge_ptep_get(pte); + pte_t huge_pte = huge_ptep_get(walk->mm, addr, pte); struct numa_maps *md; struct page *page; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 17e409ceaa33..27a3e9285fbf 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -257,7 +257,7 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, goto out; ret = false; - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep); /* * Lockless access: we're in a wait_event so it's ok if it diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h index 6dcf4d576970..594d5905f615 100644 --- a/include/asm-generic/hugetlb.h +++ b/include/asm-generic/hugetlb.h @@ -144,7 +144,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, #endif #ifndef __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get(pte_t *ptep) +static inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { return ptep_get(ptep); } diff --git a/include/linux/swapops.h b/include/linux/swapops.h index a5c560a2f8c2..cb468e418ea1 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -334,7 +334,7 @@ static inline bool is_migration_entry_dirty(swp_entry_t entry) extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address); -extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); +extern void migration_entry_wait_huge(struct vm_area_struct *vma, unsigned long addr, pte_t *pte); #else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) { @@ -359,7 +359,7 @@ static inline int is_migration_entry(swp_entry_t swp) static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address) { } static inline void migration_entry_wait_huge(struct vm_area_struct *vma, - pte_t *pte) { } + unsigned long addr, pte_t *pte) { } static inline int is_writable_migration_entry(swp_entry_t entry) { return 0; diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 381559e4a1fa..58829baf8b5d 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -339,7 +339,7 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr) { bool referenced = false; - pte_t entry = huge_ptep_get(pte); + pte_t entry = huge_ptep_get(mm, addr, pte); struct folio *folio = pfn_folio(pte_pfn(entry)); unsigned long psize = huge_page_size(hstate_vma(vma)); @@ -373,7 +373,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, pte_t entry; ptl = huge_pte_lock(h, walk->mm, pte); - entry = huge_ptep_get(pte); + entry = huge_ptep_get(walk->mm, addr, pte); if (!pte_present(entry)) goto out; @@ -509,7 +509,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, pte_t entry; ptl = huge_pte_lock(h, walk->mm, pte); - entry = huge_ptep_get(pte); + entry = huge_ptep_get(walk->mm, addr, pte); if (!pte_present(entry)) goto out; diff --git a/mm/gup.c b/mm/gup.c index 0faf07b2c51d..aaf6ef4e28ee 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -508,7 +508,7 @@ static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long sz if (pte_end < end) end = pte_end; - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(vma->vm_mm, addr, ptep); if (!pte_access_permitted(pte, flags & FOLL_WRITE)) return 0; diff --git a/mm/hmm.c b/mm/hmm.c index 93aebd9cc130..7e0229ae4a5a 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -480,7 +480,7 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, pte_t entry; ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte); - entry = huge_ptep_get(pte); + entry = huge_ptep_get(walk->mm, addr, pte); i = (start - range->start) >> PAGE_SHIFT; pfn_req_flags = range->hmm_pfns[i]; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0a69e194b517..61847f799b6a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5286,7 +5286,7 @@ static void set_huge_ptep_writable(struct vm_area_struct *vma, { pte_t entry; - entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep))); + entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(vma->vm_mm, address, ptep))); if (huge_ptep_set_access_flags(vma, address, ptep, entry, 1)) update_mmu_cache(vma, address, ptep); } @@ -5394,7 +5394,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); - entry = huge_ptep_get(src_pte); + entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte); again: if (huge_pte_none(entry)) { /* @@ -5432,7 +5432,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, set_huge_pte_at(dst, addr, dst_pte, make_pte_marker(marker), sz); } else { - entry = huge_ptep_get(src_pte); + entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte); pte_folio = page_folio(pte_page(entry)); folio_get(pte_folio); @@ -5473,7 +5473,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); - entry = huge_ptep_get(src_pte); + entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte); if (!pte_same(src_pte_old, entry)) { restore_reserve_on_error(h, dst_vma, addr, new_folio); @@ -5583,7 +5583,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, new_addr |= last_addr_mask; continue; } - if (huge_pte_none(huge_ptep_get(src_pte))) + if (huge_pte_none(huge_ptep_get(mm, old_addr, src_pte))) continue; if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) { @@ -5656,7 +5656,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, continue; } - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(mm, address, ptep); if (huge_pte_none(pte)) { spin_unlock(ptl); continue; @@ -5905,7 +5905,7 @@ static vm_fault_t hugetlb_wp(struct folio *pagecache_folio, struct vm_area_struct *vma = vmf->vma; struct mm_struct *mm = vma->vm_mm; const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; - pte_t pte = huge_ptep_get(vmf->pte); + pte_t pte = huge_ptep_get(mm, vmf->address, vmf->pte); struct hstate *h = hstate_vma(vma); struct folio *old_folio; struct folio *new_folio; @@ -6026,7 +6026,7 @@ static vm_fault_t hugetlb_wp(struct folio *pagecache_folio, vmf->pte = hugetlb_walk(vma, vmf->address, huge_page_size(h)); if (likely(vmf->pte && - pte_same(huge_ptep_get(vmf->pte), pte))) + pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), pte))) goto retry_avoidcopy; /* * race occurs while re-acquiring page table @@ -6064,7 +6064,7 @@ static vm_fault_t hugetlb_wp(struct folio *pagecache_folio, */ spin_lock(vmf->ptl); vmf->pte = hugetlb_walk(vma, vmf->address, huge_page_size(h)); - if (likely(vmf->pte && pte_same(huge_ptep_get(vmf->pte), pte))) { + if (likely(vmf->pte && pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), pte))) { pte_t newpte = make_huge_pte(vma, &new_folio->page, !unshare); /* Break COW or unshare */ @@ -6165,14 +6165,14 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_fault *vmf, * Recheck pte with pgtable lock. Returns true if pte didn't change, or * false if pte changed or is changing. */ -static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, +static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t old_pte) { spinlock_t *ptl; bool same; ptl = huge_pte_lock(h, mm, ptep); - same = pte_same(huge_ptep_get(ptep), old_pte); + same = pte_same(huge_ptep_get(mm, addr, ptep), old_pte); spin_unlock(ptl); return same; @@ -6233,7 +6233,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, * never happen on the page after UFFDIO_COPY has * correctly installed the page and returned. */ - if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) { ret = 0; goto out; } @@ -6262,7 +6262,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, * here. Before returning error, get ptl and make * sure there really is no pte entry. */ - if (hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) + if (hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) ret = vmf_error(PTR_ERR(folio)); else ret = 0; @@ -6311,7 +6311,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, folio_unlock(folio); folio_put(folio); /* See comment in userfaultfd_missing() block above */ - if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { + if (!hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) { ret = 0; goto out; } @@ -6338,7 +6338,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, vmf->ptl = huge_pte_lock(h, mm, vmf->pte); ret = 0; /* If pte changed from under us, retry */ - if (!pte_same(huge_ptep_get(vmf->pte), vmf->orig_pte)) + if (!pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), vmf->orig_pte)) goto backout; if (anon_rmap) @@ -6459,7 +6459,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, return VM_FAULT_OOM; } - vmf.orig_pte = huge_ptep_get(vmf.pte); + vmf.orig_pte = huge_ptep_get(mm, vmf.address, vmf.pte); if (huge_pte_none_mostly(vmf.orig_pte)) { if (is_pte_marker(vmf.orig_pte)) { pte_marker marker = @@ -6500,7 +6500,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * be released there. */ mutex_unlock(&hugetlb_fault_mutex_table[hash]); - migration_entry_wait_huge(vma, vmf.pte); + migration_entry_wait_huge(vma, vmf.address, vmf.pte); return 0; } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte))) ret = VM_FAULT_HWPOISON_LARGE | @@ -6533,11 +6533,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, vmf.ptl = huge_pte_lock(h, mm, vmf.pte); /* Check for a racing update before calling hugetlb_wp() */ - if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte)))) + if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(mm, vmf.address, vmf.pte)))) goto out_ptl; /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) && + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(mm, vmf.address, vmf.pte)) && (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { if (!userfaultfd_wp_async(vma)) { spin_unlock(vmf.ptl); @@ -6665,7 +6665,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, ptl = huge_pte_lock(h, dst_mm, dst_pte); /* Don't overwrite any existing PTEs (even markers) */ - if (!huge_pte_none(huge_ptep_get(dst_pte))) { + if (!huge_pte_none(huge_ptep_get(dst_mm, dst_addr, dst_pte))) { spin_unlock(ptl); return -EEXIST; } @@ -6801,7 +6801,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, * page backing it, then access the page. */ ret = -EEXIST; - if (!huge_pte_none_mostly(huge_ptep_get(dst_pte))) + if (!huge_pte_none_mostly(huge_ptep_get(dst_mm, dst_addr, dst_pte))) goto out_release_unlock; if (folio_in_pagecache) @@ -6922,7 +6922,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, address |= last_addr_mask; continue; } - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(mm, address, ptep); if (unlikely(is_hugetlb_entry_hwpoisoned(pte))) { /* Nothing to do. */ } else if (unlikely(is_hugetlb_entry_migration(pte))) { diff --git a/mm/memory-failure.c b/mm/memory-failure.c index ac661a0e967c..0cb1b7bea9a5 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -835,7 +835,7 @@ static int hwpoison_hugetlb_range(pte_t *ptep, unsigned long hmask, struct mm_walk *walk) { struct hwpoison_walk *hwp = walk->private; - pte_t pte = huge_ptep_get(ptep); + pte_t pte = huge_ptep_get(walk->mm, addr, ptep); struct hstate *h = hstate_vma(walk->vma); return check_hwpoisoned_entry(pte, addr, huge_page_shift(h), diff --git a/mm/mempolicy.c b/mm/mempolicy.c index f73acb01ad45..f8703feb68b7 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -624,7 +624,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask, pte_t entry; ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte); - entry = huge_ptep_get(pte); + entry = huge_ptep_get(walk->mm, addr, pte); if (!pte_present(entry)) { if (unlikely(is_hugetlb_entry_migration(entry))) qp->nr_failed++; diff --git a/mm/migrate.c b/mm/migrate.c index c64d867f239d..e10d2445fbd8 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -338,14 +338,14 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, * * This function will release the vma lock before returning. */ -void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *ptep) +void migration_entry_wait_huge(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, ptep); pte_t pte; hugetlb_vma_assert_locked(vma); spin_lock(ptl); - pte = huge_ptep_get(ptep); + pte = huge_ptep_get(vma->vm_mm, addr, ptep); if (unlikely(!is_hugetlb_entry_migration(pte))) { spin_unlock(ptl); diff --git a/mm/mincore.c b/mm/mincore.c index e31cf1bde614..d6bd19e520fc 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -33,7 +33,7 @@ static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr, * Hugepages under user process are always in RAM and never * swapped out, but theoretically it needs to be checked. */ - present = pte && !huge_pte_none_mostly(huge_ptep_get(pte)); + present = pte && !huge_pte_none_mostly(huge_ptep_get(walk->mm, addr, pte)); for (; addr != end; vec++, addr += PAGE_SIZE) *vec = present; walk->private = vec; diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 8dedaec00486..e54e5c8907fa 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -587,7 +587,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( } if (!uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE) && - !huge_pte_none_mostly(huge_ptep_get(dst_pte))) { + !huge_pte_none_mostly(huge_ptep_get(dst_mm, dst_addr, dst_pte))) { err = -EEXIST; hugetlb_vma_unlock_read(dst_vma); mutex_unlock(&hugetlb_fault_mutex_table[hash]); From patchwork Tue Jul 2 13:51:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ACA2C3064D for ; Tue, 2 Jul 2024 13:52:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 341F56B00BE; Tue, 2 Jul 2024 09:52:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F2436B00BF; Tue, 2 Jul 2024 09:52:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 198396B00C0; Tue, 2 Jul 2024 09:52:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EB0786B00BE for ; Tue, 2 Jul 2024 09:52:09 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A949716010F for ; Tue, 2 Jul 2024 13:52:09 +0000 (UTC) X-FDA: 82294951578.07.32C3B14 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf04.hostedemail.com (Postfix) with ESMTP id 700C740013 for ; Tue, 2 Jul 2024 13:52:07 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=issq62jO0r3sQWK/I4We0PvHHoM3Fd8NVMPhHbGHD/U=; b=eB45tCBDCcketfLd/7ysF3XfJE7ToHUO012ukM0ZimuMxHJ3UdKH6dxpEnHbIX115X17xV 3k4Qftl4M328Zn/Rpx7CMumSOTQAFwlN1STYTheJmm7oaXuqizCtWFacVPx+TVFyzsHe2U Fd2fkI+MTG2VOdQkvu5AxFEKoaS0WGA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928310; a=rsa-sha256; cv=none; b=ROKe9530oUw6LM64ahR4DFx2m93TgXA8dhTCgfUpca1pNpZKYCSosDpCbkyVJPcFho6PCk jiGhCXCqRJeIaBTK7mlaP6+DjNo1lmGNZqYnJutjlrjMa44XDi85+VwaZy4mBFEOyTYBfs 6jS3e+/R5yjR82UYr0RPLIts0vHKKUs= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48616p3z9v6y; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id q5V0x4RsUWT7; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4816rW9z9v2F; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E719F8B775; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id 1X1oHFkZy9am; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6981D8B774; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 09/23] powerpc/mm: Remove _PAGE_PSIZE Date: Tue, 2 Jul 2024 15:51:21 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=3260; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=cm0Ph/xp1X0SRFsHS2XbjMZ7j+bK1MCB+6twQ18qnAY=; b=wKCfe3P3AAC+AJggcgkbesGaq/R5QimE5BC+cTvL/N7+H8zTid2Owy0Iq3Lerz2Z/xZECy/6+ WDUwzjvYtrTAQmdL02KM8AK8EXJucNmt8XXEOwdN3Ypl3szJoFnZgx0 X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Stat-Signature: wyaojsazgcai9fuoi4bktyg4n9zp71hy X-Rspam-User: X-Rspamd-Queue-Id: 700C740013 X-Rspamd-Server: rspam02 X-HE-Tag: 1719928327-358313 X-HE-Meta: U2FsdGVkX1+H5h7ogXRT39lTxtAIYEHylMIo3j3s5YGiFKX82it3eVlXsGJxG6Usp/M6VB9ujbq3ppqtVcbHj3qsNsf9f4z6auaH4JWl6EPfBMQfzhTmtXSMUssTC9UzPknWnes0XdEaSfzeMGZbu+5yPF63ZrATSfhSF1xoMxvA/T4QTwKfy+wFPG7g9wkbRtfWOReahOAVD9q3zxVsCbvZ//Zsawgom2NsiBYcLlrqfuQ6hGuyNjABBOzZ8gHKKU8WfYPrMGQvw/WwnBEFfTqE7/bbIdsKCrYeEyGLsy9yWj9UuASyyCOmAnu/0kfy3CJxDU6TUPUcfaeUyUTiVHugB42rXo3uwdOpbJOezKvREha/XOS0CISgpsdcfX/ZbFAEwjzt+SY4nYA6Wx2JeqVK2SXNiRwbsFlQodFm3qi0XkOAaQh387CsStwI4WNjjAGTC2iQ3AlTHCRa8Kjq1zjjsuc1Lh6Gz+Yur9YWDacBDjs8pH1UySDf7QrkiY4FGJd/BanVD+KYIv5Y9PMdMaRrdZrNnZmgBfBlxrJ/rCNxy0FEuLmn5W3Xepr/G4nnuLikQ8w/8G4RtRWDLUFLNbeYeK0vXvesJ2Ef8mA0rvzAFXi5+mTrlKRWNrcLU2m9nwyrNRkm9RSDfpasnqlsRBcpU02/IoBuxDpYYgGcPcSOYVG9+8LDlJ2biFE+3uw4PMnCSxYhD5wJqTrSZIH42V1AQ1gz4mNnNANTTIu4pw0alg+bwIQsX8v60Crd1SpwQ2KWaZ51kUUNJcOIp/xBPSMI8UYScAijNnmZ5TeD1EIdBWjYbtkqqnekgkrgLLX95lixQCjldABDRaDbt8PfarKxF6UH5XTiotoP7WdLhY07CXQLp75aQem0r2WAwWTi4aCGpZxJSH/Gj6hntEZDLD3L/lDqTezrOKS+PLGQQJ8PzSEjLavTkSUerc9RzEOFQTXjytHtj76WKcDnG5f lpbkjElu LC1QsV5FW7Hk8erBd2wdbZAlqStpN+MBr78JzXM7hf9JNSFs8/OvTrTMAG6rjy4vrV5thRABJUju8RMpFHUZmdInLCmoh+Dw2nEp1Wv478XkShI5QRDV1s/vtoOeGzfkVIeN987216ieEH0u4vDr1zOYM0J8PYgy271NOqD8blgJTzBQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: _PAGE_PSIZE macro is never used outside the place it is defined and is used only on 8xx and e500. Remove indirection, remove it and use its content directly. Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- v6: Removed the change to pte-40x.h to avoid conflict with the removal of 40x --- arch/powerpc/include/asm/nohash/32/pte-44x.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-85xx.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 5 ++--- arch/powerpc/include/asm/nohash/pte-e500.h | 4 +--- 4 files changed, 3 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/include/asm/nohash/32/pte-44x.h b/arch/powerpc/include/asm/nohash/32/pte-44x.h index 851813725237..da0469928273 100644 --- a/arch/powerpc/include/asm/nohash/32/pte-44x.h +++ b/arch/powerpc/include/asm/nohash/32/pte-44x.h @@ -75,9 +75,6 @@ #define _PAGE_NO_CACHE 0x00000400 /* H: I bit */ #define _PAGE_WRITETHRU 0x00000800 /* H: W bit */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 - /* TODO: Add large page lowmem mapping support */ #define _PMD_PRESENT 0 #define _PMD_PRESENT_MASK (PAGE_MASK) diff --git a/arch/powerpc/include/asm/nohash/32/pte-85xx.h b/arch/powerpc/include/asm/nohash/32/pte-85xx.h index 653a342d3b25..14d64b4f3f14 100644 --- a/arch/powerpc/include/asm/nohash/32/pte-85xx.h +++ b/arch/powerpc/include/asm/nohash/32/pte-85xx.h @@ -31,9 +31,6 @@ #define _PAGE_WRITETHRU 0x00400 /* H: W bit */ #define _PAGE_SPECIAL 0x00800 /* S: Special page */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 - #define _PMD_PRESENT 0 #define _PMD_PRESENT_MASK (PAGE_MASK) #define _PMD_BAD (~PAGE_MASK) diff --git a/arch/powerpc/include/asm/nohash/32/pte-8xx.h b/arch/powerpc/include/asm/nohash/32/pte-8xx.h index 137dc3c84e45..625c31d6ce5c 100644 --- a/arch/powerpc/include/asm/nohash/32/pte-8xx.h +++ b/arch/powerpc/include/asm/nohash/32/pte-8xx.h @@ -74,12 +74,11 @@ #define _PTE_NONE_MASK 0 #ifdef CONFIG_PPC_16K_PAGES -#define _PAGE_PSIZE _PAGE_SPS +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_SPS) #else -#define _PAGE_PSIZE 0 +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED) #endif -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE) #define _PAGE_BASE (_PAGE_BASE_NC) #include diff --git a/arch/powerpc/include/asm/nohash/pte-e500.h b/arch/powerpc/include/asm/nohash/pte-e500.h index f516f0b5b7a8..975facc7e38e 100644 --- a/arch/powerpc/include/asm/nohash/pte-e500.h +++ b/arch/powerpc/include/asm/nohash/pte-e500.h @@ -65,8 +65,6 @@ #define _PAGE_SPECIAL _PAGE_SW0 -/* Base page size */ -#define _PAGE_PSIZE _PAGE_PSIZE_4K #define PTE_RPN_SHIFT (24) #define PTE_WIMGE_SHIFT (19) @@ -89,7 +87,7 @@ * pages. We always set _PAGE_COHERENT when SMP is enabled or * the processor might need it for DMA coherency. */ -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE) +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE_4K) #if defined(CONFIG_SMP) #define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT) #else From patchwork Tue Jul 2 13:51:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19C88C3064D for ; Tue, 2 Jul 2024 13:52:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6E096B00C0; Tue, 2 Jul 2024 09:52:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1DF16B00C1; Tue, 2 Jul 2024 09:52:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 897D16B00C2; Tue, 2 Jul 2024 09:52:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6B2916B00C0 for ; Tue, 2 Jul 2024 09:52:14 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 24D761201A7 for ; Tue, 2 Jul 2024 13:52:14 +0000 (UTC) X-FDA: 82294951788.15.F53C2C7 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf28.hostedemail.com (Postfix) with ESMTP id C9A79C0019 for ; Tue, 2 Jul 2024 13:52:11 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf28.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928306; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wEsVJ8gVSgYk+6dtILVTpKnQo4KMpLrJnCRKpJLSQLY=; b=QuP46Eor6IlDd2hu0gxX/J1m6qX2369U/YAnq/xz8ZeNzGNb1d885p8pG0t6oO+Ot4drqJ HLYXLgiA8tXv5brEwBfnVrFeGDH3+sURAF5u715MgRUJi7nB0wcdK13+A2LER0r9VJ7AYJ aU+Dev+lLg9IJIHmgKiFvuNnkiuTiCY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928306; a=rsa-sha256; cv=none; b=L2NTQbsQdOvAvAg8r3Xu47SkQB+NeQS2YrG5NPtZM3b3zJd/e54j9AVrU+qbdLnHQ9Enwo 9JDSGfiBbYlcn2m1DiF/nictnGztTD8b+R9YCuMpvZIZhK+7Msma7JRT3rpAQkTXILDqBI 4900i62E0GCETFXFXFSwCExUZHE24G8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf28.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4870pVTz9v7S; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KPV8d17Xh4fc; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4823NW6z9v5M; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6FD2D8B775; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id 65qlaemG0b6Y; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E25778B764; Tue, 2 Jul 2024 15:51:29 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 10/23] powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries Date: Tue, 2 Jul 2024 15:51:22 +0200 Message-ID: <50a3cfbab5b11890a0da027de5cb011a9d47ba89.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=5393; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=CWlk/wMDjpU6vF1ScRB8cosnmevy/J5Gz15W8u/6MPI=; b=3rzRbfyA2AlCNkJeCu2XiBl3uAg8/KojfaYSwhWKf1aOso4kMNeTT76UWD2iqp7CSCaTVAn+J PZzpMH9IhD5DnmKkWg4HENtp4bE9BcNeYHH6MduhpWLtludDoQLmZvQ X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C9A79C0019 X-Stat-Signature: pay6gpj754juk7dz6dk13g548zrqdh6i X-Rspam-User: X-HE-Tag: 1719928331-89602 X-HE-Meta: U2FsdGVkX18GTivvrb94ZedlHasQpkuS67+ilo0y6TF2gn2+YXg9WknF+2iLNcJbrJWAVqXzuEQbXLSL4HjxyJOnkHVtxX/IXrzy7x7YmCpo/vjNMHQ5bJw0zZqwqGDOn7uANaQVHNBT+sC3dx+DyXgmPhj0GAgW6IPG4AuukL+5vGFE4mGt0OJLzGIUsZFrEnY+twM9sv8esMECyUZT/QeJZS6cNK0HDnLZxwjiMPl1g1npZS9f5gj3zxg3VEqpO1hKxERDgmJbYb23wF9M9NssAs58Xzf8Fbu7fewveDVKLlHXa6sVPaUKzXMXp7Axk1rAfF6FUD19EBI0jrg9C9TfAOMftTGQIqdPyR0nE1uxm+8tA1DS69rcVQg7T6UgZ0sRh3uxBfMXDE/y2hoXnH+WTkyjKdhS/F4ydT3Vx0pWD5SHSlo88uV3VlgVXitGT0ezPom6uG6Wkf+1TC9WHC0kiVSZijiubx3DEuqYxAM6NtdtJ5Id13uoB/O3lnTds4aDZ4Mur1DZ32YhjzolQjAg1B7Iuj/9X6ARIq3RXpNuypVSdPRcsIYWV7eYXjoJSdFxYUIZBUP5kaV2pZ+FaQgIYfJLstgpBI+R+ENjsDbVWjgy+cwPfBETc5+UTFWwrcxRIlNKPy5Dl+dIw7jRbsG2xpcDuN2TriHqZxhQJMOq6abO5HLXM4MVfwYzAINiefdyrnWCkd2DECIxPQDiWATzrhOUiIS4l2Qin/QIRbWQk9/0zrsM7BuQMseWWNZ/9MB2d4Jf3GaprE64ChPM7Bvlsdh52m/+mGkiuJcOht9Lx62ovvSuQRfth/JZI6sP2f6xnzx5CFrqi9zTL8gjZzjCuNWfrSFrR7jbbBXIPV2qb78d2q4ZoeO/xISHoLqG/7/Xy9KAgiDBzP43mDGGHdwPBWqHnoCoyphFubdcvv8F7ZksPgOHwCoOYqpPgXtuRtzGj4qHst0bEImKfEv 8qa7u/V3 5ezh6JQmqngsXiFuBufSFcJqkCDLtsnyvnXVmVH6/YaRel0LhCnDoa4nscIhc345u0UFWujIy0PdjPnZt7bYf2E1KiqyiBx0uL/r8fPbQfnnBd/m2zYYfaP8JYHWr1Ytw5M1cHnveNrYEKkB6wM3ZJDj+8aEq8jO7O4MmEi+tmp75jfo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Building on 32 bits with pmd_leaf() not returning always false leads to the following error: CC arch/powerpc/mm/pgtable.o arch/powerpc/mm/pgtable.c: In function '__find_linux_pte': arch/powerpc/mm/pgtable.c:506:1: error: function may return address of local variable [-Werror=return-local-addr] 506 | } | ^ arch/powerpc/mm/pgtable.c:394:15: note: declared here 394 | pud_t pud, *pudp; | ^~~ arch/powerpc/mm/pgtable.c:394:15: note: declared here This is due to pmd_offset() being a no-op in that case. So rework it for powerpc/32 so that pXd_offset() are used on real pointers and not on on-stack copies. Behind fixing the problem, it also has the advantage of simplifying __find_linux_pte() including the removal of stack frame: After this patch: 00000018 <__find_linux_pte>: 18: 2c 06 00 00 cmpwi r6,0 1c: 41 82 00 0c beq 28 <__find_linux_pte+0x10> 20: 39 20 00 00 li r9,0 24: 91 26 00 00 stw r9,0(r6) 28: 2f 85 00 00 cmpwi cr7,r5,0 2c: 41 9e 00 0c beq cr7,38 <__find_linux_pte+0x20> 30: 39 20 00 00 li r9,0 34: 99 25 00 00 stb r9,0(r5) 38: 54 89 65 3a rlwinm r9,r4,12,20,29 3c: 7c 63 48 2e lwzx r3,r3,r9 40: 2f 83 00 00 cmpwi cr7,r3,0 44: 41 9e 00 30 beq cr7,74 <__find_linux_pte+0x5c> 48: 54 69 07 3a rlwinm r9,r3,0,28,29 4c: 2f 89 00 0c cmpwi cr7,r9,12 50: 54 63 00 26 clrrwi r3,r3,12 54: 54 84 b5 36 rlwinm r4,r4,22,20,27 58: 3c 63 c0 00 addis r3,r3,-16384 5c: 7c 63 22 14 add r3,r3,r4 60: 4c be 00 20 bnelr+ cr7 64: 4d 82 00 20 beqlr 68: 39 20 00 17 li r9,23 6c: 91 26 00 00 stw r9,0(r6) 70: 4e 80 00 20 blr 74: 38 60 00 00 li r3,0 78: 4e 80 00 20 blr Before this patch: 00000018 <__find_linux_pte>: 18: 2c 06 00 00 cmpwi r6,0 1c: 94 21 ff e0 stwu r1,-32(r1) 20: 41 82 00 0c beq 2c <__find_linux_pte+0x14> 24: 39 20 00 00 li r9,0 28: 91 26 00 00 stw r9,0(r6) 2c: 2f 85 00 00 cmpwi cr7,r5,0 30: 41 9e 00 0c beq cr7,3c <__find_linux_pte+0x24> 34: 39 20 00 00 li r9,0 38: 99 25 00 00 stb r9,0(r5) 3c: 54 89 65 3a rlwinm r9,r4,12,20,29 40: 7c 63 48 2e lwzx r3,r3,r9 44: 54 69 07 3a rlwinm r9,r3,0,28,29 48: 2f 89 00 0c cmpwi cr7,r9,12 4c: 90 61 00 0c stw r3,12(r1) 50: 41 9e 00 4c beq cr7,9c <__find_linux_pte+0x84> 54: 80 61 00 0c lwz r3,12(r1) 58: 54 69 07 3a rlwinm r9,r3,0,28,29 5c: 2f 89 00 0c cmpwi cr7,r9,12 60: 90 61 00 08 stw r3,8(r1) 64: 41 9e 00 38 beq cr7,9c <__find_linux_pte+0x84> 68: 80 61 00 08 lwz r3,8(r1) 6c: 2f 83 00 00 cmpwi cr7,r3,0 70: 41 9e 00 54 beq cr7,c4 <__find_linux_pte+0xac> 74: 54 69 07 3a rlwinm r9,r3,0,28,29 78: 2f 89 00 0c cmpwi cr7,r9,12 7c: 54 69 00 26 clrrwi r9,r3,12 80: 54 8a b5 36 rlwinm r10,r4,22,20,27 84: 3c 69 c0 00 addis r3,r9,-16384 88: 7c 63 52 14 add r3,r3,r10 8c: 54 84 93 be srwi r4,r4,14 90: 41 9e 00 14 beq cr7,a4 <__find_linux_pte+0x8c> 94: 38 21 00 20 addi r1,r1,32 98: 4e 80 00 20 blr 9c: 54 69 00 26 clrrwi r9,r3,12 a0: 54 84 93 be srwi r4,r4,14 a4: 3c 69 c0 00 addis r3,r9,-16384 a8: 54 84 25 36 rlwinm r4,r4,4,20,27 ac: 7c 63 22 14 add r3,r3,r4 b0: 41 a2 ff e4 beq 94 <__find_linux_pte+0x7c> b4: 39 20 00 17 li r9,23 b8: 91 26 00 00 stw r9,0(r6) bc: 38 21 00 20 addi r1,r1,32 c0: 4e 80 00 20 blr c4: 38 60 00 00 li r3,0 c8: 38 21 00 20 addi r1,r1,32 cc: 4e 80 00 20 blr Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- v3: Removed p4dp and pudp locals for PPC32 and add a comment. v4: Properly set pdshift on PPC32 case v5: Enhanced commit message --- arch/powerpc/mm/pgtable.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 9e7ba9c3851f..bce8a8619589 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -382,8 +382,10 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, bool *is_thp, unsigned *hpage_shift) { pgd_t *pgdp; +#ifdef CONFIG_PPC64 p4d_t p4d, *p4dp; pud_t pud, *pudp; +#endif pmd_t pmd, *pmdp; pte_t *ret_pte; hugepd_t *hpdp = NULL; @@ -401,8 +403,12 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, * page fault or a page unmap. The return pte_t * is still not * stable. So should be checked there for above conditions. * Top level is an exception because it is folded into p4d. + * + * On PPC32, P4D/PUD/PMD are folded into PGD so go straight to + * PMD level. */ pgdp = pgdir + pgd_index(ea); +#ifdef CONFIG_PPC64 p4dp = p4d_offset(pgdp, ea); p4d = READ_ONCE(*p4dp); pdshift = P4D_SHIFT; @@ -442,8 +448,11 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out_huge; } - pdshift = PMD_SHIFT; pmdp = pmd_offset(&pud, ea); +#else + pmdp = pmd_offset(pud_offset(p4d_offset(pgdp, ea), ea), ea); +#endif + pdshift = PMD_SHIFT; pmd = READ_ONCE(*pmdp); /* From patchwork Tue Jul 2 13:51:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCF29C30658 for ; Tue, 2 Jul 2024 13:52:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 518936B00C3; Tue, 2 Jul 2024 09:52:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C7856B00C4; Tue, 2 Jul 2024 09:52:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 368C06B00C5; Tue, 2 Jul 2024 09:52:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 169C36B00C3 for ; Tue, 2 Jul 2024 09:52:19 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B4DFA1201E1 for ; Tue, 2 Jul 2024 13:52:18 +0000 (UTC) X-FDA: 82294951956.20.CE8E2DA Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf13.hostedemail.com (Postfix) with ESMTP id 995C82000B for ; Tue, 2 Jul 2024 13:52:16 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf13.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xnap6QF/MKf6QAqysV3GvrdUulr8iCU56pdyhGUPre8=; b=j2KC0BVYks+PA1/FyHwd2VB6oNg1x7dTZJFMCNxSlvo+HWZSgzKLkBSAe7OeyI1qFxPCtD kD3qrv7/VNMAbO7cDhnQXKpBEW4QB/zdXvjlxfPrLsPS4OR3a9ko5MyUJyIXgz13a+pgQi xIDFCNjhCcSeTxCBri18KuS0vovIsG8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928315; a=rsa-sha256; cv=none; b=Jzv/K5EylAP8QYaWnL82BCDt9FYNMdXP3F2IYwRqfV/SfyrrRJrJl2Fo5HOWQYjWr2C0c0 vdrxxxH0SXFiVvX2EynDvKv5tIcOYBFQ5zcaueGH1JUJ1NFqYw+JKEZA6tjbhT/5y5AqsR xHfvtkXivIxLnHNsLPv78yqVrY93lRk= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf13.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4884Xg2z9v7m; Tue, 2 Jul 2024 15:51:36 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ohgMrFKa0jB3; Tue, 2 Jul 2024 15:51:36 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4826yDHz9v49; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E9E718B775; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id YnHixkjwPp5Q; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6834B8B774; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 11/23] powerpc/mm: Allow hugepages without hugepd Date: Tue, 2 Jul 2024 15:51:23 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928274; l=6864; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=OHpbr2EW8Ehxc4cGWtKirj+0lOuzXeICYvz9BBmzSDA=; b=UUTS4m+0kGxoR7F+40eCTAcfPSd1Ierr8L2y6YOzUEMTf+3WVm1xVUuo4A9pnvI9UqY1rcnsx Wv/fSraJnj1AAABHMvT9B9A+aqHDTkbNzsSIH94CoVRgqUm2zSCuNvv X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Queue-Id: 995C82000B X-Stat-Signature: yny3hrsdik117p4ckeqckymx53eiqxte X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1719928336-827088 X-HE-Meta: U2FsdGVkX19zkBmX8DWg68hF6OfTtLM0Tgs+o83EMLXh8ZzT5d9DkTS2VKiDNPYoj3g50dfpdE9ijn+0p0OvmtI0vK1cvLHt7P0/NPOgekL0qnB0uHAZ9kekujzeAhSutBuUSdYhShnPm/4gx737vvg+0DgWiTS0+n4dGX6uPvz48Luqz7ikFe/8ql4THnv+SvLEbQsi0cX53zhLeY4i9jD6Dgx+CmkvDBaFEHD4MteIPPVaj6kqsuig8Pd84eaRv811mUOscJSYxTzag7la1WXg2JNITqMTPdi3ltzcT7ex7d+RvP40fyinyt9uAKFV7FuOTMg+rLO0GUG6OVoptfaAMU/ICZZtKaDJrgYU+18LCZBFGXZIQEl8HRHjHOorrEUZ1MpkFKARZ6z28XDfjspZ2UbMx5PtKmB9FICoLOlpjta2nzDxUI7DrW3Y6c+5I1PDJavAPoJpHYGEHfKQfuwvtNclD8CQwUSXFneFUmqHrZV63OTW6Rr56b36Bs7pb9+5lRNKJAGEMfKXzbQcJSvu6ZEEK5P7jin6uXOwN1wUfHPFPQ0iRiKHWdg5SksRPRj5qlSlTiUAENijZbEqvw7w12am7PKkAt5zHKVfbJNspopFB/ZdOd+ky3tBRAeMQNSykrcNZQUUXrgBNmUTf/7bYWCILku0LdlmfqMAGq+ivu5MGJaCLMAHxxTQ6eq+mrY3UDeHuXOqFP84pdhJKHYVha2iI9ilSYhqKpm1Gn6Fkn9jeQZNUQ9p0NhyZ75xUtCytn3MHQ5aKIL1K/9hDL+/4L9iDtdL5rivCwQfGBgKuiAs+7RwNWaBOkeK/jJj9iYx2L7scX1zwat2kXyZwLhZEkiQlyBTfljqk6Clda7ftlSWp2vpjFlUq6eGBM2btN0Gfs7HGU6j0Df+JD8Bf1273wDu5uIbAyqWM2UbdNT1nHBbYoJMpM1zkZC6Ib9ZhzRyiNe/hWtwyUD+usL L8awyqFc Oz9w6MKqNEI+AzOdgoxQ8AnBh2Dnyo6lUxHxuAMlewVShw4SSEDF2Ei2ub2zVa/MSwCywcR4PlPHe/1S2KhJXPR07UCH6ksbvXsFZguqh9+Vjv046Ki5w50WkbqzYh3W8KQSMhk3CsBhwhv/pSWyieIC+2rxPHRtQGpwY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation of implementing huge pages on powerpc 8xx without hugepd, enclose hugepd related code inside an ifdef CONFIG_ARCH_HAS_HUGEPD This also allows removing some stubs. Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- v3: - Prepare huge_pte_alloc() for full standard topology, not only for 2-level - Reordered last part of huge_pte_alloc() v4: - Rebased of v6.10-rc1 v5: - Moved the Kconfig split in this patch. --- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 -- arch/powerpc/include/asm/hugetlb.h | 10 ++---- arch/powerpc/include/asm/nohash/pgtable.h | 2 +- arch/powerpc/mm/hugetlbpage.c | 33 ++++++++++++++++++++ arch/powerpc/mm/pgtable.c | 2 ++ arch/powerpc/platforms/Kconfig.cputype | 3 ++ 7 files changed, 41 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c88c6d46a5bc..b60b6e991227 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -135,7 +135,6 @@ config PPC select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_HAS_KCOV select ARCH_HAS_KERNEL_FPU_SUPPORT if PPC64 && PPC_FPU select ARCH_HAS_MEMBARRIER_CALLBACKS diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h index dc5c039eb28e..dd4eb3063175 100644 --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h @@ -47,8 +47,6 @@ static inline void pgtable_free(void *table, unsigned index_size) } } -#define get_hugepd_cache_index(x) (x) - static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) { diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h index ea71f7245a63..79176a499763 100644 --- a/arch/powerpc/include/asm/hugetlb.h +++ b/arch/powerpc/include/asm/hugetlb.h @@ -30,10 +30,12 @@ static inline int is_hugepage_only_range(struct mm_struct *mm, } #define is_hugepage_only_range is_hugepage_only_range +#ifdef CONFIG_ARCH_HAS_HUGEPD #define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); +#endif #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm, @@ -67,14 +69,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, { } -#define hugepd_shift(x) 0 -static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, - unsigned pdshift) -{ - return NULL; -} - - static inline void __init gigantic_hugetlb_cma_reserve(void) { } diff --git a/arch/powerpc/include/asm/nohash/pgtable.h b/arch/powerpc/include/asm/nohash/pgtable.h index f5f39d4f03c8..e7fc1314c23e 100644 --- a/arch/powerpc/include/asm/nohash/pgtable.h +++ b/arch/powerpc/include/asm/nohash/pgtable.h @@ -340,7 +340,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, #define pgprot_writecombine pgprot_noncached_wc -#ifdef CONFIG_HUGETLB_PAGE +#ifdef CONFIG_ARCH_HAS_HUGEPD static inline int hugepd_ok(hugepd_t hpd) { #ifdef CONFIG_PPC_8xx diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 594a4b7b2ca2..20fad59ff9f5 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -42,6 +42,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long s return __find_linux_pte(mm->pgd, addr, NULL, NULL); } +#ifdef CONFIG_ARCH_HAS_HUGEPD static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, unsigned long address, unsigned int pdshift, unsigned int pshift, spinlock_t *ptl) @@ -193,6 +194,36 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, return hugepte_offset(*hpdp, addr, pdshift); } +#else +pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long addr, unsigned long sz) +{ + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + addr &= ~(sz - 1); + + p4d = p4d_offset(pgd_offset(mm, addr), addr); + if (!mm_pud_folded(mm) && sz >= P4D_SIZE) + return (pte_t *)p4d; + + pud = pud_alloc(mm, p4d, addr); + if (!pud) + return NULL; + if (!mm_pmd_folded(mm) && sz >= PUD_SIZE) + return (pte_t *)pud; + + pmd = pmd_alloc(mm, pud, addr); + if (!pmd) + return NULL; + + if (sz >= PMD_SIZE) + return (pte_t *)pmd; + + return pte_alloc_huge(mm, pmd, addr); +} +#endif #ifdef CONFIG_PPC_BOOK3S_64 /* @@ -248,6 +279,7 @@ int __init alloc_bootmem_huge_page(struct hstate *h, int nid) return __alloc_bootmem_huge_page(h, nid); } +#ifdef CONFIG_ARCH_HAS_HUGEPD #ifndef CONFIG_PPC_BOOK3S_64 #define HUGEPD_FREELIST_SIZE \ ((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t)) @@ -505,6 +537,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, } } while (addr = next, addr != end); } +#endif bool __init arch_hugetlb_valid_size(unsigned long size) { diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index bce8a8619589..9010973f036c 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -496,8 +496,10 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, if (!hpdp) return NULL; +#ifdef CONFIG_ARCH_HAS_HUGEPD ret_pte = hugepte_offset(*hpdp, ea, pdshift); pdshift = hugepd_shift(*hpdp); +#endif out: if (hpage_shift) *hpage_shift = pdshift; diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index b2d8c0da2ad9..0c3c69ebf19f 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -37,6 +37,7 @@ config PPC_85xx config PPC_8xx bool "Freescale 8xx" + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select FSL_SOC select PPC_KUEP @@ -98,6 +99,7 @@ config PPC_BOOK3S_64 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING select HAVE_MOVE_PMD @@ -290,6 +292,7 @@ config PPC_BOOK3S config PPC_E500 select FSL_EMB_PERFMON bool + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 select PPC_SMP_MUXED_IPI select PPC_DOORBELL From patchwork Tue Jul 2 13:51:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB45CC3064D for ; Tue, 2 Jul 2024 13:52:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D9456B00C6; Tue, 2 Jul 2024 09:52:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 785E66B00C7; Tue, 2 Jul 2024 09:52:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64ECB6B00C8; Tue, 2 Jul 2024 09:52:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 44A676B00C6 for ; Tue, 2 Jul 2024 09:52:23 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 044991C0884 for ; Tue, 2 Jul 2024 13:52:22 +0000 (UTC) X-FDA: 82294952166.25.B042A77 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf08.hostedemail.com (Postfix) with ESMTP id EC395160021 for ; Tue, 2 Jul 2024 13:52:20 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf08.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928322; a=rsa-sha256; cv=none; b=uc+GzKyq930J/aekCLYN6RFN5fW/V2hgVB2cKZoi6n1FxyaBRQ0Q0H78BnCREQgY9IF6xS 9rtWSMsLqsehpoogjLrH13n+B9A68CsDq5qbgqM0C4WIFgKWHTubW5ymtT0CDJnKGCW6w7 y0+yK564bIKRvyRqLD/amnNoXv9eLlw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf08.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FK7xq7vg1hFjlKdNBvhx1wOeVtV+MZPtFyG/AWrqhmI=; b=TKP93e3OZTYMdLtpSH58IC5vXzyDe7c7VteCnsdaT3D07gJt+bPW89NfdqUvdcoutXbtaj AqqpA5zt9PbGAhc3nvtIiJVTDzV8u+b2ECp7rF9PW6A5oO8/7iBMbVKfgpT9b+A2RZEyHc Zi2ygGtmhbEUaoCZS2H0MehErGcuhYo= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD4890hHzz9v49; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6zAmKaaQ0jHm; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4833KVRz9tZl; Tue, 2 Jul 2024 15:51:31 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6CAF78B774; Tue, 2 Jul 2024 15:51:31 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id zXEONt1F524O; Tue, 2 Jul 2024 15:51:31 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E3F168B764; Tue, 2 Jul 2024 15:51:30 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 12/23] powerpc/8xx: Fix size given to set_huge_pte_at() Date: Tue, 2 Jul 2024 15:51:24 +0200 Message-ID: <97f2090011e25d99b6b0aae73e22e1b921c5d1fb.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=967; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=D5c7rc2WApQ3tCbzTNEVQ0Gk3Td9OpyCeDWt3oeDJWA=; b=uQUtdIUq84SS96MK3Yyi7eFh/R82rncNurgunjBL3cKhAN7hd4/KdduSVyWl3x7FwJ20kvEBT AAWK0LcoeyYCOlx3G8nxXwuG1GtSCMyrGnv/4xWUvckGvRl6DFLaV/A X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: EC395160021 X-Stat-Signature: 36yexdiec8zmeti4pmmtfrnrb35cerea X-Rspam-User: X-HE-Tag: 1719928340-245602 X-HE-Meta: U2FsdGVkX18Mg+VPzgO37RYkQqKAorUl/Z/KABzeZz3R4huQK2ri0XSFCcKR+uZIaOeArC7Cy16fxvqwnDVfOti6Zft93U08H+1YhmRnKFGdQhe9ijEimHkB1EUJIygOKtO1wLsexMQf1vy43MZbRBbD3AcWnw6ACpS8Xc/uW8WgTTUpc2Dm3f/7b6gvcoz1+i96itA4tLyJui8ADvJ4NPFCIfK+XjyHxQ329Y+rMavFmslvjj3PLhEvQdWhBjcyhSWe0owpi5B6yNTn9p7Vn7MDUwqzQr2BxIXa3I3hLModacUVVPAexcv5XOf4Ub6B/UuFOjtEY3ao7ueI0D46ywIrAa5xMyY+00dZPOi7/Ba/MPyuxa/seWrPZUwn/xAr0mfYEaO72PK5Yqon8Trmh2Ib2mg8YlGoMGu9seGBOJiR30NK9lncP+9Y7y+jC4HlohdyYMjkmqHAgPMRuRdy00EgOBw916tnFabsxoxtWV+/7Sbkt7VfVn0l6hP8lb6Uh5Ng+lctQ4KtdNN/hcfJ1k20S95D6OR0hFzhx5Yf6k+VqNuOt63amy7eNHlgE4fljU8CV815ilSHZG0RWJ7CXCUdEktb2reBUJEIuuatFVTXnP0VoFHQa6yBwxH5xieDFeLAvaljogYD2jjcXqCGr01oFiimFRs4G1+8sFfubvxuui7jnILeBCuHtUPJdTqZAGLkHTqRot/qIOtUTXXmYVdcagRlaKJcP/Gta6lb+KgZkOzIjHOohX+VpnV8ixWHWH0nQz4icexIiwZxcvXpp/0Tn4T+UyC+oTrAI6niFj1R0NArEteq+z8k81bfgKsMJPLE2xxrAIpp0fqRRQOkY37bjrN0ITn7Yb6UavYkj3GKRrsVmD4EN+2NRI2UVceZ0mb8eXDrV963iUFP5tKJUqdLVYmcKMrYbmJ9A3o0/dFOtyfeLJGBBVQNgE2v/SyQHmbXbCSQ6nT2t5sYop6 2S+X5HkL AjVDcfZE2mgSAz4ElqIZ7dHeE0BGD6Sjjq/z6t9gUAzZJMN1iUmugVuh22jX0FyllSfbBGlqRrPrd0zEox1iC6r2cX/twmwGHElnESaFpSBA+tQvwxxyxDKyHlRbo9qnDulaeJ3WfOneezkkTERo+a1JBTVQTQGTfC+BmS7JcsLOWjNk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: set_huge_pte_at() expects the size of the hugepage as an int, not the psize which is the index of the page definition in table mmu_psize_defs[] Fixes: 935d4f0c6dc8 ("mm: hugetlb: add huge page size param to set_huge_pte_at()") Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- arch/powerpc/mm/nohash/8xx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/mm/nohash/8xx.c b/arch/powerpc/mm/nohash/8xx.c index 43d4842bb1c7..d93433e26ded 100644 --- a/arch/powerpc/mm/nohash/8xx.c +++ b/arch/powerpc/mm/nohash/8xx.c @@ -94,7 +94,8 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa, return -EINVAL; set_huge_pte_at(&init_mm, va, ptep, - pte_mkhuge(pfn_pte(pa >> PAGE_SHIFT, prot)), psize); + pte_mkhuge(pfn_pte(pa >> PAGE_SHIFT, prot)), + 1UL << mmu_psize_to_shift(psize)); return 0; } From patchwork Tue Jul 2 13:51:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 823F3C3064D for ; Tue, 2 Jul 2024 13:52:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0ACE46B0089; Tue, 2 Jul 2024 09:52:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05C2A6B00C9; Tue, 2 Jul 2024 09:52:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF0ED6B00CA; Tue, 2 Jul 2024 09:52:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BB24D6B00C8 for ; Tue, 2 Jul 2024 09:52:27 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7008F1A01DE for ; Tue, 2 Jul 2024 13:52:27 +0000 (UTC) X-FDA: 82294952334.18.9152B37 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf08.hostedemail.com (Postfix) with ESMTP id 2C83D160018 for ; Tue, 2 Jul 2024 13:52:24 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZM6f68Ny0QOf7kfrW57eolnhNCt08Shk9qvdiv5N5tQ=; b=cWPahEU+5uKhAHrqHW09ll0vaPubrEWzklxJK9vVI94MhRHJXCHqvAu0FfROLYsJpVaSEs VQi2hQO+E9wceJVUYRoD54q3dvx7H9DKGTEc5RzKcN796EVQ1zFvmXSxIZtnIIhyxySDo+ L7UG0sCkEyBc5Tv9Hp2iH7be54aOkP0= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928315; a=rsa-sha256; cv=none; b=KDMXRTcj2uDiOanHnLYZC+9x++L93mFoskrrTKgjkgV6GbZx5JffOXbicIWpQn4gPuYFuT VceBwge2U2s/bJ+kZlCCJsmmZq+NsOctkU1NaHLthqf/2qGWGxfprtMHNqq02q1zKFk0fm N0btlSzXDjWvV+JAs3dRy+WfpEhFeps= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48B3SnZz9v2V; Tue, 2 Jul 2024 15:51:38 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3LXnyvYgdd0O; Tue, 2 Jul 2024 15:51:38 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4840lTHz9v6D; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 14B5A8B774; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id jUWgBYeMB-7J; Tue, 2 Jul 2024 15:51:31 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6EF3D8B775; Tue, 2 Jul 2024 15:51:31 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 13/23] powerpc/8xx: Rework support for 8M pages using contiguous PTE entries Date: Tue, 2 Jul 2024 15:51:25 +0200 Message-ID: <8693d9a0408371043ca63bf9e4a9c140667af63e.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=18915; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=cAPYqUBYshpLkzOaz4AEKHKXfgJcSbmgHaFhwK2BmII=; b=rZ1il+UyN4Njo0tUoIqAO6czpcv7rNX1NjeDAs9+WHsLGwmOPx6x47p24btAcSWNyA5sahn7i +uOaRC4Hj9yD691CuaAcCePdrd6uSrtIMmAhL7q3vSXD+9D5mlMfgpX X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2C83D160018 X-Stat-Signature: mk9ryabbc1xsrdk1cewd3en5yut6cfpz X-Rspam-User: X-HE-Tag: 1719928344-155982 X-HE-Meta: U2FsdGVkX1+NfOdhvbVNhJ1JqWUJhbxqHe71z3SJE5fDW5KMS2U8//7KmfCWpuF8SEWJC2Zql3A2hMkzlpfDFSTobh0CXn33HyhqFulUsVATq+ltQQM1FDno4MEjNEYWSHY+Oa1mhsHtSehJH8RhmxItBmiui4+WB2qMcrfZQKJtVz8czvC95zsHAza7zca7ahEGnxP2p4pLaW2Pv9+MCgrFAf4MAb7Pi70v4VPezRt325ft0NXuggvMxvKPIA6CGtVMWGwUHs9KF2yNP1xyO+ftYXktdzu1B7wkApOKhEFNEyMEo7CA3Rq6ZgDe63lQPNGV/L/wD3IZi7gHVi1yDlpEbG1Ttd7XfhG+uKE6aV6T8CGz5qq/M2WgTmvJKmE9QhITvdb+rXsT2asc2+VclUagNGN1EPfAEN1V+TgqufBtzmKdc/5yWyv/YlwIO+1slviq2BWc5+YrEZk1vMwPF6Q4lKZtaKJ1dglQOFibf13eU8o3Vc3jCt3cEWMH7oCtxG4i3tOz52eiY+1OBsqmNhASU8qXXRW3bU5qBtXfdT7pUewp+/KiISOd1nBedumN3DuGgyZsDA9gFzxluUHX+/fagtwVKVHDfBL1VecArmsTCgUzNv/TYOVcMXH1kHxj90VwipzrfsQ8t3I5+QTkUr/b6XzDHrUHVomnDADTq9FRStZPpsQViZ3L5zlb5V6LScfuVAxmFMwi/8+sw8fAQe1WFR7uAwXO11zSp+BZgjBeAQ8hbCWSQ4SlimIaqW2GusaXjuRrnFvZ8qp7qr50aKBPUEtlihvMCNX+Am3QWh3N1/Q3ANoXOsiLQ5WHnA9C5gZtWbeSQSKksEvK8tp3c6Lvuyy6OvqWZ8jsQCtm1CUS9XJJ0sHy1C8y0T2qBhjX2wEOAdKWSqpozEUxIMU9h1RtIFO76L5YZiCJbWg4kHb1zFp4gCJ++lVE5bUgxUXO5zWtAwB19Ql7iAtJ3R8 We/7r69J f70u+gYS1dCCBNKnQJ/NdPJqa5O4NIFiXuvv3mGcm2I4QBsaojmJsFqaEbNBeq1VUMss/StePli3DPJL97ufSKx87/1UUpcqopJgRhPFzdxT0cWDGXMM8lm2xHZ0avqRHh5Vf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to fit better with standard Linux page tables layout, add support for 8M pages using contiguous PTE entries in a standard page table. Page tables will then be populated with 1024 similar entries and two PMD entries will point to that page table. The PMD entries also get a flag to tell it is addressing an 8M page, this is required for the HW tablewalk assistance. Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- v3: - Move huge_ptep_get() for a more readable commit diff - Flag PMD as 8Mbytes in set_huge_pte_at() - Define __pte_leaf_size() - Change pte_update() instead of all huge callers of pte_update() - Added ptep_is_8m_pmdp() helper - Fixed kasan early memory 8M allocation v5: - In huge_ptep_get(), change pte_offset_kernel((pmd_t *)ptep, 0) to pte_offset_kernel((pmd_t *)ptep, ALIGN_DOWN(addr, SZ_8M)) which is more correct allthough not different. --- .../include/asm/nohash/32/hugetlb-8xx.h | 38 +++---------- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 53 ++++++++++++------- arch/powerpc/include/asm/nohash/pgtable.h | 4 -- arch/powerpc/include/asm/page.h | 5 -- arch/powerpc/include/asm/pgtable.h | 3 ++ arch/powerpc/kernel/head_8xx.S | 10 +--- arch/powerpc/mm/hugetlbpage.c | 21 +++++--- arch/powerpc/mm/kasan/8xx.c | 21 +++++--- arch/powerpc/mm/nohash/8xx.c | 40 +++++++------- arch/powerpc/mm/pgtable.c | 27 +++++++--- arch/powerpc/mm/pgtable_32.c | 2 +- arch/powerpc/platforms/Kconfig.cputype | 1 - 12 files changed, 111 insertions(+), 114 deletions(-) diff --git a/arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h b/arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h index 92df40c6cc6b..014799557f60 100644 --- a/arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h +++ b/arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h @@ -4,42 +4,12 @@ #define PAGE_SHIFT_8M 23 -static inline pte_t *hugepd_page(hugepd_t hpd) -{ - BUG_ON(!hugepd_ok(hpd)); - - return (pte_t *)__va(hpd_val(hpd) & ~HUGEPD_SHIFT_MASK); -} - -static inline unsigned int hugepd_shift(hugepd_t hpd) -{ - return PAGE_SHIFT_8M; -} - -static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, - unsigned int pdshift) -{ - unsigned long idx = (addr & (SZ_4M - 1)) >> PAGE_SHIFT; - - return hugepd_page(hpd) + idx; -} - static inline void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr) { flush_tlb_page(vma, vmaddr); } -static inline void hugepd_populate(hugepd_t *hpdp, pte_t *new, unsigned int pshift) -{ - *hpdp = __hugepd(__pa(new) | _PMD_USER | _PMD_PRESENT | _PMD_PAGE_8M); -} - -static inline void hugepd_populate_kernel(hugepd_t *hpdp, pte_t *new, unsigned int pshift) -{ - *hpdp = __hugepd(__pa(new) | _PMD_PRESENT | _PMD_PAGE_8M); -} - static inline int check_and_get_huge_psize(int shift) { return shift_to_mmu_psize(shift); @@ -49,6 +19,14 @@ static inline int check_and_get_huge_psize(int shift) void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_GET +static inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) +{ + if (ptep_is_8m_pmdp(mm, addr, ptep)) + ptep = pte_offset_kernel((pmd_t *)ptep, ALIGN_DOWN(addr, SZ_8M)); + return ptep_get(ptep); +} + #define __HAVE_ARCH_HUGE_PTE_CLEAR static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz) diff --git a/arch/powerpc/include/asm/nohash/32/pte-8xx.h b/arch/powerpc/include/asm/nohash/32/pte-8xx.h index 625c31d6ce5c..54ebb91dbdcf 100644 --- a/arch/powerpc/include/asm/nohash/32/pte-8xx.h +++ b/arch/powerpc/include/asm/nohash/32/pte-8xx.h @@ -119,7 +119,7 @@ static inline pte_t pte_mkhuge(pte_t pte) #define pte_mkhuge pte_mkhuge -static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p, +static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long clr, unsigned long set, int huge); static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -141,19 +141,12 @@ static inline void __ptep_set_access_flags(struct vm_area_struct *vma, pte_t *pt } #define __ptep_set_access_flags __ptep_set_access_flags -static inline unsigned long pgd_leaf_size(pgd_t pgd) -{ - if (pgd_val(pgd) & _PMD_PAGE_8M) - return SZ_8M; - return SZ_4M; -} - -#define pgd_leaf_size pgd_leaf_size - -static inline unsigned long pte_leaf_size(pte_t pte) +static inline unsigned long __pte_leaf_size(pmd_t pmd, pte_t pte) { pte_basic_t val = pte_val(pte); + if (pmd_val(pmd) & _PMD_PAGE_8M) + return SZ_8M; if (val & _PAGE_HUGE) return SZ_512K; if (val & _PAGE_SPS) @@ -161,31 +154,38 @@ static inline unsigned long pte_leaf_size(pte_t pte) return SZ_4K; } -#define pte_leaf_size pte_leaf_size +#define __pte_leaf_size __pte_leaf_size /* * On the 8xx, the page tables are a bit special. For 16k pages, we have * 4 identical entries. For 512k pages, we have 128 entries as if it was * 4k pages, but they are flagged as 512k pages for the hardware. - * For other page sizes, we have a single entry in the table. + * For 8M pages, we have 1024 entries as if it was 4M pages (PMD_SIZE) + * but they are flagged as 8M pages for the hardware. + * For 4k pages, we have a single entry in the table. */ static pmd_t *pmd_off(struct mm_struct *mm, unsigned long addr); -static int hugepd_ok(hugepd_t hpd); +static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address); + +static inline bool ptep_is_8m_pmdp(struct mm_struct *mm, unsigned long addr, pte_t *ptep) +{ + return (pmd_t *)ptep == pmd_off(mm, ALIGN_DOWN(addr, SZ_8M)); +} static inline int number_of_cells_per_pte(pmd_t *pmd, pte_basic_t val, int huge) { if (!huge) return PAGE_SIZE / SZ_4K; - else if (hugepd_ok(*((hugepd_t *)pmd))) - return 1; + else if ((pmd_val(*pmd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M) + return SZ_4M / SZ_4K; else if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !(val & _PAGE_HUGE)) return SZ_16K / SZ_4K; else return SZ_512K / SZ_4K; } -static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p, - unsigned long clr, unsigned long set, int huge) +static inline pte_basic_t __pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p, + unsigned long clr, unsigned long set, int huge) { pte_basic_t *entry = (pte_basic_t *)p; pte_basic_t old = pte_val(*p); @@ -197,7 +197,7 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p for (i = 0; i < num; i += PAGE_SIZE / SZ_4K, new += PAGE_SIZE) { *entry++ = new; - if (IS_ENABLED(CONFIG_PPC_16K_PAGES) && num != 1) { + if (IS_ENABLED(CONFIG_PPC_16K_PAGES)) { *entry++ = new; *entry++ = new; *entry++ = new; @@ -207,6 +207,21 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p return old; } +static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + unsigned long clr, unsigned long set, int huge) +{ + pte_basic_t old; + + if (huge && ptep_is_8m_pmdp(mm, addr, ptep)) { + pmd_t *pmdp = (pmd_t *)ptep; + + old = __pte_update(mm, addr, pte_offset_kernel(pmdp, 0), clr, set, huge); + __pte_update(mm, addr, pte_offset_kernel(pmdp + 1, 0), clr, set, huge); + } else { + old = __pte_update(mm, addr, ptep, clr, set, huge); + } + return old; +} #define pte_update pte_update #ifdef CONFIG_PPC_16K_PAGES diff --git a/arch/powerpc/include/asm/nohash/pgtable.h b/arch/powerpc/include/asm/nohash/pgtable.h index e7fc1314c23e..90d6a0943b35 100644 --- a/arch/powerpc/include/asm/nohash/pgtable.h +++ b/arch/powerpc/include/asm/nohash/pgtable.h @@ -343,12 +343,8 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, #ifdef CONFIG_ARCH_HAS_HUGEPD static inline int hugepd_ok(hugepd_t hpd) { -#ifdef CONFIG_PPC_8xx - return ((hpd_val(hpd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M); -#else /* We clear the top bit to indicate hugepd */ return (hpd_val(hpd) && (hpd_val(hpd) & PD_HUGE) == 0); -#endif } #define is_hugepd(hpd) (hugepd_ok(hpd)) diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h index e411e5a70ea3..018c3d55232c 100644 --- a/arch/powerpc/include/asm/page.h +++ b/arch/powerpc/include/asm/page.h @@ -293,13 +293,8 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) /* * Some number of bits at the level of the page table that points to * a hugepte are used to encode the size. This masks those bits. - * On 8xx, HW assistance requires 4k alignment for the hugepte. */ -#ifdef CONFIG_PPC_8xx -#define HUGEPD_SHIFT_MASK 0xfff -#else #define HUGEPD_SHIFT_MASK 0x3f -#endif #ifndef __ASSEMBLY__ diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h index 239709a2f68e..264a6c09517a 100644 --- a/arch/powerpc/include/asm/pgtable.h +++ b/arch/powerpc/include/asm/pgtable.h @@ -106,6 +106,9 @@ unsigned long vmalloc_to_phys(void *vmalloc_addr); void pgtable_cache_add(unsigned int shift); +#ifdef CONFIG_PPC32 +void __init *early_alloc_pgtable(unsigned long size); +#endif pte_t *early_pte_alloc_kernel(pmd_t *pmdp, unsigned long va); #if defined(CONFIG_STRICT_KERNEL_RWX) || defined(CONFIG_PPC32) diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S index edc479a7c2bc..ac74321b1192 100644 --- a/arch/powerpc/kernel/head_8xx.S +++ b/arch/powerpc/kernel/head_8xx.S @@ -415,14 +415,13 @@ FixupDAR:/* Entry point for dcbx workaround. */ oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha 3: lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + rlwinm r11, r11, 0, ~_PMD_PAGE_8M mtspr SPRN_MD_TWC, r11 - mtcrf 0x01, r11 mfspr r11, SPRN_MD_TWC lwz r11, 0(r11) /* Get the pte */ - bt 28,200f /* bit 28 = Large page (8M) */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT, 31 -201: lwz r11,0(r11) + lwz r11,0(r11) /* Check if it really is a dcbx instruction. */ /* dcbt and dcbtst does not generate DTLB Misses/Errors, * no need to include them here */ @@ -441,11 +440,6 @@ FixupDAR:/* Entry point for dcbx workaround. */ 141: mfspr r10,SPRN_M_TW b DARFixed /* Nope, go back to normal TLB processing */ -200: - /* concat physical page address(r11) and page offset(r10) */ - rlwimi r11, r10, 0, 32 - PAGE_SHIFT_8M, 31 - b 201b - 144: mfspr r10, SPRN_DSISR rlwinm r10, r10,0,7,5 /* Clear store bit for buggy dcbst insn */ mtspr SPRN_DSISR, r10 diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 20fad59ff9f5..394846197974 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -183,9 +183,6 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, if (!hpdp) return NULL; - if (IS_ENABLED(CONFIG_PPC_8xx) && pshift < PMD_SHIFT) - return pte_alloc_huge(mm, (pmd_t *)hpdp, addr); - BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp)); if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, @@ -218,8 +215,18 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, if (!pmd) return NULL; - if (sz >= PMD_SIZE) + if (sz >= PMD_SIZE) { + /* On 8xx, all hugepages are handled as contiguous PTEs */ + if (IS_ENABLED(CONFIG_PPC_8xx)) { + int i; + + for (i = 0; i < sz / PMD_SIZE; i++) { + if (!pte_alloc_huge(mm, pmd + i, addr)) + return NULL; + } + } return (pte_t *)pmd; + } return pte_alloc_huge(mm, pmd, addr); } @@ -617,10 +624,8 @@ static int __init hugetlbpage_init(void) * use pgt cache for hugepd. */ if (pdshift > shift) { - if (!IS_ENABLED(CONFIG_PPC_8xx)) - pgtable_cache_add(pdshift - shift); - } else if (IS_ENABLED(CONFIG_PPC_E500) || - IS_ENABLED(CONFIG_PPC_8xx)) { + pgtable_cache_add(pdshift - shift); + } else if (IS_ENABLED(CONFIG_PPC_E500)) { pgtable_cache_add(PTE_T_ORDER); } diff --git a/arch/powerpc/mm/kasan/8xx.c b/arch/powerpc/mm/kasan/8xx.c index 2784224054f8..989d6cdf4141 100644 --- a/arch/powerpc/mm/kasan/8xx.c +++ b/arch/powerpc/mm/kasan/8xx.c @@ -6,28 +6,33 @@ #include #include +#include + static int __init kasan_init_shadow_8M(unsigned long k_start, unsigned long k_end, void *block) { pmd_t *pmd = pmd_off_k(k_start); unsigned long k_cur, k_next; - for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd += 2, block += SZ_8M) { - pte_basic_t *new; + for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++, block += SZ_4M) { + pte_t *ptep; + int i; k_next = pgd_addr_end(k_cur, k_end); - k_next = pgd_addr_end(k_next, k_end); if ((void *)pmd_page_vaddr(*pmd) != kasan_early_shadow_pte) continue; - new = memblock_alloc(sizeof(pte_basic_t), SZ_4K); - if (!new) + ptep = memblock_alloc(PTE_FRAG_SIZE, PTE_FRAG_SIZE); + if (!ptep) return -ENOMEM; - *new = pte_val(pte_mkhuge(pfn_pte(PHYS_PFN(__pa(block)), PAGE_KERNEL))); + for (i = 0; i < PTRS_PER_PTE; i++) { + pte_t pte = pte_mkhuge(pfn_pte(PHYS_PFN(__pa(block + i * PAGE_SIZE)), PAGE_KERNEL)); - hugepd_populate_kernel((hugepd_t *)pmd, (pte_t *)new, PAGE_SHIFT_8M); - hugepd_populate_kernel((hugepd_t *)pmd + 1, (pte_t *)new, PAGE_SHIFT_8M); + __set_pte_at(&init_mm, k_cur, ptep + i, pte, 1); + } + pmd_populate_kernel(&init_mm, pmd, ptep); + *pmd = __pmd(pmd_val(*pmd) | _PMD_PAGE_8M); } return 0; } diff --git a/arch/powerpc/mm/nohash/8xx.c b/arch/powerpc/mm/nohash/8xx.c index d93433e26ded..388bba0ab3e7 100644 --- a/arch/powerpc/mm/nohash/8xx.c +++ b/arch/powerpc/mm/nohash/8xx.c @@ -11,6 +11,7 @@ #include #include +#include #include @@ -48,20 +49,6 @@ unsigned long p_block_mapped(phys_addr_t pa) return 0; } -static pte_t __init *early_hugepd_alloc_kernel(hugepd_t *pmdp, unsigned long va) -{ - if (hpd_val(*pmdp) == 0) { - pte_t *ptep = memblock_alloc(sizeof(pte_basic_t), SZ_4K); - - if (!ptep) - return NULL; - - hugepd_populate_kernel((hugepd_t *)pmdp, ptep, PAGE_SHIFT_8M); - hugepd_populate_kernel((hugepd_t *)pmdp + 1, ptep, PAGE_SHIFT_8M); - } - return hugepte_offset(*(hugepd_t *)pmdp, va, PGDIR_SHIFT); -} - static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa, pgprot_t prot, int psize, bool new) { @@ -75,24 +62,33 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa, if (WARN_ON(slab_is_available())) return -EINVAL; - if (psize == MMU_PAGE_512K) + if (psize == MMU_PAGE_512K) { ptep = early_pte_alloc_kernel(pmdp, va); - else - ptep = early_hugepd_alloc_kernel((hugepd_t *)pmdp, va); + /* The PTE should never be already present */ + if (WARN_ON(pte_present(*ptep) && pgprot_val(prot))) + return -EINVAL; + } else { + if (WARN_ON(!pmd_none(*pmdp) || !pmd_none(*(pmdp + 1)))) + return -EINVAL; + + ptep = early_alloc_pgtable(PTE_FRAG_SIZE); + pmd_populate_kernel(&init_mm, pmdp, ptep); + + ptep = early_alloc_pgtable(PTE_FRAG_SIZE); + pmd_populate_kernel(&init_mm, pmdp + 1, ptep); + + ptep = (pte_t *)pmdp; + } } else { if (psize == MMU_PAGE_512K) ptep = pte_offset_kernel(pmdp, va); else - ptep = hugepte_offset(*(hugepd_t *)pmdp, va, PGDIR_SHIFT); + ptep = (pte_t *)pmdp; } if (WARN_ON(!ptep)) return -ENOMEM; - /* The PTE should never be already present */ - if (new && WARN_ON(pte_present(*ptep) && pgprot_val(prot))) - return -EINVAL; - set_huge_pte_at(&init_mm, va, ptep, pte_mkhuge(pfn_pte(pa >> PAGE_SHIFT, prot)), 1UL << mmu_psize_to_shift(psize)); diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 9010973f036c..294775c793ab 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -297,11 +297,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, } #if defined(CONFIG_PPC_8xx) -void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, - pte_t pte, unsigned long sz) +static void __set_huge_pte_at(pmd_t *pmd, pte_t *ptep, pte_basic_t val) { - pmd_t *pmd = pmd_off(mm, addr); - pte_basic_t val; pte_basic_t *entry = (pte_basic_t *)ptep; int num, i; @@ -311,15 +308,29 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, */ VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep)); - pte = set_pte_filter(pte, addr); - - val = pte_val(pte); - num = number_of_cells_per_pte(pmd, val, 1); for (i = 0; i < num; i++, entry++, val += SZ_4K) *entry = val; } + +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte, unsigned long sz) +{ + pmd_t *pmdp = pmd_off(mm, addr); + + pte = set_pte_filter(pte, addr); + + if (sz == SZ_8M) { /* Flag both PMD entries as 8M and fill both page tables */ + *pmdp = __pmd(pmd_val(*pmdp) | _PMD_PAGE_8M); + *(pmdp + 1) = __pmd(pmd_val(*(pmdp + 1)) | _PMD_PAGE_8M); + + __set_huge_pte_at(pmdp, pte_offset_kernel(pmdp, 0), pte_val(pte)); + __set_huge_pte_at(pmdp, pte_offset_kernel(pmdp + 1, 0), pte_val(pte) + SZ_4M); + } else { + __set_huge_pte_at(pmdp, ptep, pte_val(pte)); + } +} #endif #endif /* CONFIG_HUGETLB_PAGE */ diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c index cfd622ebf774..787b22206386 100644 --- a/arch/powerpc/mm/pgtable_32.c +++ b/arch/powerpc/mm/pgtable_32.c @@ -48,7 +48,7 @@ notrace void __init early_ioremap_init(void) early_ioremap_setup(); } -static void __init *early_alloc_pgtable(unsigned long size) +void __init *early_alloc_pgtable(unsigned long size) { void *ptr = memblock_alloc(size, size); diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index 0c3c69ebf19f..fa4bb096b3ae 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -37,7 +37,6 @@ config PPC_85xx config PPC_8xx bool "Freescale 8xx" - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select FSL_SOC select PPC_KUEP From patchwork Tue Jul 2 13:51:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF26BC3064D for ; Tue, 2 Jul 2024 13:52:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42D626B00CB; Tue, 2 Jul 2024 09:52:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B5946B00CC; Tue, 2 Jul 2024 09:52:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 257836B00CD; Tue, 2 Jul 2024 09:52:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 03DE56B00CB for ; Tue, 2 Jul 2024 09:52:31 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9124040155 for ; Tue, 2 Jul 2024 13:52:31 +0000 (UTC) X-FDA: 82294952502.06.B52ABDA Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf07.hostedemail.com (Postfix) with ESMTP id 6E0C540017 for ; Tue, 2 Jul 2024 13:52:29 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; spf=pass (imf07.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ry7wjRmAbrCChnbrq+gG2BW8QdTQw9qwIvifIWQPxK0=; b=zTc5OX45+V22aK6SW5dKy6wLNRy8i702sh0rEifGZfleg/mi/j4b14N+3x8weZWzlAvkYA WzKd9g6yWLsgC33wrat21EoAlUqBJV/fsXtxddrIBe6mntKe+mQYodzjg6hX1rCRrinKuV sFLTzgCXtbOTvtYHr96EPb36fzvWq14= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928327; a=rsa-sha256; cv=none; b=kQvPUJVwl87oCMOfiMBoByfayUp9Bp78+tIdonqzKykYqF99A2yjiCI6FxsQJwyKtz80Lp yyxVNLlXKlDT+Rfrrghxuc6E1NurcMstkTvG/N4YG00JJaxaA1QXmj6Eul3v26ttFQiWIo 8wcTE/hezMzpzZ7SEMEcTY65KJmWM1o= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; spf=pass (imf07.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48C2f56z9v6D; Tue, 2 Jul 2024 15:51:39 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WlZ9TyEo5Dgx; Tue, 2 Jul 2024 15:51:39 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4844960z9v4J; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 8AF8F8B774; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id B0dbSiqMZunM; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 0E1988B764; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 14/23] powerpc/8xx: Simplify struct mmu_psize_def Date: Tue, 2 Jul 2024 15:51:26 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=1334; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=foRpSg705p76Fv1R91+1LawwFjSx303hjdxd+UuPeuU=; b=7ve/BvHdIJxzyweKh0VcZc2SKlNEWuWVtKP7D0YWuJZRMM79S4BQXvO4BgF6GsEMy3tUx7KzS I3U/N1LXGBGCK2Kfv+EQRWRK3z5XCXLnsOZAeet8T5FniIfLbs8pKb7 X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6E0C540017 X-Stat-Signature: 1w3zwqp4d8kxeftgfa5gggd6khu9hir4 X-HE-Tag: 1719928349-795347 X-HE-Meta: U2FsdGVkX1/bXFfqvLuh4aJlAzKvpL85CJPeyrGU/g+XSkzkBiFNDDxYlf6xptNStRQ+GcGKQCx4RXvJ2IRsywFHkW8pSGaIjKL7Mfsz4rl9/Gur58L6701lvAlKuJ0BR74nSWOxUjZAD+nyYokKp61NXHNwrqaux8RmGFfw5Cw0BFOYaVg4YwDWHJpHa+XqwARJkymycNb9da9G6fA8kMDtR+q98DQl7g9EtH9TVFtms5S7OcBQd5V0wfTVN1c1t4Y5oKUyl5luD+43T/Yz51t7eZtWQ1NHqVr35Um8YNwj0ZgC8ByA5vudiQME2WMHkFHQZ/mxZuX7zIEDF1HPZpdUr39GG3Fz0bFhxWotBdIwBixO9rvFTtBC63UlJFOtItfeNEHQbw2EtQQtQmBlxerMBohjieB8u2bETiyL1TEKSeCQ6c0s3vsPsYMlUk/2Y3R88RVR2C1GEQP/mKDCagrADO1X8mzN8ZHNChG4NQSxf099zUs3dhgaI4zVJRgXmVhGENWGHqc/R1IeWSHlDtA+LCJqxcubfJFTQu16JAJe/gMtmgf98cZ/10W4VBORTEInsPtDdH/HK+P7OIDrizrmBtp2NaF9+Bl9UQS12raOUlEDWPpSt+0oSw7AN5e5CBU++/EHAWYWr92lAGSiClfTL3sYEido3mIykJbTcqo4NdZHb0L4qI6v1+0eCMYnswHLB/sivdQ/5cgX9Mx84m1ONcZ00ArogcRkKvvivMIU1ZRQyk6Zp0+KejCnaCHneRmo+iwY0QTsiYGfbojBAs9Y4Tk8oWhGstK2zDDr05FIlRRe+BU9HgiucuR9k3qLYS5R3rxEPzmnrHjpGumuUMlHey8RuA7cAU37qK1kOtrqqdSeV1PIIuuF7Uqj758IdU6AhZvU3PZPf1ikTJHWiuMeetFmd7+alogUBHBHNThQr377ZGeodrNB9sBzIRuWJ9rNhtOZSsO55OJ+I3O HOber5ID W7G4jIPdJZ1+YgnJTungW0cAEWHrv1iFrvMQ15amNdaUvgJyjAh6Ks9k5EzjCaPKhYmcPyieb45JjsNrNG2v8fYmvB6HP3py3GKwvQe9XeTTvJId9u7Glj8yYRG30TUJjI1LcgqLmsAJaICj+vXB0Xlc4QDuKDqkj/01WQhZjs69Q5AY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 8xx, only the shift field is used in struct mmu_psize_def Remove other fields and related macros. Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/include/asm/nohash/32/mmu-8xx.h b/arch/powerpc/include/asm/nohash/32/mmu-8xx.h index 141d82e249a8..a756a1e59c54 100644 --- a/arch/powerpc/include/asm/nohash/32/mmu-8xx.h +++ b/arch/powerpc/include/asm/nohash/32/mmu-8xx.h @@ -189,19 +189,14 @@ typedef struct { #define PHYS_IMMR_BASE (mfspr(SPRN_IMMR) & 0xfff80000) -/* Page size definitions, common between 32 and 64-bit +/* + * Page size definitions for 8xx * * shift : is the "PAGE_SHIFT" value for that page size - * penc : is the pte encoding mask * */ struct mmu_psize_def { unsigned int shift; /* number of bits */ - unsigned int enc; /* PTE encoding */ - unsigned int ind; /* Corresponding indirect page size shift */ - unsigned int flags; -#define MMU_PAGE_SIZE_DIRECT 0x1 /* Supported as a direct size */ -#define MMU_PAGE_SIZE_INDIRECT 0x2 /* Supported as an indirect size */ }; extern struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT]; From patchwork Tue Jul 2 13:51:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAD91C30658 for ; Tue, 2 Jul 2024 13:52:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 762E06B00CE; Tue, 2 Jul 2024 09:52:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 711D56B00CF; Tue, 2 Jul 2024 09:52:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58A846B00D0; Tue, 2 Jul 2024 09:52:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3A62E6B00CE for ; Tue, 2 Jul 2024 09:52:36 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E7A131601AB for ; Tue, 2 Jul 2024 13:52:35 +0000 (UTC) X-FDA: 82294952670.22.C02F323 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf27.hostedemail.com (Postfix) with ESMTP id 94E6040013 for ; Tue, 2 Jul 2024 13:52:33 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; spf=pass (imf27.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GUwsHog4uuWwb0OErwYz1eYxTd3EDOsz1QgWZ06a0N0=; b=skj4MbEsE0n5Xv1POQB9SMWBxdHxKOqUsP3rmJ11kIejpQ0x62FaYvosPBulQR0TBVOTEN WC93R9NCaTGUXR3HLMGku/dk/MN+BWF9mQ5HXGANKRchesySLp4zu2aCV6ETF8s/ZddhYU hXLpNdkIbhX/LrmZzr45tPQBXB8Uctg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; spf=pass (imf27.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928343; a=rsa-sha256; cv=none; b=IhN76KGmf29SEWZk76cdFnHb6XEP+QUuD9ufc0am3qQVnnBdE5hWTSQK3YFjsfoGYOMnp6 hWWbuciLPBWhwbh/zML2t4ynhu17H/qjY0LoM7PXUCB3ERrC+jLYbP55Bb6vZ58bsU6/2y jqzD3ZarnKUnqBc2LoMS5zOiRaNyH9I= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48D0xTmz9v86; Tue, 2 Jul 2024 15:51:40 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9B-ocB9gqIgs; Tue, 2 Jul 2024 15:51:40 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4852k4Vz9v4N; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 588358B775; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id lNIOyt1GS2mJ; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 903A18B764; Tue, 2 Jul 2024 15:51:32 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 15/23] powerpc/e500: Remove enc and ind fields from struct mmu_psize_def Date: Tue, 2 Jul 2024 15:51:27 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=3777; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=GvUuKmINqllZ54PyQ+lDpy16CYLzBdm/Cr1rg49spF8=; b=Hq6AokvcEQX3KXHKd1MjTChdvR3DS0zwpXsD9x0ejqMg2bkdbI+1N7nb2He8rvXKXUn5mV0a7 sanxVNVG2jzCW4FrWfzM4A943aYO0AI9wKWCW6tI/sfYPbWjUOG8xwR X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 94E6040013 X-Stat-Signature: 9iwu1sorjh4adtdbuzj1yjmka8b85t8b X-HE-Tag: 1719928353-365525 X-HE-Meta: U2FsdGVkX19nNdfwBbk3SYqFHLcPBx7a9jaWdAZlAVASBQQGwqNIf4Uq90hSi7/vuaYCkY0wblXxm4Z2pwfSYaWw7Q8nLKk9l03zb1AiVHQIv3CFq+csfsM6uT2Ak3epFspBIIBV6+MY/H4JA77QUDsFU0u+EQn3kZwrPp4codr+mB8u1TziSiTecqucqY360wV/TJekkbgkzKjOpHkNCiLmYSupXl6/ZUglBP22DGnO/VmHkCj2MdLgxWPOYQBl8Pc2eUgHtlnR6bt2R6fohdNnHpPZO7fjvcKmJyPcHi7osp8FQryVMOiUxwjGsL9eHgvbgtZQr50d1DstEmfwQGgb1Omh0bM27yTnzb1eD/d/rp5uXMNQTe7vE4e+FUyjATch10WUFo2H5kiy+hV3wCc9hjHqAqjLmZEe4nPLG7nn+WQELeo1sYTUt6Jl3mTEC3r639pUfBn2i4wRVDxjpavxIjZIG9h48hgI0F8UGljgZBd/fjLKI7pX1mpyHDXEvdL8wIrFf3fJTU5JQYQIbzSWd3zcjkXTvrchs4ol7jHJ/DrjDRUPKzTd9oc7QM2qZPp+oJaHz+XapjDpjbLWJYOdXUpXtkHR8i8iMytGnvYIAJjDDcueEEDwXIaGFkDzElEhiBJLU/VqKDZhsukNpTvY4wxTgTcM71hIBSUyp9GeUGj/E8Ww0EWwiWcdNvA2X6kgVvkBYHPcD2LbvsI2PnsCH3oR+wMXz+CDo+Tnq/+E2xbUm0cjRVOKu46ACFv18N+KFAY4TnrVxTp1631Td69iHmJHQwTYyfsGYPUSjx6wMqXEWveVWp65R3CwGlhGVBSHYXev/U+6Vn2Tqr6sxMxQvVPXI/mDPbI70wzt0dZQzuCpge505bw9FaVOroaFKk7kBr/iDKXR8kwN0lK/qosCRbWz5DwvC8GsFndaz1Z+eGfmhn0zn/1Bk2vFU3Wu07xjZh6i7go/vjS5ANY y8bMnUFc PUCVxtUdXGKhqPvxYBCedF4vpIduTcZMm7g8AtmJde4FGbKWOFRz0ET1llIZy+Tmsr9nRSCboO8SYBe2mtCu+8QpBEEYIny04/seogHpYFAI4ta4C8sSJrDfv8aggBzW36uzskExEoFAyW1JGjU3hms/Q4aInhJIgFN4BUUPIOb2XkZqjQ68cYBEXiw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: enc field is hidden behind BOOK3E_PAGESZ_XX macros, and when you look closer you realise that this field is nothing else than the value of shift minus ten. So remove enc field and calculate tsize from shift field. Also remove inc field which is unused. Signed-off-by: Christophe Leroy Reviewed-by: Oscar Salvador --- arch/powerpc/include/asm/nohash/mmu-e500.h | 3 --- arch/powerpc/mm/nohash/book3e_pgtable.c | 4 ++-- arch/powerpc/mm/nohash/tlb.c | 9 +-------- arch/powerpc/mm/nohash/tlb_64e.c | 2 +- 4 files changed, 4 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/include/asm/nohash/mmu-e500.h b/arch/powerpc/include/asm/nohash/mmu-e500.h index 7dc24b8632d7..b281d9eeaf1e 100644 --- a/arch/powerpc/include/asm/nohash/mmu-e500.h +++ b/arch/powerpc/include/asm/nohash/mmu-e500.h @@ -244,14 +244,11 @@ typedef struct { /* Page size definitions, common between 32 and 64-bit * * shift : is the "PAGE_SHIFT" value for that page size - * penc : is the pte encoding mask * */ struct mmu_psize_def { unsigned int shift; /* number of bits */ - unsigned int enc; /* PTE encoding */ - unsigned int ind; /* Corresponding indirect page size shift */ unsigned int flags; #define MMU_PAGE_SIZE_DIRECT 0x1 /* Supported as a direct size */ #define MMU_PAGE_SIZE_INDIRECT 0x2 /* Supported as an indirect size */ diff --git a/arch/powerpc/mm/nohash/book3e_pgtable.c b/arch/powerpc/mm/nohash/book3e_pgtable.c index 1c5e4ecbebeb..ad2a7c26f2a0 100644 --- a/arch/powerpc/mm/nohash/book3e_pgtable.c +++ b/arch/powerpc/mm/nohash/book3e_pgtable.c @@ -29,10 +29,10 @@ int __meminit vmemmap_create_mapping(unsigned long start, _PAGE_KERNEL_RW; /* PTEs only contain page size encodings up to 32M */ - BUG_ON(mmu_psize_defs[mmu_vmemmap_psize].enc > 0xf); + BUG_ON(mmu_psize_defs[mmu_vmemmap_psize].shift - 10 > 0xf); /* Encode the size in the PTE */ - flags |= mmu_psize_defs[mmu_vmemmap_psize].enc << 8; + flags |= (mmu_psize_defs[mmu_vmemmap_psize].shift - 10) << 8; /* For each PTE for that area, map things. Note that we don't * increment phys because all PTEs are of the large size and diff --git a/arch/powerpc/mm/nohash/tlb.c b/arch/powerpc/mm/nohash/tlb.c index f57dc721d063..b653a7be4cb1 100644 --- a/arch/powerpc/mm/nohash/tlb.c +++ b/arch/powerpc/mm/nohash/tlb.c @@ -53,37 +53,30 @@ struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { [MMU_PAGE_4K] = { .shift = 12, - .enc = BOOK3E_PAGESZ_4K, }, [MMU_PAGE_2M] = { .shift = 21, - .enc = BOOK3E_PAGESZ_2M, }, [MMU_PAGE_4M] = { .shift = 22, - .enc = BOOK3E_PAGESZ_4M, }, [MMU_PAGE_16M] = { .shift = 24, - .enc = BOOK3E_PAGESZ_16M, }, [MMU_PAGE_64M] = { .shift = 26, - .enc = BOOK3E_PAGESZ_64M, }, [MMU_PAGE_256M] = { .shift = 28, - .enc = BOOK3E_PAGESZ_256M, }, [MMU_PAGE_1G] = { .shift = 30, - .enc = BOOK3E_PAGESZ_1GB, }, }; static inline int mmu_get_tsize(int psize) { - return mmu_psize_defs[psize].enc; + return mmu_psize_defs[psize].shift - 10; } #else static inline int mmu_get_tsize(int psize) diff --git a/arch/powerpc/mm/nohash/tlb_64e.c b/arch/powerpc/mm/nohash/tlb_64e.c index 053128a5636c..7988238496d7 100644 --- a/arch/powerpc/mm/nohash/tlb_64e.c +++ b/arch/powerpc/mm/nohash/tlb_64e.c @@ -53,7 +53,7 @@ int extlb_level_exc; */ void tlb_flush_pgtable(struct mmu_gather *tlb, unsigned long address) { - int tsize = mmu_psize_defs[mmu_pte_psize].enc; + int tsize = mmu_psize_defs[mmu_pte_psize].shift - 10; if (book3e_htw_mode != PPC_HTW_NONE) { unsigned long start = address & PMD_MASK; From patchwork Tue Jul 2 13:51:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23156C3064D for ; Tue, 2 Jul 2024 13:52:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A49E56B00D0; Tue, 2 Jul 2024 09:52:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F9786B00D1; Tue, 2 Jul 2024 09:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8735F6B00D2; Tue, 2 Jul 2024 09:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5D7AC6B00D0 for ; Tue, 2 Jul 2024 09:52:40 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1AAC91A01DA for ; Tue, 2 Jul 2024 13:52:40 +0000 (UTC) X-FDA: 82294952880.09.8624A1E Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf01.hostedemail.com (Postfix) with ESMTP id DD83940016 for ; Tue, 2 Jul 2024 13:52:37 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RTSOW/EbnmE8HR3k1nhtRsaSJvc+qkVW57PW03rpQg8=; b=UNXMR7zcgMP9T18dEDUdWgdfrShYVoJOAoQXEzwYfWsFwSl8f/TskAnZvlCNfj6iFyBEVK QVfricb5xb5TwYbk6UpyKK91e39TPrCRlvCjiGy1dEgE8Rxzq/yaqDt/mKEqKUemT+hUp1 zqqxA1XkL4AGVVZP6d3Pd+T+taOsGU4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928327; a=rsa-sha256; cv=none; b=knqBvkFi+8ayo19Ac5cgMByXdf1GlUg0syWshRm9imVw+/8ss8yr4FWpnnlhUFP6wOe+33 mveyK7huT+SjWw/t4H85+RM2XPf+S61eDQJOOszMooe5UgakIriDz3W5vdr++hs5ic58qc rd1TMFI5ZKDdECdnfhJG0KYps6Pl1rY= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48F0cdQz9v4N; Tue, 2 Jul 2024 15:51:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZTLqpHpbv2Is; Tue, 2 Jul 2024 15:51:41 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4855vHtz9v6q; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id C667F8B764; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id lj50rkcB16g6; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 52B598B774; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 16/23] powerpc/e500: Switch to 64 bits PGD on 85xx (32 bits) Date: Tue, 2 Jul 2024 15:51:28 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=2287; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=s4/AO36tt/CSDTJwqMkp0Ne29mDvmColJNi/FwYddUk=; b=+NpDIeX7nxezV8uqAJ+f57QTY8UArzmZcZxJfj50jBMGOKDntXyi6N/EbhjHi0DWOtnmkfflf MC8J1A9UT7YBjgzvu0eLLJeFz4PIekERDRRdcjgnRQgwYn84jNHhjaP X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DD83940016 X-Stat-Signature: pkoqcyk51ashth5n8on96ydcpw5e5cb5 X-Rspam-User: X-HE-Tag: 1719928357-222246 X-HE-Meta: U2FsdGVkX193ZI/11txbQxcMUYt4ncasmMYMJZNE7AbyGyc2Vv5OlN6IidYaRkUv7P3AQc9Y6tZM1EudvHs0KfwPW53I53pKXxgBeDycxjnTKKAVvL+wQa4VNFtgqAey6KpWN1BeEwWoe2Yg+kh7Jkwuo+v41W8PSCoJFGEWiJlGsSg7msCTNKjTCByDqUakGeDtggdkUQk9MeG/oxMG65r1yMjJspkzABuo8uKYnMkedVH8muHyaG5gsZjKl1LHfAq9617BXwjoTNqvLK4d4kqyywRVbtJUt7fQoAlfPKD6Uyp3jNmDNBv7LAQzMdXwyVzLRaTmDxgYIypnV4r4jTJrRxxscWfpvvxJwaU08JrzQqaOdZsBsoStBKx4kykB7wsJFmYjIUwg1+d18UG2bNF1doT9HDg2WqEMEVYyyyRyAIxQ8QYp/IxjYqMdNIwak+dwI/saQGJrFkGPH/RkiaX/PjmZ0TUu+n3wSYN6jX8Pcvk9Xh+oAvhRhbx78FLGlVqtRV9iALDQ84/q0pMh8NUSCU/7tiIr7bA+yp0FsYrd/9o8s8lgJ1xvbzmidrsCqaHyvW5FGaCS53Q4TJ9ofn5Oue+xqFStcJQm+MuLac4Fh858i1o0Txwl8gUBtehvh4fD4gWSDKTqh05NsfHd9J/1nF58AgS49ul7FaOOwjBLYqbGLHLhL8tkwgn+8TEQBw8nWo1mW4YP/kRpx42ZiM4Ec3hRDZ9OnivYRRRtvLA/Oaij4UUh4LcHuuYP9GJ1ju+HhhfhmhtUMfHz/7hHL3hGkUxeov8kI5Vf1Tneyjg4SJojFaFHlUK8g8NDbjeOwE4uvdv0K/yMLQ5+lEtsuzY7Z3Xn34iOt0A8g+Q3htwtBysgPsPxDcUGJc5tHReqtsrG1Flb1L1RYLvIURcuQm8NXmL2IMZ4R1S+XEtDPftX2D/3o5sWjsPwqiTtC5yA8VlT9e4oMiVUclK5cOX +7u4CwJL oHvoQ0Khz2YBowTHdTcOA7S83n1KPqvnYEHQYjsS/d9HjpD94Ykz7zAK1w9heTrNyoihPOxwd941ISWL9u/Q9kAP+PXP9WJOT5LKWHnLK8WNor29cCIYeq5cPok67Rm8fmObu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: At the time being when CONFIG_PTE_64BIT is selected, PTE entries are 64 bits but PGD entries are still 32 bits. In order to allow leaf PMD entries, switch the PGD to 64 bits entries. Signed-off-by: Christophe Leroy --- arch/powerpc/include/asm/pgtable-types.h | 4 ++++ arch/powerpc/kernel/head_85xx.S | 10 ++++++---- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/include/asm/pgtable-types.h b/arch/powerpc/include/asm/pgtable-types.h index 082c85cc09b1..db965d98e0ae 100644 --- a/arch/powerpc/include/asm/pgtable-types.h +++ b/arch/powerpc/include/asm/pgtable-types.h @@ -49,7 +49,11 @@ static inline unsigned long pud_val(pud_t x) #endif /* CONFIG_PPC64 */ /* PGD level */ +#if defined(CONFIG_PPC_E500) && defined(CONFIG_PTE_64BIT) +typedef struct { unsigned long long pgd; } pgd_t; +#else typedef struct { unsigned long pgd; } pgd_t; +#endif #define __pgd(x) ((pgd_t) { (x) }) static inline unsigned long pgd_val(pgd_t x) { diff --git a/arch/powerpc/kernel/head_85xx.S b/arch/powerpc/kernel/head_85xx.S index 39724ff5ae1f..a305244afc9f 100644 --- a/arch/powerpc/kernel/head_85xx.S +++ b/arch/powerpc/kernel/head_85xx.S @@ -307,8 +307,9 @@ set_ivor: #ifdef CONFIG_PTE_64BIT #ifdef CONFIG_HUGETLB_PAGE #define FIND_PTE \ - rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ - lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ + rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + add r12, r11, r12; \ + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ blt 1000f; /* Normal non-huge page */ \ beq 2f; /* Bail if no table */ \ @@ -321,8 +322,9 @@ set_ivor: 1001: lwz r11, 4(r12); /* Get pte entry */ #else #define FIND_PTE \ - rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ - lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ + rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + add r12, r11, r12; \ + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ beq 2f; /* Bail if no table */ \ rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ From patchwork Tue Jul 2 13:51:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ED4CC3064D for ; Tue, 2 Jul 2024 13:52:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95D8C6B00D3; Tue, 2 Jul 2024 09:52:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 90D976B00D4; Tue, 2 Jul 2024 09:52:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7ADC76B00D5; Tue, 2 Jul 2024 09:52:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5C2896B00D3 for ; Tue, 2 Jul 2024 09:52:44 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 15C2CC01E3 for ; Tue, 2 Jul 2024 13:52:44 +0000 (UTC) X-FDA: 82294953048.13.C5AF64A Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf14.hostedemail.com (Postfix) with ESMTP id DB837100025 for ; Tue, 2 Jul 2024 13:52:41 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf14.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928343; a=rsa-sha256; cv=none; b=79jMFczAf5p6I2YTFLiIi8TyMDHOK86Fok0hw2+2l8x51ABQlRYazQuTYofCbQp7t2i4pw d3XoNXcDSsjGYdeFkUG9M7Ki3L54bfRmfWPQNFpSZrjuPd8LKJViKx5FBdba72Ds14BUud guS5kukMn3P8/zN8I3clrinEUSZBSXU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf14.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7h45N1Bvr9rYegB3gn0n2HPHUmwTraaZKJByBJPFID4=; b=5ZItrMyVeMErHrGjo2oQowiytgepwaHo+xuxYYRVXEhYcHqBiW0ovPktj589+/cq0uSsBv SoCjSNY/o+hNn/3kCa3cJE4IP2pQ7f5jRhHjfCcSRmxPPItxkTUrLkuR8A98ZhFUEhdGWX UBgRmwJ/Tgot700Cw1+iRVRJxk4NZ1g= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48K12dVz9v2F; Tue, 2 Jul 2024 15:51:45 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CxQ474nA3jKF; Tue, 2 Jul 2024 15:51:45 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4862d6Bz9v7L; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 55D388B774; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id A5pzDnRs-Jou; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id C91538B775; Tue, 2 Jul 2024 15:51:33 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 17/23] powerpc/e500: Encode hugepage size in PTE bits Date: Tue, 2 Jul 2024 15:51:29 +0200 Message-ID: <6f7ce82fa8c381d55f65342d77060fc55802e612.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=4066; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=ynQfm9JBiFGAxo+DROotbCi4y0+yhN5d6b0AA4yoL0Q=; b=QlbwuOh8CkIKRxk1Au5CyczWq573GEKpqCwwgW7AMQgt2eEOL8L6hmPsA5PNJoTqDOTWjwumi 7MRu1yorX68DIIBViNV11hHb6SZmkTrroIUiiqrvbJl9pPA9MJhHdpu X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DB837100025 X-Stat-Signature: 4wy3q38opp648xs7r4p6is4mp4ubs6yw X-Rspam-User: X-HE-Tag: 1719928361-338811 X-HE-Meta: U2FsdGVkX1/bAXUSnd3WZB8XkRkNUq4UzWxvW6M6N/brHkojaRq1pzSRyisrqLI+nmXPddj1vOAPYUw2FQqEUPQOi8yqGKSNaSS72mseDERvqLnPLOkgL0gXSzix83pYu0gfreF27RxOEg1QzgJDzA5lJeY9J0DxgbsYEyRFyZ13BdmOgjihraO036Yop+D33NujePNwEIkA+GFJA9zOABD20uypcyCoVk7uE0APaDkx71Eus18vEf5TkXFopr4NWgASZ52CfSdMlxv5Qa9Jbvo6ccMRTBSz0P5Is6a3L8zpOAL+iEHCkVmMVUUR7CsYXrlx5QW3oyp90Mu4Ju/rRsBTW3XwVpS5KzsgrEtlUu3EbuzY1LYiaes0V7AR9YRffxVpyAtNWmVHq4kC79g/HO1kml/irX7m71C+7AaUHh8lKdDlsoWn0mhW4BDqtbFldRAMx9Q15OOxUBhXItE08+D33l/nAcbvO2rC0JlbVQXxF147IMjh0fknsy4XlArszt9PKWu2rfb79PLkU/tTMAqeHSY9ifCqqZdp+7KazzDzn2hUsanVy4PdeED2TWi6wyH7gDS/2PTSeaCvZ83RvImXcEEAUoG53etp0mpYWEVZkAowhQdTA/0lGAwiDErmYlYdGg2dRJQyYpB1rhQQ11oD5udn0DCjStLU5RtODp+HItBmP67OzcnIUaB4MCP5vKlUTlrJdOUY01tqYQ3qabMbcUJgkreE/4fo/KpLXQsi1d02+Yp6unZDwjNR1Gg8D4ro52J1w97oFyY55hBg097/qntD7gy/d6XL3SSSi8J0mIrjElLIR1FKEGBdtGIax8t1LFflFQZn660zyMOD69O/dwIri8H6PkmVogi30sG6Mn1DJYWL4C+h06vVPTX+TGZJG2mBYUCHPA9zzkwa7yDdcyZ+s8U/GB4cs1/OWR3QWHGI2tHxKQp3QvL909u0fiudfkdAPqOr1mXyR1y lMFkwb+d PiYFCfP02Pka5d22P6jCmAiFklUMGUKCysYeecPujKl47nWZByFe8O3OptLbMAI9vStz51oh4e9U65NrvcZndaj3m3xTzLSjmPfZgP4vMJ/MZfh2P/p0yqLaolXw9cGsX3bkv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use PTE page size bits to encode hugepage size with the following format corresponding to the values expected in bits 52-55 in MAS1 register. Those bits are called TSIZE: 0001 4 Kbyte 0010 16 Kbyte 0011 64 Kbyte 0100 256 Kbyte 0101 1 Mbyte 0110 4 Mbyte 0111 16 Mbyte 1000 64 Mbyte 1001 256 Mbyte 1010 1 Gbyte 1011 4 Gbyte 1100 16 Gbyte 1101 64 Gbyte 1110 256 Gbyte 1111 1 Tbyte It corresponds to shift value minus 10 with lowest bit removed. It is not the value expected in the PTE in that field, but only e6500 performs HW based TLB loading and the e6500 reference manual explicitely says that this field is ignored. Also add pte_huge_size() which will be used later. Signed-off-by: Christophe Leroy --- v5: Use PAGE SIZE field instead of U0-U3 because on some HW U2-U3 are used for something else. --- .../powerpc/include/asm/nohash/hugetlb-e500.h | 9 ++++++ arch/powerpc/include/asm/nohash/pte-e500.h | 28 +++++++++---------- 2 files changed, 22 insertions(+), 15 deletions(-) diff --git a/arch/powerpc/include/asm/nohash/hugetlb-e500.h b/arch/powerpc/include/asm/nohash/hugetlb-e500.h index 8f04ad20e040..c6a0938e86fd 100644 --- a/arch/powerpc/include/asm/nohash/hugetlb-e500.h +++ b/arch/powerpc/include/asm/nohash/hugetlb-e500.h @@ -42,4 +42,13 @@ static inline int check_and_get_huge_psize(int shift) return shift_to_mmu_psize(shift); } +static inline pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) +{ + unsigned int tsize = shift - _PAGE_PSIZE_SHIFT_OFFSET; + pte_basic_t val = (tsize << _PAGE_PSIZE_SHIFT) & _PAGE_PSIZE_MSK; + + return __pte((pte_val(entry) & ~(pte_basic_t)_PAGE_PSIZE_MSK) | val); +} +#define arch_make_huge_pte arch_make_huge_pte + #endif /* _ASM_POWERPC_NOHASH_HUGETLB_E500_H */ diff --git a/arch/powerpc/include/asm/nohash/pte-e500.h b/arch/powerpc/include/asm/nohash/pte-e500.h index 975facc7e38e..6dac1c0a6937 100644 --- a/arch/powerpc/include/asm/nohash/pte-e500.h +++ b/arch/powerpc/include/asm/nohash/pte-e500.h @@ -19,20 +19,7 @@ #define _PAGE_BAP_SX 0x000040 #define _PAGE_BAP_UX 0x000080 #define _PAGE_PSIZE_MSK 0x000f00 -#define _PAGE_PSIZE_4K 0x000200 -#define _PAGE_PSIZE_8K 0x000300 -#define _PAGE_PSIZE_16K 0x000400 -#define _PAGE_PSIZE_32K 0x000500 -#define _PAGE_PSIZE_64K 0x000600 -#define _PAGE_PSIZE_128K 0x000700 -#define _PAGE_PSIZE_256K 0x000800 -#define _PAGE_PSIZE_512K 0x000900 -#define _PAGE_PSIZE_1M 0x000a00 -#define _PAGE_PSIZE_2M 0x000b00 -#define _PAGE_PSIZE_4M 0x000c00 -#define _PAGE_PSIZE_8M 0x000d00 -#define _PAGE_PSIZE_16M 0x000e00 -#define _PAGE_PSIZE_32M 0x000f00 +#define _PAGE_TSIZE_4K 0x000100 #define _PAGE_DIRTY 0x001000 /* C: page changed */ #define _PAGE_SW0 0x002000 #define _PAGE_U3 0x004000 @@ -46,6 +33,9 @@ #define _PAGE_NO_CACHE 0x400000 /* I: cache inhibit */ #define _PAGE_WRITETHRU 0x800000 /* W: cache write-through */ +#define _PAGE_PSIZE_SHIFT 7 +#define _PAGE_PSIZE_SHIFT_OFFSET 10 + /* "Higher level" linux bit combinations */ #define _PAGE_EXEC (_PAGE_BAP_SX | _PAGE_BAP_UX) /* .. and was cache cleaned */ #define _PAGE_READ (_PAGE_BAP_SR | _PAGE_BAP_UR) /* User read permission */ @@ -87,7 +77,7 @@ * pages. We always set _PAGE_COHERENT when SMP is enabled or * the processor might need it for DMA coherency. */ -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE_4K) +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_TSIZE_4K) #if defined(CONFIG_SMP) #define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT) #else @@ -103,6 +93,14 @@ static inline pte_t pte_mkexec(pte_t pte) } #define pte_mkexec pte_mkexec +static inline unsigned long pte_huge_size(pte_t pte) +{ + pte_basic_t val = pte_val(pte); + + return 1UL << (((val & _PAGE_PSIZE_MSK) >> _PAGE_PSIZE_SHIFT) + _PAGE_PSIZE_SHIFT_OFFSET); +} +#define pte_huge_size pte_huge_size + #endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */ From patchwork Tue Jul 2 13:51:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72CD0C3065C for ; Tue, 2 Jul 2024 13:52:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0840A6B00D6; Tue, 2 Jul 2024 09:52:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0354B6B00D7; Tue, 2 Jul 2024 09:52:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E17136B00D8; Tue, 2 Jul 2024 09:52:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BF2516B00D6 for ; Tue, 2 Jul 2024 09:52:48 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 76E4B1201E6 for ; Tue, 2 Jul 2024 13:52:48 +0000 (UTC) X-FDA: 82294953216.29.D348850 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf14.hostedemail.com (Postfix) with ESMTP id 10CC0100030 for ; Tue, 2 Jul 2024 13:52:45 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IeJ2s5BHwiaIokCY9RGe1Kq8pNvNZAfr2J7palaWQuc=; b=1vcORaYJR+vPa6h7ei7lQhC4LeoThemE0hFi0sCx6EfYVXwj5U1qJStyALqeZOJONnP1Bm vRA90WgYDx7SuZEqfHp8kGtKCfQCK0w1N65xJH4QPYN3aOd6E5BawOlG66g3SwXhjURKOW pApH+kpqv7nFic4hct4aLMO2s3niUSk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928354; a=rsa-sha256; cv=none; b=c08nzUQ4QY6cBcVwVD7mtrmb+pxdjPLleUmjakl8M8y4EA8uXq80cWoKdhf4YbeOHWXMfr eq0SRtOH+wVhYhXTnJvFGMxreIu/NB8CwC8RMZTZV6zINh2tCEqEgCyuIUNYbWBrZ5twbo aGbtbYF65vNfJYLXrFbgEPDKaV+xKxg= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48L4787z9v3s; Tue, 2 Jul 2024 15:51:46 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tD71Vn51syID; Tue, 2 Jul 2024 15:51:46 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4865kqdz9v7N; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id C0D818B774; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id pWxGaw-G7BNY; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 4D51F8B764; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 18/23] powerpc/e500: Don't pre-check write access on data TLB error Date: Tue, 2 Jul 2024 15:51:30 +0200 Message-ID: <8525518e1657d6032b7e980c1888102828d66950.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=1564; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=lXkJFaNOZfYs39NvzB2t7aMNQd5E4EiqpUUgx2rbgYk=; b=UfaTLsrzSij3pYXiv+ejHeSoRAerQa7OpK+kFzmvGET2VlKw+SFhPJdwxpiVLp8BycMHe75De Wk++EkSYFizC/zAfHU9AfP7o7tpmfuohonnxIu+luqjMTdYqXgBexcN X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 10CC0100030 X-Stat-Signature: h38ac8d5ej5tqh5nzki1tfczssnqq5ua X-Rspam-User: X-HE-Tag: 1719928365-741788 X-HE-Meta: U2FsdGVkX19xZmsC1HK2EAIg7+4TlmrOZ3w6CYZSZ/1SflPmyASaqitCxShMN8fy60GX2bGGYGVuzcrVlzXe4AB3isJ21bOtkC2SI3gETpE4q62QzUhx2tw4Px7LuIKUvdFFQdwI9hiIyQzjd1pWG9t+UF8APlEbmJgnFN2L5Sk2M55ckw7IptWoaorSKSIO5qj6o1GIjYEYhksNuyc7xPTFRzMTaI3eQkxtz6lGYghW8nHWj3OiyvvkOiIxNQGIhBhFW60idzcXyGaeGAmO/XJDBtBXfeTDA6yPB2lg55ooCpzTGLJQPeJh+BqDdxxqXKmseOmeCVz70c+tgmd4vLziL1NH+kJZ5UMdrdKHzr0XueKByoGgziyJs1p95g8hSD/y1U/JoKFW+4pf0PohhbiP6PRbcqu43MAJp87JBxmGrXtEk7oekhFQ9Q5fc8Q1xVMmoQFToTwZT3j3INIW77b4TE7GrMzgqOm4rAMqxlPmQP3eC/xGY74h9t+tedU7/6uNhMNuYtLLts7n0HJIAooAx0+jNriDi4S3Xty4QAnDd4iQVaO+CvEt39C3tLFnrKJ3vqNXHGgOiF2VKeMK2Ah0ubeQu4gQ0XOHpx5TXwzYZdNMN70mak9fvvcFgVznJsRX0Y+rP2GbPNO74BOQF2VxtsrhExt1cVeUs6qoHapAtyJ9ihQMspHuCo0OD70eL2eMkFp0snQgSN1kzRMQ+qFjtPaLCXDwjG4+DtNiP4btBeT8Zum2geHvFEE50UhtniHDPL/OQdDzys8Y/GP7kDwcsQDPhsRsV8T4t4doB59gEv1AepFCNidM/g7uyUNpeSRCptxTOl2ewp+HFQEzQgIKn1izUSwAUYIuVQxcgLF8NjZK5jOZ5KMbU5CngWD8iTTDDjUA9lE0JYXECMveaN0hCAZKCr7RAy4FpDvY/0NsFhoKJbjP1P0IZSCYueOKn8w+WZqhMIKoXjO004w Q6eQPa6a 6CJfOLl9QQvWVkmzk1Gmzj+J9FHsVKKFS6aCoOY/NMgYwrRXS4n8nl/kHEwpup7bbIkcMdGUYM3v/dxv3deZ2Km1j6Bp39mwXJmYcrD9Z/uiUIBZvsuyspmSx2eynEs+x7AKC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Don't pre-check write access on read-only pages on data TLB error. Load the TLB anyway and take a DSI exception when it happens. This avoids reading SPRN_ESR at every data TLB error exception. Signed-off-by: Christophe Leroy --- v5: New --- arch/powerpc/kernel/head_85xx.S | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/arch/powerpc/kernel/head_85xx.S b/arch/powerpc/kernel/head_85xx.S index a305244afc9f..282e49c51deb 100644 --- a/arch/powerpc/kernel/head_85xx.S +++ b/arch/powerpc/kernel/head_85xx.S @@ -472,27 +472,12 @@ END_BTB_FLUSH_SECTION #endif 4: - /* Mask of required permission bits. Note that while we - * do copy ESR:ST to _PAGE_WRITE position as trying to write - * to an RO page is pretty common, we don't do it with - * _PAGE_DIRTY. We could do it, but it's a fairly rare - * event so I'd rather take the overhead when it happens - * rather than adding an instruction here. We should measure - * whether the whole thing is worth it in the first place - * as we could avoid loading SPRN_ESR completely in the first - * place... - * - * TODO: Is it worth doing that mfspr & rlwimi in the first - * place or can we save a couple of instructions here ? - */ - mfspr r12,SPRN_ESR #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT|_PAGE_BAP_SR oris r13,r13,_PAGE_ACCESSED@h #else li r13,_PAGE_PRESENT|_PAGE_READ|_PAGE_ACCESSED #endif - rlwimi r13,r12,11,29,29 FIND_PTE andc. r13,r13,r11 /* Check permission */ From patchwork Tue Jul 2 13:51:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 812D2C30658 for ; Tue, 2 Jul 2024 13:53:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16E696B00D9; Tue, 2 Jul 2024 09:53:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 11EDC6B00DA; Tue, 2 Jul 2024 09:53:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F01A56B00DB; Tue, 2 Jul 2024 09:53:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D12A46B00D9 for ; Tue, 2 Jul 2024 09:53:03 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 96BFD14014B for ; Tue, 2 Jul 2024 13:53:03 +0000 (UTC) X-FDA: 82294953846.15.54ED566 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf07.hostedemail.com (Postfix) with ESMTP id 6209840007 for ; Tue, 2 Jul 2024 13:53:00 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf07.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928359; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5YhBJVVFRl3EoQKKpKhbq1vJKrkfH8JAW5ygc4FL7lU=; b=Q/Xgy+CDiaDc6PFBanbBwKum/m72vivyLm5VePleQDfZogzxP+kBpBkCbkByBYlTPEHac7 umEAmwQI9QotqKqhJ7HnkvqkeHNfY7mKL1IneZhXpv6Ya702pIYB5bLWYzbDQnbS4J7yqT Vye15xom8ssJC4we5an6XdjDjgn+qUM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928359; a=rsa-sha256; cv=none; b=ha2xFvu94RoMITi6eeZdNT99MAR6ju9VHcrmBD80+B/ejzXbWY+j0YptdkzjOA14EQ7R0x 0Yt8ZrlnV8mwZnWHm1EichdhVVauYqUtOJJW4BNTWp5Hf7Fsng0WVbzgF6+ZVJcU+zH0Pb 92SVXJgaHRfujXeADE+Mjt9vDyCYXZc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf07.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48M2G8bz9v6q; Tue, 2 Jul 2024 15:51:47 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F1PYD05orrHY; Tue, 2 Jul 2024 15:51:47 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4872gVjz9v5M; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 5733E8B774; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id LSTeqtZMNkMb; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id C12BE8B775; Tue, 2 Jul 2024 15:51:34 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 19/23] powerpc/e500: Free r10 for FIND_PTE Date: Tue, 2 Jul 2024 15:51:31 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=4544; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=CGeNYqnnvnjPnGCbM1yMAoxPPNyoWmWcvtJY5TjAVy0=; b=M1QWIpnPCtnELqMpfZZElEuTb8maaGQCCyTc3Ar8cokovO/DoM4+jhTmObMqHUeG6WC1Y96nA N3ZYvNGaCG7BCNl4TfIiCDNRKy//BQir7dqlmDq1vhcHqqDVpGjudhr X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Queue-Id: 6209840007 X-Stat-Signature: zijp4ywcbdqtwp5s3mpcwirqh6ak4mqk X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1719928380-268502 X-HE-Meta: U2FsdGVkX1/CWM4867Ct5tCFxDx4KqiWminBRLuGnfpv1UfLOOUfsTOX4dvaYk/PuEyOduem5Cny3w8Vf0FTClTkKXMCDssLt3Qau81T2z1mPZly2b2qHPQzC1idzBHVqNnH6loBXustfzMGu5nqDTYCz249vOV6EJ1pSHylNXJqy8eBKJQvl1IpwIEOnVmCGrv6dPyh5jlBh2ywvW9KADWeRdkS+g/Amhz4lv+qklNtJWLLz0npVwVyUc2pDFvTQGYzuR/3QlMvh4dM0wb9YMd1k6wQoGMXxartI7ND2xi3e7cn29OEp6OfwjaJeVVFHzXSaLxreV5e0cfsXjT2BYQbBdRsF7aZmV1aHZvI/+vwy98+jKy6thPHGKSjEVVWsj679GznAvOrWLcSAWOz8fQviTK9yqJ5smnLh9qUkqo6Wbgo+nGroEsAvDcgi8Xsv6dmzGAwlhk+ftV9TzXB9evzntHKI//Qxxbl9YaD0VpT+iIV2SwFFamU8i5hQRW3T8a8NN4gslvz8TtNe3flYQQicPb2It7jWuYjTyuCQwi6KKJrrN0yzifDv5rGvUkayv3JoAGFfATIYbRO4yqPmsiiYVeUvsRhN7h5K4FW6pdcUt9GB64zcrM8XmXbBOz9TzfiW88Wgpr07Jz5qcPbjGZEpqdS+TivypStxmySPxjMtHcGqhSDLAKUGMRMveRWPEPZtee+Oi1oKiMGYWMNcRgh5Tv4DF7TTJ+++GANVqNofikZcC+HrB0InqkIArhy4u6fagqXxwrxw7x60sKbnoSHHUXA3xF33zvlrZO+DetdJzWcWHfZaaW5Xt7t99G8RU+j4RYUo5qfWKYEURMJTDg+wdNuOdGWYp/lQE1++deajyq9341mRNOz1gwjdeCNrPcJmtquj9katfVLP21S/xNAjPaPGq6BhvIZc8Z6tMKpQgy4/Tdj0yCRnDWRX9b++TrWPRlrwmXNCeHBGJo HRW3N7+D CdT/TOjrcBD/v08YUi6MNLbeFnmzQornSNucbRY+vAz+nsM2AZkeOHF4F6gB5YeZOavdBf1RgOeC//NOkvbkUtkfZlSzYrBHVR8ZeGnlu+6kh3KREcoa84MEIhS/g3u0hUeWgM63FkwWQI5Xq4Xw7nifmFhWvLZGSxra1n0i1Tbi+Zcl2MLkcH1dtpNqS2aD8PQlUoEjbNPeVnhY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move r13 load after the call to FIND_PTE, and use r13 instead of r10 for storing fault address. This will allow using r10 freely in FIND_PTE in following patch to handle hugepage size. Signed-off-by: Christophe Leroy --- v5: New --- arch/powerpc/kernel/head_85xx.S | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/kernel/head_85xx.S b/arch/powerpc/kernel/head_85xx.S index 282e49c51deb..226f88e77d6d 100644 --- a/arch/powerpc/kernel/head_85xx.S +++ b/arch/powerpc/kernel/head_85xx.S @@ -294,9 +294,10 @@ set_ivor: /* Macros to hide the PTE size differences * * FIND_PTE -- walks the page tables given EA & pgdir pointer - * r10 -- EA of fault + * r10 -- free * r11 -- PGDIR pointer * r12 -- free + * r13 -- EA of fault * label 2: is the bailout case * * if we find the pte (fall through): @@ -307,7 +308,7 @@ set_ivor: #ifdef CONFIG_PTE_64BIT #ifdef CONFIG_HUGETLB_PAGE #define FIND_PTE \ - rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ add r12, r11, r12; \ lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ @@ -317,26 +318,26 @@ set_ivor: andi. r10, r11, HUGEPD_SHIFT_MASK@l; /* extract size field */ \ xor r12, r10, r11; /* drop size bits from pointer */ \ b 1001f; \ -1000: rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ +1000: rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ li r10, 0; /* clear r10 */ \ 1001: lwz r11, 4(r12); /* Get pte entry */ #else #define FIND_PTE \ - rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ add r12, r11, r12; \ lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ beq 2f; /* Bail if no table */ \ - rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ + rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ lwz r11, 4(r12); /* Get pte entry */ #endif /* HUGEPAGE */ #else /* !PTE_64BIT */ #define FIND_PTE \ - rlwimi r11, r10, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ + rlwimi r11, r13, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ lwz r11, 0(r11); /* Get L1 entry */ \ rlwinm. r12, r11, 0, 0, 19; /* Extract L2 (pte) base address */ \ beq 2f; /* Bail if no table */ \ - rlwimi r12, r10, 22, 20, 29; /* Compute PTE address */ \ + rlwimi r12, r13, 22, 20, 29; /* Compute PTE address */ \ lwz r11, 0(r12); /* Get Linux PTE */ #endif @@ -443,13 +444,13 @@ START_BTB_FLUSH_SECTION BTB_FLUSH(r10) 1: END_BTB_FLUSH_SECTION - mfspr r10, SPRN_DEAR /* Get faulting address */ + mfspr r13, SPRN_DEAR /* Get faulting address */ /* If we are faulting a kernel address, we have to use the * kernel page tables. */ lis r11, PAGE_OFFSET@h - cmplw 5, r10, r11 + cmplw 5, r13, r11 blt 5, 3f lis r11, swapper_pg_dir@h ori r11, r11, swapper_pg_dir@l @@ -472,14 +473,14 @@ END_BTB_FLUSH_SECTION #endif 4: + FIND_PTE + #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT|_PAGE_BAP_SR oris r13,r13,_PAGE_ACCESSED@h #else li r13,_PAGE_PRESENT|_PAGE_READ|_PAGE_ACCESSED #endif - - FIND_PTE andc. r13,r13,r11 /* Check permission */ #ifdef CONFIG_PTE_64BIT @@ -536,13 +537,13 @@ START_BTB_FLUSH_SECTION 1: END_BTB_FLUSH_SECTION - mfspr r10, SPRN_SRR0 /* Get faulting address */ + mfspr r13, SPRN_SRR0 /* Get faulting address */ /* If we are faulting a kernel address, we have to use the * kernel page tables. */ lis r11, PAGE_OFFSET@h - cmplw 5, r10, r11 + cmplw 5, r13, r11 blt 5, 3f lis r11, swapper_pg_dir@h ori r11, r11, swapper_pg_dir@l @@ -551,6 +552,7 @@ END_BTB_FLUSH_SECTION rlwinm r12,r12,0,16,1 mtspr SPRN_MAS1,r12 + FIND_PTE /* Make up the required permissions for kernel code */ #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT | _PAGE_BAP_SX @@ -571,6 +573,7 @@ END_BTB_FLUSH_SECTION beq 2f /* KUAP fault */ #endif + FIND_PTE /* Make up the required permissions for user code */ #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT | _PAGE_BAP_UX @@ -580,7 +583,6 @@ END_BTB_FLUSH_SECTION #endif 4: - FIND_PTE andc. r13,r13,r11 /* Check permission */ #ifdef CONFIG_PTE_64BIT From patchwork Tue Jul 2 13:51:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3A30C3064D for ; Tue, 2 Jul 2024 13:53:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FF0A6B00AA; Tue, 2 Jul 2024 09:53:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AD526B00DD; Tue, 2 Jul 2024 09:53:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 627E76B00DE; Tue, 2 Jul 2024 09:53:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 423D26B00AA for ; Tue, 2 Jul 2024 09:53:18 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D7BA416019C for ; Tue, 2 Jul 2024 13:53:17 +0000 (UTC) X-FDA: 82294954434.09.AC92A4C Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf20.hostedemail.com (Postfix) with ESMTP id B75951C001D for ; Tue, 2 Jul 2024 13:53:15 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LoU8tTVE09OmHRCUtdoK62hh/xfi54W96qQ/5fNKkTA=; b=tzAJLAgONkyp0tzVugB/M8hflbuBvLEnYgKSpDNNSk7/bXH24l1k9pemup5ZY7Uelm9RnT n8wbNlE0fSQAkPFu/veptP1EftVMPfrQKx4oczIlon9/nyNiu5EzaAHghhmDxBVE3AzKlT k9gNb41yEElreQszIqhNoGknUiLC4ic= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928378; a=rsa-sha256; cv=none; b=b0PLqYSkeiotzPt/xxNQl3HtAG6SInrsgMr9+JXKVvCxLzx8HA2OJSyyuj9RXG2bNafjed iYzTAl8DLhNNQlBaeJWrwdbarmPAqnSck9HQ/CmWnO05lOHTtxML1V6giKBe4Q5LyGiyvT Uk7DqKszG0f/+xX5l+2ps8MHxpUP2L8= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48N0kWvz9v5M; Tue, 2 Jul 2024 15:51:48 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AqptdD5-fxit; Tue, 2 Jul 2024 15:51:48 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4876gQmz9v7b; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id E11F08B775; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id 8CnXM6-Udnbx; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 52DCF8B764; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 20/23] powerpc/e500: Use contiguous PMD instead of hugepd Date: Tue, 2 Jul 2024 15:51:32 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=13017; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=RCt/w1jfgjS75jeurc9KnG66BmGWNtIsWZG9ZvTwSvE=; b=Stp60MgxAZsdIe+81s34typ5shHU/SA+kLFD437+FePuPnRzV7k4NoN3mHut9D+MqWLqlkadp Z3g4+n+r6HQBXX/n81KXOR0eu5GYRJGN5SbDe4sbc6J/uHMranoIlXU X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Stat-Signature: cydspqr8r5kz84da6qi39xj1tc4y6hjd X-Rspam-User: X-Rspamd-Queue-Id: B75951C001D X-Rspamd-Server: rspam02 X-HE-Tag: 1719928395-740026 X-HE-Meta: U2FsdGVkX19rmPNMeYXTak4FdchFrN/tKQGBPIgRRPpkWdZKqTRednO8zHX1hbq+xJ4822WKc1xMwo+ZAu+hNXLNfk2WIQyeVdAk+//w69fmN6RA296AJP5gVsxRG7fC/X6WGql68+HWuuSuEMvBju1HjW4VvbVYyPolhK3eay/PrCBw0SaSwIAWRqz59A5eVEtCGRoOXTgDO65vWtvUH0R89AcPx5XkTh2bGgMSX6hMJVlBg4sa128IqMTERzp7NWEGlKokTvRCBT/Id7pO4W4kg9a81NwXZEoBw2lV+BpQHpkf/ONx8Z9dqhdTLDNtzUR2lBEtKIjgS82oXqi9M8aggvozKs1TcQNZbY1dYeuLoloLYW2z6sqayms25pk3mOoyIo2d5UGazzPVyr/aJskHZ587sXV2Yw84NAxoz1kVhfP3kuDM9vjeCuptSLg4BMueQu8nCNbCDhnb/bzYn+Dlk0qPa1gYyNhps59rGk+XAM2T3Pwrxh/DUxCUU2m0yIF6+s87cKHll0A9d2SVUna67aKZKciiWdWgeRdd1w7B8oEpfZg5IwgmmD+1hpx5RxnNsce+DCOHwapl3RPdR9GNV7qH+1vix1A/WiyGdr+RbVkuzQkq81qstNn2KQe4vsjCktuVnNH9/Vr966uQ5Kz1Wb6IsyoRDJ0rL3PtmXmioBGpLSXAnxfgyE4MMHzQjHX1NBchlmujfPB4PV1Y5HkQTOLsybbWhW38BqIF5DHKy5hXm7f8kyXKwyrz+TSgt+2Z2FrGqSpwfoMk/tGyIFylKcaE3N7aHqhJ67SWD4zMOX2dkoyzB2YkAZS82dGbjfP39H/orFvaKTczQi8Q+HkcrWhnpRFGbWris+FqyYWrtRT3iYuz6SChs++xjYdAymB4/77iwYdtoGlcY+ZE4lJbBRWKJU4la5acWtrFcXYrq5EZNFBr/h9/mv8o8OrGjn0iq8i3UqBBMjpsAk8 dgCd8q2s 6waI3RohuTW1kte1B8sr7Ff9SLbys+tCZvDg0Z/0xT/r1HxRYsZgRhx+DbZ95ntwGDaPQ4hffblb2cn0BK23Wjp6uJrTJuKYUq8ukbvf9AB1R6VO8glXdoF1OXcCKsqo2Bvwr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: e500 supports many page sizes among which the following size are implemented in the kernel at the time being: 4M, 16M, 64M, 256M, 1G. On e500, TLB miss for hugepages is exclusively handled by SW even on e6500 which has HW assistance for 4k pages, so there are no constraints like on the 8xx. On e500/32, all are at PGD/PMD level and can be handled as cont-PMD. On e500/64, smaller ones are on PMD while bigger ones are on PUD. Again, they can easily be handled as cont-PMD and cont-PUD instead of hugepd. On e500/32, use the pagesize bits in PTE to know if it is a PMD or a leaf entry. This works because the pagesize bits are in the last 12 bits and page tables are 4k aligned. On e500/64, use highest bit which is always 1 on PxD (Because PxD contains virtual address of a kernel memory) and always 0 on PTEs because not all bits of RPN are used/possible. Signed-off-by: Christophe Leroy --- v3: Add missing pmd_leaf_size() and pud_leaf_size() v4: Rebased of v6.10-rc1 : pmd_huge() and pud_huge() are gone v5: - Define pte_huge_size() to reduce redundant and handle the case of mpc85xx with 32 bits PTE. - Replace hard-coded page shift offset (10) with new macro _PAGE_PSIZE_SHIFT_OFFSET - Use TSIZE encoding in SIZE field in PTE - On 64 bits, use highest bit to know if it is a PUD/PMD or leaf entry. Avoids problem with page table alignment. --- .../powerpc/include/asm/nohash/hugetlb-e500.h | 32 ++------------ arch/powerpc/include/asm/nohash/pgalloc.h | 2 - arch/powerpc/include/asm/nohash/pgtable.h | 42 ++++++++++++++----- arch/powerpc/include/asm/nohash/pte-e500.h | 33 +++++++++++++++ arch/powerpc/include/asm/page.h | 15 +------ arch/powerpc/kernel/head_85xx.S | 21 ++++------ arch/powerpc/mm/hugetlbpage.c | 2 - arch/powerpc/mm/nohash/tlb_low_64e.S | 7 +--- arch/powerpc/mm/pgtable.c | 31 ++++++++++++++ arch/powerpc/platforms/Kconfig.cputype | 1 - 10 files changed, 107 insertions(+), 79 deletions(-) diff --git a/arch/powerpc/include/asm/nohash/hugetlb-e500.h b/arch/powerpc/include/asm/nohash/hugetlb-e500.h index c6a0938e86fd..812c71fc5eb1 100644 --- a/arch/powerpc/include/asm/nohash/hugetlb-e500.h +++ b/arch/powerpc/include/asm/nohash/hugetlb-e500.h @@ -2,38 +2,12 @@ #ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H #define _ASM_POWERPC_NOHASH_HUGETLB_E500_H -static inline pte_t *hugepd_page(hugepd_t hpd) -{ - if (WARN_ON(!hugepd_ok(hpd))) - return NULL; - - return (pte_t *)((hpd_val(hpd) & ~HUGEPD_SHIFT_MASK) | PD_HUGE); -} - -static inline unsigned int hugepd_shift(hugepd_t hpd) -{ - return hpd_val(hpd) & HUGEPD_SHIFT_MASK; -} - -static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, - unsigned int pdshift) -{ - /* - * On FSL BookE, we have multiple higher-level table entries that - * point to the same hugepte. Just use the first one since they're all - * identical. So for that case, idx=0. - */ - return hugepd_page(hpd); -} +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte, unsigned long sz); void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr); -static inline void hugepd_populate(hugepd_t *hpdp, pte_t *new, unsigned int pshift) -{ - /* We use the old format for PPC_E500 */ - *hpdp = __hugepd(((unsigned long)new & ~PD_HUGE) | pshift); -} - static inline int check_and_get_huge_psize(int shift) { if (shift & 1) /* Not a power of 4 */ diff --git a/arch/powerpc/include/asm/nohash/pgalloc.h b/arch/powerpc/include/asm/nohash/pgalloc.h index 4b62376318e1..d06efac6d7aa 100644 --- a/arch/powerpc/include/asm/nohash/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/pgalloc.h @@ -44,8 +44,6 @@ static inline void pgtable_free(void *table, int shift) } } -#define get_hugepd_cache_index(x) (x) - static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) { unsigned long pgf = (unsigned long)table; diff --git a/arch/powerpc/include/asm/nohash/pgtable.h b/arch/powerpc/include/asm/nohash/pgtable.h index 90d6a0943b35..8d1f0b7062eb 100644 --- a/arch/powerpc/include/asm/nohash/pgtable.h +++ b/arch/powerpc/include/asm/nohash/pgtable.h @@ -31,6 +31,13 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p extern int icache_44x_need_flush; +#ifndef pte_huge_size +static inline unsigned long pte_huge_size(pte_t pte) +{ + return PAGE_SIZE; +} +#endif + /* * PTE updates. This function is called whenever an existing * valid PTE is updated. This does -not- include set_pte_at() @@ -52,11 +59,34 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p { pte_basic_t old = pte_val(*p); pte_basic_t new = (old & ~(pte_basic_t)clr) | set; + unsigned long sz; + unsigned long pdsize; + int i; if (new == old) return old; - *p = __pte(new); + if (huge) + sz = pte_huge_size(__pte(old)); + else + sz = PAGE_SIZE; + + if (sz < PMD_SIZE) + pdsize = PAGE_SIZE; + else if (sz < PUD_SIZE) + pdsize = PMD_SIZE; + else if (sz < P4D_SIZE) + pdsize = PUD_SIZE; + else if (sz < PGDIR_SIZE) + pdsize = P4D_SIZE; + else + pdsize = PGDIR_SIZE; + + for (i = 0; i < sz / pdsize; i++, p++) { + *p = __pte(new); + if (new) + new += (unsigned long long)(pdsize / PAGE_SIZE) << PTE_RPN_SHIFT; + } if (IS_ENABLED(CONFIG_44x) && !is_kernel_addr(addr) && (old & _PAGE_EXEC)) icache_44x_need_flush = 1; @@ -340,16 +370,6 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, #define pgprot_writecombine pgprot_noncached_wc -#ifdef CONFIG_ARCH_HAS_HUGEPD -static inline int hugepd_ok(hugepd_t hpd) -{ - /* We clear the top bit to indicate hugepd */ - return (hpd_val(hpd) && (hpd_val(hpd) & PD_HUGE) == 0); -} - -#define is_hugepd(hpd) (hugepd_ok(hpd)) -#endif - int map_kernel_page(unsigned long va, phys_addr_t pa, pgprot_t prot); void unmap_kernel_page(unsigned long va); diff --git a/arch/powerpc/include/asm/nohash/pte-e500.h b/arch/powerpc/include/asm/nohash/pte-e500.h index 6dac1c0a6937..cb78392494da 100644 --- a/arch/powerpc/include/asm/nohash/pte-e500.h +++ b/arch/powerpc/include/asm/nohash/pte-e500.h @@ -101,6 +101,39 @@ static inline unsigned long pte_huge_size(pte_t pte) } #define pte_huge_size pte_huge_size +static inline int pmd_leaf(pmd_t pmd) +{ + if (IS_ENABLED(CONFIG_PPC64)) + return (long)pmd_val(pmd) > 0; + else + return pmd_val(pmd) & _PAGE_PSIZE_MSK; +} +#define pmd_leaf pmd_leaf + +static inline unsigned long pmd_leaf_size(pmd_t pmd) +{ + return pte_huge_size(__pte(pmd_val(pmd))); +} +#define pmd_leaf_size pmd_leaf_size + +#ifdef CONFIG_PPC64 +static inline int pud_leaf(pud_t pud) +{ + if (IS_ENABLED(CONFIG_PPC64)) + return (long)pud_val(pud) > 0; + else + return pud_val(pud) & _PAGE_PSIZE_MSK; +} +#define pud_leaf pud_leaf + +static inline unsigned long pud_leaf_size(pud_t pud) +{ + return pte_huge_size(__pte(pud_val(pud))); +} +#define pud_leaf_size pud_leaf_size + +#endif + #endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */ diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h index 018c3d55232c..7d3c3bc40e6a 100644 --- a/arch/powerpc/include/asm/page.h +++ b/arch/powerpc/include/asm/page.h @@ -269,20 +269,7 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) #define is_kernel_addr(x) ((x) >= TASK_SIZE) #endif -#ifndef CONFIG_PPC_BOOK3S_64 -/* - * Use the top bit of the higher-level page table entries to indicate whether - * the entries we point to contain hugepages. This works because we know that - * the page tables live in kernel space. If we ever decide to support having - * page tables at arbitrary addresses, this breaks and will have to change. - */ -#ifdef CONFIG_PPC64 -#define PD_HUGE 0x8000000000000000UL -#else -#define PD_HUGE 0x80000000 -#endif - -#else /* CONFIG_PPC_BOOK3S_64 */ +#ifdef CONFIG_PPC_BOOK3S_64 /* * Book3S 64 stores real addresses in the hugepd entries to * avoid overlaps with _PAGE_PRESENT and _PAGE_PTE. diff --git a/arch/powerpc/kernel/head_85xx.S b/arch/powerpc/kernel/head_85xx.S index 226f88e77d6d..f9a73fae6464 100644 --- a/arch/powerpc/kernel/head_85xx.S +++ b/arch/powerpc/kernel/head_85xx.S @@ -311,16 +311,14 @@ set_ivor: rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ add r12, r11, r12; \ lwz r11, 4(r12); /* Get pgd/pmd entry */ \ + rlwinm. r10, r11, 32 - _PAGE_PSIZE_SHIFT, 0x1e; /* get tsize*/ \ + bne 1000f; /* Huge page (leaf entry) */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ - blt 1000f; /* Normal non-huge page */ \ beq 2f; /* Bail if no table */ \ - oris r11, r11, PD_HUGE@h; /* Put back address bit */ \ - andi. r10, r11, HUGEPD_SHIFT_MASK@l; /* extract size field */ \ - xor r12, r10, r11; /* drop size bits from pointer */ \ - b 1001f; \ -1000: rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ + rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ li r10, 0; /* clear r10 */ \ -1001: lwz r11, 4(r12); /* Get pte entry */ + lwz r11, 4(r12); /* Get pte entry */ \ +1000: #else #define FIND_PTE \ rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ @@ -735,17 +733,12 @@ finish_tlb_load: lwz r15, 0(r14) 100: stw r15, 0(r17) - /* - * Calc MAS1_TSIZE from r10 (which has pshift encoded) - * tlb_enc = (pshift - 10). - */ - subi r15, r10, 10 mfspr r16, SPRN_MAS1 - rlwimi r16, r15, 7, 20, 24 + rlwimi r16, r10, MAS1_TSIZE_SHIFT, MAS1_TSIZE_MASK mtspr SPRN_MAS1, r16 /* copy the pshift for use later */ - mr r14, r10 + addi r14, r10, _PAGE_PSIZE_SHIFT_OFFSET /* fall through */ diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 394846197974..9e692d96f9e1 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -625,8 +625,6 @@ static int __init hugetlbpage_init(void) */ if (pdshift > shift) { pgtable_cache_add(pdshift - shift); - } else if (IS_ENABLED(CONFIG_PPC_E500)) { - pgtable_cache_add(PTE_T_ORDER); } configured = true; diff --git a/arch/powerpc/mm/nohash/tlb_low_64e.S b/arch/powerpc/mm/nohash/tlb_low_64e.S index a54e7d6c3d0b..de568297d5c5 100644 --- a/arch/powerpc/mm/nohash/tlb_low_64e.S +++ b/arch/powerpc/mm/nohash/tlb_low_64e.S @@ -450,11 +450,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_SMT) tlb_miss_huge_e6500: beq tlb_miss_fault_e6500 - li r10,1 - andi. r15,r14,HUGEPD_SHIFT_MASK@l /* r15 = psize */ - rldimi r14,r10,63,0 /* Set PD_HUGE */ - xor r14,r14,r15 /* Clear size bits */ - ldx r14,0,r14 + rlwinm r15,r14,32-_PAGE_PSIZE_SHIFT,0x1e /* * Now we build the MAS for a huge page. @@ -465,7 +461,6 @@ tlb_miss_huge_e6500: * MAS 2,3+7: Needs to be redone similar to non-tablewalk handler */ - subi r15,r15,10 /* Convert psize to tsize */ mfspr r10,SPRN_MAS1 rlwinm r10,r10,0,~MAS1_IND rlwimi r10,r15,MAS1_TSIZE_SHIFT,MAS1_TSIZE_MASK diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 294775c793ab..6498454959f3 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -331,6 +331,37 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, __set_huge_pte_at(pmdp, ptep, pte_val(pte)); } } +#elif defined(CONFIG_PPC_E500) +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte, unsigned long sz) +{ + unsigned long pdsize; + int i; + + pte = set_pte_filter(pte, addr); + + /* + * Make sure hardware valid bit is not set. We don't do + * tlb flush for this update. + */ + VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep)); + + if (sz < PMD_SIZE) + pdsize = PAGE_SIZE; + else if (sz < PUD_SIZE) + pdsize = PMD_SIZE; + else if (sz < P4D_SIZE) + pdsize = PUD_SIZE; + else if (sz < PGDIR_SIZE) + pdsize = P4D_SIZE; + else + pdsize = PGDIR_SIZE; + + for (i = 0; i < sz / pdsize; i++, ptep++, addr += pdsize) { + __set_pte_at(mm, addr, ptep, pte, 0); + pte = __pte(pte_val(pte) + ((unsigned long long)pdsize / PAGE_SIZE << PFN_PTE_SHIFT)); + } +} #endif #endif /* CONFIG_HUGETLB_PAGE */ diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index fa4bb096b3ae..30a78e99663e 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -291,7 +291,6 @@ config PPC_BOOK3S config PPC_E500 select FSL_EMB_PERFMON bool - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 select PPC_SMP_MUXED_IPI select PPC_DOORBELL From patchwork Tue Jul 2 13:51:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77A8DC3064D for ; Tue, 2 Jul 2024 13:53:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1496F6B0082; Tue, 2 Jul 2024 09:53:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F7E26B00A0; Tue, 2 Jul 2024 09:53:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB3DE6B00DE; Tue, 2 Jul 2024 09:53:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C54236B0082 for ; Tue, 2 Jul 2024 09:53:31 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 79E6F1C0810 for ; Tue, 2 Jul 2024 13:53:31 +0000 (UTC) X-FDA: 82294955022.28.4634EDB Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf11.hostedemail.com (Postfix) with ESMTP id 440504002D for ; Tue, 2 Jul 2024 13:53:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928397; a=rsa-sha256; cv=none; b=PvWv2oiBvAJ+OrzJhwjxI4NuBTlD1xi2WlaM2ccqmHEqTo3/AzmaVChcxN4xTDVmS9rUJv +sVFN3I7eIune2SHFOIk3Qi5dShUPjPdzQrLjNmnCPrvk1K0Y/xrWGtlsk6doffwu2MkAR 79d7aWN/LSibd647gfXqdSbx8XZU3mc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928397; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UdJhpo4T3ccdZ7pp4meDX0xWh/JbUZ5z6qHEY+qL4KU=; b=KwLzG4C4FdGmdPmBQiYg9lnITY0RZBtlBsaIhp7aYY7pK9yxlnTfTbeKTKwRZ+P04VqFgU ZCaq+xRmIR38ZnC3jbloHpy9NpWFtL0ri/tI3beLYP74AHeYeW4NWJ5l66Mzb+nUmk/8Mg tJo1qVbtE7E9gMbh0GDP9QilHk2IQ4Y= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48P18qWz9v7L; Tue, 2 Jul 2024 15:51:49 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BcwQTwacX8AA; Tue, 2 Jul 2024 15:51:49 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4883x3Tz9v7h; Tue, 2 Jul 2024 15:51:36 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 82C888B775; Tue, 2 Jul 2024 15:51:36 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id DJAGjaOAeV4z; Tue, 2 Jul 2024 15:51:36 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id DD5DA8B774; Tue, 2 Jul 2024 15:51:35 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 21/23] powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD Date: Tue, 2 Jul 2024 15:51:33 +0200 Message-ID: <7448f60a9b3efd396595f4f735d1e0babc5ae379.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=18279; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=2G8oJpcNtkRBG4yqaxrMfZ/soZsLaajV0a3G73z3lRk=; b=0csDV/KGH7+sBc/SYMKVY22MpAHPsV5G2PXVQre8Qgz/Yc0cBfG9WMJYQFaV3glH4WpuDZvcd 2i6EilDIp6oCQSEg1NWjHFTVG8zYmYBLa4ZhKyWW4yiJvCM8IGmoiEY X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Stat-Signature: or45w9jbdsgdcqontiapgzqu4zisnawm X-Rspamd-Queue-Id: 440504002D X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1719928408-2499 X-HE-Meta: U2FsdGVkX1+WFbRxSS/kyZA6BcROjAfi6KQViYRxcxt58M3sJeE5aFfIuXkhcfawGLQrkz4p6gB6zVMWWkafUsNTMKdwmSC/lZMFn5EZooklyif3NHU/SRJbed5UUZhkZoygScdJtX49/uKa5+5IkoyZwYUect9Xfe9Is4vJg5XTXb5OEz3ew3ZSC4IH7A/+Nwr8GbDYedcRjmV58o0OWGRGR2e93ontFCumJQ0r+qYrov30jdkpTPDdWhhkcNTpO7hutDDo4DM+6F+gr8+ybPovrluzA/cCebOwVjOGyuxGcMh9GvFePGLbVoW/pOEhEH8Ts4D3YYz5KMcgoCFXuDxJdA+0ArI9OJwK4gFnXiyrZegW/xrC/VtaYVJ8qm4bpUB6wQRbyZRguPiu8/waCFMOApFiIL6Er0DXNbWR//S8mGotvzCoQCeSdiuYekJVUR+HC6yU3UUNSUp6GfWzzdVWD5TdXdOC+2r5mG+InaajBclgaciiRiQiFX1JNRFEUHeNQwYgNm/IrNj74wOAINMNtg1+ng6h3w/3h6aXQaSmq5FfwznEIjMYjMTwFmRxG5PaI0dbX6w2LIdASsADrEWcHyy+8r5A8FP+DM/twxxHqU9kmm5THFppO4KIoV3eGGzX3SCxmau5qhnAlY++B3e4ujSrAv78BKgZyFvn8o+tMmllQag9gY6EDYU+x7PJkBI5axX57uFjaYxIUnenFOPil+KNyNHk9rMLlUckrTeaJ/x15HhzUmcujXfa1ojAWyPUJAr0G76z5TxVuQXS+y0pwUV6WedZl3gPEHL1oAogf7xp1XBbT9XwiIijG+fka73jly5kyrtsD+nyjw2JSTOvRPB9KRSo4er6TFOl/XSmPbzbu7zgCW0x1NkrqCPlOaVc/Nkq7xlULIhIMyrzh2F/WeeadVzBFWlACoSGq+U8EduWzwdxMgwtlUO6fwQB090S5fc7qErAPcLCjJW hXAtthFn XZorrHyu6TK8JeCcE5wiEtr8BwrLbu90OLSCAfJyjJ9JTZmmvlzP4zOvaI+hZUbGq81WVTnnB2oZlAfHi8FnISOV4zXMjY34jPt/UEgEfHKLFIOuMv2sSLwzMs/abjGml5P73 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On book3s/64, the only user of hugepd is hash in 4k mode. All other setups (hash-64, radix-4, radix-64) use leaf PMD/PUD. Rework hash-4k to use contiguous PMD and PUD instead. In that setup there are only two huge page sizes: 16M and 16G. 16M sits at PMD level and 16G at PUD level. pte_update doesn't know page size, lets use the same trick as hpte_need_flush() to get page size from segment properties. That's not the most efficient way but let's do that until callers of pte_update() provide page size instead of just a huge flag. Signed-off-by: Christophe Leroy Acked-by: Michael Ellerman (powerpc) --- v3: - Add missing pmd_leaf_size() and pud_leaf_size() - More cleanup in hugetlbpage_init() - Take a page fault when DIRTY or ACCESSED is missing on hash-4 hugepage v4: Rebased on v6.10-rc1 v6: Added a WARN_ON_ONCE() in hash__pte_update() in case the pagesize is unexpected. --- arch/powerpc/include/asm/book3s/64/hash-4k.h | 15 ------ arch/powerpc/include/asm/book3s/64/hash.h | 40 +++++++++++++--- arch/powerpc/include/asm/book3s/64/hugetlb.h | 38 --------------- .../include/asm/book3s/64/pgtable-4k.h | 47 ------------------- .../include/asm/book3s/64/pgtable-64k.h | 20 -------- arch/powerpc/include/asm/book3s/64/pgtable.h | 22 +++++++-- arch/powerpc/include/asm/hugetlb.h | 4 ++ .../powerpc/include/asm/nohash/hugetlb-e500.h | 4 -- arch/powerpc/include/asm/page.h | 8 ---- arch/powerpc/mm/book3s64/hash_utils.c | 11 +++-- arch/powerpc/mm/book3s64/hugetlbpage.c | 10 ++++ arch/powerpc/mm/book3s64/pgtable.c | 12 ----- arch/powerpc/mm/hugetlbpage.c | 26 ---------- arch/powerpc/mm/pgtable.c | 2 +- arch/powerpc/platforms/Kconfig.cputype | 1 - 15 files changed, 74 insertions(+), 186 deletions(-) delete mode 100644 arch/powerpc/include/asm/book3s/64/pgtable-4k.h diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h index 6472b08fa1b0..c654c376ef8b 100644 --- a/arch/powerpc/include/asm/book3s/64/hash-4k.h +++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h @@ -74,21 +74,6 @@ #define remap_4k_pfn(vma, addr, pfn, prot) \ remap_pfn_range((vma), (addr), (pfn), PAGE_SIZE, (prot)) -#ifdef CONFIG_HUGETLB_PAGE -static inline int hash__hugepd_ok(hugepd_t hpd) -{ - unsigned long hpdval = hpd_val(hpd); - /* - * if it is not a pte and have hugepd shift mask - * set, then it is a hugepd directory pointer - */ - if (!(hpdval & _PAGE_PTE) && (hpdval & _PAGE_PRESENT) && - ((hpdval & HUGEPD_SHIFT_MASK) != 0)) - return true; - return false; -} -#endif - /* * 4K PTE format is different from 64K PTE format. Saving the hash_slot is just * a matter of returning the PTE bits that need to be modified. On 64K PTE, diff --git a/arch/powerpc/include/asm/book3s/64/hash.h b/arch/powerpc/include/asm/book3s/64/hash.h index faf3e3b4e4b2..0755f2567021 100644 --- a/arch/powerpc/include/asm/book3s/64/hash.h +++ b/arch/powerpc/include/asm/book3s/64/hash.h @@ -4,6 +4,7 @@ #ifdef __KERNEL__ #include +#include /* * Common bits between 4K and 64K pages in a linux-style PTE. @@ -161,14 +162,10 @@ extern void hpte_need_flush(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long pte, int huge); unsigned long htab_convert_pte_flags(unsigned long pteflags, unsigned long flags); /* Atomic PTE updates */ -static inline unsigned long hash__pte_update(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, unsigned long clr, - unsigned long set, - int huge) +static inline unsigned long hash__pte_update_one(pte_t *ptep, unsigned long clr, + unsigned long set) { __be64 old_be, tmp_be; - unsigned long old; __asm__ __volatile__( "1: ldarx %0,0,%3 # pte_update\n\ @@ -182,11 +179,40 @@ static inline unsigned long hash__pte_update(struct mm_struct *mm, : "r" (ptep), "r" (cpu_to_be64(clr)), "m" (*ptep), "r" (cpu_to_be64(H_PAGE_BUSY)), "r" (cpu_to_be64(set)) : "cc" ); + + return be64_to_cpu(old_be); +} + +static inline unsigned long hash__pte_update(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long clr, + unsigned long set, + int huge) +{ + unsigned long old; + + old = hash__pte_update_one(ptep, clr, set); + + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && huge) { + unsigned int psize = get_slice_psize(mm, addr); + int nb, i; + + if (psize == MMU_PAGE_16M) + nb = SZ_16M / PMD_SIZE; + else if (psize == MMU_PAGE_16G) + nb = SZ_16G / PUD_SIZE; + else + nb = 1; + + WARN_ON_ONCE(nb == 1); /* Should never happen */ + + for (i = 1; i < nb; i++) + hash__pte_update_one(ptep + i, clr, set); + } /* huge pages use the old page table lock */ if (!huge) assert_pte_locked(mm, addr); - old = be64_to_cpu(old_be); if (old & H_PAGE_HASHPTE) hpte_need_flush(mm, addr, ptep, old, huge); diff --git a/arch/powerpc/include/asm/book3s/64/hugetlb.h b/arch/powerpc/include/asm/book3s/64/hugetlb.h index aa1c67c8bfc8..f0bba9c5f9c3 100644 --- a/arch/powerpc/include/asm/book3s/64/hugetlb.h +++ b/arch/powerpc/include/asm/book3s/64/hugetlb.h @@ -49,9 +49,6 @@ static inline bool gigantic_page_runtime_supported(void) return true; } -/* hugepd entry valid bit */ -#define HUGEPD_VAL_BITS (0x8000000000000000UL) - #define huge_ptep_modify_prot_start huge_ptep_modify_prot_start extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); @@ -60,29 +57,7 @@ extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); -/* - * This should work for other subarchs too. But right now we use the - * new format only for 64bit book3s - */ -static inline pte_t *hugepd_page(hugepd_t hpd) -{ - BUG_ON(!hugepd_ok(hpd)); - /* - * We have only four bits to encode, MMU page size - */ - BUILD_BUG_ON((MMU_PAGE_COUNT - 1) > 0xf); - return __va(hpd_val(hpd) & HUGEPD_ADDR_MASK); -} - -static inline unsigned int hugepd_mmu_psize(hugepd_t hpd) -{ - return (hpd_val(hpd) & HUGEPD_SHIFT_MASK) >> 2; -} -static inline unsigned int hugepd_shift(hugepd_t hpd) -{ - return mmu_psize_to_shift(hugepd_mmu_psize(hpd)); -} static inline void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr) { @@ -90,19 +65,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, return radix__flush_hugetlb_page(vma, vmaddr); } -static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, - unsigned int pdshift) -{ - unsigned long idx = (addr & ((1UL << pdshift) - 1)) >> hugepd_shift(hpd); - - return hugepd_page(hpd) + idx; -} - -static inline void hugepd_populate(hugepd_t *hpdp, pte_t *new, unsigned int pshift) -{ - *hpdp = __hugepd(__pa(new) | HUGEPD_VAL_BITS | (shift_to_mmu_psize(pshift) << 2)); -} - void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr); static inline int check_and_get_huge_psize(int shift) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable-4k.h b/arch/powerpc/include/asm/book3s/64/pgtable-4k.h deleted file mode 100644 index baf934578c3a..000000000000 --- a/arch/powerpc/include/asm/book3s/64/pgtable-4k.h +++ /dev/null @@ -1,47 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H -#define _ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H -/* - * hash 4k can't share hugetlb and also doesn't support THP - */ -#ifndef __ASSEMBLY__ -#ifdef CONFIG_HUGETLB_PAGE -/* - * With radix , we have hugepage ptes in the pud and pmd entries. We don't - * need to setup hugepage directory for them. Our pte and page directory format - * enable us to have this enabled. - */ -static inline int hugepd_ok(hugepd_t hpd) -{ - if (radix_enabled()) - return 0; - return hash__hugepd_ok(hpd); -} -#define is_hugepd(hpd) (hugepd_ok(hpd)) - -/* - * 16M and 16G huge page directory tables are allocated from slab cache - * - */ -#define H_16M_CACHE_INDEX (PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE - 24) -#define H_16G_CACHE_INDEX \ - (PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE + H_PUD_INDEX_SIZE - 34) - -static inline int get_hugepd_cache_index(int index) -{ - switch (index) { - case H_16M_CACHE_INDEX: - return HTLB_16M_INDEX; - case H_16G_CACHE_INDEX: - return HTLB_16G_INDEX; - default: - BUG(); - } - /* should not reach */ -} - -#endif /* CONFIG_HUGETLB_PAGE */ - -#endif /* __ASSEMBLY__ */ - -#endif /*_ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H */ diff --git a/arch/powerpc/include/asm/book3s/64/pgtable-64k.h b/arch/powerpc/include/asm/book3s/64/pgtable-64k.h index 6ac73da7b80e..4d8d7b4ea16b 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable-64k.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable-64k.h @@ -5,26 +5,6 @@ #ifndef __ASSEMBLY__ #ifdef CONFIG_HUGETLB_PAGE -/* - * With 64k page size, we have hugepage ptes in the pgd and pmd entries. We don't - * need to setup hugepage directory for them. Our pte and page directory format - * enable us to have this enabled. - */ -static inline int hugepd_ok(hugepd_t hpd) -{ - return 0; -} - -#define is_hugepd(pdep) 0 - -/* - * This should never get called - */ -static __always_inline int get_hugepd_cache_index(int index) -{ - BUILD_BUG(); -} - #endif /* CONFIG_HUGETLB_PAGE */ static inline int remap_4k_pfn(struct vm_area_struct *vma, unsigned long addr, diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 8f9432e3855a..519b1743a0f4 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -274,6 +274,24 @@ static inline bool pud_leaf(pud_t pud) { return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE)); } + +#define pmd_leaf_size pmd_leaf_size +static inline unsigned long pmd_leaf_size(pmd_t pmd) +{ + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled()) + return SZ_16M; + else + return PMD_SIZE; +} + +#define pud_leaf_size pud_leaf_size +static inline unsigned long pud_leaf_size(pud_t pud) +{ + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled()) + return SZ_16G; + else + return PUD_SIZE; +} #endif /* __ASSEMBLY__ */ #include @@ -285,11 +303,9 @@ static inline bool pud_leaf(pud_t pud) #define MAX_PHYSMEM_BITS R_MAX_PHYSMEM_BITS #endif - +/* hash 4k can't share hugetlb and also doesn't support THP */ #ifdef CONFIG_PPC_64K_PAGES #include -#else -#include #endif #include diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h index 79176a499763..e959c26c0b52 100644 --- a/arch/powerpc/include/asm/hugetlb.h +++ b/arch/powerpc/include/asm/hugetlb.h @@ -37,6 +37,10 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long ceiling); #endif +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte, unsigned long sz); + #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/arch/powerpc/include/asm/nohash/hugetlb-e500.h b/arch/powerpc/include/asm/nohash/hugetlb-e500.h index 812c71fc5eb1..cab0e1f1eea0 100644 --- a/arch/powerpc/include/asm/nohash/hugetlb-e500.h +++ b/arch/powerpc/include/asm/nohash/hugetlb-e500.h @@ -2,10 +2,6 @@ #ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H #define _ASM_POWERPC_NOHASH_HUGETLB_E500_H -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, - pte_t pte, unsigned long sz); - void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr); static inline int check_and_get_huge_psize(int shift) diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h index 7d3c3bc40e6a..c0af246a64ff 100644 --- a/arch/powerpc/include/asm/page.h +++ b/arch/powerpc/include/asm/page.h @@ -269,14 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) #define is_kernel_addr(x) ((x) >= TASK_SIZE) #endif -#ifdef CONFIG_PPC_BOOK3S_64 -/* - * Book3S 64 stores real addresses in the hugepd entries to - * avoid overlaps with _PAGE_PRESENT and _PAGE_PTE. - */ -#define HUGEPD_ADDR_MASK (0x0ffffffffffffffful & ~HUGEPD_SHIFT_MASK) -#endif /* CONFIG_PPC_BOOK3S_64 */ - /* * Some number of bits at the level of the page table that points to * a hugepte are used to encode the size. This masks those bits. diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 01c3b4b65241..6727a15ab94f 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -1233,10 +1233,6 @@ void __init hash__early_init_mmu(void) __pmd_table_size = H_PMD_TABLE_SIZE; __pud_table_size = H_PUD_TABLE_SIZE; __pgd_table_size = H_PGD_TABLE_SIZE; - /* - * 4k use hugepd format, so for hash set then to - * zero - */ __pmd_val_bits = HASH_PMD_VAL_BITS; __pud_val_bits = HASH_PUD_VAL_BITS; __pgd_val_bits = HASH_PGD_VAL_BITS; @@ -1546,6 +1542,13 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, goto bail; } + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled()) { + if (hugeshift == PMD_SHIFT && psize == MMU_PAGE_16M) + hugeshift = mmu_psize_defs[MMU_PAGE_16M].shift; + if (hugeshift == PUD_SHIFT && psize == MMU_PAGE_16G) + hugeshift = mmu_psize_defs[MMU_PAGE_16G].shift; + } + /* * Add _PAGE_PRESENT to the required access perm. If there are parallel * updates to the pte that can possibly clear _PAGE_PTE, catch that too. diff --git a/arch/powerpc/mm/book3s64/hugetlbpage.c b/arch/powerpc/mm/book3s64/hugetlbpage.c index 5a2e512e96db..83c3361b358b 100644 --- a/arch/powerpc/mm/book3s64/hugetlbpage.c +++ b/arch/powerpc/mm/book3s64/hugetlbpage.c @@ -53,6 +53,16 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, /* If PTE permissions don't match, take page fault */ if (unlikely(!check_pte_access(access, old_pte))) return 1; + /* + * If hash-4k, hugepages use seeral contiguous PxD entries + * so bail out and let mm make the page young or dirty + */ + if (IS_ENABLED(CONFIG_PPC_4K_PAGES)) { + if (!(old_pte & _PAGE_ACCESSED)) + return 1; + if ((access & _PAGE_WRITE) && !(old_pte & _PAGE_DIRTY)) + return 1; + } /* * Try to lock the PTE, add ACCESSED and DIRTY if it was diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 2975ea0841ba..f4d8d3c40e5c 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -461,18 +461,6 @@ static inline void pgtable_free(void *table, int index) case PUD_INDEX: __pud_free(table); break; -#if defined(CONFIG_PPC_4K_PAGES) && defined(CONFIG_HUGETLB_PAGE) - /* 16M hugepd directory at pud level */ - case HTLB_16M_INDEX: - BUILD_BUG_ON(H_16M_CACHE_INDEX <= 0); - kmem_cache_free(PGT_CACHE(H_16M_CACHE_INDEX), table); - break; - /* 16G hugepd directory at the pgd level */ - case HTLB_16G_INDEX: - BUILD_BUG_ON(H_16G_CACHE_INDEX <= 0); - kmem_cache_free(PGT_CACHE(H_16G_CACHE_INDEX), table); - break; -#endif /* We don't free pgd table via RCU callback */ default: BUG(); diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 9e692d96f9e1..1fe2843f5b12 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -592,40 +592,14 @@ static int __init hugetlbpage_init(void) for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { unsigned shift; - unsigned pdshift; if (!mmu_psize_defs[psize].shift) continue; shift = mmu_psize_to_shift(psize); -#ifdef CONFIG_PPC_BOOK3S_64 - if (shift > PGDIR_SHIFT) - continue; - else if (shift > PUD_SHIFT) - pdshift = PGDIR_SHIFT; - else if (shift > PMD_SHIFT) - pdshift = PUD_SHIFT; - else - pdshift = PMD_SHIFT; -#else - if (shift < PUD_SHIFT) - pdshift = PMD_SHIFT; - else if (shift < PGDIR_SHIFT) - pdshift = PUD_SHIFT; - else - pdshift = PGDIR_SHIFT; -#endif - if (add_huge_page_size(1ULL << shift) < 0) continue; - /* - * if we have pdshift and shift value same, we don't - * use pgt cache for hugepd. - */ - if (pdshift > shift) { - pgtable_cache_add(pdshift - shift); - } configured = true; } diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 6498454959f3..218792cb2c47 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -331,7 +331,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, __set_huge_pte_at(pmdp, ptep, pte_val(pte)); } } -#elif defined(CONFIG_PPC_E500) +#else void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz) { diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index 30a78e99663e..b2d8c0da2ad9 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -98,7 +98,6 @@ config PPC_BOOK3S_64 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING select HAVE_MOVE_PMD From patchwork Tue Jul 2 13:51:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9012CC3064D for ; Tue, 2 Jul 2024 13:53:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2320B6B00E1; Tue, 2 Jul 2024 09:53:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E2B26B00E2; Tue, 2 Jul 2024 09:53:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2A4B6B00E3; Tue, 2 Jul 2024 09:53:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CD6D56B00E1 for ; Tue, 2 Jul 2024 09:53:35 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 71DE71C088F for ; Tue, 2 Jul 2024 13:53:35 +0000 (UTC) X-FDA: 82294955190.20.D3E025C Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf04.hostedemail.com (Postfix) with ESMTP id 4C6F64001D for ; Tue, 2 Jul 2024 13:53:33 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928390; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m2F7tkbpd3atoPfgexgtYQlFOxdiSoE6e+B8hIZCQ64=; b=pQDHcolyH6ikvDKmuL7WoWShg3PDBVVNLOSL4bA3EDGE9PcbJQ0ohhl1PHOJGwjlPaN2Hp rQx0bQLPnJAGJGNyv5en0ESbvkphkFoXt5tv6SGNCKLCqtpK6VYD4dP2YlwsURUPtWzbDQ QOWeeBZ3+RhJd7Hu7cX59iX9zgc1Wlk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928390; a=rsa-sha256; cv=none; b=npxYpp3/G5wpE+i7HtFEQPNVeycfj4BMXgz0+63eiUBtpUHkIaKxeKkmE9uxI77t/59Hmn r1EmDbpR1uSfexmHRiIakgjtV7Ebmjw/Ng9+Hqiz7MTZ3ujKS1ax31ck77hMfVwovipyme xiK+FTLQjN3NG96+dACjcIXnqAlEFEA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48Q1q3Xz9v4H; Tue, 2 Jul 2024 15:51:50 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1aogaygGae49; Tue, 2 Jul 2024 15:51:50 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD4890zSLz9tZl; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 1CAB18B775; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id KrwMFsqEwazi; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 790BC8B764; Tue, 2 Jul 2024 15:51:36 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 22/23] powerpc/mm: Remove hugepd leftovers Date: Tue, 2 Jul 2024 15:51:34 +0200 Message-ID: <39c0d0adee6790fc42cee9f458e05fb95136c3dd.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928275; l=18716; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=hRfDLIZLnNuYyLyTuuY10TaUMZmvM7CY+ONZE5z7zuE=; b=mWwzBBaiNM3igxxIKRqIYsMUG+ENPCanPxJ2URvsFUHYCowrxPG9Y+se2vsAp8QPPpBw88+eU EuoAIdr5/NxDj2yqEKR2Ll9w6GbrpMRDZZgVIYCdZXFw0phqSYc6VVA X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4C6F64001D X-Stat-Signature: hp6hirjuoe813ubpzt7jfa3z1gwuwhuj X-HE-Tag: 1719928413-227932 X-HE-Meta: U2FsdGVkX19y3uWIDruX0x9NwjRSlo4s7Lzegs3PTkEkeaYLu9X+DiacgdQXk1tHtQOCCmGOXjeB8E/0A+fZWoseu6D0/hJYTV6qKzB3YrEZ6iJ1wsPIBMJxugYlzMj1yn/IfbkgTOLsIVDFY7eD1eL3D2glI+ZL9kQp69X1wepSew93ZDa+JHDL0hqHeX9AvIaYEqej6GfdH2f3Jgq+B2iImEnFVaAD09K5cyzSKjo9LoncIpa0wtkwqmmqoDGouvL1qNP7nkOdfHEiWCUMzRAnRWAeOBA7o03S6UWuZGhwD/NNr4L+QX+No0cfHesmj4AmC0yzMvikDMh4SFzp1xnp7ldirzZzh0IACwUvhe29QoJSgZtEvL6MpAIBtM8SqMBnS423ezlA3oXcC4DXaZurSrZZjYQGvhgEOFf/iikQuCYJjspViFA01QiK3s8aewyp15IDZKT/XCSlI6jCL8UUWxZAaZec97TIskpo4HvMLMFWYlo/8SygcBEaq48WYZiOWmvbEbC27eE6NOAytE3uf9Zmb6gbR2gNxEmnRGOt2S7XQjTK2LPeq+JLE8PIJvsafmBCNMz6DOugAzpX3gp+qDWVp0n72+snDDJbkB4RAYZS3mlNjV/CBVgp998KmZ/FI98ZCz2Dvc8wWj4plFyvkyUowPoHckqIiHcsn3edon/ntBw4MU/739vpNVJuu+AU87xKaQhVIwMBlmImx3A6UzY3CDvOX7aSHFmI1j4IIiE5y3ADBC7YKPosX3LgaOfmGCDWdE/doiT4NBPUyEGOlaqtxeF493BTSGjQczfqM6U4kUysJOAPHDHmg739WgJpSLH5TH65xXrEE6S1WHN7B/PRtJdhQC55qfZ/bbJrt7j8tm0iJ1qV/5JbeymrQUehYQ8ZxYL680NOVZ6RGXtQ7o09IxN2Q6piRaMnUk1QD979O83ioM0BEUMCwbsGxpntgHXLI9pZr6/7lOs UhMw5dBO bo60RRArBBmdkToeQ3WrGbjBCGzySYJkz7YQXHjIi2aX3x3OYmJli0yBV+Bwte5r8zK6rUccg9Dg+RXmSjxCr3WVTOZtLOGnqVYBc+XntV88BKWNGmMakunEn8y739xKezf/9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All targets have now opted out of CONFIG_ARCH_HAS_HUGEPD so remove left over code. Signed-off-by: Christophe Leroy Acked-by: Oscar Salvador --- v5: Fix a forgotten #endif which ended up in following patch --- arch/powerpc/include/asm/hugetlb.h | 7 - arch/powerpc/include/asm/page.h | 6 - arch/powerpc/include/asm/pgtable-be-types.h | 10 - arch/powerpc/include/asm/pgtable-types.h | 9 - arch/powerpc/mm/hugetlbpage.c | 413 -------------------- arch/powerpc/mm/init-common.c | 8 +- arch/powerpc/mm/pgtable.c | 27 +- 7 files changed, 3 insertions(+), 477 deletions(-) diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h index e959c26c0b52..18a3028ac3b6 100644 --- a/arch/powerpc/include/asm/hugetlb.h +++ b/arch/powerpc/include/asm/hugetlb.h @@ -30,13 +30,6 @@ static inline int is_hugepage_only_range(struct mm_struct *mm, } #define is_hugepage_only_range is_hugepage_only_range -#ifdef CONFIG_ARCH_HAS_HUGEPD -#define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE -void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, - unsigned long end, unsigned long floor, - unsigned long ceiling); -#endif - #define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz); diff --git a/arch/powerpc/include/asm/page.h b/arch/powerpc/include/asm/page.h index c0af246a64ff..83d0a4fc5f75 100644 --- a/arch/powerpc/include/asm/page.h +++ b/arch/powerpc/include/asm/page.h @@ -269,12 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) #define is_kernel_addr(x) ((x) >= TASK_SIZE) #endif -/* - * Some number of bits at the level of the page table that points to - * a hugepte are used to encode the size. This masks those bits. - */ -#define HUGEPD_SHIFT_MASK 0x3f - #ifndef __ASSEMBLY__ #ifdef CONFIG_PPC_BOOK3S_64 diff --git a/arch/powerpc/include/asm/pgtable-be-types.h b/arch/powerpc/include/asm/pgtable-be-types.h index 82633200b500..6bd8f89b25dc 100644 --- a/arch/powerpc/include/asm/pgtable-be-types.h +++ b/arch/powerpc/include/asm/pgtable-be-types.h @@ -101,14 +101,4 @@ static inline bool pmd_xchg(pmd_t *pmdp, pmd_t old, pmd_t new) return pmd_raw(old) == prev; } -#ifdef CONFIG_ARCH_HAS_HUGEPD -typedef struct { __be64 pdbe; } hugepd_t; -#define __hugepd(x) ((hugepd_t) { cpu_to_be64(x) }) - -static inline unsigned long hpd_val(hugepd_t x) -{ - return be64_to_cpu(x.pdbe); -} -#endif - #endif /* _ASM_POWERPC_PGTABLE_BE_TYPES_H */ diff --git a/arch/powerpc/include/asm/pgtable-types.h b/arch/powerpc/include/asm/pgtable-types.h index db965d98e0ae..7b3d4c592a10 100644 --- a/arch/powerpc/include/asm/pgtable-types.h +++ b/arch/powerpc/include/asm/pgtable-types.h @@ -87,13 +87,4 @@ static inline bool pte_xchg(pte_t *ptep, pte_t old, pte_t new) } #endif -#ifdef CONFIG_ARCH_HAS_HUGEPD -typedef struct { unsigned long pd; } hugepd_t; -#define __hugepd(x) ((hugepd_t) { (x) }) -static inline unsigned long hpd_val(hugepd_t x) -{ - return x.pd; -} -#endif - #endif /* _ASM_POWERPC_PGTABLE_TYPES_H */ diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 1fe2843f5b12..6b043180220a 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -28,8 +28,6 @@ bool hugetlb_disabled = false; -#define hugepd_none(hpd) (hpd_val(hpd) == 0) - #define PTE_T_ORDER (__builtin_ffs(sizeof(pte_basic_t)) - \ __builtin_ffs(sizeof(void *))) @@ -42,156 +40,6 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long s return __find_linux_pte(mm->pgd, addr, NULL, NULL); } -#ifdef CONFIG_ARCH_HAS_HUGEPD -static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, - unsigned long address, unsigned int pdshift, - unsigned int pshift, spinlock_t *ptl) -{ - struct kmem_cache *cachep; - pte_t *new; - int i; - int num_hugepd; - - if (pshift >= pdshift) { - cachep = PGT_CACHE(PTE_T_ORDER); - num_hugepd = 1 << (pshift - pdshift); - } else { - cachep = PGT_CACHE(pdshift - pshift); - num_hugepd = 1; - } - - if (!cachep) { - WARN_ONCE(1, "No page table cache created for hugetlb tables"); - return -ENOMEM; - } - - new = kmem_cache_alloc(cachep, pgtable_gfp_flags(mm, GFP_KERNEL)); - - BUG_ON(pshift > HUGEPD_SHIFT_MASK); - BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK); - - if (!new) - return -ENOMEM; - - /* - * Make sure other cpus find the hugepd set only after a - * properly initialized page table is visible to them. - * For more details look for comment in __pte_alloc(). - */ - smp_wmb(); - - spin_lock(ptl); - /* - * We have multiple higher-level entries that point to the same - * actual pte location. Fill in each as we go and backtrack on error. - * We need all of these so the DTLB pgtable walk code can find the - * right higher-level entry without knowing if it's a hugepage or not. - */ - for (i = 0; i < num_hugepd; i++, hpdp++) { - if (unlikely(!hugepd_none(*hpdp))) - break; - hugepd_populate(hpdp, new, pshift); - } - /* If we bailed from the for loop early, an error occurred, clean up */ - if (i < num_hugepd) { - for (i = i - 1 ; i >= 0; i--, hpdp--) - *hpdp = __hugepd(0); - kmem_cache_free(cachep, new); - } else { - kmemleak_ignore(new); - } - spin_unlock(ptl); - return 0; -} - -/* - * At this point we do the placement change only for BOOK3S 64. This would - * possibly work on other subarchs. - */ -pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long addr, unsigned long sz) -{ - pgd_t *pg; - p4d_t *p4; - pud_t *pu; - pmd_t *pm; - hugepd_t *hpdp = NULL; - unsigned pshift = __ffs(sz); - unsigned pdshift = PGDIR_SHIFT; - spinlock_t *ptl; - - addr &= ~(sz-1); - pg = pgd_offset(mm, addr); - p4 = p4d_offset(pg, addr); - -#ifdef CONFIG_PPC_BOOK3S_64 - if (pshift == PGDIR_SHIFT) - /* 16GB huge page */ - return (pte_t *) p4; - else if (pshift > PUD_SHIFT) { - /* - * We need to use hugepd table - */ - ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)p4; - } else { - pdshift = PUD_SHIFT; - pu = pud_alloc(mm, p4, addr); - if (!pu) - return NULL; - if (pshift == PUD_SHIFT) - return (pte_t *)pu; - else if (pshift > PMD_SHIFT) { - ptl = pud_lockptr(mm, pu); - hpdp = (hugepd_t *)pu; - } else { - pdshift = PMD_SHIFT; - pm = pmd_alloc(mm, pu, addr); - if (!pm) - return NULL; - if (pshift == PMD_SHIFT) - /* 16MB hugepage */ - return (pte_t *)pm; - else { - ptl = pmd_lockptr(mm, pm); - hpdp = (hugepd_t *)pm; - } - } - } -#else - if (pshift >= PGDIR_SHIFT) { - ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)p4; - } else { - pdshift = PUD_SHIFT; - pu = pud_alloc(mm, p4, addr); - if (!pu) - return NULL; - if (pshift >= PUD_SHIFT) { - ptl = pud_lockptr(mm, pu); - hpdp = (hugepd_t *)pu; - } else { - pdshift = PMD_SHIFT; - pm = pmd_alloc(mm, pu, addr); - if (!pm) - return NULL; - ptl = pmd_lockptr(mm, pm); - hpdp = (hugepd_t *)pm; - } - } -#endif - if (!hpdp) - return NULL; - - BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp)); - - if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, - pdshift, pshift, ptl)) - return NULL; - - return hugepte_offset(*hpdp, addr, pdshift); -} -#else pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { @@ -230,7 +78,6 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, return pte_alloc_huge(mm, pmd, addr); } -#endif #ifdef CONFIG_PPC_BOOK3S_64 /* @@ -286,266 +133,6 @@ int __init alloc_bootmem_huge_page(struct hstate *h, int nid) return __alloc_bootmem_huge_page(h, nid); } -#ifdef CONFIG_ARCH_HAS_HUGEPD -#ifndef CONFIG_PPC_BOOK3S_64 -#define HUGEPD_FREELIST_SIZE \ - ((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t)) - -struct hugepd_freelist { - struct rcu_head rcu; - unsigned int index; - void *ptes[]; -}; - -static DEFINE_PER_CPU(struct hugepd_freelist *, hugepd_freelist_cur); - -static void hugepd_free_rcu_callback(struct rcu_head *head) -{ - struct hugepd_freelist *batch = - container_of(head, struct hugepd_freelist, rcu); - unsigned int i; - - for (i = 0; i < batch->index; i++) - kmem_cache_free(PGT_CACHE(PTE_T_ORDER), batch->ptes[i]); - - free_page((unsigned long)batch); -} - -static void hugepd_free(struct mmu_gather *tlb, void *hugepte) -{ - struct hugepd_freelist **batchp; - - batchp = &get_cpu_var(hugepd_freelist_cur); - - if (atomic_read(&tlb->mm->mm_users) < 2 || - mm_is_thread_local(tlb->mm)) { - kmem_cache_free(PGT_CACHE(PTE_T_ORDER), hugepte); - put_cpu_var(hugepd_freelist_cur); - return; - } - - if (*batchp == NULL) { - *batchp = (struct hugepd_freelist *)__get_free_page(GFP_ATOMIC); - (*batchp)->index = 0; - } - - (*batchp)->ptes[(*batchp)->index++] = hugepte; - if ((*batchp)->index == HUGEPD_FREELIST_SIZE) { - call_rcu(&(*batchp)->rcu, hugepd_free_rcu_callback); - *batchp = NULL; - } - put_cpu_var(hugepd_freelist_cur); -} -#else -static inline void hugepd_free(struct mmu_gather *tlb, void *hugepte) {} -#endif - -/* Return true when the entry to be freed maps more than the area being freed */ -static bool range_is_outside_limits(unsigned long start, unsigned long end, - unsigned long floor, unsigned long ceiling, - unsigned long mask) -{ - if ((start & mask) < floor) - return true; - if (ceiling) { - ceiling &= mask; - if (!ceiling) - return true; - } - return end - 1 > ceiling - 1; -} - -static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshift, - unsigned long start, unsigned long end, - unsigned long floor, unsigned long ceiling) -{ - pte_t *hugepte = hugepd_page(*hpdp); - int i; - - unsigned long pdmask = ~((1UL << pdshift) - 1); - unsigned int num_hugepd = 1; - unsigned int shift = hugepd_shift(*hpdp); - - /* Note: On fsl the hpdp may be the first of several */ - if (shift > pdshift) - num_hugepd = 1 << (shift - pdshift); - - if (range_is_outside_limits(start, end, floor, ceiling, pdmask)) - return; - - for (i = 0; i < num_hugepd; i++, hpdp++) - *hpdp = __hugepd(0); - - if (shift >= pdshift) - hugepd_free(tlb, hugepte); - else - pgtable_free_tlb(tlb, hugepte, - get_hugepd_cache_index(pdshift - shift)); -} - -static void hugetlb_free_pte_range(struct mmu_gather *tlb, pmd_t *pmd, - unsigned long addr, unsigned long end, - unsigned long floor, unsigned long ceiling) -{ - pgtable_t token = pmd_pgtable(*pmd); - - if (range_is_outside_limits(addr, end, floor, ceiling, PMD_MASK)) - return; - - pmd_clear(pmd); - pte_free_tlb(tlb, token, addr); - mm_dec_nr_ptes(tlb->mm); -} - -static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud, - unsigned long addr, unsigned long end, - unsigned long floor, unsigned long ceiling) -{ - pmd_t *pmd; - unsigned long next; - unsigned long start; - - start = addr; - do { - unsigned long more; - - pmd = pmd_offset(pud, addr); - next = pmd_addr_end(addr, end); - if (!is_hugepd(__hugepd(pmd_val(*pmd)))) { - if (pmd_none_or_clear_bad(pmd)) - continue; - - /* - * if it is not hugepd pointer, we should already find - * it cleared. - */ - WARN_ON(!IS_ENABLED(CONFIG_PPC_8xx)); - - hugetlb_free_pte_range(tlb, pmd, addr, end, floor, ceiling); - - continue; - } - /* - * Increment next by the size of the huge mapping since - * there may be more than one entry at this level for a - * single hugepage, but all of them point to - * the same kmem cache that holds the hugepte. - */ - more = addr + (1UL << hugepd_shift(*(hugepd_t *)pmd)); - if (more > next) - next = more; - - free_hugepd_range(tlb, (hugepd_t *)pmd, PMD_SHIFT, - addr, next, floor, ceiling); - } while (addr = next, addr != end); - - if (range_is_outside_limits(start, end, floor, ceiling, PUD_MASK)) - return; - - pmd = pmd_offset(pud, start & PUD_MASK); - pud_clear(pud); - pmd_free_tlb(tlb, pmd, start & PUD_MASK); - mm_dec_nr_pmds(tlb->mm); -} - -static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, - unsigned long addr, unsigned long end, - unsigned long floor, unsigned long ceiling) -{ - pud_t *pud; - unsigned long next; - unsigned long start; - - start = addr; - do { - pud = pud_offset(p4d, addr); - next = pud_addr_end(addr, end); - if (!is_hugepd(__hugepd(pud_val(*pud)))) { - if (pud_none_or_clear_bad(pud)) - continue; - hugetlb_free_pmd_range(tlb, pud, addr, next, floor, - ceiling); - } else { - unsigned long more; - /* - * Increment next by the size of the huge mapping since - * there may be more than one entry at this level for a - * single hugepage, but all of them point to - * the same kmem cache that holds the hugepte. - */ - more = addr + (1UL << hugepd_shift(*(hugepd_t *)pud)); - if (more > next) - next = more; - - free_hugepd_range(tlb, (hugepd_t *)pud, PUD_SHIFT, - addr, next, floor, ceiling); - } - } while (addr = next, addr != end); - - if (range_is_outside_limits(start, end, floor, ceiling, PGDIR_MASK)) - return; - - pud = pud_offset(p4d, start & PGDIR_MASK); - p4d_clear(p4d); - pud_free_tlb(tlb, pud, start & PGDIR_MASK); - mm_dec_nr_puds(tlb->mm); -} - -/* - * This function frees user-level page tables of a process. - */ -void hugetlb_free_pgd_range(struct mmu_gather *tlb, - unsigned long addr, unsigned long end, - unsigned long floor, unsigned long ceiling) -{ - pgd_t *pgd; - p4d_t *p4d; - unsigned long next; - - /* - * Because there are a number of different possible pagetable - * layouts for hugepage ranges, we limit knowledge of how - * things should be laid out to the allocation path - * (huge_pte_alloc(), above). Everything else works out the - * structure as it goes from information in the hugepd - * pointers. That means that we can't here use the - * optimization used in the normal page free_pgd_range(), of - * checking whether we're actually covering a large enough - * range to have to do anything at the top level of the walk - * instead of at the bottom. - * - * To make sense of this, you should probably go read the big - * block comment at the top of the normal free_pgd_range(), - * too. - */ - - do { - next = pgd_addr_end(addr, end); - pgd = pgd_offset(tlb->mm, addr); - p4d = p4d_offset(pgd, addr); - if (!is_hugepd(__hugepd(pgd_val(*pgd)))) { - if (p4d_none_or_clear_bad(p4d)) - continue; - hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling); - } else { - unsigned long more; - /* - * Increment next by the size of the huge mapping since - * there may be more than one entry at the pgd level - * for a single hugepage, but all of them point to the - * same kmem cache that holds the hugepte. - */ - more = addr + (1UL << hugepd_shift(*(hugepd_t *)pgd)); - if (more > next) - next = more; - - free_hugepd_range(tlb, (hugepd_t *)p4d, PGDIR_SHIFT, - addr, next, floor, ceiling); - } - } while (addr = next, addr != end); -} -#endif - bool __init arch_hugetlb_valid_size(unsigned long size) { int shift = __ffs(size); diff --git a/arch/powerpc/mm/init-common.c b/arch/powerpc/mm/init-common.c index d3a7726ecf51..024e95c62a2d 100644 --- a/arch/powerpc/mm/init-common.c +++ b/arch/powerpc/mm/init-common.c @@ -120,12 +120,8 @@ void pgtable_cache_add(unsigned int shift) /* When batching pgtable pointers for RCU freeing, we store * the index size in the low bits. Table alignment must be * big enough to fit it. - * - * Likewise, hugeapge pagetable pointers contain a (different) - * shift value in the low bits. All tables must be aligned so - * as to leave enough 0 bits in the address to contain it. */ - unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1, - HUGEPD_SHIFT_MASK + 1); + */ + unsigned long minalign = MAX_PGTABLE_INDEX_SIZE + 1; struct kmem_cache *new = NULL; /* It would be nice if this was a BUILD_BUG_ON(), but at the diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 218792cb2c47..ab0656115424 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -409,11 +409,10 @@ unsigned long vmalloc_to_phys(void *va) EXPORT_SYMBOL_GPL(vmalloc_to_phys); /* - * We have 4 cases for pgds and pmds: + * We have 3 cases for pgds and pmds: * (1) invalid (all zeroes) * (2) pointer to next table, as normal; bottom 6 bits == 0 * (3) leaf pte for huge page _PAGE_PTE set - * (4) hugepd pointer, _PAGE_PTE = 0 and bits [2..6] indicate size of table * * So long as we atomically load page table pointers we are safe against teardown, * we can follow the address down to the page and take a ref on it. @@ -430,7 +429,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, #endif pmd_t pmd, *pmdp; pte_t *ret_pte; - hugepd_t *hpdp = NULL; unsigned pdshift; if (hpage_shift) @@ -463,11 +461,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } - if (is_hugepd(__hugepd(p4d_val(p4d)))) { - hpdp = (hugepd_t *)&p4d; - goto out_huge; - } - /* * Even if we end up with an unmap, the pgtable will not * be freed, because we do an rcu free and here we are @@ -485,11 +478,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } - if (is_hugepd(__hugepd(pud_val(pud)))) { - hpdp = (hugepd_t *)&pud; - goto out_huge; - } - pmdp = pmd_offset(&pud, ea); #else pmdp = pmd_offset(pud_offset(p4d_offset(pgdp, ea), ea), ea); @@ -527,21 +515,8 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } - if (is_hugepd(__hugepd(pmd_val(pmd)))) { - hpdp = (hugepd_t *)&pmd; - goto out_huge; - } - return pte_offset_kernel(&pmd, ea); -out_huge: - if (!hpdp) - return NULL; - -#ifdef CONFIG_ARCH_HAS_HUGEPD - ret_pte = hugepte_offset(*hpdp, ea, pdshift); - pdshift = hugepd_shift(*hpdp); -#endif out: if (hpage_shift) *hpage_shift = pdshift; From patchwork Tue Jul 2 13:51:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13719699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F465C30658 for ; Tue, 2 Jul 2024 13:53:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A98AB6B00E4; Tue, 2 Jul 2024 09:53:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A20606B00E5; Tue, 2 Jul 2024 09:53:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 899D76B00E6; Tue, 2 Jul 2024 09:53:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 64B2E6B00E4 for ; Tue, 2 Jul 2024 09:53:40 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EBF611201FC for ; Tue, 2 Jul 2024 13:53:39 +0000 (UTC) X-FDA: 82294955358.03.F46EEF6 Received: from pegase1.c-s.fr (pegase1.c-s.fr [93.17.236.30]) by imf12.hostedemail.com (Postfix) with ESMTP id 9E56A40013 for ; Tue, 2 Jul 2024 13:53:37 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719928406; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GYq+BCypvXu/WL22xAa5K1h0Md3bLPWN39tMNsYTec0=; b=zt97APb4fTpPfiE7v7jOsox0m6TMlr1am0PvycRsSfxM3CvcebFpDin+jFZgDGYFws2YEG BXYc+VXabprJvlRcWlwQC6RJ60gorF6+N8QiIsuknilucdLv3Z2OYteEbvQ3at3UYzfXQ5 tvpbUcS00gZyFjhv2CHvDVah2iivBrg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.236.30 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu; dmarc=pass (policy=quarantine) header.from=csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719928406; a=rsa-sha256; cv=none; b=4A+vft3RF1RcI25iG1N8chQ+GDpuvKeyEWu0pQ5YjOcOhc1+8ciB6O6Z3irWxxsjFddBjQ 7b0B3a8xRIzwIzjPHljCFeK9vcOGO7iCqFI3ctjgBz5dzjC9FKZEVYZH83UDkxsaBvZWMX W9VaYv/R/yaVfjuQR61O4PU1DkDvkYA= Received: from localhost (mailhub3.si.c-s.fr [192.168.12.233]) by localhost (Postfix) with ESMTP id 4WD48R4XGNz9tZl; Tue, 2 Jul 2024 15:51:51 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ffimB9gxzHh3; Tue, 2 Jul 2024 15:51:51 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 4WD489500lz9v83; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id A69F18B764; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id eDwV0SP7s93s; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.233.12]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 16CC98B774; Tue, 2 Jul 2024 15:51:37 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Jason Gunthorpe , Peter Xu , Oscar Salvador , Michael Ellerman , Nicholas Piggin Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v7 23/23] mm: Remove CONFIG_ARCH_HAS_HUGEPD Date: Tue, 2 Jul 2024 15:51:35 +0200 Message-ID: <4b10c54c794780b955f3ad6c657d0199dd792146.1719928057.git.christophe.leroy@csgroup.eu> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1719928276; l=13887; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=cQLj8OVBuNgw3rK4whGZI+ACAbcY/Wjqr0CuEotzvl0=; b=we+KcHNHSm06cP801qaPqlZQIypO2XBn95c2d7vV3n1zLPz+lP7Jc+Lc+jJFmzDLBx+7l3uR0 OMJjr0bG2ewAvihtt8njI+sCwCk1/JCZ0mHBsGHP8WyggvabMmP7xuD X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 9E56A40013 X-Stat-Signature: 7e684di337yme9yrw6k7ea6zizbg4hnh X-Rspam-User: X-HE-Tag: 1719928417-376637 X-HE-Meta: U2FsdGVkX195mPD1yEVSfos45NPs4fLsZUOEpDs0V3pn/JvA0ltOsrba+Q/fhXlo3UQi/pWVg6hFjgJaO9kD8LFs1akjirLGHBweqi7Ks5idEJ7+nNoMFHLwyS2odmfRp+ChgUCQk6fYxK0iCcKFVJcKn7U92fZhYxe3/m0z4fScdxs12Pion47uUvnYWj6tnJQgIipukI9Whti69zMzJs549KivD2U2TRnJL5MHKqbCrfLVRnm2aa2FdhrA2ER1EE7XpHxjyHzDoOacvmKpCNARxVtIfIdhPdcGda6Dc2ZGUeTEmDKUJjyVvvOUhLLgldhthAHE9xW0o9Tmf3SlEdRHEJdWg9jcWEOlSYQ8M4kDrbIYEVy0Rl5b/R9hlUcJEHV4pGfP8xFqomQ6BAiDkF+I+0oJE6MMxQ3U7tSsUxxbQr4Kh7cV7+jXtNDPYgQVLiwYaeZiRSgWOZ/RwDaYKw6OE1LyXubUcZGcnIo92+WgUkbQmItlE46pVP2IpFXJ0fMg8S75Qclq1y4C7zrIlkEUCJwceGvDIKvkDqMdHvpLv5HH7CWGTJQsfjnfx7qrvfM+gukaidr5mkAuTGE8g7/fpb2331o8Z7sGKd+xwAEKVwAjXqI7HsPqZxTsDwZO4WeCokO8K0dnJAImowhzj3clMrFxm2qtpJb7pB+sV3Vdg4YRTMfZeed5ImmuTvNWJwQiUxb7yUkJ2m4is/ObEHkFlLqt6E8vRjsTmlkSxzOU4zRTCqIzraC5Ud5tNhWrLb1XPDjPyfK/j6r1alU8jzWDjkl/zgkjlNhJpye197licQ5/Z/h/VK4MrSmXk4B+YryRCcxO7jidhZ4vLXff+XO3yxhxwfXLxKDrpfnnJLgTgl5tGmlUtOPFrVp6sTGIgzMHolf1uHYlixJ6Whu+97fpqmRM0dEUPHVnGF6uBB2A8re+X+l9b8Rf3G4T+Yg8J0CB0QAlnJh0bgW92d7 FsbtZnnK L65PAgXxaa8i52pCGytI1WoS+Lg4U98Auv8NWK6sE7Dhk8YkimM3lGvSn0+AP6iPUX+HyZGc4hhXEoGAW7ImlYp8fyj+aUIkusfJ9heSq6wtuqXBlcOr7OZ/trWhkqys6Rd9zgTE61pEj/klTYL5jdjPn+gfNmjPh0gnVCX4g24LKATo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: powerpc was the only user of CONFIG_ARCH_HAS_HUGEPD and doesn't use it anymore, so remove all related code. Signed-off-by: Christophe Leroy Acked-by: Oscar Salvador --- v4: Rebased on v6.10-rc1 v7: Rebased on mm-unstable (9bb8753acdd8) --- include/linux/hugetlb.h | 6 -- mm/Kconfig | 10 --- mm/gup.c | 194 ++-------------------------------------- mm/pagewalk.c | 57 +----------- 4 files changed, 9 insertions(+), 258 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index a951c0d06061..ecd0ceeea206 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -20,12 +20,6 @@ struct user_struct; struct mmu_gather; struct node; -#ifndef CONFIG_ARCH_HAS_HUGEPD -typedef struct { unsigned long pd; } hugepd_t; -#define is_hugepd(hugepd) (0) -#define __hugepd(x) ((hugepd_t) { (x) }) -#endif - void free_huge_folio(struct folio *folio); #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/Kconfig b/mm/Kconfig index 991fa9cf6137..60796402850e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1126,16 +1126,6 @@ config DMAPOOL_TEST config ARCH_HAS_PTE_SPECIAL bool -# -# Some architectures require a special hugepage directory format that is -# required to support multiple hugepage sizes. For example a4fe3ce76 -# "powerpc/mm: Allow more flexible layouts for hugepage pagetables" -# introduced it on powerpc. This allows for a more flexible hugepage -# pagetable layouts. -# -config ARCH_HAS_HUGEPD - bool - config MAPPING_DIRTY_HELPERS bool diff --git a/mm/gup.c b/mm/gup.c index aaf6ef4e28ee..87581ce6bc45 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -462,7 +462,7 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) #ifdef CONFIG_MMU -#if defined(CONFIG_ARCH_HAS_HUGEPD) || defined(CONFIG_HAVE_GUP_FAST) +#ifdef CONFIG_HAVE_GUP_FAST static int record_subpages(struct page *page, unsigned long sz, unsigned long addr, unsigned long end, struct page **pages) @@ -476,154 +476,7 @@ static int record_subpages(struct page *page, unsigned long sz, return nr; } -#endif /* CONFIG_ARCH_HAS_HUGEPD || CONFIG_HAVE_GUP_FAST */ - -#ifdef CONFIG_ARCH_HAS_HUGEPD -static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, - unsigned long sz) -{ - unsigned long __boundary = (addr + sz) & ~(sz-1); - return (__boundary - 1 < end - 1) ? __boundary : end; -} - -/* - * Returns 1 if succeeded, 0 if failed, -EMLINK if unshare needed. - * - * NOTE: for the same entry, gup-fast and gup-slow can return different - * results (0 v.s. -EMLINK) depending on whether vma is available. This is - * the expected behavior, where we simply want gup-fast to fallback to - * gup-slow to take the vma reference first. - */ -static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long sz, - unsigned long addr, unsigned long end, unsigned int flags, - struct page **pages, int *nr, bool fast) -{ - unsigned long pte_end; - struct page *page; - struct folio *folio; - pte_t pte; - int refs; - - pte_end = (addr + sz) & ~(sz-1); - if (pte_end < end) - end = pte_end; - - pte = huge_ptep_get(vma->vm_mm, addr, ptep); - - if (!pte_access_permitted(pte, flags & FOLL_WRITE)) - return 0; - - /* hugepages are never "special" */ - VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - - page = pte_page(pte); - refs = record_subpages(page, sz, addr, end, pages + *nr); - - if (fast) { - folio = try_grab_folio_fast(page, refs, flags); - if (!folio) - return 0; - } else { - folio = page_folio(page); - if (try_grab_folio(folio, refs, flags)) - return 0; - } - - if (unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) { - gup_put_folio(folio, refs, flags); - return 0; - } - - if (!pte_write(pte) && gup_must_unshare(vma, flags, &folio->page)) { - gup_put_folio(folio, refs, flags); - return -EMLINK; - } - - *nr += refs; - folio_set_referenced(folio); - return 1; -} - -/* - * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file - * systems on Power, which does not have issue with folio writeback against - * GUP updates. When hugepd will be extended to support non-hugetlbfs or - * even anonymous memory, we need to do extra check as what we do with most - * of the other folios. See writable_file_mapping_allowed() and - * gup_fast_folio_allowed() for more information. - */ -static int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, - unsigned long addr, unsigned int pdshift, - unsigned long end, unsigned int flags, - struct page **pages, int *nr, bool fast) -{ - pte_t *ptep; - unsigned long sz = 1UL << hugepd_shift(hugepd); - unsigned long next; - int ret; - - ptep = hugepte_offset(hugepd, addr, pdshift); - do { - next = hugepte_addr_end(addr, end, sz); - ret = gup_hugepte(vma, ptep, sz, addr, end, flags, pages, nr, - fast); - if (ret != 1) - return ret; - } while (ptep++, addr = next, addr != end); - - return 1; -} - -static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, - unsigned long addr, unsigned int pdshift, - unsigned int flags, - struct follow_page_context *ctx) -{ - struct page *page; - struct hstate *h; - spinlock_t *ptl; - int nr = 0, ret; - pte_t *ptep; - - /* Only hugetlb supports hugepd */ - if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) - return ERR_PTR(-EFAULT); - - h = hstate_vma(vma); - ptep = hugepte_offset(hugepd, addr, pdshift); - ptl = huge_pte_lock(h, vma->vm_mm, ptep); - ret = gup_hugepd(vma, hugepd, addr, pdshift, addr + PAGE_SIZE, - flags, &page, &nr, false); - spin_unlock(ptl); - - if (ret == 1) { - /* GUP succeeded */ - WARN_ON_ONCE(nr != 1); - ctx->page_mask = (1U << huge_page_order(h)) - 1; - return page; - } - - /* ret can be either 0 (translates to NULL) or negative */ - return ERR_PTR(ret); -} -#else /* CONFIG_ARCH_HAS_HUGEPD */ -static inline int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, - unsigned long addr, unsigned int pdshift, - unsigned long end, unsigned int flags, - struct page **pages, int *nr, bool fast) -{ - return 0; -} - -static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, - unsigned long addr, unsigned int pdshift, - unsigned int flags, - struct follow_page_context *ctx) -{ - return NULL; -} -#endif /* CONFIG_ARCH_HAS_HUGEPD */ - +#endif /* CONFIG_HAVE_GUP_FAST */ static struct page *no_page_table(struct vm_area_struct *vma, unsigned int flags, unsigned long address) @@ -993,9 +846,6 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) return no_page_table(vma, flags, address); - if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) - return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), - address, PMD_SHIFT, flags, ctx); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); @@ -1046,9 +896,6 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = READ_ONCE(*pudp); if (!pud_present(pud)) return no_page_table(vma, flags, address); - if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) - return follow_hugepd(vma, __hugepd(pud_val(pud)), - address, PUD_SHIFT, flags, ctx); if (pud_leaf(pud)) { ptl = pud_lock(mm, pudp); page = follow_huge_pud(vma, address, pudp, flags, ctx); @@ -1074,10 +921,6 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4d = READ_ONCE(*p4dp); BUILD_BUG_ON(p4d_leaf(p4d)); - if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) - return follow_hugepd(vma, __hugepd(p4d_val(p4d)), - address, P4D_SHIFT, flags, ctx); - if (!p4d_present(p4d) || p4d_bad(p4d)) return no_page_table(vma, flags, address); @@ -1121,10 +964,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, ctx->page_mask = 0; pgd = pgd_offset(mm, address); - if (unlikely(is_hugepd(__hugepd(pgd_val(*pgd))))) - page = follow_hugepd(vma, __hugepd(pgd_val(*pgd)), - address, PGDIR_SHIFT, flags, ctx); - else if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) + if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) page = no_page_table(vma, flags, address); else page = follow_p4d_mask(vma, address, pgd, flags, ctx); @@ -3362,15 +3202,6 @@ static int gup_fast_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, pages, nr)) return 0; - } else if (unlikely(is_hugepd(__hugepd(pmd_val(pmd))))) { - /* - * architecture have different format for hugetlbfs - * pmd format and THP pmd format - */ - if (gup_hugepd(NULL, __hugepd(pmd_val(pmd)), addr, - PMD_SHIFT, next, flags, pages, nr, - true) != 1) - return 0; } else if (!gup_fast_pte_range(pmd, pmdp, addr, next, flags, pages, nr)) return 0; @@ -3397,11 +3228,6 @@ static int gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, if (!gup_fast_pud_leaf(pud, pudp, addr, next, flags, pages, nr)) return 0; - } else if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) { - if (gup_hugepd(NULL, __hugepd(pud_val(pud)), addr, - PUD_SHIFT, next, flags, pages, nr, - true) != 1) - return 0; } else if (!gup_fast_pmd_range(pudp, pud, addr, next, flags, pages, nr)) return 0; @@ -3425,13 +3251,8 @@ static int gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, if (!p4d_present(p4d)) return 0; BUILD_BUG_ON(p4d_leaf(p4d)); - if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) { - if (gup_hugepd(NULL, __hugepd(p4d_val(p4d)), addr, - P4D_SHIFT, next, flags, pages, nr, - true) != 1) - return 0; - } else if (!gup_fast_pud_range(p4dp, p4d, addr, next, flags, - pages, nr)) + if (!gup_fast_pud_range(p4dp, p4d, addr, next, flags, + pages, nr)) return 0; } while (p4dp++, addr = next, addr != end); @@ -3455,11 +3276,6 @@ static void gup_fast_pgd_range(unsigned long addr, unsigned long end, if (!gup_fast_pgd_leaf(pgd, pgdp, addr, next, flags, pages, nr)) return; - } else if (unlikely(is_hugepd(__hugepd(pgd_val(pgd))))) { - if (gup_hugepd(NULL, __hugepd(pgd_val(pgd)), addr, - PGDIR_SHIFT, next, flags, pages, nr, - true) != 1) - return; } else if (!gup_fast_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) return; diff --git a/mm/pagewalk.c b/mm/pagewalk.c index f46c80b18ce4..ae2f08ce991b 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -73,45 +73,6 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, return err; } -#ifdef CONFIG_ARCH_HAS_HUGEPD -static int walk_hugepd_range(hugepd_t *phpd, unsigned long addr, - unsigned long end, struct mm_walk *walk, int pdshift) -{ - int err = 0; - const struct mm_walk_ops *ops = walk->ops; - int shift = hugepd_shift(*phpd); - int page_size = 1 << shift; - - if (!ops->pte_entry) - return 0; - - if (addr & (page_size - 1)) - return 0; - - for (;;) { - pte_t *pte; - - spin_lock(&walk->mm->page_table_lock); - pte = hugepte_offset(*phpd, addr, pdshift); - err = ops->pte_entry(pte, addr, addr + page_size, walk); - spin_unlock(&walk->mm->page_table_lock); - - if (err) - break; - if (addr >= end - page_size) - break; - addr += page_size; - } - return err; -} -#else -static int walk_hugepd_range(hugepd_t *phpd, unsigned long addr, - unsigned long end, struct mm_walk *walk, int pdshift) -{ - return 0; -} -#endif - static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -159,10 +120,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, if (walk->vma) split_huge_pmd(walk->vma, pmd, addr); - if (is_hugepd(__hugepd(pmd_val(*pmd)))) - err = walk_hugepd_range((hugepd_t *)pmd, addr, next, walk, PMD_SHIFT); - else - err = walk_pte_range(pmd, addr, next, walk); + err = walk_pte_range(pmd, addr, next, walk); if (err) break; @@ -215,10 +173,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, if (pud_none(*pud)) goto again; - if (is_hugepd(__hugepd(pud_val(*pud)))) - err = walk_hugepd_range((hugepd_t *)pud, addr, next, walk, PUD_SHIFT); - else - err = walk_pmd_range(pud, addr, next, walk); + err = walk_pmd_range(pud, addr, next, walk); if (err) break; } while (pud++, addr = next, addr != end); @@ -250,9 +205,7 @@ static int walk_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, if (err) break; } - if (is_hugepd(__hugepd(p4d_val(*p4d)))) - err = walk_hugepd_range((hugepd_t *)p4d, addr, next, walk, P4D_SHIFT); - else if (ops->pud_entry || ops->pmd_entry || ops->pte_entry) + if (ops->pud_entry || ops->pmd_entry || ops->pte_entry) err = walk_pud_range(p4d, addr, next, walk); if (err) break; @@ -287,9 +240,7 @@ static int walk_pgd_range(unsigned long addr, unsigned long end, if (err) break; } - if (is_hugepd(__hugepd(pgd_val(*pgd)))) - err = walk_hugepd_range((hugepd_t *)pgd, addr, next, walk, PGDIR_SHIFT); - else if (ops->p4d_entry || ops->pud_entry || ops->pmd_entry || ops->pte_entry) + if (ops->p4d_entry || ops->pud_entry || ops->pmd_entry || ops->pte_entry) err = walk_p4d_range(pgd, addr, next, walk); if (err) break;