From patchwork Mon Feb 25 05:03:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10828227 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9DEBD1399 for ; Mon, 25 Feb 2019 05:04:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8970D28528 for ; Mon, 25 Feb 2019 05:04:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7BE4028610; Mon, 25 Feb 2019 05:04:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2035B28528 for ; Mon, 25 Feb 2019 05:04:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=6Ad4jmJZAX0i9FEysHYIB8WX8ieOkDK/jmc1uU0VWa4=; b=m/siBvJ+UBIliXJusVeh9K06af imo1J9PvxuJMRkcbuofNDIpjVfUs8El3L7Y3Qt6PcDHycKr4crE0MdrUJUcALYm6puy9do/qkpJWT C8MALUPkezP2ay915+Ng8hrIS2AbhifbAzrgauyp5G4h+dLnZQO2mm/i4GW4Lir3rnehEKmCTzZJa Wm3+lBZyFUqAxIQExC+zC8QuzYDxtY+vNEETQJAE+MyqjGo1+6N08wEOWoFQNfCQYD44YFlx/ndSA 6/6APZfhsJywSMOeGy4kN+31HabA1HVuVTKqHRUCN1ss+i+YEUnsqj1fKT+020LsTh+EkNj6nZzwO 0PF0Baqw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8R3-0006AA-Nh; Mon, 25 Feb 2019 05:04:13 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8R0-000698-4T for linux-arm-kernel@lists.infradead.org; Mon, 25 Feb 2019 05:04:11 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0F5C31596; Sun, 24 Feb 2019 21:04:06 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.42.159]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 307A03F5C1; Sun, 24 Feb 2019 21:04:02 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH V2 1/6] KVM: ARM: Remove pgtable standard functions from stage-2 page tables Date: Mon, 25 Feb 2019 10:33:54 +0530 Message-Id: <1551071039-20192-2-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> References: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190224_210410_174889_593D119B X-CRM114-Status: GOOD ( 11.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, yuzhao@google.com, Steve.Capper@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, suzuki.poulose@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP ARM64 standard pgtable functions are going to use pgtable_page_[ctor|dtor] constructs. Certain stage-2 page table allocations are multi order which cannot be allocated through a generic pgtable function as it does not exist right now. This prevents all pgtable allocations including multi order ones in stage-2 from moving into new ARM64 (pgtable_page_[ctor|dtor]) pgtable functions. Hence remove all generic pgtable allocation function dependency from stage-2 page tables till there is one multi-order allocation function available. Signed-off-by: Anshuman Khandual --- arch/arm/include/asm/stage2_pgtable.h | 4 ++-- arch/arm64/include/asm/stage2_pgtable.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm/include/asm/stage2_pgtable.h b/arch/arm/include/asm/stage2_pgtable.h index de2089501b8b..417a3be00718 100644 --- a/arch/arm/include/asm/stage2_pgtable.h +++ b/arch/arm/include/asm/stage2_pgtable.h @@ -32,14 +32,14 @@ #define stage2_pgd_present(kvm, pgd) pgd_present(pgd) #define stage2_pgd_populate(kvm, pgd, pud) pgd_populate(NULL, pgd, pud) #define stage2_pud_offset(kvm, pgd, address) pud_offset(pgd, address) -#define stage2_pud_free(kvm, pud) pud_free(NULL, pud) +#define stage2_pud_free(kvm, pud) free_page((unsigned long)pud) #define stage2_pud_none(kvm, pud) pud_none(pud) #define stage2_pud_clear(kvm, pud) pud_clear(pud) #define stage2_pud_present(kvm, pud) pud_present(pud) #define stage2_pud_populate(kvm, pud, pmd) pud_populate(NULL, pud, pmd) #define stage2_pmd_offset(kvm, pud, address) pmd_offset(pud, address) -#define stage2_pmd_free(kvm, pmd) pmd_free(NULL, pmd) +#define stage2_pmd_free(kvm, pmd) free_page((unsigned long)pmd) #define stage2_pud_huge(kvm, pud) pud_huge(pud) diff --git a/arch/arm64/include/asm/stage2_pgtable.h b/arch/arm64/include/asm/stage2_pgtable.h index 5412fa40825e..915809e4ac32 100644 --- a/arch/arm64/include/asm/stage2_pgtable.h +++ b/arch/arm64/include/asm/stage2_pgtable.h @@ -119,7 +119,7 @@ static inline pud_t *stage2_pud_offset(struct kvm *kvm, static inline void stage2_pud_free(struct kvm *kvm, pud_t *pud) { if (kvm_stage2_has_pud(kvm)) - pud_free(NULL, pud); + free_page((unsigned long)pud); } static inline bool stage2_pud_table_empty(struct kvm *kvm, pud_t *pudp) @@ -192,7 +192,7 @@ static inline pmd_t *stage2_pmd_offset(struct kvm *kvm, static inline void stage2_pmd_free(struct kvm *kvm, pmd_t *pmd) { if (kvm_stage2_has_pmd(kvm)) - pmd_free(NULL, pmd); + free_page((unsigned long)pmd); } static inline bool stage2_pud_huge(struct kvm *kvm, pud_t pud) From patchwork Mon Feb 25 05:03:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10828229 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6B6111399 for ; Mon, 25 Feb 2019 05:04:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5829A2AA8F for ; Mon, 25 Feb 2019 05:04:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4B17E2AA94; Mon, 25 Feb 2019 05:04:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F19AC2AA8F for ; Mon, 25 Feb 2019 05:04:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NTenXRt7wa/8I0Ah9I980mzxvytg39B+meIjO7iIAqg=; b=afW4dmYCFcUVisnHX+jAqoSS3c ZnAihv2BxlPJvWs1kAX6RTwBbwllsdC2fyt5/ZD9JEYQ+E1Kz2VOMnn2ec17tSh91AQGIl5RNSQJ5 ZSpU0JWhhBYPwSH/wkFNmPMrxGseXJqAj6ijL1bFBnSLHkXcNhaKTNGBZl5WF+utgN8UxPPAlPNjT l4U0FaCTsj2pPCtDWNcIV/e2pmJzLT4NMP6hWqRfLKo/ltq/CLbWSvtNhX61zLvp7kKRRaqizV5XI KQTD7VSoYsJbdOgKOp1vDOuXD+nM6X9P+TRkMlitxj4ohQmgTvCbh9KaPZ+t1fTKmTP7x3HVudiU7 mqT5gVTQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8RZ-0006gS-6P; Mon, 25 Feb 2019 05:04:45 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8R1-00069k-OO for linux-arm-kernel@lists.infradead.org; Mon, 25 Feb 2019 05:04:13 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6391815AD; Sun, 24 Feb 2019 21:04:09 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.42.159]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 843233F5C1; Sun, 24 Feb 2019 21:04:06 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH V2 2/6] arm64/mm: Make pgd_pgtable_alloc() call pte_alloc_one() always Date: Mon, 25 Feb 2019 10:33:55 +0530 Message-Id: <1551071039-20192-3-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> References: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190224_210411_791857_062D906E X-CRM114-Status: GOOD ( 14.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, yuzhao@google.com, Steve.Capper@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, suzuki.poulose@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP pgd_pgtable_alloc() provides page allocation function while creating all levels of page table (PGD, PUD, CONT PMD etc) for various kernel mappings. It calls __get_free_page() and initializes page with pagetable_page_ctor(). pte_alloc_one() already provides a standard interface for allocating a page table page and initializes it with pagetable_page_ctor(). This removes the redundancy and instead make it call pte_alloc_one() directly. Signed-off-by: Anshuman Khandual --- arch/arm64/mm/mmu.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b6f5aa52ac67..2dbd72319152 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -372,13 +372,22 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, static phys_addr_t pgd_pgtable_alloc(void) { - void *ptr = (void *)__get_free_page(PGALLOC_GFP); - if (!ptr || !pgtable_page_ctor(virt_to_page(ptr))) - BUG(); + + pgtable_t ptr; + + /* + * pgd_pgtable_alloc() is called while creating kernel mappings + * through __create_pgd_mapping() might not install it through + * swapper_pg_dir (&init_mm). Even then init_mm is passed here + * just to indicate that the allocation is kernel specific not + * for the user space page tables. + */ + ptr = pte_alloc_one(&init_mm); + BUG_ON(!ptr); /* Ensure the zeroed page is visible to the page table walker */ dsb(ishst); - return __pa(ptr); + return page_to_phys(ptr); } /* From patchwork Mon Feb 25 05:03:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10828231 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF9FF1823 for ; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB0212AA8F for ; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C91702AA94; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4DC182AA8F for ; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=uunBTmomxELR21JJkTonFp052bKHGJiOLpMiaQzeQdk=; b=PMM05F2ezFldqUaok2VpFFhCww CdnI7gmHIDQ+5Z7D76Raq39Nv7L9Z8VvedMWL1IG0BMFcSH9THouMpXwaAhLipAqjfVOZPjSz+pi9 wcbmvhTh3lRSsOR2Rmx+3xtIkONEJnLtn2OMWRxpoj1W7wRgMng2BH1o7McrzsR0famwlYPb+RtWN LelDbKhMn0tRqoElWW2s8txYzVSE0E7hBBebkNIlMzbB0QIL95WqaI2G1ZLA1tJsVGMvlQbUGoYcU HSfvjgZjlWTUByoensxRER0sXZhgeCyXySd8ZV5D/CxYEj6xZ5faeM8j7Hn26FWWOp1baMHWYXU8d PD43SKFA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8Rg-0006oz-DW; Mon, 25 Feb 2019 05:04:52 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8R3-0006A9-Eu for linux-arm-kernel@lists.infradead.org; Mon, 25 Feb 2019 05:04:17 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BADEE80D; Sun, 24 Feb 2019 21:04:12 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.42.159]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D79D33F5C1; Sun, 24 Feb 2019 21:04:09 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH V2 3/6] arm64/mm: Make all page table pages cycles through standard constructs Date: Mon, 25 Feb 2019 10:33:56 +0530 Message-Id: <1551071039-20192-4-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> References: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190224_210413_613170_F1DB2328 X-CRM114-Status: GOOD ( 16.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, yuzhao@google.com, Steve.Capper@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, suzuki.poulose@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Currently there are inconsistencies during page table page's allocation, initialization and destruction. Not all those pages go through standard constructors pgtable_page_ctor/pgtable_page_dtor which is very much essential for proper initialization and accounting. Not going through proper pgtable_page_ctor/pgtable_page_dtor cycle makes them vulnerable for bad page state errors during page table tear down process. This makes all levels of page table pages either for the user space or for the kernel mappings go through these standard constructs. While here pte_free() has been modified to accommodate pages which might not have been allocated through buddy system but instead came from memblock [1] directly during early boot process. Those pages must not go through the pgtable_page_dtor() and instead are freed with free_reserved_page(). PGD based page table page which is allocated from 'pgd_cache' kmem cache in certain situations (PGD_SIZE != PAGE_SIZE ) is excluded from being cycled through pgtable_page_[ctor/dtor]. This is not a problem because this slab never gets freed back to buddy allocator. [1] Page table page allocation from memblock (a) early_pgtable_alloc (passed with __create_pgd_mapping) (b) vmemmap_pgd|pud_populate (vmemmap_populate) Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgalloc.h | 42 ++++++++++++++++++++++++++------ arch/arm64/mm/pgd.c | 4 +-- 2 files changed, 36 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c73bf0..076764f66fb0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -29,17 +29,21 @@ #define PGALLOC_GFP (GFP_KERNEL | __GFP_ZERO) #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm); +static inline unsigned long pte_alloc_one_virt(struct mm_struct *mm); +static inline void pte_free(struct mm_struct *mm, pgtable_t pte); + #if CONFIG_PGTABLE_LEVELS > 2 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + return (pmd_t *)pte_alloc_one_virt(mm); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); - free_page((unsigned long)pmdp); + pte_free(mm, virt_to_page(pmdp)); } static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) @@ -62,13 +66,13 @@ static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pud_t *)__get_free_page(PGALLOC_GFP); + return (pud_t *)pte_alloc_one_virt(mm); } static inline void pud_free(struct mm_struct *mm, pud_t *pudp) { BUG_ON((unsigned long)pudp & (PAGE_SIZE-1)); - free_page((unsigned long)pudp); + pte_free(mm, virt_to_page(pudp)); } static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pudp, pgdval_t prot) @@ -93,7 +97,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); static inline pte_t * pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(PGALLOC_GFP); + return (pte_t *)pte_alloc_one_virt(mm); } static inline pgtable_t @@ -111,19 +115,41 @@ pte_alloc_one(struct mm_struct *mm) return pte; } +static inline unsigned long +pte_alloc_one_virt(struct mm_struct *mm) +{ + pgtable_t ptr; + + ptr = pte_alloc_one(mm); + if (!ptr) + return 0; + + return (unsigned long) page_to_virt(ptr); +} + /* * Free a PTE table. */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *ptep) { if (ptep) - free_page((unsigned long)ptep); + pte_free(mm, virt_to_page(ptep)); } static inline void pte_free(struct mm_struct *mm, pgtable_t pte) { - pgtable_page_dtor(pte); - __free_page(pte); + /* + * Some page table pages might have come from memblock either + * with vmemmap_alloc_block (during vmemmap_populate) or with + * early_pgtable_alloc (during __create_pgd_mapping). These + * pages should be freed with free_reserved_page() instead. + */ + if (PageReserved(pte)) { + free_reserved_page(pte); + } else { + pgtable_page_dtor(pte); + __free_page(pte); + } } static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t ptep, diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 289f9113a27a..ea588c8825e2 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -31,7 +31,7 @@ static struct kmem_cache *pgd_cache __ro_after_init; pgd_t *pgd_alloc(struct mm_struct *mm) { if (PGD_SIZE == PAGE_SIZE) - return (pgd_t *)__get_free_page(PGALLOC_GFP); + return (pgd_t *)pte_alloc_one_virt(mm); else return kmem_cache_alloc(pgd_cache, PGALLOC_GFP); } @@ -39,7 +39,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { if (PGD_SIZE == PAGE_SIZE) - free_page((unsigned long)pgd); + pte_free(mm, virt_to_page(pgd)); else kmem_cache_free(pgd_cache, pgd); } From patchwork Mon Feb 25 05:03:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10828237 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 486E817E9 for ; Mon, 25 Feb 2019 05:05:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36B4B2AA27 for ; Mon, 25 Feb 2019 05:05:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 298A42AA94; Mon, 25 Feb 2019 05:05:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9F2932AA8F for ; Mon, 25 Feb 2019 05:05:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rgsv4IAkvcAUQOqe/Im1xfUsteTrNdX7I1U3dv00FJ8=; b=tElkJiVFIB0f+rkdmSjlGSgmUR OM2LhZLU64QDBniEtMHx9F2DPRFO70NuhhTsm1ECn+scOIs83BSQTxRdFRFzOZolsx1hNW1d8ai+E OkvMAOOIyy45Z+kqFjD0oNrcBSbgwY8gMmcu1O46M1ScBo28SBkJ0vBOpb1DQwEh9XMfftr4CBt6L cnRLomqrwyTlP2vje33Tsx1LSTT5I+tP8YmZALhjaeGVr+8HL/eJcUu1SV+vR+MCOEpU89s6L99Ct E2RwXGxfYJoJOvjoBpWIcPLW3rn4gVbz2BZXlh03zuLchTSgLeGQfBiAXITXjLvc+COREFe7/1hkY huUhSmAQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8SZ-0000Ol-4d; Mon, 25 Feb 2019 05:05:47 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8RC-0006LT-KC for linux-arm-kernel@lists.infradead.org; Mon, 25 Feb 2019 05:04:31 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 26CE21596; Sun, 24 Feb 2019 21:04:16 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.42.159]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3B94C3F5C1; Sun, 24 Feb 2019 21:04:12 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH V2 4/6] arm64/mm: Call pgtable_page_dtor() for both PMD and PUD page table pages Date: Mon, 25 Feb 2019 10:33:57 +0530 Message-Id: <1551071039-20192-5-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> References: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190224_210423_252702_09FF02A3 X-CRM114-Status: UNSURE ( 9.03 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, yuzhao@google.com, Steve.Capper@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, suzuki.poulose@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP TLB flush during PMD and PUD level page table page tear down currently does not call pgtable_page_dtor() like that for PTE page table pages. Now that all the page table pages which are allocated from the buddy allocator must have been initialized through pgtable_page_ctor() they must be freed back only after being called into pgtable_page_dtor() destructor. Else these pages might run into bad page state errors while getting freed into buddy allocator. Signed-off-by: Anshuman Khandual Acked-by: Will Deacon --- arch/arm64/include/asm/tlb.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 106fdc951b6e..3ef2d7ffafee 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,6 +62,9 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { + struct page *page = virt_to_page(pmdp); + + pgtable_page_dtor(page); tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -70,6 +73,9 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { + struct page *page = virt_to_page(pudp); + + pgtable_page_dtor(page); tlb_remove_table(tlb, virt_to_page(pudp)); } #endif From patchwork Mon Feb 25 05:03:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10828233 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9B89B17E9 for ; Mon, 25 Feb 2019 05:05:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 89B322AA91 for ; Mon, 25 Feb 2019 05:05:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7DFB42AA97; Mon, 25 Feb 2019 05:05:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 29DC32AA91 for ; Mon, 25 Feb 2019 05:05:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=xZ0HfIbvWtBXfMSD3foBoZQuFjggnF5UGcxAShtIzPE=; b=Vb0wvfJvUUNmBIqqmGDaWvkcP4 qNlZlF1DWXm1xpcBt0nuAyL2DWvVZj/sVUXhrmaY/EWteM4+RNq3WWLOID/ThSj6Ly6+clymZHrgI lr2xGBGCPMc9RKFgkNPLHTSnKPiBZ8e502aLgQ3huDqgFkJxfPS5F8/rRLnwuItoay2G+z6zlvDR2 dvuduPu70yVxSFR3f1GOlYx528NsLjN0vDlPi43xQLZ8vymt0fp0jZTUJk7pzXdnqR5cy47lUbkac qFwJ2Qo5SdNfEDiZJ31zjnAue7qWgBTgwywSVwoj7OtCJht5leibAlaE23GBzBYepKHi2N2LYIj1h v/ehumSA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8Rt-0007AQ-0B; Mon, 25 Feb 2019 05:05:05 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8RA-0006Hv-G3 for linux-arm-kernel@lists.infradead.org; Mon, 25 Feb 2019 05:04:24 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 91C3C80D; Sun, 24 Feb 2019 21:04:19 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.42.159]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9C1763F5C1; Sun, 24 Feb 2019 21:04:16 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH V2 5/6] arm64/mm: Enable page table page accounting for user space Date: Mon, 25 Feb 2019 10:33:58 +0530 Message-Id: <1551071039-20192-6-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> References: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190224_210420_852876_BF2523A2 X-CRM114-Status: GOOD ( 11.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, yuzhao@google.com, Steve.Capper@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, suzuki.poulose@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Page table pages created for user space processes must be accounted against their memory control groups if initialized. This just introduces a helper function pgtable_gfp_flags() which returns PGALLOC_GFP for all kernel page table page allocations but adds __GFP_ACCOUNT for user page table pages. Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgalloc.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 076764f66fb0..a02a4d1d967d 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -100,12 +100,24 @@ pte_alloc_one_kernel(struct mm_struct *mm) return (pte_t *)pte_alloc_one_virt(mm); } +extern struct mm_struct efi_mm; + +static inline gfp_t pgtable_gfp_flags(struct mm_struct *mm) +{ + if (unlikely((mm == &init_mm) || (mm == &efi_mm))) + return PGALLOC_GFP; + else + return PGALLOC_GFP | __GFP_ACCOUNT; +} + static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; + gfp_t gfp; - pte = alloc_pages(PGALLOC_GFP, 0); + gfp = pgtable_gfp_flags(mm); + pte = alloc_pages(gfp, 0); if (!pte) return NULL; if (!pgtable_page_ctor(pte)) { From patchwork Mon Feb 25 05:03:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10828235 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 603EB1399 for ; Mon, 25 Feb 2019 05:05:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E6542AA8F for ; Mon, 25 Feb 2019 05:05:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 423792AA94; Mon, 25 Feb 2019 05:05:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CF46E2AA8F for ; Mon, 25 Feb 2019 05:05:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Q0pK7CGCdtSwiErPqes3B/29AeOfi/9eDcaSIEyevNk=; b=kD9KtB/PvsAxpkdfOGK7mRiEkv /3GO1T3oBvxvaraCYQ0+gjqFNRZMOH06IV2kLlkbuJSqkxDIUhfKkaKTexZfefFx7JamyYYzLyPzY BzFAZEjwQe4BLiYDRR8p1tY4rCl03BM3CgsfHqFp70ZyztGt9Ua8kyA7RvDeob2JA01JM0DeYkYpL afOEdEcd5d02uHiG/xpwsOV0jrrLPJ1zH5mDA6y00UZV7Nmh/pA2dIsW8JOFn82Ib7urmXMAXQC6f kKdnYTo+wnqdGqozgvOkNrFSCNKoO/ywCY7lE8mSJJauOGVIFj2C2Sps1XSr5g18/zy5Mk/L8jrfS kqHR5Sng==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8SR-0000G4-SQ; Mon, 25 Feb 2019 05:05:39 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8RD-0006Np-SA for linux-arm-kernel@lists.infradead.org; Mon, 25 Feb 2019 05:04:31 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ED4B615AD; Sun, 24 Feb 2019 21:04:22 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.42.159]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 151363F5C1; Sun, 24 Feb 2019 21:04:19 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH V2 6/6] arm64/mm: Enable ARCH_ENABLE_SPLIT_PMD_PTLOCK Date: Mon, 25 Feb 2019 10:33:59 +0530 Message-Id: <1551071039-20192-7-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> References: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190224_210424_774860_5CD5047D X-CRM114-Status: GOOD ( 14.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, yuzhao@google.com, Steve.Capper@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, suzuki.poulose@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Enabling ARCH_ENABLE_SPLIT_PMD_PTLOCK should help reduce lock contention on larger systems when the page tables are being modified concurrently. This moves locking granularity from mm_struct (mm->page_table_lock) to per pmd page table lock (page->ptl). Signed-off-by: Anshuman Khandual --- arch/arm64/Kconfig | 4 +++ arch/arm64/include/asm/pgalloc.h | 43 ++++++++++++++++++++++++++++++-- 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a4168d366127..b909e5d3b951 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -285,6 +285,10 @@ config PGTABLE_LEVELS default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47 default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48 +config ARCH_ENABLE_SPLIT_PMD_PTLOCK + def_bool y + depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE && (PGTABLE_LEVELS > 2) + config ARCH_SUPPORTS_UPROBES def_bool y diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index a02a4d1d967d..9e06238be9b6 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -35,15 +35,54 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t pte); #if CONFIG_PGTABLE_LEVELS > 2 +/* + * FIXME: _workaround might not be the right suffix here but it does + * indicate the intent behind it. Generic pgtable_pmd_page_ctor/dtor + * constructs neither do account the PMD page towards NR_PAGETABLE + * nor do they update page state with the new page type PageTable. + * Ideally pgtable_pmd_page_ctor/dtor should have been just taking care + * of page->pmd_huge_pte when applicable along with what is already + * achieved with pgtable_page_ctor/dtor constructs. Unfortunately that + * is not the case currently and changing those generic mm constructs + * might impact other archs. For now lets do the right thing here and + * drop this when generic PMD constructs accommodate required changes. + */ +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && USE_SPLIT_PMD_PTLOCKS +static inline void pgtable_pmd_page_ctor_workaround(struct page *page) +{ + page->pmd_huge_pte = NULL; +} + +static inline void pgtable_pmd_page_dtor_workaround(struct page *page) +{ + VM_BUG_ON_PAGE(page->pmd_huge_pte, page); +} +#else +static inline void pgtable_pmd_page_ctor_workaround(struct page *page) { } +static inline void pgtable_pmd_page_dtor_workaround(struct page *page) { } +#endif + static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)pte_alloc_one_virt(mm); + pgtable_t ptr; + + ptr = pte_alloc_one(mm); + if (!ptr) + return 0; + + pgtable_pmd_page_ctor_workaround(ptr); + return (pmd_t *)page_to_virt(ptr); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { + struct page *page; + BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); - pte_free(mm, virt_to_page(pmdp)); + page = virt_to_page(pmdp); + + pgtable_pmd_page_dtor_workaround(page); + pte_free(mm, page); } static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot)