From patchwork Thu Jan 24 05:20:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10778317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42FD91390 for ; Thu, 24 Jan 2019 05:20:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3386E2CF16 for ; Thu, 24 Jan 2019 05:20:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 27E4B2D34E; Thu, 24 Jan 2019 05:20:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BF5892CFDF for ; Thu, 24 Jan 2019 05:20:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BFq9SxqVUdo4zGjtsIPDres/MzlblXcpflCErNsDkEs=; b=MlOrheeeag2igRk5cnCYENlKw/ m19WvIY1B0YwA7hsGcieHZ9lSPx53nutbLF0b+te/r2mS/HpKMAeN+d4u3ByAwbQ3GTXQtCVp9J5l tMLXqYZmdB7AMzPxo7LSzfiMhFW5SqJbfce0XMH45lwyhgWxNfWQ082SuLWZq0edh/PPmpS56XXFk IIdcKc0Ssifn72165FmeICwTiAF756xzRRsFod5Jmivx/0F5tJQoUkVEd78KYILfC3Kr5nZ2CXQwV wB0I6VNUt4g6b0pX/5JFL6I9/6teIiwHXOcKKa0K9KepPOeM81nzUvUiCtlrg1ouVjg/72pxEp74L 5tWZcBtg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRg-0002c3-6k; Thu, 24 Jan 2019 05:20:56 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRQ-0002OZ-98 for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 05:20:41 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5CC4D1596; Wed, 23 Jan 2019 21:20:38 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.157]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 75D9C3F5C1; Wed, 23 Jan 2019 21:20:35 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 1/6] KVM: ARM: Remove pgtable standard functions from stage-2 page tables Date: Thu, 24 Jan 2019 10:50:15 +0530 Message-Id: <1548307220-19756-2-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> References: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190123_212040_321158_4543844F X-CRM114-Status: UNSURE ( 9.82 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, suzuki.poulose@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, Steve.Capper@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP ARM64 standard pgtable functions are going to use pgtable_page_[ctor|dtor] constructs. Certain stage-2 page table allocations are multi order which cannot be allocated through a generic pgtable function as it does not exist right now. This prevents all pgtable allocations including multi order ones in stage-2 from moving into new ARM64 (pgtable_page_[ctor|dtor]) pgtable functions. Hence remove all generic pgtable allocation function dependency from stage-2 page tables till there is one multi-order allocation function available. Signed-off-by: Anshuman Khandual --- arch/arm/include/asm/stage2_pgtable.h | 4 ++-- arch/arm64/include/asm/stage2_pgtable.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm/include/asm/stage2_pgtable.h b/arch/arm/include/asm/stage2_pgtable.h index c4b1d4f..0895e0b 100644 --- a/arch/arm/include/asm/stage2_pgtable.h +++ b/arch/arm/include/asm/stage2_pgtable.h @@ -32,14 +32,14 @@ #define stage2_pgd_present(kvm, pgd) pgd_present(pgd) #define stage2_pgd_populate(kvm, pgd, pud) pgd_populate(NULL, pgd, pud) #define stage2_pud_offset(kvm, pgd, address) pud_offset(pgd, address) -#define stage2_pud_free(kvm, pud) pud_free(NULL, pud) +#define stage2_pud_free(kvm, pud) free_page((unsigned long)pud) #define stage2_pud_none(kvm, pud) pud_none(pud) #define stage2_pud_clear(kvm, pud) pud_clear(pud) #define stage2_pud_present(kvm, pud) pud_present(pud) #define stage2_pud_populate(kvm, pud, pmd) pud_populate(NULL, pud, pmd) #define stage2_pmd_offset(kvm, pud, address) pmd_offset(pud, address) -#define stage2_pmd_free(kvm, pmd) pmd_free(NULL, pmd) +#define stage2_pmd_free(kvm, pmd) free_page((unsigned long)pmd) #define stage2_pud_huge(kvm, pud) pud_huge(pud) diff --git a/arch/arm64/include/asm/stage2_pgtable.h b/arch/arm64/include/asm/stage2_pgtable.h index 5412fa4..915809e 100644 --- a/arch/arm64/include/asm/stage2_pgtable.h +++ b/arch/arm64/include/asm/stage2_pgtable.h @@ -119,7 +119,7 @@ static inline pud_t *stage2_pud_offset(struct kvm *kvm, static inline void stage2_pud_free(struct kvm *kvm, pud_t *pud) { if (kvm_stage2_has_pud(kvm)) - pud_free(NULL, pud); + free_page((unsigned long)pud); } static inline bool stage2_pud_table_empty(struct kvm *kvm, pud_t *pudp) @@ -192,7 +192,7 @@ static inline pmd_t *stage2_pmd_offset(struct kvm *kvm, static inline void stage2_pmd_free(struct kvm *kvm, pmd_t *pmd) { if (kvm_stage2_has_pmd(kvm)) - pmd_free(NULL, pmd); + free_page((unsigned long)pmd); } static inline bool stage2_pud_huge(struct kvm *kvm, pud_t pud) From patchwork Thu Jan 24 05:20:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10778319 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D345313B4 for ; Thu, 24 Jan 2019 05:21:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C36352CEF3 for ; Thu, 24 Jan 2019 05:21:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B6ECA2D6FB; Thu, 24 Jan 2019 05:21:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 74F4A2D34E for ; Thu, 24 Jan 2019 05:21:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=g8uiul2XvzbA5TqZOlMVjHy7kgs2GvoVBeD0EhHhT5A=; b=G+Z++IMRSzBhocHmRB+xFpEsBe GaAZk8+/pf/48FkWMA5KuftZCeTUMlia1P9jcdwiTInIxXmYcn2so+btGgmmFYcItE5MAQfm9hhfI u3CHbs7/eL4l1w78mDL7dcUd944c5cZ1vkoz7P1LFDj4+yUOMmA7sM3JVc/3M9Gw8t4vPCupT8pn5 HPlZ3Hh0UIu08K0XW+epPfpA9CPMOUekOMAl4aiigknXMQldaP6H5Mcqoa4Pxx+4LjtJEqz4bvQvK ME9JESxVOEm4Oe1QQHOOi2ODRFX8C6MQdfSg7Tu8VBBXT2cYLSSaF/0dxfBYzD4DNRzLZYhAlQOls qQdJOkBQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRv-0002qG-8u; Thu, 24 Jan 2019 05:21:11 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRT-0002PW-3U for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 05:20:45 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CF106A78; Wed, 23 Jan 2019 21:20:41 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.157]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DED263F5C1; Wed, 23 Jan 2019 21:20:38 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 2/6] arm64/mm: Ensure PGD kmem cache object is actually allocated Date: Thu, 24 Jan 2019 10:50:16 +0530 Message-Id: <1548307220-19756-3-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> References: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190123_212043_292390_B03771E8 X-CRM114-Status: GOOD ( 10.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, suzuki.poulose@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, Steve.Capper@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP 'pgd_cache' is created for allocating non PAGE_SIZE based PGD pointers. In such cases ensure that the kmem cache object is allocated before proceeding further with boot. Signed-off-by: Anshuman Khandual --- arch/arm64/mm/pgd.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 289f911..8cb4e39 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -62,4 +62,5 @@ void __init pgd_cache_init(void) */ pgd_cache = kmem_cache_create("pgd_cache", PGD_SIZE, PGD_SIZE, SLAB_PANIC, NULL); + BUG_ON(!pgd_cache); } From patchwork Thu Jan 24 05:20:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10778321 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB5B113B4 for ; Thu, 24 Jan 2019 05:21:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB7CC2D961 for ; Thu, 24 Jan 2019 05:21:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9F8FF2D799; Thu, 24 Jan 2019 05:21:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 109BF2D79D for ; Thu, 24 Jan 2019 05:21:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ythFZuiySaYZJ28sPWI48i0N8j+4LMJhalnSB5SqLZ4=; b=Qjecv540K4InnF+cafSXqcMmDp otVumht/iICIYwBG+CAUurtYY9vegZj58vhAmhD+cht41X15Ip592YnP2OioBS9HBKG978Vrt2Dl3 9pf6EoV3cXoaoXaM1clOa7LwjGsIAJ0CqAZrNiwyUQEDlRLx6VYXJYJEhJxE1uJx3quNcucV/ttlL UmCRPHn8rsE6vo3BTeMGDwRZBCCw2IxfWjHEoZ8IvdPRhNBFOWQ981ZISDB91rpDC1unrOvulpK38 ZDUicWE40fznNWpRNRFffyyDcYJ4NI1zXRiegYLpXauz3Ty0I7sRxU1mlIKwPwh+zURzjubbu3m2+ u20TE2PA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXSB-00035O-I7; Thu, 24 Jan 2019 05:21:27 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRW-0002SH-CQ for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 05:20:50 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4593CA78; Wed, 23 Jan 2019 21:20:45 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.157]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5D7F83F5C1; Wed, 23 Jan 2019 21:20:42 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 3/6] arm64/mm: Make pgd_pgtable_alloc() call pte_alloc_one() always Date: Thu, 24 Jan 2019 10:50:17 +0530 Message-Id: <1548307220-19756-4-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> References: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190123_212046_491003_82BDA453 X-CRM114-Status: GOOD ( 13.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, suzuki.poulose@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, Steve.Capper@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP pgd_pgtable_alloc() provides page allocation function while creating all levels of page table (e.g PGD, PUD, CONT PMD) for various kernel mappings. It calls __get_free_page() and initializes page with pagetable_page_ctor(). pte_alloc_one() already provides a standard interface for allocating a page table page and initializes it with pagetable_page_ctor(). This removes the redundancy and instead make it call pte_alloc_one() directly. Signed-off-by: Anshuman Khandual --- arch/arm64/mm/mmu.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b6f5aa5..2dbd723 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -372,13 +372,22 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, static phys_addr_t pgd_pgtable_alloc(void) { - void *ptr = (void *)__get_free_page(PGALLOC_GFP); - if (!ptr || !pgtable_page_ctor(virt_to_page(ptr))) - BUG(); + + pgtable_t ptr; + + /* + * pgd_pgtable_alloc() is called while creating kernel mappings + * through __create_pgd_mapping() might not install it through + * swapper_pg_dir (&init_mm). Even then init_mm is passed here + * just to indicate that the allocation is kernel specific not + * for the user space page tables. + */ + ptr = pte_alloc_one(&init_mm); + BUG_ON(!ptr); /* Ensure the zeroed page is visible to the page table walker */ dsb(ishst); - return __pa(ptr); + return page_to_phys(ptr); } /* From patchwork Thu Jan 24 05:20:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10778327 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 145D913B4 for ; Thu, 24 Jan 2019 05:22:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 042F02CF03 for ; Thu, 24 Jan 2019 05:22:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EDDD32D9FB; Thu, 24 Jan 2019 05:22:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6B89F2DB16 for ; Thu, 24 Jan 2019 05:22:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=XRdDwZJXODFCL5bV/3gKlB9kwR5E4ZDciAAs/7wTEwM=; b=rYBa7uggVUhKGS9MK+VXT44y7g G9py03A6NwzPG8r0y00bLHF3lol9mJU+BC8amC7wMJqFB0ReF2TUAK2mLw+hsDuLRSwgq6w8ADSBP kFzBaH8pPcyaoKjVDelzUsy7WgX2dm9LTBHXznjX0qJaGdDVgBnfyF04XAvRgBsOcXNRx7LKjM6Dp WLJX88zY9/9dFvwGLRPTiJojljMbuzhb3AlWgL1CcZIqy0L0H2ojszU7hwHHcXxNnQylww/9zC7Nv PFo2mOM2OXLmekvnKGdicze3Zr7biAvTsAEby3UoDjtp6mk3c/2r6kTkHNS43Dgu6Xv2DmEAtwS41 fDCF2tzw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXTR-00044D-8X; Thu, 24 Jan 2019 05:22:45 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRZ-0002Vb-84 for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 05:20:52 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B04AD1650; Wed, 23 Jan 2019 21:20:48 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.157]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C7B653F5C1; Wed, 23 Jan 2019 21:20:45 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 4/6] arm64/mm: Make all page table pages cycles through standard constructs Date: Thu, 24 Jan 2019 10:50:18 +0530 Message-Id: <1548307220-19756-5-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> References: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190123_212050_026114_1F1F991E X-CRM114-Status: GOOD ( 15.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, suzuki.poulose@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, Steve.Capper@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Currently there are inconsistencies during page table page's allocation, initialization and destruction. Not all those pages go through standard constructors pgtable_page_ctor/pgtable_page_dtor which is very much essential for proper initialization and accounting. Not going through proper pgtable_page_ctor/pgtable_page_dtor cycle makes them vulnerable for bad page state errors during page table tear down process. This makes all levels of page table pages either for the user space or for the kernel mappings go through these standard constructs. While here pte_free() has been modified to accommodate pages which might not have been allocated through buddy system but instead came from memblock [1] directly during early boot process. Those pages must not go through the pgtable_page_dtor() and instead are freed with free_reserved_page(). PGD based page table page which is allocated from 'pgd_cache' kmem cache in certain situations (PGD_SIZE != PAGE_SIZE ) is excluded from being cycled through pgtable_page_[ctor/dtor]. This is not a problem because this slab never gets freed back to buddy allocator. All stage-2 page table pages must cycle through pgtable_page_[ctor/dtor] constructs for proper initialization and accounting. Use the standard pte_alloc_one() and pte_free() for their allocation and release instead of calling into the buddy allocator directly. [1] Page table page allocation from memblock (a) early_pgtable_alloc (passed with __create_pgd_mapping) (b) vmemmap_pgd|pud_populate (vmemmap_populate) Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgalloc.h | 42 ++++++++++++++++++++++++++++++++-------- arch/arm64/mm/pgd.c | 4 ++-- 2 files changed, 36 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c..076764f 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -29,17 +29,21 @@ #define PGALLOC_GFP (GFP_KERNEL | __GFP_ZERO) #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm); +static inline unsigned long pte_alloc_one_virt(struct mm_struct *mm); +static inline void pte_free(struct mm_struct *mm, pgtable_t pte); + #if CONFIG_PGTABLE_LEVELS > 2 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + return (pmd_t *)pte_alloc_one_virt(mm); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); - free_page((unsigned long)pmdp); + pte_free(mm, virt_to_page(pmdp)); } static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) @@ -62,13 +66,13 @@ static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pud_t *)__get_free_page(PGALLOC_GFP); + return (pud_t *)pte_alloc_one_virt(mm); } static inline void pud_free(struct mm_struct *mm, pud_t *pudp) { BUG_ON((unsigned long)pudp & (PAGE_SIZE-1)); - free_page((unsigned long)pudp); + pte_free(mm, virt_to_page(pudp)); } static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pudp, pgdval_t prot) @@ -93,7 +97,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); static inline pte_t * pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(PGALLOC_GFP); + return (pte_t *)pte_alloc_one_virt(mm); } static inline pgtable_t @@ -111,19 +115,41 @@ pte_alloc_one(struct mm_struct *mm) return pte; } +static inline unsigned long +pte_alloc_one_virt(struct mm_struct *mm) +{ + pgtable_t ptr; + + ptr = pte_alloc_one(mm); + if (!ptr) + return 0; + + return (unsigned long) page_to_virt(ptr); +} + /* * Free a PTE table. */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *ptep) { if (ptep) - free_page((unsigned long)ptep); + pte_free(mm, virt_to_page(ptep)); } static inline void pte_free(struct mm_struct *mm, pgtable_t pte) { - pgtable_page_dtor(pte); - __free_page(pte); + /* + * Some page table pages might have come from memblock either + * with vmemmap_alloc_block (during vmemmap_populate) or with + * early_pgtable_alloc (during __create_pgd_mapping). These + * pages should be freed with free_reserved_page() instead. + */ + if (PageReserved(pte)) { + free_reserved_page(pte); + } else { + pgtable_page_dtor(pte); + __free_page(pte); + } } static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t ptep, diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 8cb4e39..e344075 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -31,7 +31,7 @@ static struct kmem_cache *pgd_cache __ro_after_init; pgd_t *pgd_alloc(struct mm_struct *mm) { if (PGD_SIZE == PAGE_SIZE) - return (pgd_t *)__get_free_page(PGALLOC_GFP); + return (pgd_t *)pte_alloc_one_virt(mm); else return kmem_cache_alloc(pgd_cache, PGALLOC_GFP); } @@ -39,7 +39,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { if (PGD_SIZE == PAGE_SIZE) - free_page((unsigned long)pgd); + pte_free(mm, virt_to_page(pgd)); else kmem_cache_free(pgd_cache, pgd); } From patchwork Thu Jan 24 05:20:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10778323 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A438A1390 for ; Thu, 24 Jan 2019 05:21:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 94CA52D7AB for ; Thu, 24 Jan 2019 05:21:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8908B2D997; Thu, 24 Jan 2019 05:21:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 30C872D7AB for ; Thu, 24 Jan 2019 05:21:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=WSEc1Liu7iAI76eusyw1DISD7+OF43CpWU+immUgxyQ=; b=c1qc/IsnHe4kaUeuAWyQQGUcnu vILzck79X42egRFi5ZWa/CuNAtm5Nc1y/fIFhz7w4of0iGiRhT2mG5pjMKvJAmHlAc5cNAAV7IhnD 0UOMVNrpZ2mV2M7NN7hYcclYGlnxtINQj5YsyR/FN9vKA7o6JCEKpSDNpv85memAO6eZ34szi5b9A Zd/32VLzNYOXe4ZKYi58TFJs1eJez7KkMRUEC1Ko75jJTGGD2eZ6rNokoRI6S/DjwF13ctnr18/Vz y1lKZSZGdAIxD2jvjoVMbiyWPIEOpP/j+F6j6qIJ7Ts7Pe9FG07TDOeYjMg/9eB0bwvNCfKqbEfFb 4xTcOvow==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXSa-0003TY-Py; Thu, 24 Jan 2019 05:21:52 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRc-0002Yt-GU for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 05:20:54 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2552AA78; Wed, 23 Jan 2019 21:20:52 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.157]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3D2133F5C1; Wed, 23 Jan 2019 21:20:48 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 5/6] arm64/mm: Call pgtable_page_dtor() for both PMD and PUD page table pages Date: Thu, 24 Jan 2019 10:50:19 +0530 Message-Id: <1548307220-19756-6-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> References: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190123_212052_635886_79E544A1 X-CRM114-Status: UNSURE ( 9.95 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, suzuki.poulose@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, Steve.Capper@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP TLB flush during PMD and PUD level page table page tear down currently does not call pgtable_page_dtor() like that for PTE page table pages. Now that all the page table pages which are allocated from the buddy allocator must have been initialized through pgtable_page_ctor() they must be freed back only after being called into pgtable_page_dtor() destructor. Else these pages might run into bad page state errors while getting freed into buddy allocator. Signed-off-by: Anshuman Khandual Acked-by: Will Deacon --- arch/arm64/include/asm/tlb.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 106fdc9..de46ba0 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,6 +62,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { + pgtable_page_dtor(virt_to_page(pmdp)); tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -70,6 +71,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { + pgtable_page_dtor(virt_to_page(pudp)); tlb_remove_table(tlb, virt_to_page(pudp)); } #endif From patchwork Thu Jan 24 05:20:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10778325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3C0BC1390 for ; Thu, 24 Jan 2019 05:22:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 283812CF03 for ; Thu, 24 Jan 2019 05:22:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1BF5E2DA9D; Thu, 24 Jan 2019 05:22:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C4BEF2CF03 for ; Thu, 24 Jan 2019 05:22:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=x0HXROOQaY45Q76N5L5PpTQyaBtsJHyBW45WVL+i/TY=; b=eDpkBJq1GY1HlAnCpCAmsAdkSG J3IkCz0oeJDYlgqiTiFjZXuc/JYjQYJ+f5igLXuodAnq/a0wTIf08OfzjvlLTna+g0bvb9sjfhlJt L/R0PPNfSRKi5weIQVcqm3o5ZZBmfHDHAKCpviQ5OL/7fAwGpZrmxNt3tPe7tFfdWxzA6JkgDMcGW mNYUtiRy4tCuzly8ONHGWA1vt2uS1AicqznKvDIcQ+Ry+GM0bHENh/lMNaLj5xMt7mKDRSYFHWhzN 2x04cspx16yqsWrS6Pp/6CAwdhaF342EGsky7oLe995btoKPM+UfiAwpCUqWkyoc+xnjffRp3e0NC dlrG7Www==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXTA-0003pV-8K; Thu, 24 Jan 2019 05:22:28 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gmXRg-0002c8-25 for linux-arm-kernel@lists.infradead.org; Thu, 24 Jan 2019 05:21:00 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9665CA78; Wed, 23 Jan 2019 21:20:55 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.157]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A7B973F5C1; Wed, 23 Jan 2019 21:20:52 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 6/6] arm64/mm: Enable page table page accounting for user space Date: Thu, 24 Jan 2019 10:50:20 +0530 Message-Id: <1548307220-19756-7-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> References: <1548307220-19756-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190123_212056_677620_D5C1DE98 X-CRM114-Status: GOOD ( 11.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, suzuki.poulose@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, Steve.Capper@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Page table pages created for user space processes must be accounted against their memory control groups if initialized. This just introduces a helper function pgtable_gfp_flags() which returns PGALLOC_GFP for all kernel page table page allocations but adds __GFP_ACCOUNT for user page table pages. Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgalloc.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 076764f..775cde9 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -100,12 +100,22 @@ pte_alloc_one_kernel(struct mm_struct *mm) return (pte_t *)pte_alloc_one_virt(mm); } +static inline gfp_t pgtable_gfp_flags(struct mm_struct *mm) +{ + if (unlikely(mm == &init_mm)) + return PGALLOC_GFP; + else + return PGALLOC_GFP | __GFP_ACCOUNT; +} + static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; + gfp_t gfp; - pte = alloc_pages(PGALLOC_GFP, 0); + gfp = pgtable_gfp_flags(mm); + pte = alloc_pages(gfp, 0); if (!pte) return NULL; if (!pgtable_page_ctor(pte)) {