From patchwork Mon Feb 25 05:03:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10828231 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF9FF1823 for ; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB0212AA8F for ; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C91702AA94; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4DC182AA8F for ; Mon, 25 Feb 2019 05:05:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=uunBTmomxELR21JJkTonFp052bKHGJiOLpMiaQzeQdk=; b=PMM05F2ezFldqUaok2VpFFhCww CdnI7gmHIDQ+5Z7D76Raq39Nv7L9Z8VvedMWL1IG0BMFcSH9THouMpXwaAhLipAqjfVOZPjSz+pi9 wcbmvhTh3lRSsOR2Rmx+3xtIkONEJnLtn2OMWRxpoj1W7wRgMng2BH1o7McrzsR0famwlYPb+RtWN LelDbKhMn0tRqoElWW2s8txYzVSE0E7hBBebkNIlMzbB0QIL95WqaI2G1ZLA1tJsVGMvlQbUGoYcU HSfvjgZjlWTUByoensxRER0sXZhgeCyXySd8ZV5D/CxYEj6xZ5faeM8j7Hn26FWWOp1baMHWYXU8d PD43SKFA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8Rg-0006oz-DW; Mon, 25 Feb 2019 05:04:52 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gy8R3-0006A9-Eu for linux-arm-kernel@lists.infradead.org; Mon, 25 Feb 2019 05:04:17 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BADEE80D; Sun, 24 Feb 2019 21:04:12 -0800 (PST) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.42.159]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D79D33F5C1; Sun, 24 Feb 2019 21:04:09 -0800 (PST) From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org Subject: [PATCH V2 3/6] arm64/mm: Make all page table pages cycles through standard constructs Date: Mon, 25 Feb 2019 10:33:56 +0530 Message-Id: <1551071039-20192-4-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> References: <1551071039-20192-1-git-send-email-anshuman.khandual@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190224_210413_613170_F1DB2328 X-CRM114-Status: GOOD ( 16.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, yuzhao@google.com, Steve.Capper@arm.com, marc.zyngier@arm.com, Catalin.Marinas@arm.com, suzuki.poulose@arm.com, will.deacon@arm.com, james.morse@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Currently there are inconsistencies during page table page's allocation, initialization and destruction. Not all those pages go through standard constructors pgtable_page_ctor/pgtable_page_dtor which is very much essential for proper initialization and accounting. Not going through proper pgtable_page_ctor/pgtable_page_dtor cycle makes them vulnerable for bad page state errors during page table tear down process. This makes all levels of page table pages either for the user space or for the kernel mappings go through these standard constructs. While here pte_free() has been modified to accommodate pages which might not have been allocated through buddy system but instead came from memblock [1] directly during early boot process. Those pages must not go through the pgtable_page_dtor() and instead are freed with free_reserved_page(). PGD based page table page which is allocated from 'pgd_cache' kmem cache in certain situations (PGD_SIZE != PAGE_SIZE ) is excluded from being cycled through pgtable_page_[ctor/dtor]. This is not a problem because this slab never gets freed back to buddy allocator. [1] Page table page allocation from memblock (a) early_pgtable_alloc (passed with __create_pgd_mapping) (b) vmemmap_pgd|pud_populate (vmemmap_populate) Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgalloc.h | 42 ++++++++++++++++++++++++++------ arch/arm64/mm/pgd.c | 4 +-- 2 files changed, 36 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c73bf0..076764f66fb0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -29,17 +29,21 @@ #define PGALLOC_GFP (GFP_KERNEL | __GFP_ZERO) #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm); +static inline unsigned long pte_alloc_one_virt(struct mm_struct *mm); +static inline void pte_free(struct mm_struct *mm, pgtable_t pte); + #if CONFIG_PGTABLE_LEVELS > 2 static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + return (pmd_t *)pte_alloc_one_virt(mm); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); - free_page((unsigned long)pmdp); + pte_free(mm, virt_to_page(pmdp)); } static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) @@ -62,13 +66,13 @@ static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pud_t *)__get_free_page(PGALLOC_GFP); + return (pud_t *)pte_alloc_one_virt(mm); } static inline void pud_free(struct mm_struct *mm, pud_t *pudp) { BUG_ON((unsigned long)pudp & (PAGE_SIZE-1)); - free_page((unsigned long)pudp); + pte_free(mm, virt_to_page(pudp)); } static inline void __pgd_populate(pgd_t *pgdp, phys_addr_t pudp, pgdval_t prot) @@ -93,7 +97,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); static inline pte_t * pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(PGALLOC_GFP); + return (pte_t *)pte_alloc_one_virt(mm); } static inline pgtable_t @@ -111,19 +115,41 @@ pte_alloc_one(struct mm_struct *mm) return pte; } +static inline unsigned long +pte_alloc_one_virt(struct mm_struct *mm) +{ + pgtable_t ptr; + + ptr = pte_alloc_one(mm); + if (!ptr) + return 0; + + return (unsigned long) page_to_virt(ptr); +} + /* * Free a PTE table. */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *ptep) { if (ptep) - free_page((unsigned long)ptep); + pte_free(mm, virt_to_page(ptep)); } static inline void pte_free(struct mm_struct *mm, pgtable_t pte) { - pgtable_page_dtor(pte); - __free_page(pte); + /* + * Some page table pages might have come from memblock either + * with vmemmap_alloc_block (during vmemmap_populate) or with + * early_pgtable_alloc (during __create_pgd_mapping). These + * pages should be freed with free_reserved_page() instead. + */ + if (PageReserved(pte)) { + free_reserved_page(pte); + } else { + pgtable_page_dtor(pte); + __free_page(pte); + } } static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t ptep, diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c index 289f9113a27a..ea588c8825e2 100644 --- a/arch/arm64/mm/pgd.c +++ b/arch/arm64/mm/pgd.c @@ -31,7 +31,7 @@ static struct kmem_cache *pgd_cache __ro_after_init; pgd_t *pgd_alloc(struct mm_struct *mm) { if (PGD_SIZE == PAGE_SIZE) - return (pgd_t *)__get_free_page(PGALLOC_GFP); + return (pgd_t *)pte_alloc_one_virt(mm); else return kmem_cache_alloc(pgd_cache, PGALLOC_GFP); } @@ -39,7 +39,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { if (PGD_SIZE == PAGE_SIZE) - free_page((unsigned long)pgd); + pte_free(mm, virt_to_page(pgd)); else kmem_cache_free(pgd_cache, pgd); }