From patchwork Sat Mar 4 14:30:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 9603993 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7A1E060453 for ; Sat, 4 Mar 2017 14:34:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A5BB28503 for ; Sat, 4 Mar 2017 14:34:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E80C28589; Sat, 4 Mar 2017 14:34:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id ACA1B28503 for ; Sat, 4 Mar 2017 14:34:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=eHzn/WRzUoSGxw8N8LBjyfynwZKXuEHv2fh+r2S4sMM=; b=Rckvi5MhGTXshn5RLT0gLdfJmK jGsLuUTr2wnZZ7d38Q4mcHjMv96LOQziZWacawHsYeghtEE/1u3bra+LVXkVhpkdGqQLD/RixvwEd /Jpxhhv/LCAzvWEn9fQLvq3ITRMAMPFr3RKtjyxEcPSeMLdkOTvZ1zEqyZWolHwnKhzwSlM9d4Imk tRzn/rvUGnybgGxXbt21iyez9LRB44MfJiyR/r+3e9p+gY8mFkJFYVOIuvEQrQHb/mQWTvmslNGk+ ZWZqhXDRGdqZZeUMInKz7rFJFFzqwdgavk1NqlpApMjj5mq/Bq8XZKTNOUxqs9nKCa/qKZ3TYxiUM RCzo/F+g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1ckAlh-000559-6P; Sat, 04 Mar 2017 14:34:45 +0000 Received: from mail-wm0-x22c.google.com ([2a00:1450:400c:c09::22c]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1ckAiz-0002R5-Gq for linux-arm-kernel@lists.infradead.org; Sat, 04 Mar 2017 14:31:59 +0000 Received: by mail-wm0-x22c.google.com with SMTP id v186so34343179wmd.0 for ; Sat, 04 Mar 2017 06:31:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SpJHP4905EcDemB3pnlKCXX5PJUL2NurBC0f2djuuxk=; b=W5K5ztjwIzTDt1dApM/gvsp33K8AW0w1UBtdpFFtR35jbmL2SbE8bBtulj+Qeb04eS lMm8oqYq7/qROn9FQcjSRWNgUIxzvDVGFeKh4no9NjyY9W4k1VzFgylO1xvPvITRPGSp 5TYrSCJa3MUu8RvFDb4FNLsA2R5wq7Y9nmCZM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SpJHP4905EcDemB3pnlKCXX5PJUL2NurBC0f2djuuxk=; b=OGfDfqq8QslJqRBBqOjh/Vsfc34pINJnHxRx2g8lfWEgLmGWgLC0hMuTN5z5H6kYMU AvSqjvgjCop9uquMLVhvQEry9byBjbDCNzFCHgMWt+CsaW3o9ank0vByij/p2ViyQbLS Wrkf0+R5il5KZcjoxrpr7jMZ8TlM991aesQ5mWvC6VIqXwZVAT38mNljj62xgN+gVveu MuB6iDxlhjoCHgryZHELDUPkDUvp2a93i5JgcVdlN6UFO3Ao8sjwHYwnYNXdBsGRdi6z S8lkUynilebe+qd+c3wV/PU3lvrYsDu9AVhN419kNPq4kBIpyuEVfPXi1wFp2TzyTMmp 6raA== X-Gm-Message-State: AMke39mO2Xth/6psdX1O67q6b6k+21G8lgKzGErRi9V1oibqSebVW8TGCjqyHPBR2/ciAtBL X-Received: by 10.28.87.19 with SMTP id l19mr7340610wmb.95.1488637895345; Sat, 04 Mar 2017 06:31:35 -0800 (PST) Received: from localhost.localdomain ([197.130.214.201]) by smtp.gmail.com with ESMTPSA id c35sm19497267wra.1.2017.03.04.06.31.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 04 Mar 2017 06:31:34 -0800 (PST) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, kernel-hardening@lists.openwall.com, mark.rutland@arm.com, catalin.marinas@arm.com, will.deacon@arm.com, labbott@fedoraproject.org Subject: [PATCH v4 6/6] arm64: mm: set the contiguous bit for kernel mappings where appropriate Date: Sat, 4 Mar 2017 14:30:48 +0000 Message-Id: <1488637848-13588-7-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1488637848-13588-1-git-send-email-ard.biesheuvel@linaro.org> References: <1488637848-13588-1-git-send-email-ard.biesheuvel@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170304_063157_865552_E79BD55B X-CRM114-Status: GOOD ( 18.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: keescook@chromium.org, suzuki.poulose@arm.com, marc.zyngier@arm.com, andre.przywara@arm.com, Ard Biesheuvel , james.morse@arm.com, kvmarm@lists.cs.columbia.edu MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP This is the third attempt at enabling the use of contiguous hints for kernel mappings. The most recent attempt 0bfc445dec9d was reverted after it turned out that updating permission attributes on live contiguous ranges may result in TLB conflicts. So this time, the contiguous hint is not set for .rodata or for the linear alias of .text/.rodata, both of which are mapped read-write initially, and remapped read-only at a later stage. (Note that the latter region could also be unmapped and remapped again with updated permission attributes, given that the region, while live, is only mapped for the convenience of the hibernation code, but that also means the TLB footprint is negligible anyway, so why bother) This enables the following contiguous range sizes for the virtual mapping of the kernel image, and for the linear mapping: granule size | cont PTE | cont PMD | -------------+------------+------------+ 4 KB | 64 KB | 32 MB | 16 KB | 2 MB | 1 GB* | 64 KB | 2 MB | 16 GB* | * Only when built for 3 or more levels of translation. This is due to the fact that a 2 level configuration only consists of PGDs and PTEs, and the added complexity of dealing with folded PMDs is not justified considering that 16 GB contiguous ranges are likely to be ignored by the hardware (and 16k/2 levels is a niche configuration) Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 86 ++++++++++++++------ 1 file changed, 63 insertions(+), 23 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 0612573ef869..d0ae2f1f44fc 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -109,8 +109,10 @@ static bool pgattr_change_is_safe(u64 old, u64 new) static void alloc_init_pte(pmd_t *pmd, unsigned long addr, unsigned long end, unsigned long pfn, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void)) + phys_addr_t (*pgtable_alloc)(void), + bool may_use_cont) { + pgprot_t __prot = prot; pte_t *pte; BUG_ON(pmd_sect(*pmd)); @@ -128,7 +130,19 @@ static void alloc_init_pte(pmd_t *pmd, unsigned long addr, do { pte_t old_pte = *pte; - set_pte(pte, pfn_pte(pfn, prot)); + /* + * Set the contiguous bit for the subsequent group of PTEs if + * its size and alignment are appropriate. + */ + if (may_use_cont && + ((addr | PFN_PHYS(pfn)) & ~CONT_PTE_MASK) == 0) { + if (end - addr >= CONT_PTE_SIZE) + __prot = __pgprot(pgprot_val(prot) | PTE_CONT); + else + __prot = prot; + } + + set_pte(pte, pfn_pte(pfn, __prot)); pfn++; /* @@ -145,8 +159,10 @@ static void alloc_init_pte(pmd_t *pmd, unsigned long addr, static void alloc_init_pmd(pud_t *pud, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(void), - bool page_mappings_only) + bool page_mappings_only, + bool may_use_cont) { + pgprot_t __prot = prot; pmd_t *pmd; unsigned long next; @@ -173,7 +189,19 @@ static void alloc_init_pmd(pud_t *pud, unsigned long addr, unsigned long end, /* try section mapping first */ if (((addr | next | phys) & ~SECTION_MASK) == 0 && !page_mappings_only) { - pmd_set_huge(pmd, phys, prot); + /* + * Set the contiguous bit for the subsequent group of + * PMDs if its size and alignment are appropriate. + */ + if (may_use_cont && + ((addr | phys) & ~CONT_PMD_MASK) == 0) { + if (end - addr >= CONT_PMD_SIZE) + __prot = __pgprot(pgprot_val(prot) | + PTE_CONT); + else + __prot = prot; + } + pmd_set_huge(pmd, phys, __prot); /* * After the PMD entry has been populated once, we @@ -183,7 +211,7 @@ static void alloc_init_pmd(pud_t *pud, unsigned long addr, unsigned long end, pmd_val(*pmd))); } else { alloc_init_pte(pmd, addr, next, __phys_to_pfn(phys), - prot, pgtable_alloc); + prot, pgtable_alloc, may_use_cont); BUG_ON(pmd_val(old_pmd) != 0 && pmd_val(old_pmd) != pmd_val(*pmd)); @@ -209,7 +237,8 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, static void alloc_init_pud(pgd_t *pgd, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(void), - bool page_mappings_only) + bool page_mappings_only, + bool may_use_cont) { pud_t *pud; unsigned long next; @@ -242,7 +271,8 @@ static void alloc_init_pud(pgd_t *pgd, unsigned long addr, unsigned long end, pud_val(*pud))); } else { alloc_init_pmd(pud, addr, next, phys, prot, - pgtable_alloc, page_mappings_only); + pgtable_alloc, page_mappings_only, + may_use_cont); BUG_ON(pud_val(old_pud) != 0 && pud_val(old_pud) != pud_val(*pud)); @@ -257,11 +287,14 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(void), - bool page_mappings_only) + bool page_mappings_only, + bool may_use_cont) { unsigned long addr, length, end, next; pgd_t *pgd = pgd_offset_raw(pgdir, virt); + BUG_ON(page_mappings_only && may_use_cont); /* sanity check */ + /* * If the virtual and physical address don't have the same offset * within a page, we cannot map the region as the caller expects. @@ -277,7 +310,7 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, do { next = pgd_addr_end(addr, end); alloc_init_pud(pgd, addr, next, phys, prot, pgtable_alloc, - page_mappings_only); + page_mappings_only, may_use_cont); phys += next - addr; } while (pgd++, addr = next, addr != end); } @@ -306,7 +339,8 @@ static void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, &phys, virt); return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, false); + __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, false, + false); } void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, @@ -316,7 +350,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, BUG_ON(mm == &init_mm); __create_pgd_mapping(mm->pgd, phys, virt, size, prot, - pgd_pgtable_alloc, page_mappings_only); + pgd_pgtable_alloc, page_mappings_only, false); } static void update_mapping_prot(phys_addr_t phys, unsigned long virt, @@ -329,7 +363,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, } __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, - NULL, debug_pagealloc_enabled()); + NULL, debug_pagealloc_enabled(), false); /* flush the TLBs after updating live kernel mappings */ flush_tlb_kernel_range(virt, virt + size); @@ -350,7 +384,8 @@ static void __init __map_memblock(pgd_t *pgd, phys_addr_t start, phys_addr_t end __create_pgd_mapping(pgd, start, __phys_to_virt(start), end - start, PAGE_KERNEL, early_pgtable_alloc, - debug_pagealloc_enabled()); + debug_pagealloc_enabled(), + !debug_pagealloc_enabled()); return; } @@ -363,13 +398,15 @@ static void __init __map_memblock(pgd_t *pgd, phys_addr_t start, phys_addr_t end __phys_to_virt(start), kernel_start - start, PAGE_KERNEL, early_pgtable_alloc, - debug_pagealloc_enabled()); + debug_pagealloc_enabled(), + !debug_pagealloc_enabled()); if (kernel_end < end) __create_pgd_mapping(pgd, kernel_end, __phys_to_virt(kernel_end), end - kernel_end, PAGE_KERNEL, early_pgtable_alloc, - debug_pagealloc_enabled()); + debug_pagealloc_enabled(), + !debug_pagealloc_enabled()); /* * Map the linear alias of the [_text, __init_begin) interval @@ -381,7 +418,8 @@ static void __init __map_memblock(pgd_t *pgd, phys_addr_t start, phys_addr_t end */ __create_pgd_mapping(pgd, kernel_start, __phys_to_virt(kernel_start), kernel_end - kernel_start, PAGE_KERNEL, - early_pgtable_alloc, debug_pagealloc_enabled()); + early_pgtable_alloc, debug_pagealloc_enabled(), + false); } void __init mark_linear_text_alias_ro(void) @@ -428,7 +466,8 @@ void mark_rodata_ro(void) } static void __init map_kernel_segment(pgd_t *pgd, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma) + pgprot_t prot, struct vm_struct *vma, + bool may_use_cont) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -437,7 +476,8 @@ static void __init map_kernel_segment(pgd_t *pgd, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(size)); __create_pgd_mapping(pgd, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, debug_pagealloc_enabled()); + early_pgtable_alloc, debug_pagealloc_enabled(), + !debug_pagealloc_enabled() && may_use_cont); vma->addr = va_start; vma->phys_addr = pa_start; @@ -464,14 +504,14 @@ static void __init map_kernel(pgd_t *pgd) pgprot_t text_prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; - map_kernel_segment(pgd, _text, _etext, text_prot, &vmlinux_text); + map_kernel_segment(pgd, _text, _etext, text_prot, &vmlinux_text, true); map_kernel_segment(pgd, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata); + &vmlinux_rodata, false); map_kernel_segment(pgd, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext); + &vmlinux_inittext, true); map_kernel_segment(pgd, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata); - map_kernel_segment(pgd, _data, _end, PAGE_KERNEL, &vmlinux_data); + &vmlinux_initdata, true); + map_kernel_segment(pgd, _data, _end, PAGE_KERNEL, &vmlinux_data, true); if (!pgd_val(*pgd_offset_raw(pgd, FIXADDR_START))) { /*