From patchwork Thu Nov 17 19:27:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13047225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AA89C43217 for ; Thu, 17 Nov 2022 19:29:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=G68jJ4/6EbyFhjHczU5MgowzwCjCm+nsXXos6x5qWys=; b=oduYZh1CzS5hIO T9PC4wOEqm2Ml3y8LvyTxzHLRARe+0I16CQNfeEXH9OC204b4mI83P5M5ELWMe6btZ2twEdoHKjNn Mhzs5hx4+gwZggspwvSAOUWvV0vqSUjstqsS4svxM/QpuGPfb4YEWwcm3ywxXjXEHfZihdlvIWLEA Iqftsvjrt6341WIuJ5RQ/J7kdrUqSQZFHVj5CyypGjYjnPNntjQ2QE7GqsMGPi/zf5Jy/77Jy3Usc Sr2ZKsqBewUqTMW0RKc9jgSGFUIlvRDYuBgV3FlLWFLP3tLe1r5uCkDluW3c7PypDCSY03SSFQGkk Yg0MiyMcsjU2DsjyGTQA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ovkYO-00Gz6I-Ek; Thu, 17 Nov 2022 19:28:04 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ovkYL-00Gz5j-15 for linux-arm-kernel@lists.infradead.org; Thu, 17 Nov 2022 19:28:02 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BA6FAB82177; Thu, 17 Nov 2022 19:27:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1D1B2C433D6; Thu, 17 Nov 2022 19:27:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668713278; bh=vNpTMYVpufCuaabwna1XSBwtNghw6FW4VH7bkkXFVB0=; h=From:To:Cc:Subject:Date:From; b=Td5MdI1UnTJ41/LG5m1Foi6tnK70zksy1+xYi+FD70I0DznMn6dWseWZ6S4qTNH3n KGCf8prJ3tgFw69W0haGHmYLJXMrr8TIEijCdUdacACaH3KvdK2GDXVQGs2tvm/rOe KV4mTMxIGfvLRFnyd+DJVIiH1hdRCJ0XrM+KYJ9mwNCszw3MdJQg3FTh8IGAi4wuMD 7KlT9+JsE+8s8Z2Y5m/FdiAVeZJEz1ORdwtu1b185iLUWtDbuyHVeLXwo6b9s+kGZO 1CxoudZoBEEy8luUTqjZX4hx3g/qgTF20oWag7hdKiwmc7S3nle+Kfou+582ORI0y+ lAMYr8ySnwVsg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Anshuman Khandual , Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , Vincenzo Frascino Subject: [PATCH] arm64: mm: Enable KASAN for 16k/48-bit VA configurations Date: Thu, 17 Nov 2022 20:27:51 +0100 Message-Id: <20221117192751.1956624-1-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4893; i=ardb@kernel.org; h=from:subject; bh=vNpTMYVpufCuaabwna1XSBwtNghw6FW4VH7bkkXFVB0=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjdos3Qz97jAgJyt6t7qfQPtNoRhYgfaM54t1WAVdd 6pEqdveJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY3aLNwAKCRDDTyI5ktmPJJ/GC/ 9HRAo3zRO2Vku2vYsQ0IbgsiHhRzbthEjfyDXFKCjErUCMOrvkztrMl9RvImnPNWGgA2sKkUhxWrp2 YjufJdYh4V40LYYTKIdYKho3oukrwVku9nUMEv2gRBGOBmnBDVXB6yeQdhiW6fqywWEavj4RQJXft0 C+GA6c4DlSvzbdDmBDJ3Tto7gUW+2NlaLV2ZuuN00WKrpicnDSVa3DkoXLSd8wswgWmaJe6k9b/hO6 yxDC/tJ5o0tsUinjLOyGvBukNxOEZ3tweh8QqMyI9IySNcOgY3s9nUp5S4vyms0rrCRIz2/h2XyUqk NPDYmrr7iGPcnxg+pfZNVtIFPVD8P+23NdW1QbmqkfT0txMq0Edoj5Kp/W9R9lMs9CjYSsNh7bt7k9 6Nak6efRR5TI7/4AyWmFTg0J1uDUasTXQFy+NxTh/PWbrF2Vq8OxY8NmlSStDfybc8bq2F1rkyaK5M +igN5Nc9MPDv3rIIIheTh5nNu20QNUQG4MTPniyIDqscs= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221117_112801_374343_C52B7937 X-CRM114-Status: GOOD ( 21.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We never bothered with enabling KASAN on 16k pages configurations with 48-bit virtual addressing, given that you could get 47 bits of VA space with 3 levels of paging instead of 4. However, once we enable LPA2 support, 4 levels of paging are needed to access the entire 52-bit VA space, making this configuration more relevant, as it is what you get when you run such a kernel on hardware that does not implement LPA2. The KASAN init code assumes that the shadow memory subhierarchy does not share any top level PGD entries with other users of the kernel VA space, but given that 16k/48-bit only has two top level entries (as this level only translates a single bit in the VA), the init code needs some rework to deal with this specific case. So relax the build time requirement that the start of the shadow memory region is PGDIR_SIZE aligned, and if it is not, recurse one level and clone the next level page table as well as the top level one. Note that the end address of the shadow region is still guaranteed to be PGDIR_SIZE aligned, and so the shadow region is always covered by a single pgd_t entry, and therefore a single pud_t[] table. Cc: Will Deacon Cc: Mark Rutland Cc: Kees Cook Cc: Catalin Marinas Cc: Anshuman Khandual Cc: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Vincenzo Frascino Signed-off-by: Ard Biesheuvel --- arch/arm64/Kconfig | 2 +- arch/arm64/mm/kasan_init.c | 34 ++++++++++++++++++-- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 505c8a1ccbe0cd04..7a3f109bfd69dec9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -153,7 +153,7 @@ config ARM64 select HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE - select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) + select HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_VMALLOC if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index e969e68de005fd2a..2c1869102dfc7beb 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -170,8 +170,8 @@ asmlinkage void __init kasan_early_init(void) { BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), PGDIR_SIZE)); - BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), PGDIR_SIZE)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), PUD_SIZE)); + BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), PUD_SIZE)); BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, PGDIR_SIZE)); kasan_pgd_populate(KASAN_SHADOW_START, KASAN_SHADOW_END, NUMA_NO_NODE, true); @@ -211,6 +211,15 @@ static void __init clear_pgds(unsigned long start, set_pgd(pgd_offset_k(start), __pgd(0)); } +static void __init clear_puds(unsigned long start, unsigned long end) +{ + pgd_t *pgd = pgd_offset_k(start); + pud_t *pud = pud_offset_kimg(p4d_offset(pgd, start), start); + + for (; start < end; start += PUD_SIZE) + set_pud(pud++, __pud(0)); +} + static void __init kasan_init_shadow(void) { u64 kimg_shadow_start, kimg_shadow_end; @@ -235,10 +244,29 @@ static void __init kasan_init_shadow(void) * setup will be finished. */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); + if (KASAN_SHADOW_START % PGDIR_SIZE) { + /* + * We need some special handling when the shadow memory range + * shares a pgd_t entry with the linear region. This can only + * happen when running with 16k pages/48-bit VAs, in which case + * the root level table has only two entries. + */ + static pud_t pud[PTRS_PER_PUD] __initdata __aligned(PAGE_SIZE); + u64 addr = KASAN_SHADOW_START & PGDIR_MASK; + pgd_t *pgdp = pgd_offset_k(addr); + pud_t *pudp = pud_offset_kimg(p4d_offset(pgdp, addr), addr); + + memcpy(pud, pudp, sizeof(pud)); + tmp_pg_dir[0] = __pgd(__phys_to_pgd_val(__pa_symbol(pud)) | + PUD_TYPE_TABLE); + } dsb(ishst); cpu_replace_ttbr1(lm_alias(tmp_pg_dir), idmap_pg_dir); - clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); + if (KASAN_SHADOW_START % PGDIR_SIZE) + clear_puds(KASAN_SHADOW_START, KASAN_SHADOW_END); + else + clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); kasan_map_populate(kimg_shadow_start, kimg_shadow_end, early_pfn_to_nid(virt_to_pfn(lm_alias(KERNEL_START))));