From patchwork Tue Feb 4 17:33:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BCCA3C02198 for ; Tue, 4 Feb 2025 17:35:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=266Z0AdIHBLz7GPfIHnFsCFQcOTdIv7KmJ6/mc6CgOc=; b=mwRmjTEZ6rCICb e7XWzV90hf9EUYSy6+Kd72Or2FvJuKBEb3dR8ZOsyVYQjZCEeLea2iB1Kd30+Tw7yh27zgT3PPcwm uLptXGNzZx+3a0JXJtuI59uOGJnuq/jQviMzvBpAA0cr8zdPUwhq3P9H0TDOs+3SL67ninkfbTnnq CQsTN/RHKYXR4ngxGA183WhkPQESfflvK7OQULmmvnZCQ6TQvL14uI2uUMINryysljk1n/LjEcf1D PiMmHDr1rn1t2pjsdsx7JgaRTMHJd7+fRXNQ/LKJpe+03NWVjmCrJAzLjlNa6ITT+e+/4febMYTcG +rWnmJkalUY9+VPNvQUg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMq0-0000000174o-0rFI; Tue, 04 Feb 2025 17:35:52 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMoz-000000016fx-30yw; Tue, 04 Feb 2025 17:34:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690489; x=1770226489; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LrFP/S58o6ICJkrRaOG0zWteTvtIiSjG9jGn/c23LW4=; b=Ie/fh6xiqA66AZcYQ4Y6orvCz4F+I7OgQtmMq3hkvXhFg0wI6qGc1M0l 16382plnKRpFu35/2VUtIw8fLA0L0AO4pOfxriHivoBuXqooHcjtcYlzU PrgKOqI0VL9F/WYaqj8pGDZo2vNjQHTazzFSjhwmWx5ST7uCaqYnYvR2S isXiBe4usojrGFk2FQ0xAMTsmwRthgr5JSGN4sFcurUYZe0yiTNErIbeR qfwabJrGAwTf3HF1clPTzspGTBHGA2gBiDT5F8gx/FIeSDi3hjnnLF5Pi vWQB8qNxEMS7npas28kV5fDS5kS1n31hvv7hQA/YNkDgZn+2KBqqC6M0S Q==; X-CSE-ConnectionGUID: 7jFYQz1fSqaUsTg2rtrOOg== X-CSE-MsgGUID: YCZqZDfwTj6A+Mp2M4ZQPw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930327" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930327" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:34:49 -0800 X-CSE-ConnectionGUID: hsBgrPAEQzS1k8EGLwQyaA== X-CSE-MsgGUID: MO+fRiNSTeubDAn41XyOaA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866143" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:34:37 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 01/15] kasan: Allocation enhancement for dense tag-based mode Date: Tue, 4 Feb 2025 18:33:42 +0100 Message-ID: <808cc6516f47d5f5e811d2c237983767952f3743.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093449_826765_1BC56646 X-CRM114-Status: GOOD ( 21.26 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Tag-based KASAN (on arm64) works by generating a random 8-bit tag and putting it in both the top byte of the pointer (that points to the allocated memory) and into all bytes of shadow memory that correspond to the chunk of allocated regular memory. Each byte of shadow memory covers a 16 byte chunk of allocated memory - a value called KASAN granularity. This means that out-of-bounds memory accesses that happen inside the 16 bytes can't be caught. The dense mode offers reducing the tag width from 8 to 4 bits and storing two tags in one byte of shadow memory - one in the upper 4 bits of the byte and one in the lower 4. This way one byte of shadow memory can cover 32 bytes of allocated memory while still keeping the "16 bytes per one tag" granularity. The lower 4 bits of each shadow byte map bytes of memory with offsets 0-15 and the upper 4 bits map offsets 16-31. Example: The example below shows how the shadow memory looks like after allocating 48 bytes of memory in both normal tag-based mode and the dense mode. The contents of shadow memory are overlaid onto address offsets that they relate to in the allocated kernel memory. Each cell | | symbolizes one byte of shadow memory. = The regular tag based mode: - Randomly generated 8-bit tag equals 0xAB. - 0xFE is the tag that symbolizes unallocated memory. Shadow memory contents: | 0xAB | 0xAB | 0xAB | 0xFE | Shadow memory address offsets: 0 1 2 3 4 Allocated memory address offsets: 0 16 32 48 64 = The dense tag based mode: - Randomly generated 4-bit tag equals 0xC. - 0xE is the tag that symbolizes unallocated memory. Shadow memory contents: |0xC 0xC |0xC 0xE |0xE 0xE |0xE 0xE | Shadow memory address offsets: 0 1 2 3 4 Allocated memory address offsets: 0 32 64 96 128 Add a new config option and defines that can override the standard system of one tag per one shadow byte. Add alternative version of the kasan_poison() that deals with tags not being aligned to byte size in shadow memory. Signed-off-by: Maciej Wieczor-Retman --- include/linux/kasan.h | 18 ++++++++++++++++++ lib/Kconfig.kasan | 21 +++++++++++++++++++++ mm/kasan/kasan.h | 4 +--- mm/kasan/shadow.c | 33 ++++++++++++++++++++++++++++++--- 4 files changed, 70 insertions(+), 6 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 03b440658817..ea0f5acd875b 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -35,6 +35,24 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t; /* Software KASAN implementations use shadow memory. */ +#ifdef CONFIG_KASAN_SW_TAGS_DENSE +#define KASAN_GRANULE_SHIFT (KASAN_SHADOW_SCALE_SHIFT - 1) +#define KASAN_SHADOW_SCALE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT) +static inline u8 kasan_dense_tag(u8 tag) +{ + return (tag << KASAN_TAG_WIDTH | tag); +} +#else +#define KASAN_GRANULE_SHIFT KASAN_SHADOW_SCALE_SHIFT +#define KASAN_SHADOW_SCALE_SIZE (1UL << KASAN_GRANULE_SHIFT) +static inline u8 kasan_dense_tag(u8 tag) +{ + return tag; +} +#endif + +#define KASAN_GRANULE_SIZE (1UL << KASAN_GRANULE_SHIFT) + #ifdef CONFIG_KASAN_SW_TAGS /* This matches KASAN_TAG_INVALID. */ #define KASAN_SHADOW_INIT 0xFE diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan index 98016e137b7f..d08b4e9bf477 100644 --- a/lib/Kconfig.kasan +++ b/lib/Kconfig.kasan @@ -19,6 +19,13 @@ config ARCH_DISABLE_KASAN_INLINE Disables both inline and stack instrumentation. Selected by architectures that do not support these instrumentation types. +config ARCH_HAS_KASAN_SW_TAGS_DENSE + bool + help + Enables option to compile tag-based KASAN with densely packed tags - + two 4-bit tags per one byte of shadow memory. Set on architectures + that have 4-bit tag macros. + config CC_HAS_KASAN_GENERIC def_bool $(cc-option, -fsanitize=kernel-address) @@ -223,4 +230,18 @@ config KASAN_EXTRA_INFO boot parameter, it will add 8 * stack_ring_size bytes of additional memory consumption. +config KASAN_SW_TAGS_DENSE + bool "Two 4-bit tags in one shadow memory byte" + depends on KASAN_SW_TAGS + depends on ARCH_HAS_KASAN_SW_TAGS_DENSE + help + Enables packing two tags into one shadow byte to half the memory usage + compared to normal tag-based mode. + + After setting this option, tag width macro is set to 4 and size macros + are adjusted based on used KASAN_SHADOW_SCALE_SHIFT. + + ARCH_HAS_KASAN_SW_TAGS_DENSE is needed for this option since the + special tag macros need to be properly set for 4-bit wide tags. + endif # KASAN diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 72da5ddcceaa..0e04c5e2c405 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -128,9 +128,7 @@ static inline bool kasan_requires_meta(void) #endif /* CONFIG_KASAN_GENERIC */ -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) -#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT) -#else +#ifdef CONFIG_KASAN_HW_TAGS #include #define KASAN_GRANULE_SIZE MTE_GRANULE_SIZE #endif diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index d6210ca48dda..368503f54b87 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -123,7 +123,8 @@ EXPORT_SYMBOL(__hwasan_memcpy); void kasan_poison(const void *addr, size_t size, u8 value, bool init) { - void *shadow_start, *shadow_end; + u8 *shadow_start, *shadow_end, *shadow_start_aligned, *shadow_end_aligned, tag; + u64 addr64, addr_start_aligned, addr_end_aligned; if (!kasan_arch_is_ready()) return; @@ -134,16 +135,42 @@ void kasan_poison(const void *addr, size_t size, u8 value, bool init) * addresses to this function. */ addr = kasan_reset_tag(addr); + addr64 = (u64)addr; - if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + if (WARN_ON(addr64 & KASAN_GRANULE_MASK)) return; if (WARN_ON(size & KASAN_GRANULE_MASK)) return; shadow_start = kasan_mem_to_shadow(addr); shadow_end = kasan_mem_to_shadow(addr + size); + addr_start_aligned = round_up(addr64, KASAN_SHADOW_SCALE_SIZE); + addr_end_aligned = round_down(addr64 + size, KASAN_SHADOW_SCALE_SIZE); + shadow_start_aligned = kasan_mem_to_shadow((void *)addr_start_aligned); + shadow_end_aligned = kasan_mem_to_shadow((void *)addr_end_aligned); + + /* If size is empty just return. */ + if (!size) + return; - __memset(shadow_start, value, shadow_end - shadow_start); + /* Memset the first unaligned tag in shadow memory. */ + if (addr64 % KASAN_SHADOW_SCALE_SIZE) { + tag = *shadow_start & KASAN_TAG_MASK; + tag |= value << KASAN_TAG_WIDTH; + *shadow_start = tag; + } + + /* Memset the middle aligned part in shadow memory. */ + tag = kasan_dense_tag(value); + __memset(shadow_start_aligned, tag, shadow_end_aligned - shadow_start_aligned); + + /* Memset the last unaligned tag in shadow memory. */ + if ((addr64 + size) % KASAN_SHADOW_SCALE_SIZE) { + tag = KASAN_TAG_MASK << KASAN_TAG_WIDTH; + tag &= *shadow_end; + tag |= value; + *shadow_end = tag; + } } EXPORT_SYMBOL_GPL(kasan_poison); From patchwork Tue Feb 4 17:33:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F13E1C02193 for ; Tue, 4 Feb 2025 17:35:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mtUbipf6E5trEUN4gltyrniLzBOrLHYPamENTgBi/VQ=; b=NazkMeizoUCcqv 11T0m57J6VikBs4kMyzkAHZO25XyWdGUum/XrDerHMBJbNoTA4lu5VaHr99PZ0tM9p2BwOrXKrOqS Lp+MENxjtBfpjcDInTcdi08n1n8GkNfc6H/bsbxSZKefwvZG4kcnpLj67BsGr6D/+306LzboE4GHq YJwYhAeuVQulxgd0kKtlYjjEkf5zbQ9EFATPa3PVm3trYRBFBbIIttXhVjoUzJE6QbqGBWhc1bUNv pyZFPfuJzG2HE8jDFi/CWn4jmS95hsCPrP0HTZdrfUazOpos7biOSUAZz59KRPLB1npxxBy6GdQqD LIz2r+Nkj2QhrL0NX0jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMq0-00000001759-3hH8; Tue, 04 Feb 2025 17:35:52 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMpG-000000016n6-0EMs; Tue, 04 Feb 2025 17:35:07 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690506; x=1770226506; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aHGhpjph+eDn968NL6XwKtEp1WbZxR8Rwtm1kIwOPTc=; b=Rufk0rAbWg2sNISQsQphLLXIzm50vQe2J5XKJ47wS9ocHJVCOz1dc2RV a3p30EuwmX6PAv9QhFiy+tNnjk2MF+gZs8I6dvJvnX/j0GESFppww3LAj s+l68OXJW4LrJXyMpA/5lG0vz70+lxlgNJ7jP1oXOPhAaNOb6FDwfhu+h TJPxPOQSC1Bsw3IWCvkNCfi4n00u/a758N6LyOqOmcTd8QS659Yz+IBBY gwa40HkYNni2MxC4OiQyjXhjFgDAzW0Y6VtvKIy9B89jOWPRPZHrgOoMc ygTZcJNK/CapE0pJod+by0Wtg02AiGo+iBg8NBxdGKnYtbSHElAutvgEK g==; X-CSE-ConnectionGUID: LxMqgJH+Qa6m9zsKd4RPUQ== X-CSE-MsgGUID: vopx7qgpRqa6j0gTRDKR7Q== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930394" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930394" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:04 -0800 X-CSE-ConnectionGUID: eBbq5MWaSZONJhS+dGPi8A== X-CSE-MsgGUID: mjasgO5WTcSBmxRL24BE0w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866217" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:34:49 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 02/15] kasan: Tag checking with dense tag-based mode Date: Tue, 4 Feb 2025 18:33:43 +0100 Message-ID: <8f790bb7e166c1ea2e5003318149eb1d7aba3596.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093506_214183_9B3AEFF8 X-CRM114-Status: GOOD ( 28.09 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In KASAN's tag-based mode (arm64) when a memory access occurs, the tag stored in the top 8 bits of the pointer is compared with tags saved in the region of the shadow memory that maps to memory the pointer points to. If any of the tags in the shadow memory region do not match the one stored in the pointer an error report is generated. With the introduction of the dense mode, tags won't necessarily occupy whole bytes of shadow memory if the previously allocated memory wasn't aligned to 32 bytes - which is the coverage of one shadow byte. Add an alternative implementation of kasan_check_range() that performs special checks on first and last bytes of shadow memory ranges if the originally allocated memory wasn't aligned to 32 bytes. Signed-off-by: Maciej Wieczor-Retman --- include/linux/kasan.h | 47 +++++++++++++++------- mm/kasan/Makefile | 3 ++ mm/kasan/dense.c | 83 +++++++++++++++++++++++++++++++++++++++ mm/kasan/kasan.h | 2 +- mm/kasan/report.c | 2 +- mm/kasan/report_sw_tags.c | 12 ++---- mm/kasan/sw_tags.c | 8 ++++ 7 files changed, 133 insertions(+), 24 deletions(-) create mode 100644 mm/kasan/dense.c diff --git a/include/linux/kasan.h b/include/linux/kasan.h index ea0f5acd875b..5a3e9bec21c2 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -33,6 +33,20 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t; #include +#ifndef kasan_mem_to_shadow +static inline void *kasan_mem_to_shadow(const void *addr) +{ + void *scaled; + + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT); + else + scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT); + + return KASAN_SHADOW_OFFSET + scaled; +} +#endif + /* Software KASAN implementations use shadow memory. */ #ifdef CONFIG_KASAN_SW_TAGS_DENSE @@ -53,6 +67,25 @@ static inline u8 kasan_dense_tag(u8 tag) #define KASAN_GRANULE_SIZE (1UL << KASAN_GRANULE_SHIFT) +#ifdef CONFIG_KASAN_SW_TAGS_DENSE +static inline u8 kasan_get_shadow_tag(const void *ptr) +{ + u8 shadow_byte = *(u8 *)kasan_mem_to_shadow(ptr); + unsigned long addr = (unsigned long)ptr; + int shift; + + shift = !!(addr & KASAN_GRANULE_SIZE) * KASAN_TAG_WIDTH; + shadow_byte >>= shift; + + return shadow_byte & KASAN_TAG_KERNEL; +} +#else +static inline u8 kasan_get_shadow_tag(const void *addr) +{ + return (*(u8 *)kasan_mem_to_shadow(addr)); +} +#endif + #ifdef CONFIG_KASAN_SW_TAGS /* This matches KASAN_TAG_INVALID. */ #define KASAN_SHADOW_INIT 0xFE @@ -73,20 +106,6 @@ extern p4d_t kasan_early_shadow_p4d[MAX_PTRS_PER_P4D]; int kasan_populate_early_shadow(const void *shadow_start, const void *shadow_end); -#ifndef kasan_mem_to_shadow -static inline void *kasan_mem_to_shadow(const void *addr) -{ - void *scaled; - - if (IS_ENABLED(CONFIG_KASAN_GENERIC)) - scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT); - else - scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT); - - return KASAN_SHADOW_OFFSET + scaled; -} -#endif - int kasan_add_zero_shadow(void *start, unsigned long size); void kasan_remove_zero_shadow(void *start, unsigned long size); diff --git a/mm/kasan/Makefile b/mm/kasan/Makefile index b88543e5c0cc..3a460abd4c18 100644 --- a/mm/kasan/Makefile +++ b/mm/kasan/Makefile @@ -5,6 +5,7 @@ KCOV_INSTRUMENT := n # Disable ftrace to avoid recursion. CFLAGS_REMOVE_common.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_dense.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_generic.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_init.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_quarantine.o = $(CC_FLAGS_FTRACE) @@ -24,6 +25,7 @@ CC_FLAGS_KASAN_RUNTIME += -fno-stack-protector CC_FLAGS_KASAN_RUNTIME += -DDISABLE_BRANCH_PROFILING CFLAGS_common.o := $(CC_FLAGS_KASAN_RUNTIME) +CFLAGS_dense.o := $(CC_FLAGS_KASAN_RUNTIME) CFLAGS_generic.o := $(CC_FLAGS_KASAN_RUNTIME) CFLAGS_init.o := $(CC_FLAGS_KASAN_RUNTIME) CFLAGS_quarantine.o := $(CC_FLAGS_KASAN_RUNTIME) @@ -49,6 +51,7 @@ RUSTFLAGS_kasan_test_rust.o := $(RUSTFLAGS_KASAN) CFLAGS_kasan_test_module.o := $(CFLAGS_KASAN_TEST) obj-y := common.o report.o +obj-$(CONFIG_KASAN_SW_TAGS_DENSE) += dense.o obj-$(CONFIG_KASAN_GENERIC) += init.o generic.o report_generic.o shadow.o quarantine.o obj-$(CONFIG_KASAN_HW_TAGS) += hw_tags.o report_hw_tags.o tags.o report_tags.o obj-$(CONFIG_KASAN_SW_TAGS) += init.o report_sw_tags.o shadow.o sw_tags.o tags.o report_tags.o diff --git a/mm/kasan/dense.c b/mm/kasan/dense.c new file mode 100644 index 000000000000..306bbbfdce29 --- /dev/null +++ b/mm/kasan/dense.c @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "kasan.h" + +static __always_inline bool kasan_check_range_inline(const void *addr, + size_t size, bool write, + unsigned long ret_ip) +{ + u8 *shadow_first, *shadow_last, *shadow, *shadow_first_aligned, *shadow_last_aligned; + u64 addr_start_aligned, addr_end_aligned; + u8 tag, kasan_granule_offset; + size_t aligned_size; + void *untagged_addr; + + if (unlikely(size == 0)) + return true; + + if (unlikely(addr + size < addr)) + return !kasan_report(addr, size, write, ret_ip); + + tag = get_tag((const void *)addr); + + /* + * Ignore accesses for pointers tagged with native kernel + * pointer tag to suppress false positives caused by kmap. + * + * Some kernel code was written to account for archs that don't keep + * high memory mapped all the time, but rather map and unmap particular + * pages when needed. Instead of storing a pointer to the kernel memory, + * this code saves the address of the page structure and offset within + * that page for later use. Those pages are then mapped and unmapped + * with kmap/kunmap when necessary and virt_to_page is used to get the + * virtual address of the page. For arm64 (that keeps the high memory + * mapped all the time), kmap is turned into a page_address call. + + * The issue is that with use of the page_address + virt_to_page + * sequence the top byte value of the original pointer gets lost (gets + * set to KASAN_TAG_KERNEL). + */ + if (tag == KASAN_TAG_KERNEL) + return true; + + untagged_addr = kasan_reset_tag((void *)round_down((u64)addr, KASAN_GRANULE_SIZE)); + if (unlikely(!addr_has_metadata(untagged_addr))) + return !kasan_report(addr, size, write, ret_ip); + + kasan_granule_offset = ((u64)addr & KASAN_GRANULE_MASK); + aligned_size = round_up(size + kasan_granule_offset, KASAN_GRANULE_SIZE); + shadow_first = kasan_mem_to_shadow(untagged_addr); + shadow_last = kasan_mem_to_shadow(untagged_addr + aligned_size); + addr_start_aligned = round_up((u64)untagged_addr, KASAN_SHADOW_SCALE_SIZE); + addr_end_aligned = round_down((u64)untagged_addr + aligned_size, KASAN_SHADOW_SCALE_SIZE); + shadow_first_aligned = kasan_mem_to_shadow((void *)addr_start_aligned); + shadow_last_aligned = kasan_mem_to_shadow((void *)addr_end_aligned); + + /* Check the first unaligned tag in shadow memory. */ + if ((u64)untagged_addr % KASAN_SHADOW_SCALE_SIZE) { + if (unlikely((*shadow_first >> KASAN_TAG_WIDTH) != tag)) + return !kasan_report(addr, size, write, ret_ip); + } + + /* Check the middle aligned part in shadow memory. */ + for (shadow = shadow_first_aligned; shadow < shadow_last_aligned; shadow++) { + if (unlikely(*shadow != ((tag << KASAN_TAG_WIDTH) | tag))) + return !kasan_report(addr, size, write, ret_ip); + } + + /* Check the last unaligned tag in shadow memory. */ + if (((u64)untagged_addr + aligned_size) % KASAN_SHADOW_SCALE_SIZE) { + if (unlikely((*shadow_last & KASAN_TAG_MASK) != tag)) + return !kasan_report(addr, size, write, ret_ip); + } + + return true; +} + +#if IS_ENABLED(CONFIG_KASAN_SW_TAGS_DENSE) +bool kasan_check_range(const void *addr, size_t size, bool write, + unsigned long ret_ip) +{ + return kasan_check_range_inline(addr, size, write, ret_ip); +} +#endif diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 0e04c5e2c405..d29bd0e65020 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -183,7 +183,7 @@ static inline bool kasan_requires_meta(void) #define META_BYTES_PER_BLOCK 1 #define META_BLOCKS_PER_ROW 16 #define META_BYTES_PER_ROW (META_BLOCKS_PER_ROW * META_BYTES_PER_BLOCK) -#define META_MEM_BYTES_PER_ROW (META_BYTES_PER_ROW * KASAN_GRANULE_SIZE) +#define META_MEM_BYTES_PER_ROW (META_BYTES_PER_ROW * KASAN_SHADOW_SCALE_SIZE) #define META_ROWS_AROUND_ADDR 2 #define KASAN_STACK_DEPTH 64 diff --git a/mm/kasan/report.c b/mm/kasan/report.c index c08097715686..ee9e406b0cdb 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -436,7 +436,7 @@ static int meta_pointer_offset(const void *row, const void *addr) * plus 1 byte for space. */ return 3 + (BITS_PER_LONG / 8) * 2 + - (addr - row) / KASAN_GRANULE_SIZE * 3 + 1; + (addr - row) / KASAN_SHADOW_SCALE_SIZE * 3 + 1; } static void print_memory_metadata(const void *addr) diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c index 689e94f9fe3c..1ac5c7a9011d 100644 --- a/mm/kasan/report_sw_tags.c +++ b/mm/kasan/report_sw_tags.c @@ -39,7 +39,7 @@ const void *kasan_find_first_bad_addr(const void *addr, size_t size) if (!addr_has_metadata(p)) return p; - while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p)) + while (p < end && tag == kasan_get_shadow_tag(p)) p += KASAN_GRANULE_SIZE; return p; @@ -48,7 +48,6 @@ const void *kasan_find_first_bad_addr(const void *addr, size_t size) size_t kasan_get_alloc_size(void *object, struct kmem_cache *cache) { size_t size = 0; - u8 *shadow; /* * Skip the addr_has_metadata check, as this function only operates on @@ -59,13 +58,11 @@ size_t kasan_get_alloc_size(void *object, struct kmem_cache *cache) * The loop below returns 0 for freed objects, for which KASAN cannot * calculate the allocation size based on the metadata. */ - shadow = (u8 *)kasan_mem_to_shadow(object); while (size < cache->object_size) { - if (*shadow != KASAN_TAG_INVALID) + if (kasan_get_shadow_tag(object + size) != KASAN_TAG_INVALID) size += KASAN_GRANULE_SIZE; else return size; - shadow++; } return cache->object_size; @@ -78,9 +75,8 @@ void kasan_metadata_fetch_row(char *buffer, void *row) void kasan_print_tags(u8 addr_tag, const void *addr) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(addr); - - pr_err("Pointer tag: [%02x], memory tag: [%02x]\n", addr_tag, *shadow); + pr_err("Pointer tag: [%02x], memory tag: [%02x]\n", addr_tag, + kasan_get_shadow_tag(addr)); } #ifdef CONFIG_KASAN_STACK diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c index 32435d33583a..7a6b8ea9bf78 100644 --- a/mm/kasan/sw_tags.c +++ b/mm/kasan/sw_tags.c @@ -79,6 +79,7 @@ u8 __hwasan_generate_tag(void) } EXPORT_SYMBOL(__hwasan_generate_tag); +#if !IS_ENABLED(CONFIG_KASAN_SW_TAGS_DENSE) bool kasan_check_range(const void *addr, size_t size, bool write, unsigned long ret_ip) { @@ -127,17 +128,24 @@ bool kasan_check_range(const void *addr, size_t size, bool write, return true; } +#endif bool kasan_byte_accessible(const void *addr) { u8 tag = get_tag(addr); void *untagged_addr = kasan_reset_tag(addr); u8 shadow_byte; + int shift; if (!addr_has_metadata(untagged_addr)) return false; shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(untagged_addr)); + if (IS_ENABLED(CONFIG_KASAN_SW_TAGS_DENSE)) { + shift = !!((u64)addr & BIT(KASAN_TAG_WIDTH)) * KASAN_TAG_WIDTH; + shadow_byte = (shadow_byte >> shift) & KASAN_TAG_KERNEL; + } + return tag == KASAN_TAG_KERNEL || tag == shadow_byte; } From patchwork Tue Feb 4 17:33:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 76CECC0219A for ; Tue, 4 Feb 2025 17:35:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ocFItDamMALhgRIOyn78H/sejLZFdevS8GLWgce6ALQ=; b=F9f3J55t4+BdsS hWEVuYOdhkgt9ndWHstv4wShCzbjPpoE3E8fOprVA4KtUEjpVUN5rWBdblxXEVGRBIWnGFawP9Vld 4sm0C/1vW7mWWcOFCNth99baUYc8+W6Zh8lxE5Mx56hJfbBKc5P9G89lstZfqxDcTQinXFVkkBQRe W2mndR/nkRFbMQ5EbFV1g6ML4GBRPJzP7aTYCYnJ4ghG88KZOKKR0QSgiFTw4XxkA6yWMXrhCCJ+F kM+gf6lmHaL9kH9Qrj6Qa8LYGJGVay9NTj4Fcam2HwiRY2f6v0yGz5ZVmNBMwowu7ZtKhdJZ7aZiX l+EijpE2YBCkwJq4Oypw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMq1-0000000176S-2V7h; Tue, 04 Feb 2025 17:35:53 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMpP-000000016rF-3H6Y; Tue, 04 Feb 2025 17:35:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690515; x=1770226515; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=edAJD92hFjQk8fjbvzxioatc9um2IkZNk+/tzcDM8p4=; b=P97rc45zonc1zwCjsnnsZZX+u6SdiysHnon46Y5Xz0gfBFvwzAobHBNe ahl8LI9wvjLuuVmERlzsyVAXJgkGPAeVMV7GPirxyhfG3ePMnaC8SFDBX IzsqMQFUHNz5gma9YG2B3H8pIKQM9XHxKgFgi2dzfo5hxnNEv9HYoTL86 e0eMsdrpHpPjdeL+0D2o3agFJbr0pmjjhHyJJT42qpGfAcA7gdhDwfoH9 M09cXgCAn8sQtHcJ5XVGV3pgMOX0rZkez8FFD0UND/BCKhOdRGGIkb+wF zjfpCzwZqs1bsQcVeU9uVMaPOkArSM7CNNHQEN6dk1UDkz8IALfGzlfn/ g==; X-CSE-ConnectionGUID: kSR95XdURfOg5YrwK9wgzA== X-CSE-MsgGUID: pQ2NyfcjS8ycPaZ6PKP0OQ== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930454" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930454" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:14 -0800 X-CSE-ConnectionGUID: GXavZLsESKizsbio/E/Srw== X-CSE-MsgGUID: e0b52Ax9SOSm68zuxVJ9gQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866342" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:02 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 03/15] kasan: Vmalloc dense tag-based mode support Date: Tue, 4 Feb 2025 18:33:44 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093515_896374_44687D34 X-CRM114-Status: GOOD ( 12.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org To use KASAN with the vmalloc allocator multiple functions are implemented that deal with full pages of memory. Many of these functions are hardcoded to deal with byte aligned shadow memory regions by using __memset(). With the introduction of the dense mode, tags won't necessarily occupy whole bytes of shadow memory if the previously allocated memory wasn't aligned to 32 bytes - which is the coverage of one shadow byte. Change __memset() calls to kasan_poison(). With dense tag-based mode enabled that will take care of any unaligned tags in shadow memory. Signed-off-by: Maciej Wieczor-Retman --- mm/kasan/kasan.h | 2 +- mm/kasan/shadow.c | 14 ++++++-------- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index d29bd0e65020..a56aadd51485 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -135,7 +135,7 @@ static inline bool kasan_requires_meta(void) #define KASAN_GRANULE_MASK (KASAN_GRANULE_SIZE - 1) -#define KASAN_MEMORY_PER_SHADOW_PAGE (KASAN_GRANULE_SIZE << PAGE_SHIFT) +#define KASAN_MEMORY_PER_SHADOW_PAGE (KASAN_SHADOW_SCALE_SIZE << PAGE_SHIFT) #ifdef CONFIG_KASAN_GENERIC #define KASAN_PAGE_FREE 0xFF /* freed page */ diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 368503f54b87..94f51046e6ae 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -332,7 +332,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, if (!page) return -ENOMEM; - __memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE); + kasan_poison((void *)page, PAGE_SIZE, KASAN_VMALLOC_INVALID, false); pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL); spin_lock(&init_mm.page_table_lock); @@ -357,9 +357,6 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size) if (!is_vmalloc_or_module_addr((void *)addr)) return 0; - shadow_start = (unsigned long)kasan_mem_to_shadow((void *)addr); - shadow_end = (unsigned long)kasan_mem_to_shadow((void *)addr + size); - /* * User Mode Linux maps enough shadow memory for all of virtual memory * at boot, so doesn't need to allocate more on vmalloc, just clear it. @@ -368,12 +365,12 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size) * reason. */ if (IS_ENABLED(CONFIG_UML)) { - __memset((void *)shadow_start, KASAN_VMALLOC_INVALID, shadow_end - shadow_start); + kasan_poison((void *)addr, size, KASAN_VMALLOC_INVALID, false); return 0; } - shadow_start = PAGE_ALIGN_DOWN(shadow_start); - shadow_end = PAGE_ALIGN(shadow_end); + shadow_start = PAGE_ALIGN_DOWN((unsigned long)kasan_mem_to_shadow((void *)addr)); + shadow_end = PAGE_ALIGN((unsigned long)kasan_mem_to_shadow((void *)addr + size)); ret = apply_to_page_range(&init_mm, shadow_start, shadow_end - shadow_start, @@ -546,7 +543,8 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end, if (shadow_end > shadow_start) { size = shadow_end - shadow_start; if (IS_ENABLED(CONFIG_UML)) { - __memset(shadow_start, KASAN_SHADOW_INIT, shadow_end - shadow_start); + kasan_poison((void *)region_start, region_start - region_end, + KASAN_VMALLOC_INVALID, false); return; } apply_to_existing_page_range(&init_mm, From patchwork Tue Feb 4 17:33:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959504 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A4A15C0219B for ; Tue, 4 Feb 2025 17:35:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hjrfz4XncdIy8qkP4c2cK+WnimtsWJF9TeaGtomms8E=; b=r0xjcOfKY3CorT xxeBo/CIHrTKO+dlKdo/Bj/3MGYYFeG4+qGj/zMs+2cROek65lCg7p5VybI0Kxzn0QlMwz5sYt4Gi 0CvI0GDvsEeQUZkmcHZYa+Hs+gQn2cHFIB1G9u0VFapG9xCR0Fud7lDwSm0M5IEKLKNN2jaDLGFjA qUbUg0De6sDd6OOv2HKjI/SxaCxXQqbKAACXneSMaaLHnHrlBmnpXBan0UbfJbo/13L1PFJUHWVIl +EBQEbYV8dcnnlC8w0V1hAj6L6rUhZKpyR0BrwGEHy9XdkWfHbkaYv9RM17tBw921K7CSzDJFUwj1 lF0tPi5CaLm1AtIShSZg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMq2-0000000177c-1PZF; Tue, 04 Feb 2025 17:35:54 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMpb-000000016vn-0FHQ; Tue, 04 Feb 2025 17:35:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690527; x=1770226527; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7SPlp9QEiWP+nB12GdV91yF1dqiTepZdICL7iZuq4h4=; b=mpFAEIXAsrrIMBAcnIvCKJz0uXT/PV3jYGdsm2nGrycSggR92krYDUDV z60eDyZ65u18zUxm1kRCJcpBr3p+37+AlhMOZ0EEIoRxJ26Mb+qdvqGbc YA/Mof5ZB8+G+LYxLDwTGj4mODCPKpqA2+eQqGH050Su+kuJD7qhZrpBR 6l4dmgLhYm8P+2Yf4UJoZhByx7cJweIzinBUzOOiF2NCADNz6VVF2dPtL ENh8gKGRbgQBLcnB0plRnGQwdOa0s2XrL8oysy4Jyxi7WlC+rG2d+HdRy uMxJtn774w/BI47RG4J1YLPe5nTY7l44Vh5BRxkmYQCuZnQ1ZnZocaT0n g==; X-CSE-ConnectionGUID: CvfOLJpaSb+3INz2lxi8Mg== X-CSE-MsgGUID: REgNd3IdS+ecdE2qEUowtA== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930504" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930504" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:25 -0800 X-CSE-ConnectionGUID: /sEOqI4kSuqYrX4FgQuozA== X-CSE-MsgGUID: zBO7dwN1RyyrRSQp4bH21Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866447" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:14 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 04/15] kasan: arm64: x86: risc-v: Make special tags arch specific Date: Tue, 4 Feb 2025 18:33:45 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093527_180432_E97A40CF X-CRM114-Status: GOOD ( 19.54 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org KASAN's tag-based mode defines multiple special tag values. They're reserved for: - Native kernel value. On arm64 it's 0xFF and it causes an early return in the tag checking function. - Invalid value. 0xFE marks an area as freed / unallocated. It's also the value that is used to initialize regions of shadow memory. - Max value. 0xFD is the highest value that can be randomly generated for a new tag. Metadata macro is also defined: - Tag width equal to 8. Tag-based mode on x86 is going to use 4 bit wide tags so all the above values need to be changed accordingly. Make tags arch specific for x86, risc-v and arm64. On x86 the values just lose the top 4 bits. Replace hardcoded kernel tag value and tag width with macros in KASAN's non-arch specific code. Signed-off-by: Maciej Wieczor-Retman --- MAINTAINERS | 2 +- arch/arm64/include/asm/kasan-tags.h | 9 +++++++++ arch/riscv/include/asm/kasan-tags.h | 12 ++++++++++++ arch/riscv/include/asm/kasan.h | 4 ---- arch/x86/include/asm/kasan-tags.h | 9 +++++++++ include/linux/kasan-tags.h | 12 +++++++++++- include/linux/kasan.h | 4 +++- include/linux/mm.h | 6 +++--- include/linux/page-flags-layout.h | 7 +------ 9 files changed, 49 insertions(+), 16 deletions(-) create mode 100644 arch/arm64/include/asm/kasan-tags.h create mode 100644 arch/riscv/include/asm/kasan-tags.h create mode 100644 arch/x86/include/asm/kasan-tags.h diff --git a/MAINTAINERS b/MAINTAINERS index b878ddc99f94..45671faa3b6f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12227,7 +12227,7 @@ L: kasan-dev@googlegroups.com S: Maintained B: https://bugzilla.kernel.org/buglist.cgi?component=Sanitizers&product=Memory%20Management F: Documentation/dev-tools/kasan.rst -F: arch/*/include/asm/*kasan.h +F: arch/*/include/asm/*kasan*.h F: arch/*/mm/kasan_init* F: include/linux/kasan*.h F: lib/Kconfig.kasan diff --git a/arch/arm64/include/asm/kasan-tags.h b/arch/arm64/include/asm/kasan-tags.h new file mode 100644 index 000000000000..9e835da95f6b --- /dev/null +++ b/arch/arm64/include/asm/kasan-tags.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_KASAN_TAGS_H +#define __ASM_KASAN_TAGS_H + +#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */ + +#define KASAN_TAG_WIDTH 8 + +#endif /* ASM_KASAN_TAGS_H */ diff --git a/arch/riscv/include/asm/kasan-tags.h b/arch/riscv/include/asm/kasan-tags.h new file mode 100644 index 000000000000..83d7dcc8af74 --- /dev/null +++ b/arch/riscv/include/asm/kasan-tags.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_KASAN_TAGS_H +#define __ASM_KASAN_TAGS_H + +#ifdef CONFIG_KASAN_SW_TAGS +#define KASAN_TAG_KERNEL 0x7f /* native kernel pointers tag */ +#endif + +#define KASAN_TAG_WIDTH 8 + +#endif /* ASM_KASAN_TAGS_H */ + diff --git a/arch/riscv/include/asm/kasan.h b/arch/riscv/include/asm/kasan.h index f6b378ba936d..27938e0d5233 100644 --- a/arch/riscv/include/asm/kasan.h +++ b/arch/riscv/include/asm/kasan.h @@ -41,10 +41,6 @@ #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) -#ifdef CONFIG_KASAN_SW_TAGS -#define KASAN_TAG_KERNEL 0x7f /* native kernel pointers tag */ -#endif - #define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) #define arch_kasan_reset_tag(addr) __tag_reset(addr) #define arch_kasan_get_tag(addr) __tag_get(addr) diff --git a/arch/x86/include/asm/kasan-tags.h b/arch/x86/include/asm/kasan-tags.h new file mode 100644 index 000000000000..68ba385bc75c --- /dev/null +++ b/arch/x86/include/asm/kasan-tags.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_KASAN_TAGS_H +#define __ASM_KASAN_TAGS_H + +#define KASAN_TAG_KERNEL 0xF /* native kernel pointers tag */ + +#define KASAN_TAG_WIDTH 4 + +#endif /* ASM_KASAN_TAGS_H */ diff --git a/include/linux/kasan-tags.h b/include/linux/kasan-tags.h index e07c896f95d3..b4aacfa8709b 100644 --- a/include/linux/kasan-tags.h +++ b/include/linux/kasan-tags.h @@ -2,7 +2,17 @@ #ifndef _LINUX_KASAN_TAGS_H #define _LINUX_KASAN_TAGS_H -#include +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) +#include +#endif + +#ifdef CONFIG_KASAN_SW_TAGS_DENSE +#define KASAN_TAG_WIDTH 4 +#endif + +#ifndef KASAN_TAG_WIDTH +#define KASAN_TAG_WIDTH 0 +#endif #ifndef KASAN_TAG_KERNEL #define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */ diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 5a3e9bec21c2..83146367170a 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -88,7 +88,9 @@ static inline u8 kasan_get_shadow_tag(const void *addr) #ifdef CONFIG_KASAN_SW_TAGS /* This matches KASAN_TAG_INVALID. */ -#define KASAN_SHADOW_INIT 0xFE +#ifndef KASAN_SHADOW_INIT +#define KASAN_SHADOW_INIT KASAN_TAG_INVALID +#endif #else #define KASAN_SHADOW_INIT 0 #endif diff --git a/include/linux/mm.h b/include/linux/mm.h index 61fff5d34ed5..ddca2f63a5f6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1813,7 +1813,7 @@ static inline u8 page_kasan_tag(const struct page *page) if (kasan_enabled()) { tag = (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK; - tag ^= 0xff; + tag ^= KASAN_TAG_KERNEL; } return tag; @@ -1826,7 +1826,7 @@ static inline void page_kasan_tag_set(struct page *page, u8 tag) if (!kasan_enabled()) return; - tag ^= 0xff; + tag ^= KASAN_TAG_KERNEL; old_flags = READ_ONCE(page->flags); do { flags = old_flags; @@ -1845,7 +1845,7 @@ static inline void page_kasan_tag_reset(struct page *page) static inline u8 page_kasan_tag(const struct page *page) { - return 0xff; + return KASAN_TAG_KERNEL; } static inline void page_kasan_tag_set(struct page *page, u8 tag) { } diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-layout.h index 7d79818dc065..ac3576f409ad 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -3,6 +3,7 @@ #define PAGE_FLAGS_LAYOUT_H #include +#include #include /* @@ -72,12 +73,6 @@ #define NODE_NOT_IN_PAGE_FLAGS 1 #endif -#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) -#define KASAN_TAG_WIDTH 8 -#else -#define KASAN_TAG_WIDTH 0 -#endif - #ifdef CONFIG_NUMA_BALANCING #define LAST__PID_SHIFT 8 #define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1) From patchwork Tue Feb 4 17:33:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E1EAAC0219D for ; Tue, 4 Feb 2025 17:35:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cveJAn0D4t6XLwa+MD14Uup3ws5+nFVvFO51pQcT+Vk=; b=dQh3ft+PT+IICT gNaHd+2LpArmP7rGesVEQ8BfiY8OgM7VsSFL3LrLRGY4UPtGuJrG6SHsqudLCyWZz3l3gKqtPlKrn lOUQyt/vwKyIqPhsLRXGQK/JJ1Kb8zLyyCGnt4Qv0JaY4rnhJrlJPfaN+8/5rmVmsxCuzlQQhLZJV bHdMLiqFxa/flrckfbhATNAeqnJv0E2qjw66LEgA46NWHrYIgRoETBNbfUUitKY4igo2fnQmvdUco eCEKDiKXZt0hPIIjfW8MrvvGWccH+kLusiX/ktIpbv8bys96dCk/6uaP2UQjiIUvY4tZ12c4uhOcr AlbUH5ub8eA8oogRz2Nw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMq3-0000000178c-2jrD; Tue, 04 Feb 2025 17:35:55 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMpn-000000016zx-23B7; Tue, 04 Feb 2025 17:35:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690539; x=1770226539; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1nix+N5DUro8RwKo5+pglZtqohhVbR1ohGEV2E0y1dM=; b=YrldGYiJkWVWWXF5kYu19Jo1gWt5VOhdWzSu6yg/TefEg3Qif1IhKVIi 6HAqZuM1sUQMcSmX1TmbferFMZNToJKyV/9u/ht9wPSuYaTJye5U0yztV YJEEvr9AmO2OX37iDhCQ3vP1YJbLY9zvw4oDZq0RvTNxSJmh9Mv9+94Kn uLX8TXNUFC5FQq47iZVcACvsS12swwtk9kX5c9OjHBwTbWFwez+/a3D7b btjZLGa5Eb8cCJsUwSq3M8WjFxuO8vTwm8rudi3LlHn51A71vnOKPpYsD XhIk+N0XzUvF+LazP2Unb37uB78AVEyD9BHukCZlxvdQ82KABR12QXlX2 g==; X-CSE-ConnectionGUID: OThGODJNRWG1gSFnvOWEBw== X-CSE-MsgGUID: iNrktRuLT4aF8vKQ6auGXA== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930587" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930587" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:38 -0800 X-CSE-ConnectionGUID: BjNSJDU0RVuIyvdxJ63vGQ== X-CSE-MsgGUID: INVQujqZTM+d6XKry25vsA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866530" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:26 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 05/15] x86: Add arch specific kasan functions Date: Tue, 4 Feb 2025 18:33:46 +0100 Message-ID: <911ad4b9f001bca4c274b60144b1db80eab2015f.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093539_599868_7406A979 X-CRM114-Status: GOOD ( 10.04 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org KASAN's software tag-based mode needs multiple macros/functions to handle tag and pointer interactions - mainly to set and retrieve tags from the top bits of a pointer. Mimic functions currently used by arm64 but change the tag's position to bits [60:57] in the pointer. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/include/asm/kasan.h | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h index de75306b932e..8829337a75fa 100644 --- a/arch/x86/include/asm/kasan.h +++ b/arch/x86/include/asm/kasan.h @@ -3,6 +3,8 @@ #define _ASM_X86_KASAN_H #include +#include +#include #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) #define KASAN_SHADOW_SCALE_SHIFT 3 @@ -24,8 +26,33 @@ KASAN_SHADOW_SCALE_SHIFT))) #ifndef __ASSEMBLY__ +#include +#include +#include + +#define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) +#define arch_kasan_reset_tag(addr) __tag_reset(addr) +#define arch_kasan_get_tag(addr) __tag_get(addr) + +#ifdef CONFIG_KASAN_SW_TAGS + +#define __tag_shifted(tag) FIELD_PREP(GENMASK_ULL(60, 57), tag) +#define __tag_reset(addr) (sign_extend64((u64)(addr), 56)) +#define __tag_get(addr) ((u8)FIELD_GET(GENMASK_ULL(60, 57), (u64)addr)) +#else +#define __tag_shifted(tag) 0UL +#define __tag_reset(addr) (addr) +#define __tag_get(addr) 0 +#endif /* CONFIG_KASAN_SW_TAGS */ #ifdef CONFIG_KASAN + +static inline const void *__tag_set(const void *addr, u8 tag) +{ + u64 __addr = (u64)addr & ~__tag_shifted(KASAN_TAG_KERNEL); + return (const void *)(__addr | __tag_shifted(tag)); +} + void __init kasan_early_init(void); void __init kasan_init(void); void __init kasan_populate_shadow_for_vaddr(void *va, size_t size, int nid); @@ -34,8 +61,9 @@ static inline void kasan_early_init(void) { } static inline void kasan_init(void) { } static inline void kasan_populate_shadow_for_vaddr(void *va, size_t size, int nid) { } -#endif -#endif +#endif /* CONFIG_KASAN */ + +#endif /* __ASSEMBLY__ */ #endif From patchwork Tue Feb 4 17:33:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C422EC02193 for ; Tue, 4 Feb 2025 17:36:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SIrHtCi6XKxQ5fLKYqWjJ7We41e1K4V+YcyYD6oGu/8=; b=2VxCsnXTYpXpHP 9tUn3sOKRZEd6mFxblR8Dv2RW42GyKeBL7irmDLdzdRuyr8uqOW/XRXH+cNGfF4U52VzymMeJ5oWQ d2OUPg7vPc6oFhII1/aibkrUB45Y4G8Q7qr9aRS0hGRWiWLJiUdAvSTcAuOjnHfAl6EStlWZsbXFv +hLUi2+hrq1vxPtSDzih435E1e5DQAeiiv+GfCz+D4fDGEKIKl9Ue9wpZ/RmO6aaVvwWSw3CoE/RG kMFaqIBEWuMJbxl+C7Ele6SFO62yBxXsw7DyxXVRO4B9Oq1LazT54pO20QMCuXK0YdkRmlVSRGqRP XaFQQqcym7TbpcdliMqQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMq6-000000017CX-2aCo; Tue, 04 Feb 2025 17:35:58 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMpz-0000000173v-1TMW; Tue, 04 Feb 2025 17:35:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690551; x=1770226551; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GgxfVN861pKW86Az0//GU2BTNPe6s6rSZNyJJoV8Pn4=; b=jVrYBjQhO2QmuAwBzqwkPwrrzPU5Nwr4rnCsPXzPE3azWzRnsV67F5cE qNaKF77o2/MmWr94b1YMv4AcIUFwBJ534hDfJR/2DskqfHhFNoDehhdpX mWEhUhIbGxgFXSwt75w0F7bhUk+0/ET4eLWXewymsvIYOCG/bmzJAlO0m w+IyyhKVsrDy9S9K5UTe4J6Gi3d+Z+kFtbdZCQu2A6BehBIdrxNTLVC1j GMcXqKzSy4l5bVvz2kwUoW1XBKV6HG53eqSypQstIEht+w2kTSMfuf8lp TfZD7LaUD1OOyrZDye0efbG5oEuStbEk8Paq4DI6elvgaAIcTAG1cn0d8 g==; X-CSE-ConnectionGUID: fw0T5rKJSqqFHSxsDkeu4Q== X-CSE-MsgGUID: LuUZQrzJSjSLyX9TbI3dkQ== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930649" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930649" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:50 -0800 X-CSE-ConnectionGUID: EjCbNRnNQ/eG0RktDBjJIQ== X-CSE-MsgGUID: zLXQup0VQWiJeXH9lbUyzw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866602" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:38 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 06/15] x86: Reset tag for virtual to physical address conversions Date: Tue, 4 Feb 2025 18:33:47 +0100 Message-ID: <80aa9a4c633502b5330c40f8b2d4da705dca92e7.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093551_464769_F46A0F8F X-CRM114-Status: GOOD ( 13.26 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Any place where pointer arithmetic is used to convert a virtual address into a physical one can raise errors if the virtual address is tagged. Reset the pointer's tag by sign extending the tag bits in macros that do pointer arithmetic in address conversions. There will be no change in compiled code with KASAN disabled since the compiler will optimize the __tag_reset() out. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/include/asm/page.h | 17 +++++++++++++---- arch/x86/include/asm/page_64.h | 2 +- arch/x86/mm/physaddr.c | 1 + 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index 1b93ff80b43b..09c3914d8ce4 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -7,6 +7,7 @@ #ifdef __KERNEL__ #include +#include #ifdef CONFIG_X86_64 #include @@ -41,7 +42,7 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, #define __pa(x) __phys_addr((unsigned long)(x)) #endif -#define __pa_nodebug(x) __phys_addr_nodebug((unsigned long)(x)) +#define __pa_nodebug(x) __phys_addr_nodebug((unsigned long)(__tag_reset(x))) /* __pa_symbol should be used for C visible symbols. This seems to be the official gcc blessed way to do such arithmetic. */ /* @@ -65,9 +66,17 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, * virt_to_page(kaddr) returns a valid pointer if and only if * virt_addr_valid(kaddr) returns true. */ -#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT) + +#ifdef CONFIG_KASAN_SW_TAGS +#define page_to_virt(x) ({ \ + __typeof__(x) __page = x; \ + void *__addr = __va(page_to_pfn((__typeof__(x))__tag_reset(__page)) << PAGE_SHIFT); \ + (void *)__tag_set((const void *)__addr, page_kasan_tag(__page)); \ +}) +#endif +#define virt_to_page(kaddr) pfn_to_page(__pa((void *)__tag_reset(kaddr)) >> PAGE_SHIFT) extern bool __virt_addr_valid(unsigned long kaddr); -#define virt_addr_valid(kaddr) __virt_addr_valid((unsigned long) (kaddr)) +#define virt_addr_valid(kaddr) __virt_addr_valid((unsigned long)(__tag_reset(kaddr))) static __always_inline void *pfn_to_kaddr(unsigned long pfn) { @@ -81,7 +90,7 @@ static __always_inline u64 __canonical_address(u64 vaddr, u8 vaddr_bits) static __always_inline u64 __is_canonical_address(u64 vaddr, u8 vaddr_bits) { - return __canonical_address(vaddr, vaddr_bits) == vaddr; + return __canonical_address(vaddr, vaddr_bits) == __tag_reset(vaddr); } #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index f3d257c45225..6e24aeff36eb 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -33,7 +33,7 @@ static __always_inline unsigned long __phys_addr_nodebug(unsigned long x) extern unsigned long __phys_addr(unsigned long); extern unsigned long __phys_addr_symbol(unsigned long); #else -#define __phys_addr(x) __phys_addr_nodebug(x) +#define __phys_addr(x) __phys_addr_nodebug(__tag_reset(x)) #define __phys_addr_symbol(x) \ ((unsigned long)(x) - __START_KERNEL_map + phys_base) #endif diff --git a/arch/x86/mm/physaddr.c b/arch/x86/mm/physaddr.c index fc3f3d3e2ef2..7f2b11308245 100644 --- a/arch/x86/mm/physaddr.c +++ b/arch/x86/mm/physaddr.c @@ -14,6 +14,7 @@ #ifdef CONFIG_DEBUG_VIRTUAL unsigned long __phys_addr(unsigned long x) { + x = __tag_reset(x); unsigned long y = x - __START_KERNEL_map; /* use the carry flag to determine if x was < __START_KERNEL_map */ From patchwork Tue Feb 4 17:33:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF772C02194 for ; Tue, 4 Feb 2025 17:36:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YQk6qd+iF6V8tC9SnMkFjKFNDYcAvgQkJ++02YM4Egw=; b=uemXZahwCO7zyv YNQ5Ex9x7XXO5h9RgNBSn0c/bR497fZhGiHgXbY0/TQ6ewax7OMHx8UOmoJklZNE2DoVzGqWhKiHc l0quUh4fPAwJAMrzxSM2CKBIk7ug8L/aj/WbjeKFpN+z/LJHLRRstAOrUI7oX1x/r0ISXzsRp3rC4 W9KruUpVAk8mI+zTeGR7Zvgd4xozwAk3m5dax/U3CU3VIvOtGyZw8JSzn/4oHk1iEXzdKk80kIQCu /md1g7isCpBvF8CpA6GTBCDSvZTRfoZyUSlxQb6K+H2cjYCTlHoQoFmId50eWfgNE38gESxIyA8ql 0TArv3lSpytnVd+dXMCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqD-000000017Lw-1x9q; Tue, 04 Feb 2025 17:36:05 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqA-000000017IT-3WJk; Tue, 04 Feb 2025 17:36:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690563; x=1770226563; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PewXt3mq2ZzovNzL7I8Fp3xpOP5/pryHgDbcA5gr/7A=; b=Kpt7Oki/BH29+IUz11Gk2Vv+HZLsVTdbAl71awZMdoS5WYrrbXDqVh53 V+K2DsSI8oZkTLg3OUxpurrXKh72SOT5bkxHo5f4pLY7UEE3V6ts7xc5D BiDxzdLvAesTf6XwNgavX/NLgTcVLwUU4fEYPGf1mcXE0eEvbZ/BTVzbU uPckAl4hhHlyYXZMjDREq3QxApXakRFXu1GwMlt2QPUk4BnPZg48eHBkJ gfF9/8e3Y721P5KruO4rpT5OHWFuP/E0mcm94vaWCwMa1U6uG+EKGQ3rn MF5mxDXd9tnv9+9LqmWdtGuh/WCWxYoz05tS+pnGIXKk1WBcuSPWypuOK A==; X-CSE-ConnectionGUID: QE41qNROTei65Ibwqfu1Dg== X-CSE-MsgGUID: xaTlgVK8SUeqm97vBTcokQ== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930686" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930686" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:02 -0800 X-CSE-ConnectionGUID: PpX6VWM3RaGBO1JoNfSfGA== X-CSE-MsgGUID: vYDFTil1TqeMDIrSmNQ1tw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866647" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:35:50 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 07/15] mm: Pcpu chunk address tag reset Date: Tue, 4 Feb 2025 18:33:48 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093602_947419_BC6DF3D4 X-CRM114-Status: GOOD ( 13.98 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The problem presented here is related to NUMA systems and tag-based KASAN mode. Getting to it can be explained in the following points: 1. A new chunk is created with pcpu_create_chunk() and vm_structs are allocated. On systems with one NUMA node only one is allocated, but with more NUMA nodes at least a second one will be allocated too. 2. chunk->base_addr is assigned the modified value of vms[0]->addr and thus inherits the tag of this allocated structure. 3. In pcpu_alloc() for each possible cpu pcpu_chunk_addr() is executed which calculates per cpu pointers that correspond to the vms structure addresses. The calculations are based on adding an offset from a table to chunk->base_addr. Here the problem presents itself since for addresses based on vms[1] and up, the tag will be different than the ones based on vms[0] (base_addr). The tag mismatch happens and an error is reported. Reset the base_addr tag, since it will disable tag checks for pointers derived arithmetically from base_addr that would inherit its tag. Signed-off-by: Maciej Wieczor-Retman --- mm/percpu-vm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index cd69caf6aa8d..e13750d804f7 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -347,7 +347,7 @@ static struct pcpu_chunk *pcpu_create_chunk(gfp_t gfp) } chunk->data = vms; - chunk->base_addr = vms[0]->addr - pcpu_group_offsets[0]; + chunk->base_addr = kasan_reset_tag(vms[0]->addr) - pcpu_group_offsets[0]; pcpu_stats_chunk_alloc(); trace_percpu_create_chunk(chunk->base_addr); From patchwork Tue Feb 4 17:33:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 12D26C02193 for ; Tue, 4 Feb 2025 20:01:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ei19gdmt9sOjiCUri5hurw3wR7WbdJb5L3yRltEN0lU=; b=PrIU3rYj60dW7S ugGWlDxUZrYUs26ynRP00afGpeACcK7A37WgnFM5wX/XILVR1gXZSOCyi8VfBtxqORv+7/2M1fqjH vXutk5HwO7QmtC0XRzx/Gj6wy0HEZT5b7qkV7ZtbUfQnBUeZAzELoRMRqMDGld7PLw70xGn7zyHPZ 6XMbdHcCMvhCy+F0CkOsPuBCFZp2NehPw/quYsSmvBzmoYegQ/5itzKcj+8ufdAowTz9muGgO6eEM 58L4FvNjnYadWULrR1+SlsLu3MSlP29RQwTfOZtY6xNUBx9HU1LPB/WVhIXGOKqOSjUdKKW1Fs4TL M0JeaJhjEiYNFdfUk23A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6S-00000001SVk-1Rtf; Tue, 04 Feb 2025 20:01:00 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqW-000000017cc-3zdn; Tue, 04 Feb 2025 17:36:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=lQ5v1NnA6R05ztWWI7B+8ia+Rd5WZ3/fmwVsOhBY4uI=; b=Brf4U0OScI8qoWztrysxA+piOc 6Qdgrg9o5Mk9NUNVECRg8cnL8q0bsikhkEncpIbum/WAlhcKZkixkDncKx0ijbLLV0rmVhrIrAATQ G5EUtgdKnpMYn3dzTSGgDIJXW4N9WFbSurV8XYk047KbHVuCT1rs3ggd71drHfpL3XaULv/yidfQR fAhjljN1YqBN8BIZoV1jElFip6/ETKuklU6cnqctk+oBt5W2asEQW2YWjZtevmpuBaa3vDn8auH4F NyG/0ggy8b8J2sivnRF/AEI8Py8LZaqr0FQLNf0YICS3ZGDGg6EQ3mm4PCQJ/4CExH9rpUxAWm/Q5 ffEVyI9Q==; Received: from mgamail.intel.com ([198.175.65.20]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqT-0000000GLsT-130a; Tue, 04 Feb 2025 17:36:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690581; x=1770226581; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/9iE/zrVNGKs+zQWoSO6YvnUY6c4YJpTItZxJ7K+vOY=; b=GwzFVqMHV8ubytx/P97N0xpl9S7RxgUp+wQycWpBOmyBCydUA2E/sIUg knwpiNYSboY/fggekX+gRnbPAqWA+6RMsVowi3HvHUojb1nsMHYjXG8r3 R+FJ1YgP30PzmROHsgzLoHgpn/ZEdA4leoXs0+ZzIjr+bq5nSAT95r7xD idLHXfz8tDCymhqa754yBhls882nl8hqI6cL6L60XBgq9v3sE/XoreC+X 2k+XqSRVizh1Lovqt2WdvPW8CR3aNTbnPoQirJjGs0QsH7WPqYGhGu3zw UnIqinEIxyDl1F3LSrAUTxy9Q6UhkiG0UA9DvUAUcqi6yTLO3w7DcQtxF w==; X-CSE-ConnectionGUID: qlLtWe9PTlatpUmDxLepug== X-CSE-MsgGUID: f6HorramRB+DkckO5vrWgg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930749" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930749" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:14 -0800 X-CSE-ConnectionGUID: +qvwGRPBT+u0hTrJ0ps4gQ== X-CSE-MsgGUID: gKQBPywxQQq2MfdmEZxhgA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866742" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:02 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 08/15] x86: Physical address comparisons in fill_p*d/pte Date: Tue, 4 Feb 2025 18:33:49 +0100 Message-ID: <2c2a71ec844db597f30754dd79faf87c9de0b21f.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_173621_904448_C952B4AF X-CRM114-Status: UNSURE ( 8.09 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Calculating page offset returns a pointer without a tag. When comparing the calculated offset to a tagged page pointer an error is raised because they are not equal. Change pointer comparisons to physical address comparisons as to avoid issues in KASAN that pointer arithmetic would create. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/mm/init_64.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index ff253648706f..bb101412424a 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -251,7 +251,7 @@ static p4d_t *fill_p4d(pgd_t *pgd, unsigned long vaddr) if (pgd_none(*pgd)) { p4d_t *p4d = (p4d_t *)spp_getpage(); pgd_populate(&init_mm, pgd, p4d); - if (p4d != p4d_offset(pgd, 0)) + if (__pa(p4d) != __pa(p4d_offset(pgd, 0))) printk(KERN_ERR "PAGETABLE BUG #00! %p <-> %p\n", p4d, p4d_offset(pgd, 0)); } @@ -263,7 +263,7 @@ static pud_t *fill_pud(p4d_t *p4d, unsigned long vaddr) if (p4d_none(*p4d)) { pud_t *pud = (pud_t *)spp_getpage(); p4d_populate(&init_mm, p4d, pud); - if (pud != pud_offset(p4d, 0)) + if (__pa(pud) != __pa(pud_offset(p4d, 0))) printk(KERN_ERR "PAGETABLE BUG #01! %p <-> %p\n", pud, pud_offset(p4d, 0)); } @@ -275,7 +275,7 @@ static pmd_t *fill_pmd(pud_t *pud, unsigned long vaddr) if (pud_none(*pud)) { pmd_t *pmd = (pmd_t *) spp_getpage(); pud_populate(&init_mm, pud, pmd); - if (pmd != pmd_offset(pud, 0)) + if (__pa(pmd) != __pa(pmd_offset(pud, 0))) printk(KERN_ERR "PAGETABLE BUG #02! %p <-> %p\n", pmd, pmd_offset(pud, 0)); } @@ -287,7 +287,7 @@ static pte_t *fill_pte(pmd_t *pmd, unsigned long vaddr) if (pmd_none(*pmd)) { pte_t *pte = (pte_t *) spp_getpage(); pmd_populate_kernel(&init_mm, pmd, pte); - if (pte != pte_offset_kernel(pmd, 0)) + if (__pa(pte) != __pa(pte_offset_kernel(pmd, 0))) printk(KERN_ERR "PAGETABLE BUG #03!\n"); } return pte_offset_kernel(pmd, vaddr); From patchwork Tue Feb 4 17:33:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6F017C02194 for ; Tue, 4 Feb 2025 20:01:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nAiGOf4Th7dmjfrU3bdPO3+3gXF0xN1nw69RFO3KNY0=; b=qrTb88+RuSdFDc YRBU7R0PCI5LKVQeVfcuii4V6vCKOa9msAyLXBERaY+GXd7tm+71Y/wa4jvLs6fxVyKt7LPAX2imw IdVZceu+1MYEfR24kXPc24zP3HSKJrDa28c3i8iandJqnuU3yjOkFjfls3KCs1YYw82GW0nk8th5p qS+vXPEfF6OiiDoNnPOXYbF14Q9HsNrCIJkXPnLtsmcQZXVxZ7/RfexqIrmM33dyxEYpat1/EVlr/ FI4O8oZna60lwSIaRBuB6uJOs5q/xOT6DGWJLexCwZuEmj+zQGUqSyltpiROc3r0Di/GiMPzRef/e MIz5PTbrkcTYD3OyjCmg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6T-00000001SWK-2fTE; Tue, 04 Feb 2025 20:01:01 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqd-000000017gY-3ztA; Tue, 04 Feb 2025 17:36:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=OjOreTL+Ot1aK4xRTqvixmr6X+5Hdo5QfDrtzsBAFBk=; b=cF9BjhlLs5jGI6B+rtjcQ7R//+ EMtDfW371mUM0XVPARAM71Kgf/cfqAx0ui52m1W0XinQF6OIYfYiXhynoYukET4I1bbb1JukE6KuI N9ILiK63nwX3VOHwLBt8EEAaoccSzgsuHpyXPmpvjLOZKUDOB6K3GDXBJ09H/ZckvfVbgH1XuUy+p 27TLJa1wVZVFTxEPyVpkdlQINJweZ7kx1hZLBBG/Wadrt+jScjJW4kjh74W0hrPWw0/ZIKsGBPvRT 7qJhOOofklWLgzoiNZOL4+nDSa3qPiQhc0cVnRZccIbwWm+mXVlWXIcvKD9zm/ZdwymBcQcx2Dp3N JHGO6L0g==; Received: from mgamail.intel.com ([198.175.65.20]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqZ-0000000GLsT-2Kaa; Tue, 04 Feb 2025 17:36:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690587; x=1770226587; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZPDKrg9roFM67mEQ2fI48/uwWPYPBpbE1ZWp7rfkzQw=; b=OzKLEZRcGnEafPuWzqcBa2H/6WpLfuxAPAQKLEg+q0BEnRqRSm549NYD CJd00DexTfsg5MFYeUUdGd7nX9aNaXTU57Gwcx1LKymI7cM7Sib/0jzbe Y9ZbE3nOGscJs7gB7xwfuBBre2j8iA9T2jBSfUGfhd1tvFEeSV1VqdOMM TIO8+ZF3S1VXNPkSX0eEpdl9ZQD2MckykDnD2uY2ZdZW8cra2Eh5PM1pF MGwFiR+HFhadiAKiNJL4kgU9Y2rEDnxOdKHQ06PnpATfH1MKE5WVQO55x DBY+/6GIcbm3lrik/UrpKk9ylLYZ+EsDRzDcdfhO4hR7bkYQZkXS5/ROQ Q==; X-CSE-ConnectionGUID: PDqFjK0LQGOzIpkL9EBZcw== X-CSE-MsgGUID: UQRW6WyYRT++i02erQ2Gqg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930848" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930848" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:26 -0800 X-CSE-ConnectionGUID: jXIRAGRVSi6AV3rBRiAebw== X-CSE-MsgGUID: yAFfDOxmTeCrIA01iFfVXA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866806" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:14 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 09/15] x86: Physical address comparison in current_mm pgd check Date: Tue, 4 Feb 2025 18:33:50 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_173627_955976_C4C56D53 X-CRM114-Status: UNSURE ( 8.84 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org With KASAN software tag-based mode enabled PGD pointer stored in current_mm structure is tagged while the same pointer computed through __va(read_cr3_pa()) ends up with the tag space filled with ones. Use current_mm->pgd' physical address and drop the __va() so the VM_WARN_ON_ONCE can work properly and not report false positives while KASAN is enabled. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/mm/tlb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 86593d1b787d..95e3dc1fb766 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1295,7 +1295,7 @@ bool nmi_uaccess_okay(void) if (loaded_mm != current_mm) return false; - VM_WARN_ON_ONCE(current_mm->pgd != __va(read_cr3_pa())); + VM_WARN_ON_ONCE(__pa(current_mm->pgd) != read_cr3_pa()); return true; } From patchwork Tue Feb 4 17:33:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F2BEC0219A for ; Tue, 4 Feb 2025 20:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mbypDyTNTuVl4aFCAvZagHzVcr9sYdCZ5JRK/+o7CC8=; b=BrSDhLxDEWoXnI Ah6B8r0gH/E1csOhpDpN4vTBYp2/NPsLms8y8GZW358Sgdn9TFevYJAhyZUmfWA5prvh5C30aCkuN yP9vi9+8pd9hHqkvd3s9bkvFpoItTN9elJYAuRRJ1lBb8ZsspRoLqW4C4t+fahoaAM3/QN7I7gDRv 5WfCBXFN17QKhyEIkgA2Xw5Hwg4AE3Z0Ln8661Yh7DxrKvjMj77Xp9lz+QI+suXgGtAvWQKXocZTp mXgJZdEwPqPKVYDXjAEEIF0INupIPlpTU4VkPAKvYH8yxdr7Me2vn8lCqZE18GTOFTaCGN6A6R8If qLnKpvLfvoba+U/gvgZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6U-00000001SXA-3AlE; Tue, 04 Feb 2025 20:01:02 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqt-000000017oH-0v21; Tue, 04 Feb 2025 17:36:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pLFVRgBwiXf+T1hr1v6y3oXYvLLcOkmu5U12KRQmKlw=; b=P4xYveaFpJK9pc4gYL/1MF4dzW OI68Y+6LfrnXjxMEjr3ujkMVPExu3z1oH/mG9V4WeBQGdfdjabeLa9f/JZOiUhZzMOc2MVDYFBtkz 07jWanDc8eOftyKMs/oyUAQQoNQtCVZN27FoUV/vgZoHqywT8fs5BBVKEPwT/jatiKB0rPfpxX/LI crsWWcS/ekyu+xU/GgAeIhtyF0lyqNN0LOsoInCOkHNkZvozssaqCR7tVJc7rz01BsTOUrSzvlUX3 Iwj2VU1a3J3HNtXU8zQDebomjax0A1pYe/TRDPMVpGUahC6dg2fx/JcpFt3mIOsdMGY18LlhletTx YrLn4b0g==; Received: from mgamail.intel.com ([198.175.65.20]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqo-0000000GLyH-2l8W; Tue, 04 Feb 2025 17:36:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690602; x=1770226602; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=J5wZ/mcDbil10hn8JDYxev0M9sT2GI5gsQFk6+vTupM=; b=MD8u5pZwNV6nqI3Vfm0GYAZ+eL3Ow8ODROe0TRHcLWo9zrMsFhCij/8q bj8utFGBci7HoRGse9nF272zdpq6LvpfHC5ibfTWsqjEY3x+PCTdDXAAZ fzRI+qfx4qih+JhcO9yIiK8XovHShHMwSlt2ChAdl1F6AU8xLL4b4xtU+ i4541dGSOQkQjGBVNIJzB1Hp7oxV0T0ZmhspRBUEVjnK8y8Z/ub0Ep72X QhWtaYQOhy1LyRWss8vDBe2rifyrn3TLojyFx5zjcOswcgXmv2XCD6s76 mOWDOidcLXIk5ARlaTobuQx50AS0pTWbezDEt4cQ6XkinrWBCIPFXu8RQ Q==; X-CSE-ConnectionGUID: 7QsrvLh3Q0yfxDVzJOstrw== X-CSE-MsgGUID: 9AD/To2cQN6c5aT34AINbg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930931" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930931" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:39 -0800 X-CSE-ConnectionGUID: euT5Fx0UQBaZT6g1J2Ba6Q== X-CSE-MsgGUID: VDPCZKL7T1GLAmeyS4orWQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866863" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:27 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 10/15] x86: KASAN raw shadow memory PTE init Date: Tue, 4 Feb 2025 18:33:51 +0100 Message-ID: <28ddfb1694b19278405b4934f37d398794409749.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_173643_087071_BFF6CEB2 X-CRM114-Status: GOOD ( 16.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In KASAN's generic mode the default value in shadow memory is zero. During initialization of shadow memory pages they are allocated and zeroed. In KASAN's tag-based mode the default tag for the arm64 architecture is 0xFE which corresponds to any memory that should not be accessed. On x86 (where tags are 4-bit wide instead of 8-bit wide) that tag is 0xE so during the initializations all the bytes in shadow memory pages should be filled with 0xE or 0xEE if two tags should be packed in one shadow byte. Use memblock_alloc_try_nid_raw() instead of memblock_alloc_try_nid() to avoid zeroing out the memory so it can be set with the KASAN invalid tag. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/mm/kasan_init_64.c | 19 ++++++++++++++++--- include/linux/kasan.h | 25 +++++++++++++++++++++++++ mm/kasan/kasan.h | 19 ------------------- 3 files changed, 41 insertions(+), 22 deletions(-) diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 9dddf19a5571..55d468d83682 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -35,6 +35,18 @@ static __init void *early_alloc(size_t size, int nid, bool should_panic) return ptr; } +static __init void *early_raw_alloc(size_t size, int nid, bool should_panic) +{ + void *ptr = memblock_alloc_try_nid_raw(size, size, + __pa(MAX_DMA_ADDRESS), MEMBLOCK_ALLOC_ACCESSIBLE, nid); + + if (!ptr && should_panic) + panic("%pS: Failed to allocate page, nid=%d from=%lx\n", + (void *)_RET_IP_, nid, __pa(MAX_DMA_ADDRESS)); + + return ptr; +} + static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, int nid) { @@ -64,8 +76,9 @@ static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr, if (!pte_none(*pte)) continue; - p = early_alloc(PAGE_SIZE, nid, true); - entry = pfn_pte(PFN_DOWN(__pa(p)), PAGE_KERNEL); + p = early_raw_alloc(PAGE_SIZE, nid, true); + memset(p, PAGE_SIZE, kasan_dense_tag(KASAN_SHADOW_INIT)); + entry = pfn_pte(PFN_DOWN(__pa_nodebug(p)), PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } while (pte++, addr += PAGE_SIZE, addr != end); } @@ -437,7 +450,7 @@ void __init kasan_init(void) * it may contain some garbage. Now we can clear and write protect it, * since after the TLB flush no one should write to it. */ - memset(kasan_early_shadow_page, 0, PAGE_SIZE); + kasan_poison(kasan_early_shadow_page, PAGE_SIZE, KASAN_SHADOW_INIT, false); for (i = 0; i < PTRS_PER_PTE; i++) { pte_t pte; pgprot_t prot; diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 83146367170a..af8272c74409 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -151,6 +151,31 @@ static __always_inline void kasan_unpoison_range(const void *addr, size_t size) __kasan_unpoison_range(addr, size); } +#ifdef CONFIG_KASAN_HW_TAGS + +static inline void kasan_poison(const void *addr, size_t size, u8 value, bool init) +{ + if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + hw_set_mem_tag_range(kasan_reset_tag(addr), size, value, init); +} + +#else /* CONFIG_KASAN_HW_TAGS */ + +/** + * kasan_poison - mark the memory range as inaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size, must be aligned to KASAN_GRANULE_SIZE + * @value - value that's written to metadata for the range + * @init - whether to initialize the memory range (only for hardware tag-based) + */ +void kasan_poison(const void *addr, size_t size, u8 value, bool init); + +#endif /* CONFIG_KASAN_HW_TAGS */ + void __kasan_poison_pages(struct page *page, unsigned int order, bool init); static __always_inline void kasan_poison_pages(struct page *page, unsigned int order, bool init) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index a56aadd51485..2405477c5899 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -466,16 +466,6 @@ static inline u8 kasan_random_tag(void) { return 0; } #ifdef CONFIG_KASAN_HW_TAGS -static inline void kasan_poison(const void *addr, size_t size, u8 value, bool init) -{ - if (WARN_ON((unsigned long)addr & KASAN_GRANULE_MASK)) - return; - if (WARN_ON(size & KASAN_GRANULE_MASK)) - return; - - hw_set_mem_tag_range(kasan_reset_tag(addr), size, value, init); -} - static inline void kasan_unpoison(const void *addr, size_t size, bool init) { u8 tag = get_tag(addr); @@ -497,15 +487,6 @@ static inline bool kasan_byte_accessible(const void *addr) #else /* CONFIG_KASAN_HW_TAGS */ -/** - * kasan_poison - mark the memory range as inaccessible - * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size, must be aligned to KASAN_GRANULE_SIZE - * @value - value that's written to metadata for the range - * @init - whether to initialize the memory range (only for hardware tag-based) - */ -void kasan_poison(const void *addr, size_t size, u8 value, bool init); - /** * kasan_unpoison - mark the memory range as accessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE From patchwork Tue Feb 4 17:33:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D0BA1C02199 for ; Tue, 4 Feb 2025 20:01:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sAbTDyS8uNjxiSUz4c+yVNV0teMEq4PCCGjKQoiaBl0=; b=sg39lmqb7ZKynn gr4qytFFIcQRwZ/aksYZEs9rQJZgB58VeR4/83lVAkxgdZWEgQcPB/vgqTdRgl7CIlN8v1VputzH9 VkJfj14he4uc2POB/prU/tXLR6INvWTmjA0gP+YSr2GRrjHtKC7QkqfPtaG4yQwYRAih7zITnvan8 kEHgZb442I/tYuwBl/LjLuAfwVv0V1vL5FFnflb4pbqlOtc/MgqBMS7a8ahmfMNaEcBwFoURdVZLP w1Bdwz5lrCvlYq1wkR7OhhP0BYfXPnziqlQYO2gnwAZeHEd+X126Q3I2dZLCP9dO/Yyq91QdcoKH+ 8WYdW3MMx4+WpqW2s9Bg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6V-00000001SY8-3i3U; Tue, 04 Feb 2025 20:01:03 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMr2-000000017tA-2SrN; Tue, 04 Feb 2025 17:36:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wcThim39EYPHxXLHIVD/RA5qpLaRNquzUdH4zXdFJ+w=; b=YA0hyO2EoSy5o5sd3CNHcsm2k1 Ej0g1bYGP4ymlQ5hXmasRBOKebLAqNuFf0J4H6hgIXmds7T5hIZIsmalF33TUoSis5orLwQDwDVqK M78V5y3WhOPtVVVE17v/2t2D7Vj9QHElUoAtX/0Br1IBNoaTj229y9mxhgXZHNQHoDi/JNaW0bDD9 RFMZ6B8MzhdI8dTFoCqHiDyly/QfDTb0ESNIhUT1e/nDjDOls2IX9n5GKwbcNGm+a2NbvwZyQ62eC hlEzfc/cd87YIgISn5oatbgYI21KCQP/9yiigllaGY/vG9OlJk5psqwXhBqnSvZbU+Ix8WAbsrWi7 fEzJbeRA==; Received: from mgamail.intel.com ([198.175.65.20]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMqz-0000000GM17-2QF7; Tue, 04 Feb 2025 17:36:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690613; x=1770226613; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EUgvSoD6ZXvZMdQY8EXK092WIVXF++/3+F17U+GxOWU=; b=CWAB+BXHZ2GM4DhILaTk5HeAJ6vUIBQsu6QRGt6n5Eha5enrPIJ/q3/3 B655tMNN1brqCn2fCzMVZG9+qVtKJGG9sthgV9r0W9bqunWEoDEi1k6mu 3rD1yOIXWCp0pbYm6mMbh569ajBUzK3ePw5VDZYZu9wIxmY5uu0BTkGA4 wa7GxqCsi5ojt9EZ8bTrUHWsb49FITy7TRx4NyhWCKZARkeOUigwE0Ypf 37iUqmd/xNmvwn17CtlPX9Y7tJBBoaeFGZzQIyooUixVhmubEPK4P2Ilg DLe9TSDfU5vMnjSkRZNOBcQtTqadbq01CE3E0P4q6TH+YUt6+YaOnVVTx g==; X-CSE-ConnectionGUID: pVim1VIZTySXLQmIm/LYrQ== X-CSE-MsgGUID: RmhbzAmYST+HgvWubX73gw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38930979" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38930979" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:51 -0800 X-CSE-ConnectionGUID: P6du7V6VScafjuMIXvnbWA== X-CSE-MsgGUID: W7s9nIAMS+GBPujRL+zY2g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866889" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:39 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 11/15] x86: LAM initialization Date: Tue, 4 Feb 2025 18:33:52 +0100 Message-ID: <01104816cdd0d430ac843847a8056d07b8770be0.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_173653_950719_9831F02A X-CRM114-Status: GOOD ( 11.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org To make use of KASAN's tag based mode on x86 Linear Address Masking (LAM) needs to be enabled. To do that the 28th bit in CR4 needs to be set. Set the bit in early memory initialization. When launching secondary CPUs the LAM bit gets lost. To avoid this it needs to get added in a mask in head_64.S. The bit mask permits some bits of CR4 to pass from the primary CPU to the secondary CPUs without being cleared. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/kernel/head_64.S | 3 +++ arch/x86/mm/init.c | 3 +++ 2 files changed, 6 insertions(+) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 16752b8dfa89..7cdafcedbc70 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -199,6 +199,9 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) * there will be no global TLB entries after the execution." */ movl $(X86_CR4_PAE | X86_CR4_LA57), %edx +#ifdef CONFIG_ADDRESS_MASKING + orl $X86_CR4_LAM_SUP, %edx +#endif #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support. diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index eb503f53c319..4dc3679fedd1 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -756,6 +756,9 @@ void __init init_mem_mapping(void) probe_page_size_mask(); setup_pcid(); + if (boot_cpu_has(X86_FEATURE_LAM) && IS_ENABLED(CONFIG_KASAN_SW_TAGS)) + cr4_set_bits_and_update_boot(X86_CR4_LAM_SUP); + #ifdef CONFIG_X86_64 end = max_pfn << PAGE_SHIFT; #else From patchwork Tue Feb 4 17:33:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B9C5C02194 for ; Tue, 4 Feb 2025 20:01:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LJh3LiD4Mc4mnFyWyZK3NMWnNBomU78K4U8EIJgqNcM=; b=hD8Cmwckg9tFqM 8nfQ8IPE9MsRoAVsnMx8ngHIDgw/CX9BugNAqpTfE1S47ZLFO5mksz54TWdl2WzkRFgit2KvIAHun v7iWuxKEaaCVnJ8aH60ezKg0NVxluY32mqOSBH7fmfED1bPQE9Z+L8E8AZHGSXDqmCbocSDkXil8i O3vxQZOW8Rh4Cy6cTny8tqiRei7acUxW56shWxxfL9/vE3Y+FeW5JVZddT5Wdi/jmlMuzPxGZbu7K kY817LWNN6XNjV95qpN5kwxSMCGjGnrQyg5E53tJhODx/EFM61J2/WfpwPObbtPyNpLvZBgYtDB2Z nGFMy1hOri51EbyjKj9w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6W-00000001SZ1-465i; Tue, 04 Feb 2025 20:01:04 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMrH-0000000181S-1iFo; Tue, 04 Feb 2025 17:37:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9cP48FhwOS/LMk0h6akqBPktUCXlskOxYu2pQkbZsrI=; b=f7oqf4QA7cXpitqRj0ovRiqFlc yfAL9Sr9eUvmCBnqtQY7sJXn9N7kUSyLHjZAWRgmbkf33hHrXTPKQFYbOpmPqWLnKNlyuyzusiXvD 7isGVTYOjDVTfBvicroBFEeZe5/KAM+8dWtrMehiW3rhuV3CleFMwzGXoiTev8oQzwRPd0qXzNb+c vjhtiferBvO032y8cCXDdErBd5wB3dm2AZouTh3+OUi10oMLGHCCiQkjY5WTVYCugEWq/bMeu1z9N TOfqZ+3qiNXdKbeCfGDltGa9REaSCMh5QSsD5XMkq327mU82kgzKYEcbOn85TsVftj1sSdTIAgbkf jxD/kudA==; Received: from mgamail.intel.com ([198.175.65.20]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMrE-0000000GM5X-0DI9; Tue, 04 Feb 2025 17:37:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690628; x=1770226628; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=r2iiDiPZOS4/QANoocz7hbq1LBuYUYYysQaiQVX4JUg=; b=UxkB6hmdCi7+ILCyMdNoOh9xn4RYjZLedgVTY2PTsSehUmTXn/ndY2h6 rWAOVs+LM0m/SLZUOivDMZqR/zRFgSSxDjuTY010rk0rUAqHiYELB7uM8 hYnf9AjRsxOTccRnrwvh9s9VpKy810Zq2XZY/ztI/4jM/IDP3jLV5oku6 CbOB0KJdbfpkPTTkAwMnyN7pg1+SN9oo2SY/NlUbTvXL7ZxhAu7ejKxMn gS918diin6BiHK0sfKqx/BTSzkPiHrYy+fcpBSaHGlmOv25bCwgRZyuXz sWeDVB0EP31rRSpqM8JIeHlqtAp9o4FHsJ9a8sC7Rh2dyniZEHPGJUL32 A==; X-CSE-ConnectionGUID: MgYdJRj/TlSdR1/855aFCQ== X-CSE-MsgGUID: /XaAUMDsQ+e/HE+eshLT7Q== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38931050" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38931050" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:37:04 -0800 X-CSE-ConnectionGUID: F7WrZE34TJC30qp8xAUvrQ== X-CSE-MsgGUID: 1Ko21FfoQgSOsBU65eanYQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147866985" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:36:51 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 12/15] x86: Minimal SLAB alignment Date: Tue, 4 Feb 2025 18:33:53 +0100 Message-ID: <162610a0af3e04e2f42872401461b1d62ec78fbd.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_173708_488519_1617A277 X-CRM114-Status: UNSURE ( 6.71 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Adjust x86 minimal SLAB alignment to match KASAN granularity size. In tag-based mode the size changes to 16 bytes so the value needs to be 4. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/include/asm/kasan.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h index 8829337a75fa..f7a8d3763615 100644 --- a/arch/x86/include/asm/kasan.h +++ b/arch/x86/include/asm/kasan.h @@ -36,6 +36,8 @@ #ifdef CONFIG_KASAN_SW_TAGS +#define ARCH_SLAB_MINALIGN (1ULL << KASAN_GRANULE_SHIFT) + #define __tag_shifted(tag) FIELD_PREP(GENMASK_ULL(60, 57), tag) #define __tag_reset(addr) (sign_extend64((u64)(addr), 56)) #define __tag_get(addr) ((u8)FIELD_GET(GENMASK_ULL(60, 57), (u64)addr)) From patchwork Tue Feb 4 17:33:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B2391C02193 for ; Tue, 4 Feb 2025 20:01:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5dO6EOiwqyVXJ+3bZZKWT2T85Ld0iGdPYx1NTq77AZ0=; b=BeGYIdyqDPI3Kx EB8VJD+fpD5bkfwb5sjSu56Hy+1t2GfO88fR/6RsWHfcyTzM42WC1/8/SW3k4XynXkIyeqgG+0WjZ tacrsztfOyqG1U+3ehfbEXenzIQOitNGUN0JzKmFK5y4Vua2qZLaoKSiMG+wZ5MJ5sz5aavMPEH8j pK0QGB8bCPmrnFIm0Few5gKWc0eunS/wCTSQ3P9+2yBDmHT5ixtqGrVHLfdKQrFw30b8M0pOwX/2Y JcE7h6/lqPn+Zxg4tmdoaB0CXjfu0LdzatdDNuY+F0yr3Gu3HeMMwgUUlks5IHUtE6ORB4kYQIf1M tpJeJPct5O/4ivZJd41w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6X-00000001SZj-3WPO; Tue, 04 Feb 2025 20:01:05 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMrW-0000000186E-1sDz; Tue, 04 Feb 2025 17:37:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AJ+KNrmQGw87u/Lv3sW/ixn2Pc83/zgfJdm0b3G7QSs=; b=S5W63giuseUSKR3TufUpyefe70 tZQJenlhTVO3ETpUdt7KlXVCf0mRkDm5Cthms8F+tuWpUMY+qcFSGdtmeYBTp04fjxDKkU3j8jHlj n+E192WiTXCBLXltJhGXtqL0m4+t/qGxjfisP9ynorAXd953kCZZbZD7K+M7MbVMn2a7TBBgxcPgQ NbNN0Q48xcH0Ie7vhwSbHGSaDQeZR0KWm2QEOPVWuhHcZOsQU4XERUt6jxyXCRgQZNAvsAuS75J4t nYc2rVMfdgQGO/lZ5dUDmMc3BKX8gs7mSCRqpJDaP03kYc4tZI6HjcMqV6YE6C0XbcBd1zhExrdm2 0/1aiTPg==; Received: from mgamail.intel.com ([198.175.65.20]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMrS-0000000GM8P-0Mio; Tue, 04 Feb 2025 17:37:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690642; x=1770226642; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tgQnPnMq4WHkStd3vJbMqoHpzLn5Ygn4fGXf7vNlMB4=; b=mFESbTU9nXeiVEzQwpZ0873UAhu6UFcSPd8YK8UqGu4wfncKoanfqlkN pudmvobTB4jR11zXTVUS6w+hoJmZTPnAkF6ofXPDY+SWTiDTGI7Ik+Ani qphC/4zUT7lbeY6I37t27/uc0/rkCOIL+bCgcSc9RgBg/z9zpxwjjkqqw +3+V746PjlR8qn0y2mz7zeNK4LbD5qJeoD1fZq6oPZs7EXx/lgwfz+g8l w881eXWJ2zdFe1HSJkds0lHgbnuISFhJA7s7HHWKMRXTsy5BNvzt1Fk9u 96GV8y2FByJO+I9P/X6VbkVpO4CHPSrNeS43bANE60rrZ3+f5qSluqKsR g==; X-CSE-ConnectionGUID: 5rrGllTJSVuYB98NsCiYYQ== X-CSE-MsgGUID: ycg+7T+UQge8Dyj10ZoYiA== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38931123" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38931123" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:37:19 -0800 X-CSE-ConnectionGUID: OH9xu6cPRWustG/bFhP1dw== X-CSE-MsgGUID: vAuLaG4TRpmkRprgXY7/Og== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147867096" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:37:04 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 13/15] x86: runtime_const used for KASAN_SHADOW_END Date: Tue, 4 Feb 2025 18:33:54 +0100 Message-ID: <5d0f9dbd0f7c2326229f2a1f3dcedd46842a9615.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_173722_528662_CD2A8B74 X-CRM114-Status: GOOD ( 17.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On x86, generic KASAN is setup in a way that needs a single KASAN_SHADOW_OFFSET value for both 4 and 5 level paging. It's required to facilitate boot time switching and it's a compiler ABI so it can't be changed during runtime. Software tag-based mode doesn't tie shadow start and end to any linear addresses as part of the compiler ABI so it can be changed during runtime. This notion, for KASAN purposes, allows to optimize out macros such us pgtable_l5_enabled() which would otherwise be used in every single KASAN related function. Use runtime_const infrastructure with pgtable_l5_enabled() to initialize the end address of KASAN's shadow address space. It's a good choice since in software tag based mode KASAN_SHADOW_OFFSET and KASAN_SHADOW_END refer to the same value and the offset in kasan_mem_to_shadow() is a signed negative value. Setup KASAN_SHADOW_END values so that they're aligned to 4TB in 4-level paging mode and to 2PB in 5-level paging mode. Also update x86 memory map documentation. Signed-off-by: Maciej Wieczor-Retman --- Documentation/arch/x86/x86_64/mm.rst | 6 ++++-- arch/x86/Kconfig | 3 +-- arch/x86/include/asm/kasan.h | 14 +++++++++++++- arch/x86/kernel/vmlinux.lds.S | 1 + arch/x86/mm/kasan_init_64.c | 5 ++++- 5 files changed, 23 insertions(+), 6 deletions(-) diff --git a/Documentation/arch/x86/x86_64/mm.rst b/Documentation/arch/x86/x86_64/mm.rst index 35e5e18c83d0..4e8c04d71a13 100644 --- a/Documentation/arch/x86/x86_64/mm.rst +++ b/Documentation/arch/x86/x86_64/mm.rst @@ -48,7 +48,8 @@ Complete virtual memory map with 4-level page tables ffffe90000000000 | -23 TB | ffffe9ffffffffff | 1 TB | ... unused hole ffffea0000000000 | -22 TB | ffffeaffffffffff | 1 TB | virtual memory map (vmemmap_base) ffffeb0000000000 | -21 TB | ffffebffffffffff | 1 TB | ... unused hole - ffffec0000000000 | -20 TB | fffffbffffffffff | 16 TB | KASAN shadow memory + ffffec0000000000 | -20 TB | fffffbffffffffff | 16 TB | KASAN shadow memory (generic mode) + fffff80000000000 | -8 TB | fffffc0000000000 | 4 TB | KASAN shadow memory (software tag-based mode) __________________|____________|__________________|_________|____________________________________________________________ | | Identical layout to the 56-bit one from here on: @@ -107,7 +108,8 @@ Complete virtual memory map with 5-level page tables ffd2000000000000 | -11.5 PB | ffd3ffffffffffff | 0.5 PB | ... unused hole ffd4000000000000 | -11 PB | ffd5ffffffffffff | 0.5 PB | virtual memory map (vmemmap_base) ffd6000000000000 | -10.5 PB | ffdeffffffffffff | 2.25 PB | ... unused hole - ffdf000000000000 | -8.25 PB | fffffbffffffffff | ~8 PB | KASAN shadow memory + ffdf000000000000 | -8.25 PB | fffffbffffffffff | ~8 PB | KASAN shadow memory (generic mode) + ffe8000000000000 | -6 PB | fff0000000000000 | 2 PB | KASAN shadow memory (software tag-based mode) __________________|____________|__________________|_________|____________________________________________________________ | | Identical layout to the 47-bit one from here on: diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7b9a7e8f39ac..dfec7bc692d4 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -392,8 +392,7 @@ config AUDIT_ARCH config KASAN_SHADOW_OFFSET hex - depends on KASAN - default 0xdffffc0000000000 + default 0xdffffc0000000000 if KASAN_GENERIC config HAVE_INTEL_TXT def_bool y diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h index f7a8d3763615..79151356d5f2 100644 --- a/arch/x86/include/asm/kasan.h +++ b/arch/x86/include/asm/kasan.h @@ -5,7 +5,7 @@ #include #include #include -#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) + #define KASAN_SHADOW_SCALE_SHIFT 3 /* @@ -14,6 +14,8 @@ * for kernel really starts from compiler's shadow offset + * 'kernel address space start' >> KASAN_SHADOW_SCALE_SHIFT */ +#ifdef CONFIG_KASAN_GENERIC +#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) #define KASAN_SHADOW_START (KASAN_SHADOW_OFFSET + \ ((-1UL << __VIRTUAL_MASK_SHIFT) >> \ KASAN_SHADOW_SCALE_SHIFT)) @@ -24,12 +26,22 @@ #define KASAN_SHADOW_END (KASAN_SHADOW_START + \ (1ULL << (__VIRTUAL_MASK_SHIFT - \ KASAN_SHADOW_SCALE_SHIFT))) +#endif + #ifndef __ASSEMBLY__ +#include #include #include #include +#ifdef CONFIG_KASAN_SW_TAGS +extern unsigned long KASAN_SHADOW_END_RC; +#define KASAN_SHADOW_END runtime_const_ptr(KASAN_SHADOW_END_RC) +#define KASAN_SHADOW_OFFSET KASAN_SHADOW_END +#define KASAN_SHADOW_START (KASAN_SHADOW_END - ((UL(1)) << (__VIRTUAL_MASK_SHIFT - KASAN_SHADOW_SCALE_SHIFT))) +#endif + #define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) #define arch_kasan_reset_tag(addr) __tag_reset(addr) #define arch_kasan_get_tag(addr) __tag_get(addr) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index feb8102a9ca7..46183f7439c9 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -359,6 +359,7 @@ SECTIONS RUNTIME_CONST_VARIABLES RUNTIME_CONST(ptr, USER_PTR_MAX) + RUNTIME_CONST(ptr, KASAN_SHADOW_END_RC) . = ALIGN(PAGE_SIZE); diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 55d468d83682..0f8190e0e5f6 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -358,6 +358,9 @@ void __init kasan_init(void) int i; memcpy(early_top_pgt, init_top_pgt, sizeof(early_top_pgt)); + unsigned long KASAN_SHADOW_END_RC = pgtable_l5_enabled() ? 0xfff0000000000000 : 0xfffffc0000000000; + + runtime_const_init(ptr, KASAN_SHADOW_END_RC); /* * We use the same shadow offset for 4- and 5-level paging to @@ -372,7 +375,7 @@ void __init kasan_init(void) * bunch of things like kernel code, modules, EFI mapping, etc. * We need to take extra steps to not overwrite them. */ - if (pgtable_l5_enabled()) { + if (pgtable_l5_enabled() && !IS_ENABLED(CONFIG_KASAN_SW_TAGS)) { void *ptr; ptr = (void *)pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_END)); From patchwork Tue Feb 4 17:33:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52458C02193 for ; Tue, 4 Feb 2025 17:37:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pge2f/3RQVOJeDkUHhsbbpeZflVbBhnL/T8b3oT5oH4=; b=geRPqwOsHjZ4Zh qcirxe7gCkUg4TaYWAJQY8vFLeB+1tJRtjfeTFZ5vVdG2MqGTpCq9t5xZEest38CL28MvleLfWkUw G9ogYt1EuMpjSiibsSSF2EWnJlvdGDQeaEgT1J1oi1p/cFuNgJrGckVLnS6Z098H2HFvDXEAw4qjh iibyXSV2WQDnPxv3iskPrCnhzI+A2Ov3pIH0QaGp6ZVcr2jcprvcLf0XAjJYKcz90k3YPObKRjPIi 24yNuwO3lIFy7JuFoYktEivkrfnlsq8UCgZddbuiCWyMyMfs82noKmzI50a3BUYQbBcLyycne8TG7 sHSOD+ECy/FhgYCK4Lqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfMre-000000018A9-3qNU; Tue, 04 Feb 2025 17:37:34 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMrd-0000000188q-0sjW; Tue, 04 Feb 2025 17:37:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PCQPfnDRBxmJXeqGE9pZTyAVVotiYe5di+G+aFzDagA=; b=HLYfdIQLAojtFiseo7grIMUX22 reMeiUIVRYXR/0GEWxwujVpzviwq/bZK5SbGAxztVnjmob4Ak5pUCISHCc7VH3I7YGBJxW9BdGcpI Y4AG7Z5voFgdmmRP0dBg4vjdQHDdjZStk+L46SZxdgskSZZMzsNOEgYaalo7pZ4plE/jQZE6tvFJT gJubamFGmwIqHdmCQ/G47g2tbrmlNclGW+l0h8Rz36LFTCtzmxMt4fzFX7DWfTA68TgC07MjfvfkU dPlJ0ovUGwjAw5d6f3cp6jVigx2s9wXmV9u2DFOBY7TpLIIZaBTZSj3AMIwf2uUembLbaqiYZtsle 84i70LJA==; Received: from mgamail.intel.com ([198.175.65.20]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMra-0000000GM8P-0IIv; Tue, 04 Feb 2025 17:37:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690650; x=1770226650; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oiRBgvokfmZMzIN+jtwGJEePgjLFxs7ade5tHmSo4go=; b=dleCM97NLMMhqJF6uboYjgTeQ0240BEnWj7Y702GgVnQXpm/f8L6en+Z LXs8D7VuYRmkqjgjVanentYaMGZhrUShE3427/ukzGUQbEJVPgayw5hAB RpvdBF21NTojyrTlIh5tDb0hXS1lnDoeHnyEtYExXn8jJ/zW0EcZOZYZs 7vwgPFrs8NdSCl91jartCp2x78gvIvH+t+IZYjQq765UdvuhRn08bEz59 p/Wj/S229mUwhgd4qwIXY7zK5bALQg1fisoL+g4eXSZnmHnS63qkoiAl2 MMgWPYQvpBUmOpdKqoJUCkTtEmAMvNa5fB296KpTZkV6ZXTHIpy6DWN45 A==; X-CSE-ConnectionGUID: Ew1KeiVcSf2xZoIggImK5Q== X-CSE-MsgGUID: wSNJEya+TQWWisS92pEdPw== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38931169" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38931169" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:37:29 -0800 X-CSE-ConnectionGUID: VKIBhH62QgyxuyV1BEgDvg== X-CSE-MsgGUID: /T7gEKWsTDWNscbEm9C8IA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147867163" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:37:17 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 14/15] x86: Make software tag-based kasan available Date: Tue, 4 Feb 2025 18:33:55 +0100 Message-ID: <794a931acfb8e73e28c02932ef08bed9254f164e.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_173730_610192_7FC188FC X-CRM114-Status: GOOD ( 10.97 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Make CONFIG_KASAN_SW_TAGS available for x86 machines if they have ADDRESS_MASKING enabled (LAM) as that works similarly to Top-Byte Ignore (TBI) that allows the software tag-based mode on arm64 platform. Set scale macro based on KASAN mode: in software tag-based mode 32 bytes of memory map to one shadow byte and 16 in generic mode. Signed-off-by: Maciej Wieczor-Retman --- arch/x86/Kconfig | 8 ++++++++ arch/x86/boot/compressed/misc.h | 2 ++ arch/x86/include/asm/kasan.h | 2 +- arch/x86/kernel/setup.c | 2 ++ 4 files changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index dfec7bc692d4..afbcf27ad278 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -36,6 +36,7 @@ config X86_64 select ARCH_HAS_ELFCORE_COMPAT select ZONE_DMA32 select EXECMEM if DYNAMIC_FTRACE + select ARCH_HAS_KASAN_SW_TAGS_DENSE config FORCE_DYNAMIC_FTRACE def_bool y @@ -190,6 +191,7 @@ config X86 select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 select HAVE_ARCH_KASAN_VMALLOC if X86_64 + select HAVE_ARCH_KASAN_SW_TAGS if ADDRESS_MASKING select HAVE_ARCH_KFENCE select HAVE_ARCH_KMSAN if X86_64 select HAVE_ARCH_KGDB @@ -394,6 +396,12 @@ config KASAN_SHADOW_OFFSET hex default 0xdffffc0000000000 if KASAN_GENERIC +config KASAN_SHADOW_SCALE_SHIFT + int + default 5 if KASAN_SW_TAGS_DENSE + default 4 if KASAN_SW_TAGS + default 3 + config HAVE_INTEL_TXT def_bool y depends on INTEL_IOMMU && ACPI diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index dd8d1a85f671..397a70558ffa 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -13,6 +13,8 @@ #undef CONFIG_PARAVIRT_SPINLOCKS #undef CONFIG_KASAN #undef CONFIG_KASAN_GENERIC +#undef CONFIG_KASAN_SW_TAGS +#undef CONFIG_KASAN_SW_TAGS_DENSE #define __NO_FORTIFY diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h index 79151356d5f2..99ff4ae83bf7 100644 --- a/arch/x86/include/asm/kasan.h +++ b/arch/x86/include/asm/kasan.h @@ -6,7 +6,7 @@ #include #include -#define KASAN_SHADOW_SCALE_SHIFT 3 +#define KASAN_SHADOW_SCALE_SHIFT CONFIG_KASAN_SHADOW_SCALE_SHIFT /* * Compiler uses shadow offset assuming that addresses start diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index f1fea506e20f..c300274e205a 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -1121,6 +1121,8 @@ void __init setup_arch(char **cmdline_p) kasan_init(); + kasan_init_sw_tags(); + /* * Sync back kernel address range. * From patchwork Tue Feb 4 17:33:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Wieczor-Retman X-Patchwork-Id: 13959786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B1DBC02198 for ; Tue, 4 Feb 2025 20:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+FmzwlsGm+7Ei8F3uTSXMYBstg1XaByfBTyI3lyqqu8=; b=lHpy2HJmuqA9SO rBGwnqxdPhwEsQdAVgrFEyvSOEq/MS9WJy67WO7p100vV1Zabaz2VcQq2973klx/S0ghIFNili09b biRlMryJnqxdSXnsMksJk/jcYmYZCKiBhPPC2R6oG4wwQKkEfqn5drhipsbgUm4vCTxjcDnBAvaPq hHhF3o7h26HCvkOnZAfRd6yVQgEwIqgpYZTnC6r5pbJtaxGHE8gCxQsAhMrboypdgVgyyK6xGz10g aVxqMVAEnp6Otjlyx3SMS99rf2NW6A6GmntRSlcSy9C345SskgEMzgJAmHwFHbBBd+AY3p4U0Ut6V I6YqVeDcNqq2waiZUfsA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tfP6Y-00000001SaS-31vK; Tue, 04 Feb 2025 20:01:06 +0000 Received: from mgamail.intel.com ([198.175.65.20]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tfMrm-000000018Dt-0ERI; Tue, 04 Feb 2025 17:37:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738690662; x=1770226662; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ht7/TOmkT+iwiNKb8KTAnbXFoYcUy6oM0kb2gCQ20Wc=; b=F7XPcWxSQup1ygaEE4vV/PWueMPDhbOr0FzR6ddIpOXvVpoZSVPg1nE/ kcG1v8fwsa+EsIZRi0rP4HHzE1wXY+tHjxZHWq+nRwAE/oSQRCCYpDqBQ yijZXa1EbEH+U5IFB4twvX2G5i6atp0HhpKOlXXXTkOnVFh/DOerx5R1F kccGlRtx0y8D6qB9vcEAKUcdYGlcHpipjgdDkBiDQ3CpHHtopKJZKABzo 4JSJ5HgXSZ+iO5rwG0DmxYGlVNK/7O9FT5/ZFJCa5KJll48xBpoV+tarO tyVf4cv8Lm7byHgk7l/qd+uz9xXXSm+3A6rvFasEDSM1DZ3vQOy+h/v6G w==; X-CSE-ConnectionGUID: rRMZ7G2kTxm/JpOco32ztA== X-CSE-MsgGUID: 5LUiSiEgQ0K4nAEyFCHpYg== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="38931220" X-IronPort-AV: E=Sophos;i="6.13,259,1732608000"; d="scan'208";a="38931220" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:37:41 -0800 X-CSE-ConnectionGUID: UAr5/zW7Rha56MjtNiYK0Q== X-CSE-MsgGUID: 2nPTC7p0TeC/cQLyC4Ywqw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="147867266" Received: from mjarzebo-mobl1.ger.corp.intel.com (HELO wieczorr-mobl1.intel.com) ([10.245.244.61]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2025 09:37:29 -0800 From: Maciej Wieczor-Retman To: luto@kernel.org, xin@zytor.com, kirill.shutemov@linux.intel.com, palmer@dabbelt.com, tj@kernel.org, andreyknvl@gmail.com, brgerst@gmail.com, ardb@kernel.org, dave.hansen@linux.intel.com, jgross@suse.com, will@kernel.org, akpm@linux-foundation.org, arnd@arndb.de, corbet@lwn.net, maciej.wieczor-retman@intel.com, dvyukov@google.com, richard.weiyang@gmail.com, ytcoode@gmail.com, tglx@linutronix.de, hpa@zytor.com, seanjc@google.com, paul.walmsley@sifive.com, aou@eecs.berkeley.edu, justinstitt@google.com, jason.andryuk@amd.com, glider@google.com, ubizjak@gmail.com, jannh@google.com, bhe@redhat.com, vincenzo.frascino@arm.com, rafael.j.wysocki@intel.com, ndesaulniers@google.com, mingo@redhat.com, catalin.marinas@arm.com, junichi.nomura@nec.com, nathan@kernel.org, ryabinin.a.a@gmail.com, dennis@kernel.org, bp@alien8.de, kevinloughlin@google.com, morbo@google.com, dan.j.williams@intel.com, julian.stecklina@cyberus-technology.de, peterz@infradead.org, cl@linux.com, kees@kernel.org Cc: kasan-dev@googlegroups.com, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, llvm@lists.linux.dev, linux-doc@vger.kernel.org Subject: [PATCH 15/15] kasan: Add mititgation and debug modes Date: Tue, 4 Feb 2025 18:33:56 +0100 Message-ID: <450a1fe078b0e07bf2e4f3098c9110c9959c6524.1738686764.git.maciej.wieczor-retman@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250204_093742_172574_C5B52C12 X-CRM114-Status: GOOD ( 12.99 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org With smaller memory footprint KASAN could be used in production systems. One problem is that saving stacktraces slowes memory allocation substantially - with KASAN enabled up to 90% of time spent on kmalloc() is spent on saving the stacktrace. Add mitigation mode to allow the option for running KASAN focused on performance and security. In mitigation mode disable saving stacktraces and set fault mode to always panic on KASAN error as a security mechanism. Signed-off-by: Maciej Wieczor-Retman --- lib/Kconfig.kasan | 28 ++++++++++++++++++++++++++++ mm/kasan/report.c | 4 ++++ mm/kasan/tags.c | 5 +++++ 3 files changed, 37 insertions(+) diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan index d08b4e9bf477..6daa62b40dea 100644 --- a/lib/Kconfig.kasan +++ b/lib/Kconfig.kasan @@ -244,4 +244,32 @@ config KASAN_SW_TAGS_DENSE ARCH_HAS_KASAN_SW_TAGS_DENSE is needed for this option since the special tag macros need to be properly set for 4-bit wide tags. +choice + prompt "KASAN operation mode" + default KASAN_OPERATION_DEBUG + help + Choose between the mitigation or debug operation modes. + + The first one disables stacktrace saving and enables panic on error. + Faster memory allocation but less information. The second one is the + default where KASAN operates with full functionality. + +config KASAN_OPERATION_DEBUG + bool "Debug operation mode" + depends on KASAN + help + The default mode. Full functionality and all boot parameters + available. + +config KASAN_OPERATION_MITIGATION + bool "Mitigation operation mode" + depends on KASAN + help + Operation mode dedicated at faster operation at the cost of less + information collection. Disables stacktrace saving for faster + allocations and forces panic on KASAN error to mitigate malicious + attacks. + +endchoice + endif # KASAN diff --git a/mm/kasan/report.c b/mm/kasan/report.c index ee9e406b0cdb..ae989d3bd919 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -47,7 +47,11 @@ enum kasan_arg_fault { KASAN_ARG_FAULT_PANIC_ON_WRITE, }; +#ifdef CONFIG_KASAN_OPERATION_MITIGATION +static enum kasan_arg_fault kasan_arg_fault __ro_after_init = KASAN_ARG_FAULT_PANIC; +#else static enum kasan_arg_fault kasan_arg_fault __ro_after_init = KASAN_ARG_FAULT_DEFAULT; +#endif /* kasan.fault=report/panic */ static int __init early_kasan_fault(char *arg) diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c index c111d98961ed..2414cddeaaf3 100644 --- a/mm/kasan/tags.c +++ b/mm/kasan/tags.c @@ -78,6 +78,11 @@ early_param("kasan.stack_ring_size", early_kasan_flag_stack_ring_size); void __init kasan_init_tags(void) { + if (IS_ENABLED(CONFIG_KASAN_OPERATION_MITIGATION)) { + static_branch_disable(&kasan_flag_stacktrace); + return; + } + switch (kasan_arg_stacktrace) { case KASAN_ARG_STACKTRACE_DEFAULT: /* Default is specified by kasan_flag_stacktrace definition. */