From patchwork Tue Oct 22 01:57:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13844971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66FF7D17158 for ; Tue, 22 Oct 2024 01:59:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAFC66B00A6; Mon, 21 Oct 2024 21:59:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A37E76B00A7; Mon, 21 Oct 2024 21:59:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 865AB6B00A8; Mon, 21 Oct 2024 21:59:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4F8666B00A6 for ; Mon, 21 Oct 2024 21:59:36 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A4AB6120E45 for ; Tue, 22 Oct 2024 01:59:21 +0000 (UTC) X-FDA: 82699581048.10.07808FB Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf17.hostedemail.com (Postfix) with ESMTP id 0B1EA4000E for ; Tue, 22 Oct 2024 01:59:21 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=SZJtKdg0; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729562297; a=rsa-sha256; cv=none; b=gkCDiCBB8txhU/T9WfulqpNUICwsjoIUfMCNxdNzE9UNaj/GF5dKWsqb1VnyyWL97CmRH3 w6HExA5otnj9W9nnQjW24e9FpG+ZvVnZt25S02tYF9lp9zSWcr1UKNnNitQjVQFYwcoAa4 pLEQeLIAwRt6XMdOg1svvcnEc0OXcac= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=SZJtKdg0; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729562297; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=71aQqKWMHXYc2akd1hnznLqXP6CqCFxBPNX0wi4YbCg=; b=tRppwpRKOQbheGGuwTAEJW/nJavh/fjAtFk3Ft7G44IuqpbRcxrur0rxD4sJ0gNIoH73rW L9VEPPXVzQFUsFGT2TqZ5VdYpU+BKnaRgT2rdL42wXg69xoxkpYeDiXPZvdprI2ynma6sr zKe2dxGVVETfwewvkh9i2QbLD90GhMY= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-20ca1b6a80aso47740885ad.2 for ; Mon, 21 Oct 2024 18:59:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1729562373; x=1730167173; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=71aQqKWMHXYc2akd1hnznLqXP6CqCFxBPNX0wi4YbCg=; b=SZJtKdg05Z5hV3SByle1shFlegDcuZoTnbT3KWY/RCXHvz2NgrUtFug/8peh11+4RU Hl7+4zyM5RuQF4sCD45pfodvD2q1Vsp4Y5li1VT7W6Z5tumBIIxLttaum12ZFZtvMlwI TpD5G+6kcJF1REIlWu4eQoTwpnpaGI1d9ybE58cofcrbrPf/wJO76cF5HcZohmoWDFft //r671e2z1c3vuLEM6Ishd7A7WN5ZAC8oO5A26+sVk+ZP5L7zXO/en3XBqdAefFi8hf0 0O9ZTSN4HaP2MS2pXRZUXO/y30BsYDgdmFvbfI6UiqBBSi0KY9Qlg3XQNwp4BhYTaLr6 Nd2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729562373; x=1730167173; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=71aQqKWMHXYc2akd1hnznLqXP6CqCFxBPNX0wi4YbCg=; b=uTr2jip3AuvBkGB2jBijBnxwNNThQG4ReT1bWlFy/uv/DxR51lLFohd7bTZWwOC4Gc 16v1j71WSpNSgyciAPoCwoQP02OlHDwaInmgdSN8xREa63S57f7YrrbNv023KhHy4tE4 laK3r2O6LCIt3GZSdoZf4cYAKt/R46KCvLfbeLRTWLl2LHMpbqk17wZzWZh6R0R1PzCc phtuvWdfRlZrnMgQ+lKOiTT20fkng1ArCq2IevM2cDQEujtuzUhXGnOUEq3JtoGeYdmd 7hHZTuXJXSEsH7NMO3PH0o/Yh5bD3RHzP5r1U/JTdRdgi4KPsPpOriLdJkezPDVU/CpY xYVw== X-Forwarded-Encrypted: i=1; AJvYcCUHtW9ssztjkNIx9qZQiS+YO+spC9qQxpXeQJjm8Ab5WqdhZpfz2iuOVlQUKqVXa253V92jQjtGAQ==@kvack.org X-Gm-Message-State: AOJu0YylRN9TSGp7Wh2HEebKaYWe/YRuQM2/sbQry4UI5EGlKLlA9EOM XCVejUkwb9PxJmIlXfzeIGIX3nO4xyc9u+NaPfnBuRdtzmI4EdhYgpdlHBMQDOs= X-Google-Smtp-Source: AGHT+IFV2z0GV96s/ePkXg1tapAZlwikiDMmkPYrz9DE1sLSgHPnQXvAoedURgE4MOGtScwinFMLZA== X-Received: by 2002:a05:6a21:2d08:b0:1d9:1906:a68e with SMTP id adf61e73a8af0-1d92c56cd23mr19203945637.34.1729562372514; Mon, 21 Oct 2024 18:59:32 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71ec132ffdcsm3600710b3a.46.2024.10.21.18.59.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Oct 2024 18:59:32 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, Catalin Marinas , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Will Deacon , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [PATCH v2 9/9] riscv: Implement KASAN_SW_TAGS Date: Mon, 21 Oct 2024 18:57:17 -0700 Message-ID: <20241022015913.3524425-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20241022015913.3524425-1-samuel.holland@sifive.com> References: <20241022015913.3524425-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Stat-Signature: g9zzd1pyd3ayngesqxutkikctbmhu1qn X-Rspamd-Queue-Id: 0B1EA4000E X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1729562361-80478 X-HE-Meta: U2FsdGVkX19I1xK2HSqX66ttAEuBLFLcsQu5omzYDDJ8OG2btEf4HzKIVz9cmHSUZ9dRzpmWtz4N7SzgTwqyyr57Iixm+Pca6Mm+7p8JRGUkZsqJlMj+shQlw7rPB6cr8coi0I57lpKqA43HPsDM++Wc6ndOvGsqKVq+dPAQEC7Vi1o2bQemv7VY3+FCU3eviVBKJT0+VCOA5pn6XsWXTSt5550CaC17VWILu2KQsHdTgyDPq+/wLoXGmdJo5GiHiu6TOosw0beUWxCKhe5cVINxsl/S9Qa42sIqaLRavdBoib4qlnKLQPdGR/tZMN7xMZqHLw8fDCqsH5EWv1uBIUbHXJuOdJrUEzDAC853sxTYvl1vdVaeoT6zim/Rwga/Y0cc6N9RWk5g7h4rykvHAU/Rts9OEVMRXoohejHNzuNf4MnFrvGb9s+tTMhgGIk7b/Xg+ED4ewP9eVn671Nx594N3q5G6x3TnkGtC6O5O1si2xq/Ip8xIBpMtmHK+Y9I2noo+Q/GziqPZbo3F4zmA9RNzHpVA+pHV+sLZc5Ab2kFKxlPx5Aq21Kj/UaJ59fIbqVKX03O1koOrcougCIzUPQvCNomZItwoGmB/S3Hlk8T7WhBgIB8FSijGrwtw+7yV0Jx3BRApWjz5t7pZQICW+k8mCYmXJCP5Qd+LFMOSCHNIEs/mT4nLjPWnsASdxW6PB9U11U8+Tj8UtB4eKA9/w64Efa5E39B69TkpJvsgwkvbCtOhF683Z/AMMnWC1X8hHrOy/KlQtDpR5RYpZKUjWElky4jUGIlLWPCUcx9CbljiygDnGrYEipmMxsN5BxFS7rzJu+9BmQ2Hn7JrJYgmbz2v0RKy3yvbqe0b8J7ZROVWHQyI+imV+JC01Agx586wpb/tOQvaSDUfMsiFvKiy6YHcXBU5DvH5KxfpueK0fGy2ILF/c+jZ/hmsim0gPp+QJWwq1vlejbPteasjSG ZVMYHcHQ HTsriNLlmVebNjcVoQU2rrz1jLEKm3vJ86SWFT6IydfocD/IS/GAsWvF7A74ePuSJ3WfTT87hzb1DTv5PDRrGqmFW/e6Z2pGysRMo8xIzN4fhc4+2F4JDgIq8XzrEZnNGOxO4t9FSsFwHUMTopA5CAFTk6tS5Tq5nkZHONis5Lxz7wFu0NTmLJv85LstZ7wgX0sLNvqaVgg6QzIqcuDiJD4XTb4NGZM2dy9mbgt6EDdS6H7TvN+2gZjFt6rjNRsymgv/lhXF5cSr8/QG/jL2xvfbjzc5l6bWm8gFHir1awe4NGmnvkRDGe3L1uq8ns9NXyDQK9MGkDHqIAg25V4PTw/GZjxsl/iC81NGspCabk00aUdoewnDgLG2VdvQMuqtiSJl0wbVd6CyMbOEqsjqsLKNyFUHUrfN+cPaez5cqYglTDWhCGlaPhiOA0BASfTJaY0Lu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement support for software tag-based KASAN using the RISC-V pointer masking extension, which supports 7 and/or 16-bit tags. This implemen- tation uses 7-bit tags, so it is compatible with either hardware mode. Pointer masking is an optional ISA extension, and it must be enabled using an SBI call to firmware on each CPU. This SBI call must be made very early in smp_callin(), as dereferencing any tagged pointers before that point will crash the kernel. If the SBI call fails on the boot CPU, then KASAN is globally disabled, and the kernel boots normally (unless stack tagging is enabled). If the SBI call fails on any other CPU, that CPU is excluded from the system. When pointer masking is enabled for the kernel's privilege mode, it must be more careful about accepting tagged pointers from userspace. Normally, __access_ok() accepts tagged aliases of kernel memory as long as the MSB is zero, since those addresses cannot be dereferenced -- they will cause a page fault in the uaccess routines. But when the kernel is using pointer masking, those addresses are dereferenceable, so __access_ok() must specifically check the most-significant non-tag bit instead of the MSB. Pointer masking does not apply to the operands of fence instructions, so software is responsible for untagging those addresses. Signed-off-by: Samuel Holland --- Changes in v2: - Fix build error with KASAN_GENERIC - Use symbolic definitons for SBI firmware features call - Update indentation in scripts/Makefile.kasan - Use kasan_params to set hwasan-generate-tags-with-calls=1 Documentation/dev-tools/kasan.rst | 14 ++++--- arch/riscv/Kconfig | 4 +- arch/riscv/include/asm/cache.h | 4 ++ arch/riscv/include/asm/kasan.h | 20 ++++++++++ arch/riscv/include/asm/page.h | 19 ++++++++-- arch/riscv/include/asm/pgtable.h | 6 +++ arch/riscv/include/asm/tlbflush.h | 4 +- arch/riscv/kernel/setup.c | 6 +++ arch/riscv/kernel/smpboot.c | 8 +++- arch/riscv/lib/Makefile | 2 + arch/riscv/lib/kasan_sw_tags.S | 61 +++++++++++++++++++++++++++++++ arch/riscv/mm/kasan_init.c | 32 +++++++++++++++- arch/riscv/mm/physaddr.c | 4 ++ scripts/Makefile.kasan | 5 +++ 14 files changed, 174 insertions(+), 15 deletions(-) create mode 100644 arch/riscv/lib/kasan_sw_tags.S diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index d7de44f5339d..6548aebac57f 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -22,8 +22,8 @@ architectures, but it has significant performance and memory overheads. Software Tag-Based KASAN or SW_TAGS KASAN, enabled with CONFIG_KASAN_SW_TAGS, can be used for both debugging and dogfood testing, similar to userspace HWASan. -This mode is only supported for arm64, but its moderate memory overhead allows -using it for testing on memory-restricted devices with real workloads. +This mode is only supported on arm64 and riscv, but its moderate memory overhead +allows using it for testing on memory-restricted devices with real workloads. Hardware Tag-Based KASAN or HW_TAGS KASAN, enabled with CONFIG_KASAN_HW_TAGS, is the mode intended to be used as an in-field memory bug detector or as a @@ -340,12 +340,14 @@ Software Tag-Based KASAN ~~~~~~~~~~~~~~~~~~~~~~~~ Software Tag-Based KASAN uses a software memory tagging approach to checking -access validity. It is currently only implemented for the arm64 architecture. +access validity. It is currently only implemented for the arm64 and riscv +architectures. Software Tag-Based KASAN uses the Top Byte Ignore (TBI) feature of arm64 CPUs -to store a pointer tag in the top byte of kernel pointers. It uses shadow memory -to store memory tags associated with each 16-byte memory cell (therefore, it -dedicates 1/16th of the kernel memory for shadow memory). +or the pointer masking (Sspm) feature of RISC-V CPUs to store a pointer tag in +the top byte of kernel pointers. It uses shadow memory to store memory tags +associated with each 16-byte memory cell (therefore, it dedicates 1/16th of the +kernel memory for shadow memory). On each memory allocation, Software Tag-Based KASAN generates a random tag, tags the allocated memory with this tag, and embeds the same tag into the returned diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 62545946ecf4..d08b99f6bf76 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -121,6 +121,7 @@ config RISCV select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL select HAVE_ARCH_JUMP_LABEL_RELATIVE if !XIP_KERNEL select HAVE_ARCH_KASAN if MMU && 64BIT + select HAVE_ARCH_KASAN_SW_TAGS if MMU && 64BIT select HAVE_ARCH_KASAN_VMALLOC if MMU && 64BIT select HAVE_ARCH_KFENCE if MMU && 64BIT select HAVE_ARCH_KGDB if !XIP_KERNEL @@ -291,7 +292,8 @@ config PAGE_OFFSET config KASAN_SHADOW_OFFSET hex - depends on KASAN_GENERIC + depends on KASAN + default 0xffffffff00000000 if KASAN_SW_TAGS default 0xdfffffff00000000 if 64BIT default 0xffffffff if 32BIT diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h index 570e9d8acad1..232288a060c6 100644 --- a/arch/riscv/include/asm/cache.h +++ b/arch/riscv/include/asm/cache.h @@ -16,6 +16,10 @@ #define ARCH_KMALLOC_MINALIGN (8) #endif +#ifdef CONFIG_KASAN_SW_TAGS +#define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT) +#endif + /* * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that * the flat loader aligns it accordingly. diff --git a/arch/riscv/include/asm/kasan.h b/arch/riscv/include/asm/kasan.h index a4e92ce9fa31..f6b378ba936d 100644 --- a/arch/riscv/include/asm/kasan.h +++ b/arch/riscv/include/asm/kasan.h @@ -25,7 +25,11 @@ * KASAN_SHADOW_OFFSET = KASAN_SHADOW_END - * (1ULL << (64 - KASAN_SHADOW_SCALE_SHIFT)) */ +#if defined(CONFIG_KASAN_GENERIC) #define KASAN_SHADOW_SCALE_SHIFT 3 +#elif defined(CONFIG_KASAN_SW_TAGS) +#define KASAN_SHADOW_SCALE_SHIFT 4 +#endif #define KASAN_SHADOW_SIZE (UL(1) << ((VA_BITS - 1) - KASAN_SHADOW_SCALE_SHIFT)) /* @@ -37,6 +41,14 @@ #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) +#ifdef CONFIG_KASAN_SW_TAGS +#define KASAN_TAG_KERNEL 0x7f /* native kernel pointers tag */ +#endif + +#define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) +#define arch_kasan_reset_tag(addr) __tag_reset(addr) +#define arch_kasan_get_tag(addr) __tag_get(addr) + void kasan_init(void); asmlinkage void kasan_early_init(void); void kasan_swapper_init(void); @@ -48,5 +60,13 @@ void kasan_swapper_init(void); #endif /* CONFIG_KASAN */ +#ifdef CONFIG_KASAN_SW_TAGS +bool kasan_boot_cpu_enabled(void); +int kasan_cpu_enable(void); +#else +static inline bool kasan_boot_cpu_enabled(void) { return false; } +static inline int kasan_cpu_enable(void) { return 0; } +#endif + #endif #endif /* __ASM_KASAN_H */ diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 6e2f79cf77c5..43c625a4894d 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -89,6 +89,16 @@ typedef struct page *pgtable_t; #define PTE_FMT "%08lx" #endif +#ifdef CONFIG_KASAN_SW_TAGS +#define __tag_set(addr, tag) ((void *)((((u64)(addr) << 7) >> 7) | ((u64)(tag) << 57))) +#define __tag_reset(addr) ((void *)((s64)((u64)(addr) << 7) >> 7)) +#define __tag_get(addr) ((u8)((u64)(addr) >> 57)) +#else +#define __tag_set(addr, tag) (addr) +#define __tag_reset(addr) (addr) +#define __tag_get(addr) 0 +#endif + #if defined(CONFIG_64BIT) && defined(CONFIG_MMU) /* * We override this value as its generic definition uses __pa too early in @@ -168,7 +178,7 @@ phys_addr_t linear_mapping_va_to_pa(unsigned long x); #endif #define __va_to_pa_nodebug(x) ({ \ - unsigned long _x = x; \ + unsigned long _x = (unsigned long)__tag_reset(x); \ is_linear_mapping(_x) ? \ linear_mapping_va_to_pa(_x) : kernel_mapping_va_to_pa(_x); \ }) @@ -192,7 +202,10 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x); #define pfn_to_virt(pfn) (__va(pfn_to_phys(pfn))) #define virt_to_page(vaddr) (pfn_to_page(virt_to_pfn(vaddr))) -#define page_to_virt(page) (pfn_to_virt(page_to_pfn(page))) +#define page_to_virt(page) ({ \ + __typeof__(page) __page = page; \ + __tag_set(pfn_to_virt(page_to_pfn(__page)), page_kasan_tag(__page)); \ +}) #define page_to_phys(page) (pfn_to_phys(page_to_pfn(page))) #define phys_to_page(paddr) (pfn_to_page(phys_to_pfn(paddr))) @@ -209,7 +222,7 @@ static __always_inline void *pfn_to_kaddr(unsigned long pfn) #endif /* __ASSEMBLY__ */ #define virt_addr_valid(vaddr) ({ \ - unsigned long _addr = (unsigned long)vaddr; \ + unsigned long _addr = (unsigned long)__tag_reset(vaddr); \ (unsigned long)(_addr) >= PAGE_OFFSET && pfn_valid(virt_to_pfn(_addr)); \ }) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index e79f15293492..ae6fa9dba0fc 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -916,7 +916,13 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) */ #ifdef CONFIG_64BIT #define TASK_SIZE_64 (PGDIR_SIZE * PTRS_PER_PGD / 2) +/* + * When pointer masking is enabled for the kernel's privilege mode, + * __access_ok() must reject tagged aliases of kernel memory. + */ +#ifndef CONFIG_KASAN_SW_TAGS #define TASK_SIZE_MAX LONG_MAX +#endif #ifdef CONFIG_COMPAT #define TASK_SIZE_32 (_AC(0x80000000, UL) - PAGE_SIZE) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 72e559934952..68b3a85c6960 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -31,14 +31,14 @@ static inline void local_flush_tlb_all_asid(unsigned long asid) /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_SFENCE_VMA_ADDR(addr); + ALT_SFENCE_VMA_ADDR(__tag_reset(addr)); } static inline void local_flush_tlb_page_asid(unsigned long addr, unsigned long asid) { if (asid != FLUSH_TLB_NO_ASID) - ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + ALT_SFENCE_VMA_ADDR_ASID(__tag_reset(addr), asid); else local_flush_tlb_page(addr); } diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..fdc72edc4857 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -299,6 +299,12 @@ void __init setup_arch(char **cmdline_p) riscv_user_isa_enable(); } +void __init smp_prepare_boot_cpu(void) +{ + if (kasan_boot_cpu_enabled()) + kasan_init_sw_tags(); +} + bool arch_cpu_is_hotpluggable(int cpu) { return cpu_has_hotplug(cpu); diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index 0f8f1c95ac38..a1cc555691b0 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -210,7 +211,11 @@ void __init smp_cpus_done(unsigned int max_cpus) asmlinkage __visible void smp_callin(void) { struct mm_struct *mm = &init_mm; - unsigned int curr_cpuid = smp_processor_id(); + unsigned int curr_cpuid; + + /* Must be called first, before referencing any dynamic allocations */ + if (kasan_boot_cpu_enabled() && kasan_cpu_enable()) + return; if (has_vector()) { /* @@ -225,6 +230,7 @@ asmlinkage __visible void smp_callin(void) mmgrab(mm); current->active_mm = mm; + curr_cpuid = smp_processor_id(); store_cpu_topology(curr_cpuid); notify_cpu_starting(curr_cpuid); diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 8eec6b69a875..ae36616fe1f5 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -20,3 +20,5 @@ lib-$(CONFIG_RISCV_ISA_ZBC) += crc32.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o + +obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o diff --git a/arch/riscv/lib/kasan_sw_tags.S b/arch/riscv/lib/kasan_sw_tags.S new file mode 100644 index 000000000000..f7d3e0acba6a --- /dev/null +++ b/arch/riscv/lib/kasan_sw_tags.S @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 Google LLC + * Copyright (C) 2024 SiFive + */ + +#include + +/* + * Report a tag mismatch detected by tag-based KASAN. + * + * A compiler-generated thunk calls this with a custom calling convention. + * Upon entry to this function, the following registers have been modified: + * + * x1/ra: clobbered by call to this function + * x2/sp: decremented by 256 + * x6/t1: tag from shadow memory + * x7/t2: tag from pointer + * x10/a0: fault address + * x11/a1: fault description + * x28/t3: clobbered by thunk + * x29/t4: clobbered by thunk + * x30/t5: clobbered by thunk + * x31/t6: clobbered by thunk + * + * The caller has decremented the SP by 256 bytes, and stored the following + * registers in slots on the stack according to their number (sp + 8 * xN): + * + * x1/ra: return address to user code + * x8/s0/fp: saved value from user code + * x10/a0: saved value from user code + * x11/a1: saved value from user code + */ +SYM_CODE_START(__hwasan_tag_mismatch) + /* Store the remaining unclobbered caller-saved regs */ + sd t0, (8 * 5)(sp) + sd a2, (8 * 12)(sp) + sd a3, (8 * 13)(sp) + sd a4, (8 * 14)(sp) + sd a5, (8 * 15)(sp) + sd a6, (8 * 16)(sp) + sd a7, (8 * 17)(sp) + + /* a0 and a1 are already set by the thunk */ + ld a2, (8 * 1)(sp) + call kasan_tag_mismatch + + ld ra, (8 * 1)(sp) + ld t0, (8 * 5)(sp) + ld a0, (8 * 10)(sp) + ld a1, (8 * 11)(sp) + ld a2, (8 * 12)(sp) + ld a3, (8 * 13)(sp) + ld a4, (8 * 14)(sp) + ld a5, (8 * 15)(sp) + ld a6, (8 * 16)(sp) + ld a7, (8 * 17)(sp) + addi sp, sp, 256 + ret +SYM_CODE_END(__hwasan_tag_mismatch) +EXPORT_SYMBOL(__hwasan_tag_mismatch) diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index c301c8d291d2..50f0e7a03cc8 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -11,6 +11,10 @@ #include #include +#ifdef CONFIG_KASAN_SW_TAGS +static bool __kasan_boot_cpu_enabled __ro_after_init; +#endif + /* * Kasan shadow region must lie at a fixed address across sv39, sv48 and sv57 * which is right before the kernel. @@ -323,8 +327,11 @@ asmlinkage void __init kasan_early_init(void) { uintptr_t i; - BUILD_BUG_ON(KASAN_SHADOW_OFFSET != - KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != + KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + else + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END); for (i = 0; i < PTRS_PER_PTE; ++i) set_pte(kasan_early_shadow_pte + i, @@ -356,6 +363,10 @@ asmlinkage void __init kasan_early_init(void) KASAN_SHADOW_START, KASAN_SHADOW_END); local_flush_tlb_all(); + +#ifdef CONFIG_KASAN_SW_TAGS + __kasan_boot_cpu_enabled = !kasan_cpu_enable(); +#endif } void __init kasan_swapper_init(void) @@ -534,3 +545,20 @@ void __init kasan_init(void) csr_write(CSR_SATP, PFN_DOWN(__pa(swapper_pg_dir)) | satp_mode); local_flush_tlb_all(); } + +#ifdef CONFIG_KASAN_SW_TAGS +bool kasan_boot_cpu_enabled(void) +{ + return __kasan_boot_cpu_enabled; +} + +int kasan_cpu_enable(void) +{ + struct sbiret ret; + + ret = sbi_ecall(SBI_EXT_FWFT, SBI_EXT_FWFT_SET, + SBI_FWFT_POINTER_MASKING_PMLEN, 7, 0, 0, 0, 0); + + return sbi_err_map_linux_errno(ret.error); +} +#endif diff --git a/arch/riscv/mm/physaddr.c b/arch/riscv/mm/physaddr.c index 18706f457da7..6d1cf6ffd54e 100644 --- a/arch/riscv/mm/physaddr.c +++ b/arch/riscv/mm/physaddr.c @@ -8,6 +8,8 @@ phys_addr_t __virt_to_phys(unsigned long x) { + x = __tag_reset(x); + /* * Boundary checking aginst the kernel linear mapping space. */ @@ -24,6 +26,8 @@ phys_addr_t __phys_addr_symbol(unsigned long x) unsigned long kernel_start = kernel_map.virt_addr; unsigned long kernel_end = kernel_start + kernel_map.size; + x = __tag_reset(x); + /* * Boundary checking aginst the kernel image mapping. * __pa_symbol should only be used on kernel symbol addresses. diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan index 693dbbebebba..72a8c9d5fb0e 100644 --- a/scripts/Makefile.kasan +++ b/scripts/Makefile.kasan @@ -91,6 +91,11 @@ ifeq ($(call clang-min-version, 150000)$(call gcc-min-version, 130000),y) kasan_params += hwasan-kernel-mem-intrinsic-prefix=1 endif +# RISC-V requires dynamically determining if stack tagging can be enabled. +ifdef CONFIG_RISCV + kasan_params += hwasan-generate-tags-with-calls=1 +endif + endif # CONFIG_KASAN_SW_TAGS # Add all as-supported KASAN LLVM parameters requested by the configuration.