From patchwork Tue Nov 30 22:08:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 12694245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F0B67C433EF for ; Tue, 30 Nov 2021 22:23:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FE0wqXAQjfzt80Ye02Uoqtd8nHZsOMymTNLWj9uH6mQ=; b=37A0Gnmvraruan a/tUeSxp0urzw4Xr1ZDtvOBJl+f/u8cQ+PHAMYdGJcB6I0z0q9/sC3k2yZnNhpHkVd+VB+CEgTYCe Zulisad9qvasB07R2KlcXyIvpQg8cL1RI++pQ+e8OJmC+SR2NtV6zpjtXCIO/6EWDNgSvf+pzCrU7 IrHaoqmBphxGqT+QlqOJsbDaKOH4IlN1k7Cfgk5KZwGTeO+EblhOtUOXAildm9nCzyPg9paA0oylm 1UaaFKXSGw0dzAcjBTMhur7dJwCwLq5cgTTALL0WH812kWI/3WJKzxv1I0tVjfUOMUfvgj/U8LaQz s7AWCMXsp8rOyMmOs+zg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1msBVp-00702H-Vi; Tue, 30 Nov 2021 22:22:10 +0000 Received: from out1.migadu.com ([91.121.223.63]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1msBIN-006uow-Pp for linux-arm-kernel@lists.infradead.org; Tue, 30 Nov 2021 22:08:18 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1638310094; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HNHQ9BiBKyIqAtWVImrJMYlD8BjFZxN/2CktLIPvhw0=; b=e5hYOgE63daSzhorQx9xqkB3uU5LgMm60aa/Hr6EXq7ubqpCA73UnpNNs4TkF1qZhJ3UKg XPKKCif92EXNbK4AEx7/5+wTtt3Fyt49dD0+4BfOZ+Q54Mx3TIUEnFgRjqn7VBjPAzBuyi lH0orqEWB5TuHTmUnamwQqlsfWyAH2M= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko , Vincenzo Frascino , Catalin Marinas , Peter Collingbourne Cc: Andrey Konovalov , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Andrew Morton , linux-mm@kvack.org, Will Deacon , linux-arm-kernel@lists.infradead.org, Evgenii Stepanov , linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH 27/31] kasan, vmalloc: add vmalloc support to HW_TAGS Date: Tue, 30 Nov 2021 23:08:12 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211130_140816_204812_13E40B81 X-CRM114-Status: GOOD ( 26.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Andrey Konovalov This patch adds vmalloc tagging support to HW_TAGS KASAN. The key difference between HW_TAGS and the other two KASAN modes when it comes to vmalloc: HW_TAGS KASAN can only assign tags to physical memory. The other two modes have shadow memory covering every mapped virtual memory region. This patch makes __kasan_unpoison_vmalloc() for HW_TAGS KASAN: - Skip non-VM_ALLOC mappings as HW_TAGS KASAN can only tag a single mapping of normal physical memory; see the comment in the function. - Generate a random tag, tag the returned pointer and the allocation. - Propagate the tag into the page stucts to allow accesses through page_address(vmalloc_to_page()). The rest of vmalloc-related KASAN hooks are not needed: - The shadow-related ones are fully skipped. - __kasan_poison_vmalloc() is kept as a no-op with a comment. Poisoning of physical pages that are backing vmalloc() allocations is skipped via __GFP_SKIP_KASAN_UNPOISON: __kasan_unpoison_vmalloc() poisons them instead. Signed-off-by: Andrey Konovalov Co-developed-by: Vincenzo Frascino --- include/linux/kasan.h | 27 +++++++++++-- mm/kasan/hw_tags.c | 92 +++++++++++++++++++++++++++++++++++++++++++ mm/kasan/shadow.c | 8 +++- mm/vmalloc.c | 25 +++++++++--- 4 files changed, 143 insertions(+), 9 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 6a2619759e93..df1a09fb7623 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -417,19 +417,40 @@ static inline void kasan_init_hw_tags(void) { } #ifdef CONFIG_KASAN_VMALLOC +#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) + void kasan_populate_early_vm_area_shadow(void *start, unsigned long size); int kasan_populate_vmalloc(unsigned long addr, unsigned long size); void kasan_release_vmalloc(unsigned long start, unsigned long end, unsigned long free_region_start, unsigned long free_region_end); +#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */ + +static inline void kasan_populate_early_vm_area_shadow(void *start, + unsigned long size) +{ } +static inline int kasan_populate_vmalloc(unsigned long start, + unsigned long size) +{ + return 0; +} +static inline void kasan_release_vmalloc(unsigned long start, + unsigned long end, + unsigned long free_region_start, + unsigned long free_region_end) { } + +#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */ + void * __must_check __kasan_unpoison_vmalloc(const void *start, - unsigned long size); + unsigned long size, + bool vm_alloc); static __always_inline void * __must_check kasan_unpoison_vmalloc( - const void *start, unsigned long size) + const void *start, unsigned long size, + bool vm_alloc) { if (kasan_enabled()) - return __kasan_unpoison_vmalloc(start, size); + return __kasan_unpoison_vmalloc(start, size, vm_alloc); return (void *)start; } diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index 76cf2b6229c7..fd3a93dfca42 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -192,6 +192,98 @@ void __init kasan_init_hw_tags(void) kasan_stack_collection_enabled() ? "on" : "off"); } +#ifdef CONFIG_KASAN_VMALLOC + +static void unpoison_vmalloc_pages(const void *addr, u8 tag) +{ + struct vm_struct *area; + int i; + + /* + * As hardware tag-based KASAN only tags VM_ALLOC vmalloc allocations + * (see the comment in __kasan_unpoison_vmalloc), all of the pages + * should belong to a single area. + */ + area = find_vm_area((void *)addr); + if (WARN_ON(!area)) + return; + + for (i = 0; i < area->nr_pages; i++) { + struct page *page = area->pages[i]; + + page_kasan_tag_set(page, tag); + } +} + +void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, + bool vm_alloc) +{ + u8 tag; + unsigned long redzone_start, redzone_size; + + if (!is_vmalloc_or_module_addr(start)) + return (void *)start; + + /* Unpoisoning and pointer tag assignment is skipped for non-VM_ALLOC + * mappings as: + * + * 1. Unlike the software KASAN modes, hardware tag-based KASAN only + * supports tagging physical memory. Therefore, it can only tag a + * single mapping of normal physical pages. + * 2. Hardware tag-based KASAN can only tag memory mapped with special + * mapping protection bits, see arch_vmalloc_pgprot_modify(). + * As non-VM_ALLOC mappings can be mapped outside of vmalloc code, + * providing these bits would require tracking all non-VM_ALLOC + * mappers. + * + * Thus, for VM_ALLOC mappings, hardware tag-based KASAN only tags + * the first virtual mapping, which is created by vmalloc(). + * Tagging the page_alloc memory backing that vmalloc() allocation is + * skipped, see ___GFP_SKIP_KASAN_UNPOISON. + * + * For non-VM_ALLOC allocations, page_alloc memory is tagged as usual. + */ + if (!vm_alloc) + return (void *)start; + + tag = kasan_random_tag(); + start = set_tag(start, tag); + + /* + * Unpoison but don't initialize. The pages have already been + * initialized by page_alloc. + */ + kasan_unpoison(start, size, false); + + /* + * Unlike software KASAN modes, hardware tag-based KASAN doesn't + * unpoison memory when populating shadow for vmalloc() space. + * Thus, it needs to explicitly poison the in-page vmalloc() redzone. + */ + redzone_start = round_up((unsigned long)start + size, KASAN_GRANULE_SIZE); + redzone_size = round_up(redzone_start, PAGE_SIZE) - redzone_start; + kasan_poison((void *)redzone_start, redzone_size, KASAN_TAG_INVALID, false); + + /* + * Set per-page tag flags to allow accessing physical memory for the + * vmalloc() mapping through page_address(vmalloc_to_page()). + */ + unpoison_vmalloc_pages(start, tag); + + return (void *)start; +} + +void __kasan_poison_vmalloc(const void *start, unsigned long size) +{ + /* + * No tagging here. + * The physical pages backing the vmalloc() allocation are poisoned + * through the usual page_alloc paths. + */ +} + +#endif + #if IS_ENABLED(CONFIG_KASAN_KUNIT_TEST) void kasan_enable_tagging_sync(void) diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 4ca280a96fbc..f27d48c24166 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -475,8 +475,14 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end, } } -void *__kasan_unpoison_vmalloc(const void *start, unsigned long size) +void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, + bool vm_alloc) { + /* + * As software tag-based KASAN tags both VM_ALLOC and non-VM_ALLOC + * mappings, the vm_alloc argument is ignored. + */ + if (!is_vmalloc_or_module_addr(start)) return (void *)start; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 82ef1e27e2e4..409a289dec81 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2214,8 +2214,12 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node) return NULL; } - /* Mark the pages as accessible after they were mapped in. */ - mem = kasan_unpoison_vmalloc(mem, size); + /* + * Mark the pages as accessible after they were mapped in. + * With hardware tag-based KASAN, marking is skipped for + * non-VM_ALLOC mappings, see __kasan_unpoison_vmalloc(). + */ + mem = kasan_unpoison_vmalloc(mem, size, false); return mem; } @@ -2449,9 +2453,12 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, * accessible after they are mapped in. * Otherwise, as the pages can be mapped outside of vmalloc code, * mark them now as a best-effort approach. + * With hardware tag-based KASAN, marking is skipped for + * non-VM_ALLOC mappings, see __kasan_unpoison_vmalloc(). */ if (!(flags & VM_ALLOC)) - area->addr = kasan_unpoison_vmalloc(area->addr, requested_size); + area->addr = kasan_unpoison_vmalloc(area->addr, requested_size, + false); return area; } @@ -2849,6 +2856,12 @@ vm_area_alloc_pages(gfp_t gfp, int nid, struct page *page; int i; + /* + * Skip page_alloc poisoning for pages backing VM_ALLOC mappings, + * see __kasan_unpoison_vmalloc. Only effective in HW_TAGS mode. + */ + gfp &= __GFP_SKIP_KASAN_UNPOISON; + /* * For order-0 pages we make use of bulk allocator, if * the page array is partly or not at all populated due @@ -3084,7 +3097,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, * Mark the pages for VM_ALLOC mappings as accessible after they were * mapped in. */ - addr = kasan_unpoison_vmalloc(addr, real_size); + addr = kasan_unpoison_vmalloc(addr, real_size, true); /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -3784,10 +3797,12 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, * Mark allocated areas as accessible. * As the pages are mapped outside of vmalloc code, * mark them now as a best-effort approach. + * With hardware tag-based KASAN, marking is skipped for + * non-VM_ALLOC mappings, see __kasan_unpoison_vmalloc(). */ for (area = 0; area < nr_vms; area++) vms[area]->addr = kasan_unpoison_vmalloc(vms[area]->addr, - vms[area]->size); + vms[area]->size, false); kfree(vas); return vms;