From patchwork Mon Jul 29 14:21:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 11063877 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D40AE13AC for ; Mon, 29 Jul 2019 14:21:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4DC7285EE for ; Mon, 29 Jul 2019 14:21:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B8F9F286DA; Mon, 29 Jul 2019 14:21:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B9C7286B1 for ; Mon, 29 Jul 2019 14:21:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DDB28E0006; Mon, 29 Jul 2019 10:21:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 28EA48E0002; Mon, 29 Jul 2019 10:21:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A39B8E0006; Mon, 29 Jul 2019 10:21:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id D3FDF8E0002 for ; Mon, 29 Jul 2019 10:21:22 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id u21so38579987pfn.15 for ; Mon, 29 Jul 2019 07:21:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=4K6A/BJDfCnYwl6IVii41tr2UtMMY6EbkGcfzGbxqy8=; b=NsvrPYDb2MSc5yfXrcQyRKAlYA9LnGwpVHyytA1X5c93snVb3FiC9YjxKCBVtVQ6JH Cd06tEDzHEQw7Mip0tz74rb9nTd4BRwkoOneGLrp2GqjROMuY/zHCLeWR/KL8j+pA3cr J8bKCx6UMmLF/QiTKCoHNgf6b2E22jNRIBo680eSyRIgCaU2hoGiDMO/S//ReCsn6dJl gl9vdPxGKeDpeBNfUJ+Bl4n00+Jr9qZ2tREUT2QGpY5aN1S2HAYigF6K31bYg1pJRrHX 6QV9jmILwiq+KlAmd9F4RQESYtllo47PLcu7RFkREmFxBtQI6Rb2IGEibYMDc9RNEoTX SuPw== X-Gm-Message-State: APjAAAXcd+ipcrMtTvLjqFgbUZYauZf/DZBgyU6C2XlBAnPOIT+nz8+p jbUvCePbQ2srmKmQ04pYjsI/MlIGjrjRwTyst0PG/UsjTRJIucwJsRWUeq6mPJD4kgAPNywMxqv XpdVG+ROQUt9dr9++H5x6bf0vmeo/06DNAqzY+6uW++Ax93z34YIdKzzIPnZS/QOeUw== X-Received: by 2002:a63:c03:: with SMTP id b3mr40862058pgl.23.1564410082345; Mon, 29 Jul 2019 07:21:22 -0700 (PDT) X-Received: by 2002:a63:c03:: with SMTP id b3mr40861960pgl.23.1564410081046; Mon, 29 Jul 2019 07:21:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1564410081; cv=none; d=google.com; s=arc-20160816; b=bu7Jrj7DMGDMjlmrUOPkvqCTI6tfNWN6k9UlWb7kpqOIW6Rv5UwRDpgBHqehwq/BcU DwJagKq2QtLG5Dh031qtk0gG/KnkD3DVtWf0zQbfkewFVBee/gQGvTT8OQ5qhYvOClpe SFAwdiQKQ21mTONtljS8pox/4lCZ1nms4ePIYNLSqkHyGLMnfUWXgoeV7LrsN+h9YvvJ Tdyy7Zz7CGR5T9SzaTeOzxWj+5Tw6K8xlfkMrTfsQWnjfof+JhEii2U+f0oMp4kO9Vgu WHiDxyBBvB26oybRluW48VcGaPMRf1AGLv+1UHV8UmnekE/1m3CF3IR/ivKGhh57aTQY uTzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=4K6A/BJDfCnYwl6IVii41tr2UtMMY6EbkGcfzGbxqy8=; b=qk7sc4ebu6owK2Bzln65G+8f8otmGEBPQputPzBvWw3ROaZbcTEEn/1KQ94KJ77VUx U0Bm9nG7aSRtaIcudoq0/OE9fyWRQmCXEi+Q2TiW1/4g4HUkn9rD/BqQ8NeFIj0yxOv6 5tAc7xJ1L3NgpwfW61xmSWR60ruu2Nr7KOkHLQQolRJG3hL1EVprJt3ytGEad2wMY1Mj b+5E+tPtM2H42ZXzHLGXQI4mwj4vwac4fgAZAhdRlwaRNz+KGK0sX2wls3l+sLbeaUeB Ns+dKiIbKPFqDkMm3rLlMXU2AUeMR/B712rKy4d6erVd4e1aRNpNEtej407V5XI8EsHh Mjvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@axtens.net header.s=google header.b="k/JHoGQL"; spf=pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) smtp.mailfrom=dja@axtens.net Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 11sor36155665pgh.67.2019.07.29.07.21.20 for (Google Transport Security); Mon, 29 Jul 2019 07:21:21 -0700 (PDT) Received-SPF: pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@axtens.net header.s=google header.b="k/JHoGQL"; spf=pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) smtp.mailfrom=dja@axtens.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4K6A/BJDfCnYwl6IVii41tr2UtMMY6EbkGcfzGbxqy8=; b=k/JHoGQL3OaWjfKuB6ioubmbYvLKnaDcEsQSlZ0kxcHmP5VGUrtqxNo24Jr1UGk7Pg mMC/pY3m20FNhsjPW1tVbcfI++WoT1CENSZoJWSdD6AAvSgsDmfj0tgLWDoIEwvJ8a++ NwUxEPtnkRK5BNLFxvThjsSeZesfAEDgsvTuA= X-Google-Smtp-Source: APXvYqxEBTVWIFfjuDm3vIEII09WI0/WS8SQcmJ44QaatzBLcmtekytmxZx95yK8Os69M/yJP8MEZQ== X-Received: by 2002:a63:9e54:: with SMTP id r20mr70626655pgo.64.1564410080536; Mon, 29 Jul 2019 07:21:20 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id 85sm65677300pfv.130.2019.07.29.07.21.18 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 29 Jul 2019 07:21:19 -0700 (PDT) From: Daniel Axtens To: kasan-dev@googlegroups.com, linux-mm@kvack.org, x86@kernel.org, aryabinin@virtuozzo.com, glider@google.com, luto@kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, dvyukov@google.com Cc: Daniel Axtens Subject: [PATCH v2 1/3] kasan: support backing vmalloc space with real shadow memory Date: Tue, 30 Jul 2019 00:21:06 +1000 Message-Id: <20190729142108.23343-2-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190729142108.23343-1-dja@axtens.net> References: <20190729142108.23343-1-dja@axtens.net> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Hook into vmalloc and vmap, and dynamically allocate real shadow memory to back the mappings. Most mappings in vmalloc space are small, requiring less than a full page of shadow space. Allocating a full shadow page per mapping would therefore be wasteful. Furthermore, to ensure that different mappings use different shadow pages, mappings would have to be aligned to KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE. Instead, share backing space across multiple mappings. Allocate a backing page the first time a mapping in vmalloc space uses a particular page of the shadow region. Keep this page around regardless of whether the mapping is later freed - in the mean time the page could have become shared by another vmalloc mapping. This can in theory lead to unbounded memory growth, but the vmalloc allocator is pretty good at reusing addresses, so the practical memory usage grows at first but then stays fairly stable. This requires architecture support to actually use: arches must stop mapping the read-only zero page over portion of the shadow region that covers the vmalloc space and instead leave it unmapped. This allows KASAN with VMAP_STACK, and will be needed for architectures that do not have a separate module space (e.g. powerpc64, which I am currently working on). Link: https://bugzilla.kernel.org/show_bug.cgi?id=202009 Signed-off-by: Daniel Axtens --- v2: let kasan_unpoison_shadow deal with ranges that do not use a full shadow byte. --- Documentation/dev-tools/kasan.rst | 60 +++++++++++++++++++++++++++++++ include/linux/kasan.h | 16 +++++++++ lib/Kconfig.kasan | 16 +++++++++ lib/test_kasan.c | 26 ++++++++++++++ mm/kasan/common.c | 51 ++++++++++++++++++++++++++ mm/kasan/generic_report.c | 3 ++ mm/kasan/kasan.h | 1 + mm/vmalloc.c | 15 +++++++- 8 files changed, 187 insertions(+), 1 deletion(-) diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index b72d07d70239..35fda484a672 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -215,3 +215,63 @@ brk handler is used to print bug reports. A potential expansion of this mode is a hardware tag-based mode, which would use hardware memory tagging support instead of compiler instrumentation and manual shadow memory manipulation. + +What memory accesses are sanitised by KASAN? +-------------------------------------------- + +The kernel maps memory in a number of different parts of the address +space. This poses something of a problem for KASAN, which requires +that all addresses accessed by instrumented code have a valid shadow +region. + +The range of kernel virtual addresses is large: there is not enough +real memory to support a real shadow region for every address that +could be accessed by the kernel. + +By default +~~~~~~~~~~ + +By default, architectures only map real memory over the shadow region +for the linear mapping (and potentially other small areas). For all +other areas - such as vmalloc and vmemmap space - a single read-only +page is mapped over the shadow area. This read-only shadow page +declares all memory accesses as permitted. + +This presents a problem for modules: they do not live in the linear +mapping, but in a dedicated module space. By hooking in to the module +allocator, KASAN can temporarily map real shadow memory to cover +them. This allows detection of invalid accesses to module globals, for +example. + +This also creates an incompatibility with ``VMAP_STACK``: if the stack +lives in vmalloc space, it will be shadowed by the read-only page, and +the kernel will fault when trying to set up the shadow data for stack +variables. + +CONFIG_KASAN_VMALLOC +~~~~~~~~~~~~~~~~~~~~ + +With ``CONFIG_KASAN_VMALLOC``, KASAN can cover vmalloc space at the +cost of greater memory usage. Currently this is only supported on x86. + +This works by hooking into vmalloc and vmap, and dynamically +allocating real shadow memory to back the mappings. + +Most mappings in vmalloc space are small, requiring less than a full +page of shadow space. Allocating a full shadow page per mapping would +therefore be wasteful. Furthermore, to ensure that different mappings +use different shadow pages, mappings would have to be aligned to +``KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE``. + +Instead, we share backing space across multiple mappings. We allocate +a backing page the first time a mapping in vmalloc space uses a +particular page of the shadow region. We keep this page around +regardless of whether the mapping is later freed - in the mean time +this page could have become shared by another vmalloc mapping. + +This can in theory lead to unbounded memory growth, but the vmalloc +allocator is pretty good at reusing addresses, so the practical memory +usage grows at first but then stays fairly stable. + +This allows ``VMAP_STACK`` support on x86, and enables support of +architectures that do not have a fixed module region. diff --git a/include/linux/kasan.h b/include/linux/kasan.h index cc8a03cc9674..fcabc5a03fca 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -70,8 +70,18 @@ struct kasan_cache { int free_meta_offset; }; +/* + * These functions provide a special case to support backing module + * allocations with real shadow memory. With KASAN vmalloc, the special + * case is unnecessary, as the work is handled in the generic case. + */ +#ifndef CONFIG_KASAN_VMALLOC int kasan_module_alloc(void *addr, size_t size); void kasan_free_shadow(const struct vm_struct *vm); +#else +static inline int kasan_module_alloc(void *addr, size_t size) { return 0; } +static inline void kasan_free_shadow(const struct vm_struct *vm) {} +#endif int kasan_add_zero_shadow(void *start, unsigned long size); void kasan_remove_zero_shadow(void *start, unsigned long size); @@ -194,4 +204,10 @@ static inline void *kasan_reset_tag(const void *addr) #endif /* CONFIG_KASAN_SW_TAGS */ +#ifdef CONFIG_KASAN_VMALLOC +void kasan_cover_vmalloc(unsigned long requested_size, struct vm_struct *area); +#else +static inline void kasan_cover_vmalloc(unsigned long requested_size, struct vm_struct *area) {} +#endif + #endif /* LINUX_KASAN_H */ diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan index 4fafba1a923b..a320dc2e9317 100644 --- a/lib/Kconfig.kasan +++ b/lib/Kconfig.kasan @@ -6,6 +6,9 @@ config HAVE_ARCH_KASAN config HAVE_ARCH_KASAN_SW_TAGS bool +config HAVE_ARCH_KASAN_VMALLOC + bool + config CC_HAS_KASAN_GENERIC def_bool $(cc-option, -fsanitize=kernel-address) @@ -135,6 +138,19 @@ config KASAN_S390_4_LEVEL_PAGING to 3TB of RAM with KASan enabled). This options allows to force 4-level paging instead. +config KASAN_VMALLOC + bool "Back mappings in vmalloc space with real shadow memory" + depends on KASAN && HAVE_ARCH_KASAN_VMALLOC + help + By default, the shadow region for vmalloc space is the read-only + zero page. This means that KASAN cannot detect errors involving + vmalloc space. + + Enabling this option will hook in to vmap/vmalloc and back those + mappings with real shadow memory allocated on demand. This allows + for KASAN to detect more sorts of errors (and to support vmapped + stacks), but at the cost of higher memory usage. + config TEST_KASAN tristate "Module for testing KASAN for bug detection" depends on m && KASAN diff --git a/lib/test_kasan.c b/lib/test_kasan.c index b63b367a94e8..d375246f5f96 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -18,6 +18,7 @@ #include #include #include +#include /* * Note: test functions are marked noinline so that their names appear in @@ -709,6 +710,30 @@ static noinline void __init kmalloc_double_kzfree(void) kzfree(ptr); } +#ifdef CONFIG_KASAN_VMALLOC +static noinline void __init vmalloc_oob(void) +{ + void *area; + + pr_info("vmalloc out-of-bounds\n"); + + /* + * We have to be careful not to hit the guard page. + * The MMU will catch that and crash us. + */ + area = vmalloc(3000); + if (!area) { + pr_err("Allocation failed\n"); + return; + } + + ((volatile char *)area)[3100]; + vfree(area); +} +#else +static void __init vmalloc_oob(void) {} +#endif + static int __init kmalloc_tests_init(void) { /* @@ -752,6 +777,7 @@ static int __init kmalloc_tests_init(void) kasan_strings(); kasan_bitops(); kmalloc_double_kzfree(); + vmalloc_oob(); kasan_restore_multi_shot(multishot); diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 2277b82902d8..15d8f4ad581b 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -568,6 +568,7 @@ void kasan_kfree_large(void *ptr, unsigned long ip) /* The object will be poisoned by page_alloc. */ } +#ifndef CONFIG_KASAN_VMALLOC int kasan_module_alloc(void *addr, size_t size) { void *ret; @@ -603,6 +604,7 @@ void kasan_free_shadow(const struct vm_struct *vm) if (vm->flags & VM_KASAN) vfree(kasan_mem_to_shadow(vm->addr)); } +#endif extern void __kasan_report(unsigned long addr, size_t size, bool is_write, unsigned long ip); @@ -722,3 +724,52 @@ static int __init kasan_memhotplug_init(void) core_initcall(kasan_memhotplug_init); #endif + +#ifdef CONFIG_KASAN_VMALLOC +void kasan_cover_vmalloc(unsigned long requested_size, struct vm_struct *area) +{ + unsigned long shadow_alloc_start, shadow_alloc_end; + unsigned long addr; + unsigned long backing; + pgd_t *pgdp; + p4d_t *p4dp; + pud_t *pudp; + pmd_t *pmdp; + pte_t *ptep; + pte_t backing_pte; + + shadow_alloc_start = ALIGN_DOWN( + (unsigned long)kasan_mem_to_shadow(area->addr), + PAGE_SIZE); + shadow_alloc_end = ALIGN( + (unsigned long)kasan_mem_to_shadow(area->addr + area->size), + PAGE_SIZE); + + addr = shadow_alloc_start; + do { + pgdp = pgd_offset_k(addr); + p4dp = p4d_alloc(&init_mm, pgdp, addr); + pudp = pud_alloc(&init_mm, p4dp, addr); + pmdp = pmd_alloc(&init_mm, pudp, addr); + ptep = pte_alloc_kernel(pmdp, addr); + + /* + * we can validly get here if pte is not none: it means we + * allocated this page earlier to use part of it for another + * allocation + */ + if (pte_none(*ptep)) { + backing = __get_free_page(GFP_KERNEL); + backing_pte = pfn_pte(PFN_DOWN(__pa(backing)), + PAGE_KERNEL); + set_pte_at(&init_mm, addr, ptep, backing_pte); + } + } while (addr += PAGE_SIZE, addr != shadow_alloc_end); + + kasan_unpoison_shadow(area->addr, requested_size); + requested_size = round_up(requested_size, KASAN_SHADOW_SCALE_SIZE); + kasan_poison_shadow(area->addr + requested_size, + area->size - requested_size, + KASAN_VMALLOC_INVALID); +} +#endif diff --git a/mm/kasan/generic_report.c b/mm/kasan/generic_report.c index 36c645939bc9..2d97efd4954f 100644 --- a/mm/kasan/generic_report.c +++ b/mm/kasan/generic_report.c @@ -86,6 +86,9 @@ static const char *get_shadow_bug_type(struct kasan_access_info *info) case KASAN_ALLOCA_RIGHT: bug_type = "alloca-out-of-bounds"; break; + case KASAN_VMALLOC_INVALID: + bug_type = "vmalloc-out-of-bounds"; + break; } return bug_type; diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 014f19e76247..8b1f2fbc780b 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -25,6 +25,7 @@ #endif #define KASAN_GLOBAL_REDZONE 0xFA /* redzone for global variable */ +#define KASAN_VMALLOC_INVALID 0xF9 /* unallocated space in vmapped page */ /* * Stack redzone shadow values diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4fa8d84599b0..8cbcb5056c9b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2012,6 +2012,15 @@ static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, va->vm = vm; va->flags |= VM_VM_AREA; spin_unlock(&vmap_area_lock); + + /* + * If we are in vmalloc space we need to cover the shadow area with + * real memory. If we come here through VM_ALLOC, this is done + * by a higher level function that has access to the true size, + * which might not be a full page. + */ + if (is_vmalloc_addr(vm->addr) && !(vm->flags & VM_ALLOC)) + kasan_cover_vmalloc(vm->size, vm); } static void clear_vm_uninitialized_flag(struct vm_struct *vm) @@ -2483,6 +2492,8 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, if (!addr) return NULL; + kasan_cover_vmalloc(real_size, area); + /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED * flag. It means that vm_struct is not fully initialized. @@ -3324,9 +3335,11 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, spin_unlock(&vmap_area_lock); /* insert all vm's */ - for (area = 0; area < nr_vms; area++) + for (area = 0; area < nr_vms; area++) { setup_vmalloc_vm(vms[area], vas[area], VM_ALLOC, pcpu_get_vm_areas); + kasan_cover_vmalloc(sizes[area], vms[area]); + } kfree(vas); return vms; From patchwork Mon Jul 29 14:21:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 11063879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 80FB51398 for ; Mon, 29 Jul 2019 14:21:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71FA82832D for ; Mon, 29 Jul 2019 14:21:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 66113286A4; Mon, 29 Jul 2019 14:21:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBC692832D for ; Mon, 29 Jul 2019 14:21:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A58A28E0007; Mon, 29 Jul 2019 10:21:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9E0AA8E0002; Mon, 29 Jul 2019 10:21:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F7EC8E0007; Mon, 29 Jul 2019 10:21:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 5D6C88E0002 for ; Mon, 29 Jul 2019 10:21:26 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id b18so38335877pgg.8 for ; Mon, 29 Jul 2019 07:21:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=w7zQctoUbbEOIFpo2YzbJpxQDN+pRrmLCUjs8vOkAsU=; b=Zf3jc8HWC2FhBJTkFsSU0AZw5tKbElkfkf9gyrMCly7EwKD2pqBRMmxbL2L5oOOiBk xV7/7s2LIzV96W9xKWCxlh/uXc4baKaYTKkr72oRjBTRLCYFoEjDkPMaVdGBgJuDKgKY rdoA31hlXofpKN2x6KN94B1JPezEsRLJ5PFyyu26yl36tHyhrq2/hciUIfLgd+T/1lWy rLtPWKuMnW+40SB2D4hx/nuFM3+5z920mXxxEmw+vhkg7SgYoEaFLZM4tRSxZpE1/BKo o7EBH8Y4HWIpLhGBbkZQUl4USCQ+wnqhLnvVMigx0pwxe8K8qw0G/4lMVmI2qNLgsPRI khvQ== X-Gm-Message-State: APjAAAVRcurSMzV1qwM0u06p3JKbaFgPyK3L3bLYtNC04K3rVlpsGGUV w14MNyTJPG/KaGApkdjNJGSMvkXyzlaZlDzB9J2FtyA6ecHjJHTQP6Yisy7UvUmUTiO2UrH9Lea VeqjaMBV4o0b9Eq2ii8jJ0t6ll8iBXxidS6pfwSGKnqIisSWhIhLubeVtSDPWynLlZQ== X-Received: by 2002:a17:90a:ac11:: with SMTP id o17mr113774036pjq.134.1564410086015; Mon, 29 Jul 2019 07:21:26 -0700 (PDT) X-Received: by 2002:a17:90a:ac11:: with SMTP id o17mr113773978pjq.134.1564410085303; Mon, 29 Jul 2019 07:21:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1564410085; cv=none; d=google.com; s=arc-20160816; b=sxGQqQ1RbzerwQLyfl0dPzPwkqYfLlHcjj3MFxTjzx+RNRBSGQNNNuWpz2FJMOC7L6 evY2XkoZOcS/adWRZjfSsiQHfOrEOixflNGC6PPOIThDQC7mfZt9drRUc3o+EjTEtdnm rZmW4bur9dBScO6GR/TqdOUaBO9/82axeVPoeUZ9VfTTv3DHcrn/WLY99ogN/RmanJVb idEzacuAg/R0YeEf1Oy2VoUIVuSyrspp0FIBAOW4r5ZkoMabcGkA0ozFZ6yDu7L/DpRA P08LF1B0akvC78+4jvdYOBbXISmAg3ICahtsaJZTNUNfPsJKibQ+SqVW2/cDtT0iBtYg vBvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=w7zQctoUbbEOIFpo2YzbJpxQDN+pRrmLCUjs8vOkAsU=; b=eDeaih8wlCHwRZmZCWtbnSfgPIpwUwoF+W7kAy0y7CuL4fXWGrtqZdvr7d56zVV4DW 9p/AUUJvGq7JC/8bM9oy5z1CtFfcw1icveWwCdzhHbWgSweYTFVEuPavnEpbbkn9uer9 gvoIHZyIlakFS0VgrwiflxL2YlWjr75ytuyms7/ZrPfRaVAEswSGPT1tksPWLIv3Tt6t 8qpYpnkhICuQwp7QkPyQRQu4iKwq2Z2lNJx8hyWF0k/7CTpkSmT61E9CeI0s35mMobTf lMMR/TP8pH2Zf/oj5gl1SwMCgmC8iBTO3R0w4h+QFbXVxmorncT34sNHi3N/SRdNvfnH 6H0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@axtens.net header.s=google header.b=kZwz3XiA; spf=pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) smtp.mailfrom=dja@axtens.net Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id j8sor43104860pfa.53.2019.07.29.07.21.25 for (Google Transport Security); Mon, 29 Jul 2019 07:21:25 -0700 (PDT) Received-SPF: pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@axtens.net header.s=google header.b=kZwz3XiA; spf=pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) smtp.mailfrom=dja@axtens.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=w7zQctoUbbEOIFpo2YzbJpxQDN+pRrmLCUjs8vOkAsU=; b=kZwz3XiA2dsXPzcGr8BMeHi0Q8r5Ij+/PBK8Vx8c8QCcKJhFrrDBFKITSMR5uQ5Pi8 nKH417WV7u4OJz4rjzW35RiQ402Jwzfx9u4n7eFeVKMP82hZx/7ua8ZWY2bYaTDYcnh3 5FAoGTMbKLPZv0MVBB4IdczUX73bm6ZnSs15E= X-Google-Smtp-Source: APXvYqyh8hzXYVlRyPGlq4xtJAc38Emiq/syXzPg0A0KUJS5nwkYSCuqEx8dzzwpR52NEl2vN1VgIw== X-Received: by 2002:aa7:8502:: with SMTP id v2mr36039354pfn.98.1564410085043; Mon, 29 Jul 2019 07:21:25 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id x14sm78684881pfq.158.2019.07.29.07.21.23 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 29 Jul 2019 07:21:24 -0700 (PDT) From: Daniel Axtens To: kasan-dev@googlegroups.com, linux-mm@kvack.org, x86@kernel.org, aryabinin@virtuozzo.com, glider@google.com, luto@kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, dvyukov@google.com Cc: Daniel Axtens Subject: [PATCH v2 2/3] fork: support VMAP_STACK with KASAN_VMALLOC Date: Tue, 30 Jul 2019 00:21:07 +1000 Message-Id: <20190729142108.23343-3-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190729142108.23343-1-dja@axtens.net> References: <20190729142108.23343-1-dja@axtens.net> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Supporting VMAP_STACK with KASAN_VMALLOC is straightforward: - clear the shadow region of vmapped stacks when swapping them in - tweak Kconfig to allow VMAP_STACK to be turned on with KASAN Reviewed-by: Dmitry Vyukov Signed-off-by: Daniel Axtens --- arch/Kconfig | 9 +++++---- kernel/fork.c | 4 ++++ 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index a7b57dd42c26..e791196005e1 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -825,16 +825,17 @@ config HAVE_ARCH_VMAP_STACK config VMAP_STACK default y bool "Use a virtually-mapped stack" - depends on HAVE_ARCH_VMAP_STACK && !KASAN + depends on HAVE_ARCH_VMAP_STACK + depends on !KASAN || KASAN_VMALLOC ---help--- Enable this if you want the use virtually-mapped kernel stacks with guard pages. This causes kernel stack overflows to be caught immediately rather than causing difficult-to-diagnose corruption. - This is presently incompatible with KASAN because KASAN expects - the stack to map directly to the KASAN shadow map using a formula - that is incorrect if the stack is in vmalloc space. + To use this with KASAN, the architecture must support backing + virtual mappings with real shadow memory, and KASAN_VMALLOC must + be enabled. config ARCH_OPTIONAL_KERNEL_RWX def_bool n diff --git a/kernel/fork.c b/kernel/fork.c index d8ae0f1b4148..ce3150fe8ff2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -94,6 +94,7 @@ #include #include #include +#include #include #include @@ -215,6 +216,9 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node) if (!s) continue; + /* Clear the KASAN shadow of the stack. */ + kasan_unpoison_shadow(s->addr, THREAD_SIZE); + /* Clear stale pointers from reused stack. */ memset(s->addr, 0, THREAD_SIZE); From patchwork Mon Jul 29 14:21:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 11063881 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAB8D1398 for ; Mon, 29 Jul 2019 14:21:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9BA1F28849 for ; Mon, 29 Jul 2019 14:21:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8FC4828872; Mon, 29 Jul 2019 14:21:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1100428849 for ; Mon, 29 Jul 2019 14:21:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F135D8E0008; Mon, 29 Jul 2019 10:21:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EC3478E0002; Mon, 29 Jul 2019 10:21:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB21E8E0008; Mon, 29 Jul 2019 10:21:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id A66EE8E0002 for ; Mon, 29 Jul 2019 10:21:31 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id e25so38600831pfn.5 for ; Mon, 29 Jul 2019 07:21:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UahEbtdXwZNAU+KwZPeJAGjgE2lYz0Dt6XczqvKwLUQ=; b=Ah4T/QcQu7v0H/kxiWOz65Csl3Vr5DKsu5ghXMSbEcTIs3GMOYBa1KLpneX9Pgks94 zGh9F7J2T2NAE7kZm2jmtWRqhlzyEPkiAMbNWbkwN3Pi/ZwxnPRMvOx9+jrB1p4qmNTG Res9c2Viyeqf1OxMcqFemqLxlFKNsvtCtku8nmQNkEHx39vTW2DdOoEZ98pzJNeZDM3h kIeNkqZBMGah1m4GhwwjlQTVhTAadbWsX7hvneiB7r5oZiq8jXKQSKvsRCciXMC/2lG+ or/ZcVmBrZ+gfsVi+bopk0pp2Gl8SsJD/wH/FCQMe/48iH87hrPi/+FMukLdxtXDzqhp b5fg== X-Gm-Message-State: APjAAAVk0KNE5NNDPWYMYBg0E0LXg2pyHZMIHOYzDyhO+EyQkjExJ+pn 4ppafNc9GXWpj25UlMS+T01D6Z8fNTkbwDCbRpXxad9lgxF2Uxj1EZ7jEEm1ApQw51G/2gSU94t DJhLW9Gh4D4jxvf5W5R7Lkb/ZsvV836ROU3HpwPInWTy+1ZQCyTwkum1AWaGGyvxlGw== X-Received: by 2002:a17:902:968d:: with SMTP id n13mr63179961plp.257.1564410091348; Mon, 29 Jul 2019 07:21:31 -0700 (PDT) X-Received: by 2002:a17:902:968d:: with SMTP id n13mr63179903plp.257.1564410090499; Mon, 29 Jul 2019 07:21:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1564410090; cv=none; d=google.com; s=arc-20160816; b=QRKlZp2JcyOE+dAfCgvg1tHxwc98pyh1FVcBSmiXBbktSnqs2EuVobGYvv8QtzoKin 43zs6abyCyheHDmIAKuremyPrxdzaAFBATRv99jgGTjLYYVXJK5LnQ8zf3+AsFMhWeWc vGXE8jtEUQmi6MZJ738fHsiVFUSWcsoOnljfH4CP/36xJPeUoC7wCHrbut+IPzSalGVg 1a7KpmcrcApv3fvSj6c3VF7W9VeRSzs/SoJoBPgA+EYAf2ykSJ+YlPGc8Cw+cbw8Bi/H bO5VVAtZp+UJ7pFJu6dwbX5ol9EDwquBtz8UTp5SmGt9U2tIdaw5pNZKHtmb9zqNc7hT 6ICg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=UahEbtdXwZNAU+KwZPeJAGjgE2lYz0Dt6XczqvKwLUQ=; b=gR7xirBoGajf2YJ7pRkwyhjZma8wzARksbsFnhnEybO6l8QR8VvgD3s9Rwje0RR/cY HkyRPeCLNI86eOLwlUaHFduv8KOyZRNCPFXQDj0dt4bAGLbKwi9XUBToaPXobysVN81d /a8u3oTA+5l7BnUNS4EfzHlSPSWwuSoyUzScLwQuXJbYAHn2dCeAYQWA68ZHXpojBWMN UGJ3wDKGY1yMBcqZ8ZnsiVeZZxtcaYQ4iHyJ2XbiPNS/0IAeG895DMHza3tNx65UfJrw wL2ANl8rFczpj3BLbicUIFM5vtLfenTxCXZocz16oZb7VZ6ammCIwOhbJ1RqviMz3Lnn WrIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@axtens.net header.s=google header.b=fzL9e9NG; spf=pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) smtp.mailfrom=dja@axtens.net Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id n3sor28700942pgn.56.2019.07.29.07.21.30 for (Google Transport Security); Mon, 29 Jul 2019 07:21:30 -0700 (PDT) Received-SPF: pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@axtens.net header.s=google header.b=fzL9e9NG; spf=pass (google.com: domain of dja@axtens.net designates 209.85.220.65 as permitted sender) smtp.mailfrom=dja@axtens.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UahEbtdXwZNAU+KwZPeJAGjgE2lYz0Dt6XczqvKwLUQ=; b=fzL9e9NGg4Qjr2J3UpWztGNQR57AulF/3UQEiR1a9Md80VZwZTIn9TRRubWlaS/Hl4 LOCeBjBaUrIgk2w0oLdXnYFC/x89KdBH/EUp4SGo0wf7qML0Q1zbmstYKKVoAfXixnvH 1UwH3wAFUDqMh2FnvB0ouTVLkQFq2ybEtsObY= X-Google-Smtp-Source: APXvYqzsTWziV3uGfQSZhld3lbCm5Hl80o2+zdoas9rkn9wivbK+frNOV0FOh9c9mgtrfH4egyHamQ== X-Received: by 2002:a63:2026:: with SMTP id g38mr98776818pgg.172.1564410090185; Mon, 29 Jul 2019 07:21:30 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id s66sm65997285pfs.8.2019.07.29.07.21.28 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 29 Jul 2019 07:21:29 -0700 (PDT) From: Daniel Axtens To: kasan-dev@googlegroups.com, linux-mm@kvack.org, x86@kernel.org, aryabinin@virtuozzo.com, glider@google.com, luto@kernel.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, dvyukov@google.com Cc: Daniel Axtens Subject: [PATCH v2 3/3] x86/kasan: support KASAN_VMALLOC Date: Tue, 30 Jul 2019 00:21:08 +1000 Message-Id: <20190729142108.23343-4-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190729142108.23343-1-dja@axtens.net> References: <20190729142108.23343-1-dja@axtens.net> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In the case where KASAN directly allocates memory to back vmalloc space, don't map the early shadow page over it. We prepopulate pgds/p4ds for the range that would otherwise be empty. This is required to get it synced to hardware on boot, allowing the lower levels of the page tables to be filled dynamically. Acked-by: Dmitry Vyukov Signed-off-by: Daniel Axtens --- v2: move from faulting in shadow pgds to prepopulating --- arch/x86/Kconfig | 1 + arch/x86/mm/kasan_init_64.c | 61 +++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 222855cc0158..40562cc3771f 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -134,6 +134,7 @@ config X86 select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if X86_64 + select HAVE_ARCH_KASAN_VMALLOC if X86_64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 296da58f3013..2f57c4ddff61 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -245,6 +245,52 @@ static void __init kasan_map_early_shadow(pgd_t *pgd) } while (pgd++, addr = next, addr != end); } +static void __init kasan_shallow_populate_p4ds(pgd_t *pgd, + unsigned long addr, + unsigned long end, + int nid) +{ + p4d_t *p4d; + unsigned long next; + void *p; + + p4d = p4d_offset(pgd, addr); + do { + next = p4d_addr_end(addr, end); + + if (p4d_none(*p4d)) { + p = early_alloc(PAGE_SIZE, nid, true); + p4d_populate(&init_mm, p4d, p); + } + } while (p4d++, addr = next, addr != end); +} + +static void __init kasan_shallow_populate_pgds(void *start, void *end) +{ + unsigned long addr, next; + pgd_t *pgd; + void *p; + int nid = early_pfn_to_nid((unsigned long)start); + + addr = (unsigned long)start; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, (unsigned long)end); + + if (pgd_none(*pgd)) { + p = early_alloc(PAGE_SIZE, nid, true); + pgd_populate(&init_mm, pgd, p); + } + + /* + * we need to populate p4ds to be synced when running in + * four level mode - see sync_global_pgds_l4() + */ + kasan_shallow_populate_p4ds(pgd, addr, next, nid); + } while (pgd++, addr = next, addr != (unsigned long)end); +} + + #ifdef CONFIG_KASAN_INLINE static int kasan_die_handler(struct notifier_block *self, unsigned long val, @@ -352,9 +398,24 @@ void __init kasan_init(void) shadow_cpu_entry_end = (void *)round_up( (unsigned long)shadow_cpu_entry_end, PAGE_SIZE); + /* + * If we're in full vmalloc mode, don't back vmalloc space with early + * shadow pages. Instead, prepopulate pgds/p4ds so they are synced to + * the global table and we can populate the lower levels on demand. + */ +#ifdef CONFIG_KASAN_VMALLOC + kasan_shallow_populate_pgds( + kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM), + kasan_mem_to_shadow((void *)VMALLOC_END)); + + kasan_populate_early_shadow( + kasan_mem_to_shadow((void *)VMALLOC_END + 1), + shadow_cpu_entry_begin); +#else kasan_populate_early_shadow( kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM), shadow_cpu_entry_begin); +#endif kasan_populate_shadow((unsigned long)shadow_cpu_entry_begin, (unsigned long)shadow_cpu_entry_end, 0);