From patchwork Fri Jul 1 14:22:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 12903366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B702C43334 for ; Fri, 1 Jul 2022 14:23:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C9F66B0071; Fri, 1 Jul 2022 10:23:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 979B86B0081; Fri, 1 Jul 2022 10:23:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F4096B0082; Fri, 1 Jul 2022 10:23:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 70AE26B0071 for ; Fri, 1 Jul 2022 10:23:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A519334B48 for ; Fri, 1 Jul 2022 14:23:42 +0000 (UTC) X-FDA: 79638749484.10.907600F Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) by imf06.hostedemail.com (Postfix) with ESMTP id 3F294180033 for ; Fri, 1 Jul 2022 14:23:42 +0000 (UTC) Received: by mail-ed1-f73.google.com with SMTP id z19-20020a05640240d300b00437633081abso1901914edb.0 for ; Fri, 01 Jul 2022 07:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9CmKhXrgq4R//9cDj1zsNax3tT0d2FywVlyLyC3NqDY=; b=U+fVjLRG5oNjqME2B9ccBc/zjLmfwrEmyw/ej3ru79fJemqPnxYlMMrth6VkT3UGNR HYhunHxxINt63ndT6iLXlgZzqmTWlNiZWKNbO5DwAV9ASu3H9cFPMY4yM0S7lI8xD5Rd EdfdxYo8yK9PcvxDjTY8AkAfIw+DM3ORaR0VjSQ6WImxJGGIPu0tQkrosIMl1vZ0/2Ji NWuO1Z/jcEr4SrlBYEg338PXfVrC2Q6IRrJfBy14cKv5aKi1rkr6MnEskoTrso8+Vuwy cR78mRBYVCFAKZOPuO/rTahaUWaznyDz6n12VcGqjAy/jcfSNqAIl2kMglHKUYWNqXly zlCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9CmKhXrgq4R//9cDj1zsNax3tT0d2FywVlyLyC3NqDY=; b=MmGfD+B0b/C5buR6puUM0z4l0GIuKF7AAB8wWVaJiHoHloyhyAxc55Ii005y3Ktwb8 lCNWXW/6m9CXI8ofu+Pd3sReS0VxdHgvRIxPB8qCOeh+ZSVPXGcmncgffpr6n8Zc25DX diPUHdYIYTNQzzDAaw1WeKFfurjlZOg1ccUqXAvxo6v2bWnU76BBSH3EOWoecV7ScN9m 0kyxH7QXxj0d4AKedFo71g0sZzKrSrq8Y1pEiWf3fApacpnaLCsMcRGJ3nHSIUysEMsW KT7/958R3PRAzkWWSKYH9xEMz+JD6HfuMbGbdjktrW9RQPReEUzvb/JRmeqZ5hjsFkKz 7zOA== X-Gm-Message-State: AJIora+hGKUc28KHjdsaaBZIVgbnY9loQtPZ9qSUVXQ+FkL7OS1vl10B Uloguu3MskSRVnvIsBLFVSLF/Xrdrq4= X-Google-Smtp-Source: AGRyM1tC0/9ONSX2Oj+NF6gPMNaP4csfTn6MQttvUu3tGGrRxI+8L442tFZ55fo0CQX675W66GnAMJtI6Fs= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:a6f5:f713:759c:abb6]) (user=glider job=sendgmr) by 2002:a05:6402:1914:b0:437:8f32:96e5 with SMTP id e20-20020a056402191400b004378f3296e5mr19396026edz.218.1656685421053; Fri, 01 Jul 2022 07:23:41 -0700 (PDT) Date: Fri, 1 Jul 2022 16:22:34 +0200 In-Reply-To: <20220701142310.2188015-1-glider@google.com> Message-Id: <20220701142310.2188015-10-glider@google.com> Mime-Version: 1.0 References: <20220701142310.2188015-1-glider@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [PATCH v4 09/45] x86: kmsan: pgtable: reduce vmalloc space From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Alexei Starovoitov , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656685422; a=rsa-sha256; cv=none; b=FSRYsdiT27cSxU3hY4hX7QPczxNk76+tC4bZxZ7iAJIOcw+i/bvMw+y3W1JMsYRRKVvV1/ NpBxO1Yhga2F38N6x5SP8rUr+raSalHLsGMD835CJMdmY/Xk4zGqrH64vqpheaoVjKMIk0 nuXxie2VdIFt48i5CyDbVUjSz6kRxDc= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=U+fVjLRG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3bQO_YgYKCIww1ytu7w44w1u.s421y3AD-220Bqs0.47w@flex--glider.bounces.google.com designates 209.85.208.73 as permitted sender) smtp.mailfrom=3bQO_YgYKCIww1ytu7w44w1u.s421y3AD-220Bqs0.47w@flex--glider.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656685422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9CmKhXrgq4R//9cDj1zsNax3tT0d2FywVlyLyC3NqDY=; b=UQkPu0byr/3Y2fG0hQNInGQqWiDiWm3ZNn7GlCuPXWJIQb5O7nFJtqnJ7ZRWBxHBlOgIhj cf016XgIo0GoXMRLpY03IqyJXY8Y2qRMYxVmn3fOHRSWCBKSB0NdxzuXo4/5+N/Ba/Qi3M KPwm02/+OY8EbHmT3Tp+Ft8YgufMj+8= X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3F294180033 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=U+fVjLRG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3bQO_YgYKCIww1ytu7w44w1u.s421y3AD-220Bqs0.47w@flex--glider.bounces.google.com designates 209.85.208.73 as permitted sender) smtp.mailfrom=3bQO_YgYKCIww1ytu7w44w1u.s421y3AD-220Bqs0.47w@flex--glider.bounces.google.com X-Rspam-User: X-Stat-Signature: ja9er1zcz8asg6gfmmy3n1ar9ffrq9p5 X-HE-Tag: 1656685422-709936 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KMSAN is going to use 3/4 of existing vmalloc space to hold the metadata, therefore we lower VMALLOC_END to make sure vmalloc() doesn't allocate past the first 1/4. Signed-off-by: Alexander Potapenko --- v2: -- added x86: to the title Link: https://linux-review.googlesource.com/id/I9d8b7f0a88a639f1263bc693cbd5c136626f7efd --- arch/x86/include/asm/pgtable_64_types.h | 41 ++++++++++++++++++++++++- arch/x86/mm/init_64.c | 2 +- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 70e360a2e5fb7..ad6ded5b1dedf 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -139,7 +139,46 @@ extern unsigned int ptrs_per_p4d; # define VMEMMAP_START __VMEMMAP_BASE_L4 #endif /* CONFIG_DYNAMIC_MEMORY_LAYOUT */ -#define VMALLOC_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) +#define VMEMORY_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) + +#ifndef CONFIG_KMSAN +#define VMALLOC_END VMEMORY_END +#else +/* + * In KMSAN builds vmalloc area is four times smaller, and the remaining 3/4 + * are used to keep the metadata for virtual pages. The memory formerly + * belonging to vmalloc area is now laid out as follows: + * + * 1st quarter: VMALLOC_START to VMALLOC_END - new vmalloc area + * 2nd quarter: KMSAN_VMALLOC_SHADOW_START to + * VMALLOC_END+KMSAN_VMALLOC_SHADOW_OFFSET - vmalloc area shadow + * 3rd quarter: KMSAN_VMALLOC_ORIGIN_START to + * VMALLOC_END+KMSAN_VMALLOC_ORIGIN_OFFSET - vmalloc area origins + * 4th quarter: KMSAN_MODULES_SHADOW_START to KMSAN_MODULES_ORIGIN_START + * - shadow for modules, + * KMSAN_MODULES_ORIGIN_START to + * KMSAN_MODULES_ORIGIN_START + MODULES_LEN - origins for modules. + */ +#define VMALLOC_QUARTER_SIZE ((VMALLOC_SIZE_TB << 40) >> 2) +#define VMALLOC_END (VMALLOC_START + VMALLOC_QUARTER_SIZE - 1) + +/* + * vmalloc metadata addresses are calculated by adding shadow/origin offsets + * to vmalloc address. + */ +#define KMSAN_VMALLOC_SHADOW_OFFSET VMALLOC_QUARTER_SIZE +#define KMSAN_VMALLOC_ORIGIN_OFFSET (VMALLOC_QUARTER_SIZE << 1) + +#define KMSAN_VMALLOC_SHADOW_START (VMALLOC_START + KMSAN_VMALLOC_SHADOW_OFFSET) +#define KMSAN_VMALLOC_ORIGIN_START (VMALLOC_START + KMSAN_VMALLOC_ORIGIN_OFFSET) + +/* + * The shadow/origin for modules are placed one by one in the last 1/4 of + * vmalloc space. + */ +#define KMSAN_MODULES_SHADOW_START (VMALLOC_END + KMSAN_VMALLOC_ORIGIN_OFFSET + 1) +#define KMSAN_MODULES_ORIGIN_START (KMSAN_MODULES_SHADOW_START + MODULES_LEN) +#endif /* CONFIG_KMSAN */ #define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) /* The module sections ends with the start of the fixmap */ diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 39c5246964a91..5806331172361 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1287,7 +1287,7 @@ static void __init preallocate_vmalloc_pages(void) unsigned long addr; const char *lvl; - for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + for (addr = VMALLOC_START; addr <= VMEMORY_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { pgd_t *pgd = pgd_offset_k(addr); p4d_t *p4d; pud_t *pud;