From patchwork Tue Apr 26 20:32:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 12827949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF4B4C433EF for ; Tue, 26 Apr 2022 20:32:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CFB06B0074; Tue, 26 Apr 2022 16:32:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 65B8B6B0075; Tue, 26 Apr 2022 16:32:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AA5E6B0078; Tue, 26 Apr 2022 16:32:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 36E386B0074 for ; Tue, 26 Apr 2022 16:32:45 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 1B48A80F2F for ; Tue, 26 Apr 2022 20:32:45 +0000 (UTC) X-FDA: 79400178690.11.3EE8D0D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf09.hostedemail.com (Postfix) with ESMTP id 7973B140043 for ; Tue, 26 Apr 2022 20:32:41 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-2f825f8c944so23818017b3.15 for ; Tue, 26 Apr 2022 13:32:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ltoHorK2wXcd+a1S2OPAtE1WX5XYl1FW0si4xQoCroE=; b=NzL50QDr4Qx2Wmcu5b5/caaxWCFPCsWoZ8IrIkN2tZwE48j9h9dxAFyrvRmG6HzEgJ MPxYFBCQ7dcnIt6/MbMN5QuFdkeNHjxiCsJGiqUzGY+p31fGozQOjsGbHon+C8zA7fGN 9EJvF9UWojx7/MYjlzobRutT8+e+zb0BQ1jH5rvjYEULqDNznTuSO36ZzlfkWnXIVYt3 udT6td888NFbkQ361IAlXc+NZITcxoYg1BInJSgez+pJ4wbzgbtVkQjml8IpCmpuv8ff jBEFewk4xn4G3n4jEALq5fep9zwyuEY3aLiYoFRjdk5cfocjm0fn6ZBlHJqL7mpRzO9d 5ETg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ltoHorK2wXcd+a1S2OPAtE1WX5XYl1FW0si4xQoCroE=; b=hoE5KcqFym3n7saOJzPOfo0iDI6HflsjKViwJjVPIcfFnsQe57pZdH0atKb1ViHGKx l/BdO10qcU3YwMZ9aNtBhsJo3jOmnmS+Zh61vk+tOVovpo5BV/im8waLjxI8vo6n9DdR aIAq5nYEC35ONRDTpQ6D00JmCTYY+Jhtb/+jVOV/ys2yT/AFDxVwAxX3s+55szDq9yo4 Oms88R9doRhVANbWRpi4EjseVLiq7nBLSl5GeDU5EA8TJ7k5n8z3CwZM1EI20RU/fnZ1 ww+7s06n1U0wmaZe6EAJFKCNFAsJcTT401MmjqZA29fxtqHHzCmpOwlAvZcK0CWnoTIe lavA== X-Gm-Message-State: AOAM5309y026mxETvNpCkJOSWUzU4aes042NuEoFWIOeJnUy6Ee2IOp9 /USUuP/Bw7xHx63Z4DfDAVm1G+I= X-Google-Smtp-Source: ABdhPJxNY3ncovCwIEUfGZWoHEe0RfFGEUNq4+teJxTrSaV1EC/58nkZBxTEmZ5+Nh614RPM9VgIGQg= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2ce:200:709f:5eff:336a:4109]) (user=pcc job=sendgmr) by 2002:a81:9146:0:b0:2f7:da3d:8bfa with SMTP id i67-20020a819146000000b002f7da3d8bfamr13154889ywg.206.1651005163884; Tue, 26 Apr 2022 13:32:43 -0700 (PDT) Date: Tue, 26 Apr 2022 13:32:31 -0700 In-Reply-To: <20220426203231.2107365-1-pcc@google.com> Message-Id: <20220426203231.2107365-2-pcc@google.com> Mime-Version: 1.0 References: <20220426203231.2107365-1-pcc@google.com> X-Mailer: git-send-email 2.36.0.rc2.479.g8af0fa9b8e-goog Subject: [PATCH v4 2/2] mm: make minimum slab alignment a runtime property From: Peter Collingbourne To: Andrey Konovalov , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Andrew Morton , Catalin Marinas Cc: Peter Collingbourne , Linux ARM , Linux Memory Management List , Linux Kernel Mailing List , vbabka@suse.cz, penberg@kernel.org, roman.gushchin@linux.dev, iamjoonsoo.kim@lge.com, rientjes@google.com, Herbert Xu , Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , kasan-dev , Eric Biederman , Kees Cook X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7973B140043 X-Stat-Signature: bgp78ck445rtjffuswzss7e87hzag1yy Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NzL50QDr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 361ZoYgMKCFQByy2AA270.yA8749GJ-886Hwy6.AD2@flex--pcc.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=361ZoYgMKCFQByy2AA270.yA8749GJ-886Hwy6.AD2@flex--pcc.bounces.google.com X-Rspam-User: X-HE-Tag: 1651005161-26693 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When CONFIG_KASAN_HW_TAGS is enabled we currently increase the minimum slab alignment to 16. This happens even if MTE is not supported in hardware or disabled via kasan=off, which creates an unnecessary memory overhead in those cases. Eliminate this overhead by making the minimum slab alignment a runtime property and only aligning to 16 if KASAN is enabled at runtime. On a DragonBoard 845c (non-MTE hardware) with a kernel built with CONFIG_KASAN_HW_TAGS, waiting for quiescence after a full Android boot I see the following Slab measurements in /proc/meminfo (median of 3 reboots): Before: 169020 kB After: 167304 kB Link: https://linux-review.googlesource.com/id/I752e725179b43b144153f4b6f584ceb646473ead Signed-off-by: Peter Collingbourne Reviewed-by: Andrey Konovalov Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Acked-by: David Rientjes Reviewed-by: Catalin Marinas --- v4: - add a dependent patch to fix the build with CONFIG_JUMP_LABEL disabled v3: - go back to ARCH_SLAB_MINALIGN - revert changes to fs/binfmt_flat.c - update arch_slab_minalign() comment to say that it must be a power of two v2: - use max instead of max_t in flat_stack_align() arch/arm64/include/asm/cache.h | 17 ++++++++++++----- include/linux/slab.h | 12 ++++++++++++ mm/slab.c | 7 +++---- mm/slab_common.c | 3 +-- mm/slob.c | 6 +++--- 5 files changed, 31 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index a074459f8f2f..22b22dc1b1b5 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -6,6 +6,7 @@ #define __ASM_CACHE_H #include +#include #define CTR_L1IP_SHIFT 14 #define CTR_L1IP_MASK 3 @@ -49,16 +50,22 @@ */ #define ARCH_DMA_MINALIGN (128) +#ifndef __ASSEMBLY__ + +#include +#include + #ifdef CONFIG_KASAN_SW_TAGS #define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT) #elif defined(CONFIG_KASAN_HW_TAGS) -#define ARCH_SLAB_MINALIGN MTE_GRANULE_SIZE +static inline size_t arch_slab_minalign(void) +{ + return kasan_hw_tags_enabled() ? MTE_GRANULE_SIZE : + __alignof__(unsigned long long); +} +#define arch_slab_minalign() arch_slab_minalign() #endif -#ifndef __ASSEMBLY__ - -#include - #define ICACHEF_ALIASING 0 #define ICACHEF_VPIPT 1 extern unsigned long __icache_flags; diff --git a/include/linux/slab.h b/include/linux/slab.h index 373b3ef99f4e..2c7190db4cc0 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -209,6 +209,18 @@ void kmem_dump_obj(void *object); #define ARCH_SLAB_MINALIGN __alignof__(unsigned long long) #endif +/* + * Arches can define this function if they want to decide the minimum slab + * alignment at runtime. The value returned by the function must be a power + * of two and >= ARCH_SLAB_MINALIGN. + */ +#ifndef arch_slab_minalign +static inline size_t arch_slab_minalign(void) +{ + return ARCH_SLAB_MINALIGN; +} +#endif + /* * kmalloc and friends return ARCH_KMALLOC_MINALIGN aligned * pointers. kmem_cache_alloc and friends return ARCH_SLAB_MINALIGN diff --git a/mm/slab.c b/mm/slab.c index 0edb474edef1..97b756976c8b 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3009,10 +3009,9 @@ static void *cache_alloc_debugcheck_after(struct kmem_cache *cachep, objp += obj_offset(cachep); if (cachep->ctor && cachep->flags & SLAB_POISON) cachep->ctor(objp); - if (ARCH_SLAB_MINALIGN && - ((unsigned long)objp & (ARCH_SLAB_MINALIGN-1))) { - pr_err("0x%px: not aligned to ARCH_SLAB_MINALIGN=%d\n", - objp, (int)ARCH_SLAB_MINALIGN); + if ((unsigned long)objp & (arch_slab_minalign() - 1)) { + pr_err("0x%px: not aligned to arch_slab_minalign()=%d\n", objp, + (int)arch_slab_minalign()); } return objp; } diff --git a/mm/slab_common.c b/mm/slab_common.c index 2b3206a2c3b5..33cc49810a54 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -154,8 +154,7 @@ static unsigned int calculate_alignment(slab_flags_t flags, align = max(align, ralign); } - if (align < ARCH_SLAB_MINALIGN) - align = ARCH_SLAB_MINALIGN; + align = max_t(size_t, align, arch_slab_minalign()); return ALIGN(align, sizeof(void *)); } diff --git a/mm/slob.c b/mm/slob.c index 40ea6e2d4ccd..3bd2669bd690 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -478,7 +478,7 @@ static __always_inline void * __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) { unsigned int *m; - int minalign = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN); + int minalign = max_t(size_t, ARCH_KMALLOC_MINALIGN, arch_slab_minalign()); void *ret; gfp &= gfp_allowed_mask; @@ -555,7 +555,7 @@ void kfree(const void *block) sp = virt_to_folio(block); if (folio_test_slab(sp)) { - int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN); + int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, arch_slab_minalign()); unsigned int *m = (unsigned int *)(block - align); slob_free(m, *m + align); } else { @@ -584,7 +584,7 @@ size_t __ksize(const void *block) if (unlikely(!folio_test_slab(folio))) return folio_size(folio); - align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN); + align = max_t(size_t, ARCH_KMALLOC_MINALIGN, arch_slab_minalign()); m = (unsigned int *)(block - align); return SLOB_UNITS(*m) * SLOB_UNIT; }