From patchwork Wed May 29 12:38:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10967023 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 376061575 for ; Wed, 29 May 2019 12:38:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 282B828899 for ; Wed, 29 May 2019 12:38:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 25F8228987; Wed, 29 May 2019 12:38:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87C7428899 for ; Wed, 29 May 2019 12:38:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4776E6B000C; Wed, 29 May 2019 08:38:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 429C86B000D; Wed, 29 May 2019 08:38:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2EFB16B000E; Wed, 29 May 2019 08:38:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 069D76B000C for ; Wed, 29 May 2019 08:38:32 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id t11so1812424qtc.9 for ; Wed, 29 May 2019 05:38:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=NnBaq8iZzFC6qD9p6ruWp0p3PmadGDrTDSRRwu5HpM0=; b=POnYbNwROxoXAnDFVs1B3j2YSVhQa53Pgj4rJe8E1PJ9HooeEG/5Aos3gaKc7af206 fXS1zINwX5MM5oBAkl1pmbYxxQ8X36WYyKM1RkyJmbD3Ke8l3FvEB2kZqW27s7TxRtdk 3SR1/V6e6FhxpXopUTejGkt4n6mQ0MH5S7p3NyEG4T7FyJp0MfQZSRHsI5kIi7fg2Pci 131eGdheQnn2cuX8N7f8LtG46aDWaKEpzXW9qMMCBFk4YZkhtpnw3MrxSL9byCarf2hb JlMQE03DF5TRnEOddrZNAoXzl5UMV4hJq1UXEXx76T2TfHWjTKUagvMydnIV1VbQGfMD 6J0w== X-Gm-Message-State: APjAAAXaHU+B1hmtU1mciBqFK5v08qAEAYnuPZz5alD6/27dQS/LDvf2 P0qNYp5PmLIAyxczl5ye8/p+ydbh4bAtZvVYoy70XgrVfyspIkSeEEQ4yAThkqAdXIl7odQDLTP 8ydK627qBwnKEhq7Mq5cuISoL9rMPujNINwup8QGtkGdCqs8ckzuzq2oTdglqz07cqA== X-Received: by 2002:a0c:b621:: with SMTP id f33mr69925365qve.199.1559133511641; Wed, 29 May 2019 05:38:31 -0700 (PDT) X-Received: by 2002:a0c:b621:: with SMTP id f33mr69925252qve.199.1559133510267; Wed, 29 May 2019 05:38:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559133510; cv=none; d=google.com; s=arc-20160816; b=BWBH8b6EC7E+slcE0ijEPmpKYmlRrJztFwKwcf8J2e7QUktHlSd7FG2KeMFAG7eBqt vH5vYKbNPxyNdCNeQKrH5FZaXrQj2+QjCgWtRk4f7l+JQeMYefVp8hL1oqqGG62LMc9n qEbN5eoI6hsJpTSTBqxD5b2tAQ1F/JnMdm2UID9Eu7cxtUQAClK0c/CpXD+p/hAdqTR4 AdhSvrBKaO9xG/uKMtEsliuWvHEbn1V/2NhyM9qVItbi26nG47Mt19XCdk5gCxtkpPmY NNKXNPhwChJxAzjy3LAXnACLOWCVzFYAIcHJrZI2pcrnAZGLqGkVNLwcPVOzBFwYrZwY +heA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=NnBaq8iZzFC6qD9p6ruWp0p3PmadGDrTDSRRwu5HpM0=; b=saBcEMnzFWT03g5Eg8SJMs5WiU/ZtK7/iKMWVA1mHRsRVdBMMqpZjReqMr4cywO25T EwUVOvD4Bpf/X81l6535zHg5ZM/JwG22NuUqEhMeReE2M3UCoRyx8wmMFE7oDE8IKLKb 78R0HMq2M+a+03oQJ/MfGX72mbuqoewgbgkiSKBbsFaqAiUUqSCZx1ncstmeepvZ4N5y ixZp4PZ8O2w+CyvuIVg/zaoaxew1Y3A76ktnsy2uMIRo82rx6adXFseII2cAIEyvUmt3 TJ0LWbd8CmiJwSdsAkE5QDIKr9tdrx+TpSPn2T6i61hvy/wsJ76lV0C5PJKH6XrcehdO 18vw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=amhjWIX2; spf=pass (google.com: domain of 3rx3uxaykcoyotqlmzowwotm.kwutqvcf-uusdiks.wzo@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3RX3uXAYKCOYOTQLMZOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id 41sor12431677qvi.69.2019.05.29.05.38.30 for (Google Transport Security); Wed, 29 May 2019 05:38:30 -0700 (PDT) Received-SPF: pass (google.com: domain of 3rx3uxaykcoyotqlmzowwotm.kwutqvcf-uusdiks.wzo@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=amhjWIX2; spf=pass (google.com: domain of 3rx3uxaykcoyotqlmzowwotm.kwutqvcf-uusdiks.wzo@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3RX3uXAYKCOYOTQLMZOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=NnBaq8iZzFC6qD9p6ruWp0p3PmadGDrTDSRRwu5HpM0=; b=amhjWIX28V42pGaQmrFoK/VHF82lRQziq0SsPBvR2LtnzOjb1BGkrEjWECIjmsg2hp rKXgR2UZAQp7A0x7S7b3duk7t3ycPkY7bG2/ph721/OgPg2wYBdlCTiCj8phH6MSQS8t HZzjZAAgnE4GJnaJWK6phXBuWO9uvGaCd48JaCKc8ihhxVJKUqgjwXFhSlIFvXTB30R/ 2l69N4Ht9vVGJ4BARxZhsjUMG0+zdhKSKdorPE6EMVT6+RH9DVTclkfcojNkr6O+6aop 9YI8frQsV7r9Jn8TxNy6WmZ2uyISKWi1Z01nctTRO/+cShSvADDp+M2Jy0i65+vCVjCp PftQ== X-Google-Smtp-Source: APXvYqygGXbm41Pep0o6KZaLy6S22CUhjhG13B82ya/gOhm13zlxzQ3JyLLFLet2Dre7gNfT/L/cGGFfGBA= X-Received: by 2002:a0c:af3d:: with SMTP id i58mr101373891qvc.71.1559133509897; Wed, 29 May 2019 05:38:29 -0700 (PDT) Date: Wed, 29 May 2019 14:38:10 +0200 In-Reply-To: <20190529123812.43089-1-glider@google.com> Message-Id: <20190529123812.43089-2-glider@google.com> Mime-Version: 1.0 References: <20190529123812.43089-1-glider@google.com> X-Mailer: git-send-email 2.22.0.rc1.257.g3120a18244-goog Subject: [PATCH v5 1/3] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options From: Alexander Potapenko To: Andrew Morton , Christoph Lameter , Kees Cook Cc: Alexander Potapenko , Masahiro Yamada , Michal Hocko , James Morris , "Serge E. Hallyn" , Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Randy Dunlap , Jann Horn , Mark Rutland , Marco Elver , linux-mm@kvack.org, linux-security-module@vger.kernel.org, kernel-hardening@lists.openwall.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic. init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses. Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and SLAB_TYPESAFE_BY_RCU flag. Those are never zero-initialized to preserve their semantics. Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON. Slowdown for the new features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%) Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both on_alloc and on_free hooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization. Although init_on_free is rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series. Signed-off-by: Alexander Potapenko To: Andrew Morton To: Christoph Lameter To: Kees Cook Cc: Masahiro Yamada Cc: Michal Hocko Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: Marco Elver Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com Acked-by: Kees Cook --- v2: - unconditionally initialize pages in kernel_init_free_pages() - comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening v3: - don't call kernel_init_free_pages() from memblock_free_pages() - adopted some Kees' comments for the patch description v4: - use NULL instead of 0 in slab_alloc_node() (found by kbuild test robot) - don't write to NULL object in slab_alloc_node() (found by Android testing) v5: - adjusted documentation wording as suggested by Kees - disable SLAB_POISON if auto-initialization is on - don't wipe RCU cache allocations made without __GFP_ZERO - dropped SLOB support --- .../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 +++++++ kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 63 ++++++++++++++++--- mm/slab.c | 16 ++++- mm/slab.h | 19 ++++++ mm/slub.c | 33 ++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 +++++++++ 11 files changed, 180 insertions(+), 19 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 138f6664b2e2..84ee1121a2b9 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1673,6 +1673,15 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can diff --git a/drivers/infiniband/core/uverbs_ioctl.c b/drivers/infiniband/core/uverbs_ioctl.c index 829b0c6944d8..61758201d9b2 100644 --- a/drivers/infiniband/core/uverbs_ioctl.c +++ b/drivers/infiniband/core/uverbs_ioctl.c @@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used = ALIGN(new_used, sizeof(*pbundle->internal_buffer)); - if (flags & __GFP_ZERO) + if (want_init_on_alloc(flags)) memset(res, 0, size); return res; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 0e8834ac32b7..7733a341c0c4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2685,6 +2685,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE(init_on_alloc); +#endif +static inline bool want_init_on_alloc(gfp_t flags) +{ + if (static_branch_unlikely(&init_on_alloc)) + return true; + return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_free); +#else +DECLARE_STATIC_KEY_FALSE(init_on_free); +#endif +static inline bool want_init_on_free(void) +{ + return static_branch_unlikely(&init_on_free); +} + extern bool _debug_pagealloc_enabled; static inline bool debug_pagealloc_enabled(void) diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index fd5c95ff9251..2f75dd0d0d81 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) arch_kexec_post_alloc_pages(page_address(pages), count, gfp_mask); - if (gfp_mask & __GFP_ZERO) + if (want_init_on_alloc(gfp_mask)) for (i = 0; i < count; i++) clear_highpage(pages + i); } diff --git a/mm/dmapool.c b/mm/dmapool.c index 76a160083506..493d151067cb 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, #endif spin_unlock_irqrestore(&pool->lock, flags); - if (mem_flags & __GFP_ZERO) + if (want_init_on_alloc(mem_flags)) memset(retval, 0, pool->size); return retval; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d66bc8abe0af..50a3b104a491 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -136,6 +136,48 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_alloc); +#else +DEFINE_STATIC_KEY_FALSE(init_on_alloc); +#endif +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_free); +#else +DEFINE_STATIC_KEY_FALSE(init_on_free); +#endif + +static int __init early_init_on_alloc(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_alloc); + else + static_branch_disable(&init_on_alloc); + return ret; +} +early_param("init_on_alloc", early_init_on_alloc); + +static int __init early_init_on_free(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_free); + else + static_branch_disable(&init_on_free); + return ret; +} +early_param("init_on_free", early_init_on_free); /* * A cached value of the page's pageblock's migratetype, used when the page is @@ -1090,6 +1132,14 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) return ret; } +static void kernel_init_free_pages(struct page *page, int numpages) +{ + int i; + + for (i = 0; i < numpages; i++) + clear_highpage(page + i); +} + static __always_inline bool free_pages_prepare(struct page *page, unsigned int order, bool check_free) { @@ -1142,6 +1192,8 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); + if (want_init_on_free()) + kernel_init_free_pages(page, 1 << order); if (debug_pagealloc_enabled()) kernel_map_pages(page, 1 << order, 0); @@ -2020,8 +2072,8 @@ static inline int check_new_page(struct page *page) static inline bool free_pages_prezeroed(void) { - return IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && - page_poisoning_enabled(); + return (IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && + page_poisoning_enabled()) || want_init_on_free(); } #ifdef CONFIG_DEBUG_VM @@ -2075,13 +2127,10 @@ inline void post_alloc_hook(struct page *page, unsigned int order, static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags) { - int i; - post_alloc_hook(page, order, gfp_flags); - if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO)) - for (i = 0; i < (1 << order); i++) - clear_highpage(page + i); + if (!free_pages_prezeroed() && want_init_on_alloc(gfp_flags)) + kernel_init_free_pages(page, 1 << order); if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); diff --git a/mm/slab.c b/mm/slab.c index f7117ad9b3a3..98a89d7c922d 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1830,6 +1830,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num = 0; + /* + * If slab auto-initialization on free is enabled, store the freelist + * off-slab, so that its contents don't end up in one of the allocated + * objects. + */ + if (unlikely(slab_want_init_on_free(cachep))) + return false; + if (cachep->ctor || flags & SLAB_TYPESAFE_BY_RCU) return false; @@ -3263,7 +3271,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, local_irq_restore(save_flags); ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller); - if (unlikely(flags & __GFP_ZERO) && ptr) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &ptr); @@ -3320,7 +3328,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller); prefetchw(objp); - if (unlikely(flags & __GFP_ZERO) && objp) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &objp); @@ -3441,6 +3449,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, struct array_cache *ac = cpu_cache_get(cachep); check_irq_off(); + if (unlikely(slab_want_init_on_free(cachep))) + memset(objp, 0, cachep->object_size); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); @@ -3528,7 +3538,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_); /* Clear memory outside IRQ disabled section */ - if (unlikely(flags & __GFP_ZERO)) + if (unlikely(slab_want_init_on_alloc(flags, s))) for (i = 0; i < size; i++) memset(p[i], 0, s->object_size); diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..31032d488b29 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -524,4 +524,23 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep, static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { } #endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc)) { + if (c->ctor) + return false; + if (c->flags & SLAB_TYPESAFE_BY_RCU) + return flags & __GFP_ZERO; + return true; + } + return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free(struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_free)) + return !(c->ctor || (c->flags & SLAB_TYPESAFE_BY_RCU)); + return false; +} + #endif /* MM_SLAB_H */ diff --git a/mm/slub.c b/mm/slub.c index cd04dbd2b5d0..9c4a8b9a955c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1279,6 +1279,12 @@ static int __init setup_slub_debug(char *str) if (*str == ',') slub_debug_slabs = str + 1; out: + if ((static_branch_unlikely(&init_on_alloc) || + static_branch_unlikely(&init_on_free)) && + (slub_debug & SLAB_POISON)) { + pr_warn("disabling SLAB_POISON: can't be used together with memory auto-initialization\n"); + slub_debug &= ~SLAB_POISON; + } return 1; } @@ -1424,6 +1430,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x) static inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail) { + + void *object; + void *next = *head; + void *old_tail = *tail ? *tail : *head; + + if (slab_want_init_on_free(s)) + do { + object = next; + next = get_freepointer(s, object); + memset(object, 0, s->size); + set_freepointer(s, object, next); + } while (object != old_tail); + /* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here. @@ -1433,9 +1452,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, defined(CONFIG_DEBUG_OBJECTS_FREE) || \ defined(CONFIG_KASAN) - void *object; - void *next = *head; - void *old_tail = *tail ? *tail : *head; + next = *head; /* Head and tail of the reconstructed freelist */ *head = NULL; @@ -2741,8 +2758,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, prefetch_freepointer(s, next_object); stat(s, ALLOC_FASTPATH); } + /* + * If the object has been wiped upon free, make sure it's fully + * initialized by zeroing out freelist pointer. + */ + if (unlikely(slab_want_init_on_free(s)) && object) + *(void **)object = NULL; - if (unlikely(gfpflags & __GFP_ZERO) && object) + if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object) memset(object, 0, s->object_size); slab_post_alloc_hook(s, gfpflags, 1, &object); @@ -3163,7 +3186,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, local_irq_enable(); /* Clear memory outside IRQ disabled fastpath loop */ - if (unlikely(flags & __GFP_ZERO)) { + if (unlikely(slab_want_init_on_alloc(flags, s))) { int j; for (j = 0; j < i; j++) diff --git a/net/core/sock.c b/net/core/sock.c index 75b1c950b49f..9ceb90c875bc 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1602,7 +1602,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO); if (!sk) return sk; - if (priority & __GFP_ZERO) + if (want_init_on_alloc(priority)) sk_prot_clear_nulls(sk, prot->obj_size); } else sk = kmalloc(prot->obj_size, priority); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index c6cb2d9b2905..a1ffe2eb4d5f 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -160,6 +160,35 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Enable heap memory zeroing on allocation by default" + help + This has the effect of setting "init_on_alloc=1" on the kernel + command line. This can be disabled with "init_on_alloc=0". + When "init_on_alloc" is enabled, all page allocator and slab + allocator memory will be zeroed when allocated, eliminating + many kinds of "uninitialized heap memory" flaws, especially + heap content exposures. The performance impact varies by + workload, but most cases see <1% impact. Some synthetic + workloads have measured as high as 7%. + +config INIT_ON_FREE_DEFAULT_ON + bool "Enable heap memory zeroing on free by default" + help + This has the effect of setting "init_on_free=1" on the kernel + command line. This can be disabled with "init_on_free=0". + Similar to "init_on_alloc", when "init_on_free" is enabled, + all page allocator and slab allocator memory will be zeroed + when freed, eliminating many kinds of "uninitialized heap memory" + flaws, especially heap content exposures. The primary difference + with "init_on_free" is that data lifetime in memory is reduced, + as anything freed is wiped immediately, making live forensics or + cold boot memory attacks unable to recover freed memory contents. + The performance impact varies by workload, but is more expensive + than "init_on_alloc" due to the negative cache effects of + touching "cold" memory areas. Most cases see 3-5% impact. Some + synthetic workloads have measured as high as 8%. + endmenu endmenu From patchwork Wed May 29 12:38:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10967025 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 42216933 for ; Wed, 29 May 2019 12:38:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D6D628989 for ; Wed, 29 May 2019 12:38:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2B3462899B; Wed, 29 May 2019 12:38:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 90E9A28989 for ; Wed, 29 May 2019 12:38:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7111F6B000D; Wed, 29 May 2019 08:38:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6C29F6B000E; Wed, 29 May 2019 08:38:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B18F6B0010; Wed, 29 May 2019 08:38:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 35DA46B000D for ; Wed, 29 May 2019 08:38:34 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id 28so1802832qtw.5 for ; Wed, 29 May 2019 05:38:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=OTSVci+9HQFGuKiIh59lqpYGkMYL4TSES3kfTgmqHdU=; b=b7Z2YnGDWnJwOWGLcxxB2LHRCe5TQHof1pQenMkcfjJYZslQROkJps0rdaCMvb5On/ ZROZt8y5SovcZYhQRi/yVBPOH51uNjpJjywmSRaecuZAKTyTztvBCIlLREmc+6FKdpXK MKytKrFLYl4dhebzEFby/Pbg7PASnRfqTp649gaUY2ShtyZhRUvVhDuhY8/Y3LvTpC1O yWqXf8n7ATAWb3Jp7krMLjDJ44G1zJerKtpTOOsCpKyY9uReqKhgi0hMNJ47r9VTkJ4M Ou1vbA3IiKzk6opm1yt6jr2KHDqeEUWApPAI+KwQs7ogHLXvTXaQ2hoTplE2hH4W0Qg4 LYWw== X-Gm-Message-State: APjAAAXlb/U3+iCn98o3i74N/N0GHr9BGCvDwJip5oGvKJ0zpgpjfK6d weaekG3Co6aug5FoTvqB+qiguYVwAJgcmrUNRqbRsk05fm4qD9t1OfPAQSU07PdRTNKvd5oWd8P 8zNkBMEAu95mw8PUjdwDS7k8exjN4GsHkdQdSaFAKecShidkz9cJeVcpnxp8iNfAlUg== X-Received: by 2002:aed:2383:: with SMTP id j3mr10796766qtc.313.1559133513985; Wed, 29 May 2019 05:38:33 -0700 (PDT) X-Received: by 2002:aed:2383:: with SMTP id j3mr10796721qtc.313.1559133513393; Wed, 29 May 2019 05:38:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559133513; cv=none; d=google.com; s=arc-20160816; b=Y2Qhe/Iz9pLRpsrUV0YZkL4P8cBeTP46RfqCVtQlCUDOHrVaOP+mEDJR95MdiLugCj RJxmnWhZ9ZdXdwsphQVJbSrRAtUk8bO369BoxSoKEKvflN9OvF2pYUQbU56kagv44BM3 unbCwfIJv4jEiOlIZax6vJ0GyegiRjwI6+NF+JZaKL957/ySYzOfXhJlpCnWdu+iKnfR Ki+tdy/REUF2v60iYy7XDvld+1Cm48DJS4q6NElV9mdisdjXu0MoKdofzgzPozrq2aCN JAMjiGxzCaueJrVAlD2jigF/sCnuS4K+w0gnj99KmFqflSkTgdk5mVSv/ATx6V1Y5GQM OmHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=OTSVci+9HQFGuKiIh59lqpYGkMYL4TSES3kfTgmqHdU=; b=HfVsj850a34o2Gjdk88TIcTEPs5ec/Mh/Jwat4nsAA03dRPBih6O3pbWdZHraCb1dK Y1a3Al1lfW/xP0vLZoLUTPjtKjC2bMTnlG13ahDKPOwIAqxWAYWwL/mUniRqLec4gD9j K/wBxIpss0EIkwOQwKtJCvfIJejZefO2X9RctbSgy1UrEMAxtCK+ZHFt/qfSs2Dxi9V8 xt1+rJ4QsKxcCtXTREOKXWWQ6IgDbJBapHd9mbrGkoy0jHlYykyRJYugt8xTqqOutkBQ qmnvkUNFoGY4mOhn2fe9yWDys+hAFJo4OE/zmF3h0kWRGyY+H4fyvEshomZ1mVG/qfAf kqPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=QVUrmXjB; spf=pass (google.com: domain of 3sx3uxaykcoosxupqdsaasxq.oayxuzgj-yywhmow.ads@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3SX3uXAYKCOoSXUPQdSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id m13sor8948553qkg.139.2019.05.29.05.38.33 for (Google Transport Security); Wed, 29 May 2019 05:38:33 -0700 (PDT) Received-SPF: pass (google.com: domain of 3sx3uxaykcoosxupqdsaasxq.oayxuzgj-yywhmow.ads@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=QVUrmXjB; spf=pass (google.com: domain of 3sx3uxaykcoosxupqdsaasxq.oayxuzgj-yywhmow.ads@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3SX3uXAYKCOoSXUPQdSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OTSVci+9HQFGuKiIh59lqpYGkMYL4TSES3kfTgmqHdU=; b=QVUrmXjBtd4xAGA041VU9vn2Wo24J0G/ryDcfbSzyIaXXA9eT1e/5dNi0QJEzMBCWq fN3WkgM5gqMR9l0iwxBy9vH+tbNaCYmzc3eMgJngHj1kSa5AtgjWh5+5HjO/QfCZpvY6 Jb+OfcJFPlWhCrVMcT9BX7VIzn5XFynOaCCkftBB3mbVd1JQuJaHm0m7kyXHACmQ9cgW 9zAnA35gH0p0YX9UacAq6LYdKrQ3NUIsIStqgl+m5S9sxABy/LK6SgMc6qA6s4YfiEYT SjwPmB5DRuLJ/T8J4jYw4vuR8yqEzw06eiiDOGT9MuQeYfiVfkXQYF9mVBXb/uZL2taU nQYA== X-Google-Smtp-Source: APXvYqyuj6WNrP123438HFtnTx32oR/o63x7jglpOtKNMGuuQ+g2SgRyW1Nxym/uXkKBa07I2QYAnrLim8k= X-Received: by 2002:a05:620a:141a:: with SMTP id d26mr6791202qkj.32.1559133513103; Wed, 29 May 2019 05:38:33 -0700 (PDT) Date: Wed, 29 May 2019 14:38:11 +0200 In-Reply-To: <20190529123812.43089-1-glider@google.com> Message-Id: <20190529123812.43089-3-glider@google.com> Mime-Version: 1.0 References: <20190529123812.43089-1-glider@google.com> X-Mailer: git-send-email 2.22.0.rc1.257.g3120a18244-goog Subject: [PATCH v5 2/3] mm: init: report memory auto-initialization features at boot time From: Alexander Potapenko To: Andrew Morton , Christoph Lameter Cc: Alexander Potapenko , Kees Cook , Dmitry Vyukov , James Morris , Jann Horn , Kostya Serebryany , Laura Abbott , Mark Rutland , Masahiro Yamada , Matthew Wilcox , Nick Desaulniers , Randy Dunlap , Sandeep Patil , "Serge E. Hallyn" , Souptick Joarder , Marco Elver , kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-security-module@vger.kernel.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Print the currently enabled stack and heap initialization modes. The possible options for stack are: - "all" for CONFIG_INIT_STACK_ALL; - "byref_all" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL; - "byref" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF; - "__user" for CONFIG_GCC_PLUGIN_STRUCTLEAK_USER; - "off" otherwise. Depending on the values of init_on_alloc and init_on_free boottime options we also report "heap alloc" and "heap free" as "on"/"off". In the init_on_free mode initializing pages at boot time may take some time, so print a notice about that as well. Signed-off-by: Alexander Potapenko Suggested-by: Kees Cook To: Andrew Morton To: Christoph Lameter Cc: Dmitry Vyukov Cc: James Morris Cc: Jann Horn Cc: Kostya Serebryany Cc: Laura Abbott Cc: Mark Rutland Cc: Masahiro Yamada Cc: Matthew Wilcox Cc: Nick Desaulniers Cc: Randy Dunlap Cc: Sandeep Patil Cc: "Serge E. Hallyn" Cc: Souptick Joarder Cc: Marco Elver Cc: kernel-hardening@lists.openwall.com Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Acked-by: Kees Cook --- init/main.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/init/main.c b/init/main.c index 66a196c5e4c3..9d63ff1d48f3 100644 --- a/init/main.c +++ b/init/main.c @@ -520,6 +520,29 @@ static inline void initcall_debug_enable(void) } #endif +/* Report memory auto-initialization states for this boot. */ +void __init report_meminit(void) +{ + const char *stack; + + if (IS_ENABLED(CONFIG_INIT_STACK_ALL)) + stack = "all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)) + stack = "byref_all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF)) + stack = "byref"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER)) + stack = "__user"; + else + stack = "off"; + + pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", + stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", + want_init_on_free() ? "on" : "off"); + if (want_init_on_free()) + pr_info("Clearing system memory may take some time...\n"); +} + /* * Set up kernel memory allocators */ @@ -530,6 +553,7 @@ static void __init mm_init(void) * bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem(); + report_meminit(); mem_init(); kmem_cache_init(); pgtable_init(); From patchwork Wed May 29 12:38:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10967029 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 54A0F1575 for ; Wed, 29 May 2019 12:38:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 482452842D for ; Wed, 29 May 2019 12:38:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3B79D28658; Wed, 29 May 2019 12:38:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 26A8F289AA for ; Wed, 29 May 2019 12:38:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 933176B000E; Wed, 29 May 2019 08:38:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8E3C56B0010; Wed, 29 May 2019 08:38:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78A6D6B0266; Wed, 29 May 2019 08:38:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) by kanga.kvack.org (Postfix) with ESMTP id 409FF6B000E for ; Wed, 29 May 2019 08:38:38 -0400 (EDT) Received: by mail-ot1-f72.google.com with SMTP id w110so1040241otb.10 for ; Wed, 29 May 2019 05:38:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Av06FkFyTGfvOvGo6A9zKZw+CBXEF6FDDT5YdORZtrs=; b=BxwZ3SWr7fH3NEyA2p1lzdv0hJCA4549p+J17JseMo+fIy+NZUUlAfzjXghXIhYvXg X61alUFHHzO0aHigPxC6LDasOYa5Fcg5ecEulvVLmxBOfGHcF+jq7FbgkZ7yT4Ry1hHE qdPqeWPtWLAodMZCyglhB1sHcvYSFvg25Wja1/Cl6FhYRSUDZLiZDZL400sqpQVjPByj DAxPAlZKufqi/dp/fHRQFONHIWwc5cOOfp4jAo2J/USGor/ENcvc7x421tAzkLjAOLc2 mXuWueeFcGT6GpZDvHclruQPyZAFBH8cfiYazrGgIc2a1cXEKcUqt3wjZ1SG7Gjal14A VYeg== X-Gm-Message-State: APjAAAUhWY+EtJGe9xpWZ5eVryXCiEJEj4PSlP2BpqaCD7lB9hfMdY+0 CWIWkc9Pw+9e5SLThI+MVm9Svo7ykwlLYiQje9XVz4XcC/EPJjtYLH/Beik5q59sOHfewz4wZnL QkSYKsgFfzpPog9GoQb1rXkSyWTSbjDfl8fSS1sucFRlxhrrCGRIA47sVURz1aLE62w== X-Received: by 2002:a9d:7d9a:: with SMTP id j26mr1452265otn.102.1559133517819; Wed, 29 May 2019 05:38:37 -0700 (PDT) X-Received: by 2002:a9d:7d9a:: with SMTP id j26mr1452203otn.102.1559133516859; Wed, 29 May 2019 05:38:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559133516; cv=none; d=google.com; s=arc-20160816; b=VT0bytT8MgZVkomPwpxzGY0vGP4MMEUXreHFpJC7w6mCW7W4vtAcgH9D9WEezBV2J9 qIjfeVJt8Jjl9vVLxcc7P0s8hLm2v9odWSwt455O3gnKJ8K/inSgWXBcq2rpRM/PkyM9 F7kT/+Kybr1YVlEtJKKOBKK39UO1e4gXuaXjmYAnoY8MbwqqD4kYSazQryMy/wCgxH4n yHQv4ofaYHl697IIkwOXcbPG1OvNhmgm8hnrknUa/2tQIoITtINnTEQMnLltCcFAm7fO kMXOlxaNYiWB/pzO/qYKZDJqR+0crLiX4IlJBu6aVcDI+kyO/25HxgC0sc3uX2wjTj7t rApQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=Av06FkFyTGfvOvGo6A9zKZw+CBXEF6FDDT5YdORZtrs=; b=yqudBV1VXdUancdCRbk7LWBdKkFZeIavgL7d7Sxf3xFMEAwTc+WJB0xASTOESxveM8 GaQLXP6vDm69INAbPVFer0NSW/M+Nq/sNuN7ESq2E34cIXEE+alyNZm5Ky6LcjoMrsCN RasfDc713Sj+H4bnRzluZ4p82z88mSvQTC5soHPdwpINIUmUdcwhIaa7X91lB3pNFrSW Cn5OPJGsbeBIpKdk16t03YegRzcBOEXea76Hj0lbL4p3aeEo8cq/yD5b+JxevosWAjNe Uj0ETh9ICYzLdGLaTgd5hLV/pURj44H9jwiyJyPw2++gRY/CReMOAfAgUz261ARxC94k FsKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=sw4Ift6Q; spf=pass (google.com: domain of 3th3uxaykco0vaxstgvddvat.rdbaxcjm-bbzkprz.dgv@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3TH3uXAYKCO0VaXSTgVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id m67sor6334461oif.15.2019.05.29.05.38.36 for (Google Transport Security); Wed, 29 May 2019 05:38:36 -0700 (PDT) Received-SPF: pass (google.com: domain of 3th3uxaykco0vaxstgvddvat.rdbaxcjm-bbzkprz.dgv@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=sw4Ift6Q; spf=pass (google.com: domain of 3th3uxaykco0vaxstgvddvat.rdbaxcjm-bbzkprz.dgv@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3TH3uXAYKCO0VaXSTgVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Av06FkFyTGfvOvGo6A9zKZw+CBXEF6FDDT5YdORZtrs=; b=sw4Ift6QkIQDVlTsm4rKEpyHSZaLhULHZE/12qIXwDyvy6galnSCmqE3hV+WQAVYPP Vm0Ka4mGYbpQysCXTVy7n6xEe8LVpmH2r8qwqIH4j1fUfvef2bDy4YM8fw5Xz3z92yry 5eOyYqRoYuBn690KcRqre2j3sTffG3tnN856BwTpTU1wIvqc1AceNvcU045en+jHKJ4D P/rybP6B1tMo/CUP1tLGXd0S+l44oaXzVAA/eAQPZ50sL5mmBcekFBPC/mZfQcpnPorO Q83tK9eM1adlpl2ld1NqJGk4cium0VTiio51DR3nbpCciC3l3CrwGkL1YQjiYSoOb8h9 2xRg== X-Google-Smtp-Source: APXvYqwSC38bt7d+wyDQ/RiPd1KAVrVX9T7m1Y/u/pZl9RfAookLsQSNAuw0CuKISwNXubwsUZEAEoIDXig= X-Received: by 2002:aca:5ad7:: with SMTP id o206mr583301oib.65.1559133516337; Wed, 29 May 2019 05:38:36 -0700 (PDT) Date: Wed, 29 May 2019 14:38:12 +0200 In-Reply-To: <20190529123812.43089-1-glider@google.com> Message-Id: <20190529123812.43089-4-glider@google.com> Mime-Version: 1.0 References: <20190529123812.43089-1-glider@google.com> X-Mailer: git-send-email 2.22.0.rc1.257.g3120a18244-goog Subject: [PATCH v5 3/3] lib: introduce test_meminit module From: Alexander Potapenko To: Kees Cook , Andrew Morton , Christoph Lameter Cc: Alexander Potapenko , Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Jann Horn , Marco Elver , linux-mm@kvack.org, linux-security-module@vger.kernel.org, kernel-hardening@lists.openwall.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add tests for heap and pagealloc initialization. These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization. Expected test output in the case the kernel provides heap initialization (e.g. when running with either init_on_alloc=1 or init_on_free=1): test_meminit: all 10 tests in test_pages passed test_meminit: all 40 tests in test_kvmalloc passed test_meminit: all 60 tests in test_kmemcache passed test_meminit: all 10 tests in test_rcu_persistent passed test_meminit: all 120 tests passed! Signed-off-by: Alexander Potapenko To: Kees Cook To: Andrew Morton To: Christoph Lameter Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Jann Horn Cc: Marco Elver Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com Acked-by: Kees Cook --- v3: - added example test output to the description - fixed a missing include spotted by kbuild test robot - added a missing MODULE_LICENSE - call do_kmem_cache_size() with size >= sizeof(void*) to unbreak debug builds v5: - added tests for RCU slabs and __GFP_ZERO --- lib/Kconfig.debug | 8 + lib/Makefile | 1 + lib/test_meminit.c | 362 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 371 insertions(+) create mode 100644 lib/test_meminit.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index cbdfae379896..085711f14abf 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2040,6 +2040,14 @@ config TEST_STACKINIT If unsure, say N. +config TEST_MEMINIT + tristate "Test heap/page initialization" + help + Test if the kernel is zero-initializing heap and page allocations. + This can be useful to test init_on_alloc and init_on_free features. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTEST diff --git a/lib/Makefile b/lib/Makefile index fb7697031a79..05980c802500 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -91,6 +91,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o +obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/ diff --git a/lib/test_meminit.c b/lib/test_meminit.c new file mode 100644 index 000000000000..ed7efec1387b --- /dev/null +++ b/lib/test_meminit.c @@ -0,0 +1,362 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for SL[AOU]B/page initialization at alloc/free time. + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include +#include + +#define GARBAGE_INT (0x09A7BA9E) +#define GARBAGE_BYTE (0x9E) + +#define REPORT_FAILURES_IN_FN() \ + do { \ + if (failures) \ + pr_info("%s failed %d out of %d times\n", \ + __func__, failures, num_tests); \ + else \ + pr_info("all %d tests in %s passed\n", \ + num_tests, __func__); \ + } while (0) + +/* Calculate the number of uninitialized bytes in the buffer. */ +static int __init count_nonzero_bytes(void *ptr, size_t size) +{ + int i, ret = 0; + unsigned char *p = (unsigned char *)ptr; + + for (i = 0; i < size; i++) + if (p[i]) + ret++; + return ret; +} + +/* Fill a buffer with garbage, skipping |skip| first bytes. */ +static void __init fill_with_garbage_skip(void *ptr, size_t size, size_t skip) +{ + unsigned int *p = (unsigned int *)ptr; + int i = 0; + + if (skip) { + WARN_ON(skip > size); + p += skip; + } + while (size >= sizeof(*p)) { + p[i] = GARBAGE_INT; + i++; + size -= sizeof(*p); + } + if (size) + memset(&p[i], GARBAGE_BYTE, size); +} + +static void __init fill_with_garbage(void *ptr, size_t size) +{ + fill_with_garbage_skip(ptr, size, 0); +} + +static int __init do_alloc_pages_order(int order, int *total_failures) +{ + struct page *page; + void *buf; + size_t size = PAGE_SIZE << order; + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + fill_with_garbage(buf, size); + __free_pages(page, order); + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + __free_pages(page, order); + return 1; +} + +/* Test the page allocator by calling alloc_pages with different orders. */ +static int __init test_pages(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i; + + for (i = 0; i < 10; i++) + num_tests += do_alloc_pages_order(i, &failures); + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +/* Test kmalloc() with given parameters. */ +static int __init do_kmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = kmalloc(size, GFP_KERNEL); + fill_with_garbage(buf, size); + kfree(buf); + + buf = kmalloc(size, GFP_KERNEL); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + kfree(buf); + return 1; +} + +/* Test vmalloc() with given parameters. */ +static int __init do_vmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = vmalloc(size); + fill_with_garbage(buf, size); + vfree(buf); + + buf = vmalloc(size); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + vfree(buf); + return 1; +} + +/* Test kmalloc()/vmalloc() by allocating objects of different sizes. */ +static int __init test_kvmalloc(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 20; i++) { + size = 1 << i; + num_tests += do_kmalloc_size(size, &failures); + num_tests += do_vmalloc_size(size, &failures); + } + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +#define CTOR_BYTES (sizeof(unsigned int)) +#define CTOR_PATTERN (0x41414141) +/* Initialize the first 4 bytes of the object. */ +static void test_ctor(void *obj) +{ + *(unsigned int *)obj = CTOR_PATTERN; +} + +/* + * Check the invariants for the buffer allocated from a slab cache. + * If the cache has a test constructor, the first 4 bytes of the object must + * always remain equal to CTOR_PATTERN. + * If the cache isn't an RCU-typesafe one, or if the allocation is done with + * __GFP_ZERO, then the object contents must be zeroed after allocation. + * If the cache is an RCU-typesafe one, the object contents must never be + * zeroed after the first use. This is checked by memcmp() in + * do_kmem_cache_size(). + */ +static bool __init check_buf(void *buf, int size, bool want_ctor, + bool want_rcu, bool want_zero) +{ + int bytes; + bool fail = false; + + bytes = count_nonzero_bytes(buf, size); + WARN_ON(want_ctor && want_zero); + if (want_zero) + return bytes; + if (want_ctor) { + if (*(unsigned int *)buf != CTOR_PATTERN) + fail = 1; + } else { + if (bytes) + fail = !want_rcu; + } + return fail; +} + +/* + * Test kmem_cache with given parameters: + * want_ctor - use a constructor; + * want_rcu - use SLAB_TYPESAFE_BY_RCU; + * want_zero - use __GFP_ZERO. + */ +static int __init do_kmem_cache_size(size_t size, bool want_ctor, + bool want_rcu, bool want_zero, + int *total_failures) +{ + struct kmem_cache *c; + int iter; + bool fail = false; + gfp_t alloc_mask = GFP_KERNEL | (want_zero ? __GFP_ZERO : 0); + void *buf, *buf_copy; + + c = kmem_cache_create("test_cache", size, 1, + want_rcu ? SLAB_TYPESAFE_BY_RCU : 0, + want_ctor ? test_ctor : NULL); + for (iter = 0; iter < 10; iter++) { + buf = kmem_cache_alloc(c, alloc_mask); + /* Check that buf is zeroed, if it must be. */ + fail = check_buf(buf, size, want_ctor, want_rcu, want_zero); + fill_with_garbage_skip(buf, size, want_ctor ? CTOR_BYTES : 0); + /* + * If this is an RCU cache, use a critical section to ensure we + * can touch objects after they're freed. + */ + if (want_rcu) { + rcu_read_lock(); + /* + * Copy the buffer to check that it's not wiped on + * free(). + */ + buf_copy = kmalloc(size, GFP_KERNEL); + if (buf_copy) + memcpy(buf_copy, buf, size); + } + kmem_cache_free(c, buf); + if (want_rcu) { + /* + * Check that |buf| is intact after kmem_cache_free(). + * |want_zero| is false, because we wrote garbage to + * the buffer already. + */ + fail |= check_buf(buf, size, want_ctor, want_rcu, + false); + if (buf_copy) { + fail |= (bool)memcmp(buf, buf_copy, size); + kfree(buf_copy); + } + rcu_read_unlock(); + } + } + kmem_cache_destroy(c); + + *total_failures += fail; + return 1; +} + +/* + * Check that the data written to an RCU-allocated object survives + * reallocation. + */ +static int __init do_kmem_cache_rcu_persistent(int size, int *total_failures) +{ + struct kmem_cache *c; + void *buf, *buf_contents, *saved_ptr; + void **used_objects; + int i, iter, maxiter = 1024; + bool fail = false; + + c = kmem_cache_create("test_cache", size, size, SLAB_TYPESAFE_BY_RCU, + NULL); + buf = kmem_cache_alloc(c, GFP_KERNEL); + saved_ptr = buf; + fill_with_garbage(buf, size); + buf_contents = kmalloc(size, GFP_KERNEL); + if (!buf_contents) + goto out; + used_objects = kmalloc_array(maxiter, sizeof(void *), GFP_KERNEL); + if (!used_objects) { + kfree(buf_contents); + goto out; + } + memcpy(buf_contents, buf, size); + kmem_cache_free(c, buf); + /* + * Run for a fixed number of iterations. If we never hit saved_ptr, + * assume the test passes. + */ + for (iter = 0; iter < maxiter; iter++) { + buf = kmem_cache_alloc(c, GFP_KERNEL); + used_objects[iter] = buf; + if (buf == saved_ptr) { + fail = memcmp(buf_contents, buf, size); + for (i = 0; i <= iter; i++) + kmem_cache_free(c, used_objects[i]); + goto free_out; + } + } + +free_out: + kmem_cache_destroy(c); + kfree(buf_contents); + kfree(used_objects); +out: + *total_failures += fail; + return 1; +} + +/* + * Test kmem_cache allocation by creating caches of different sizes, with and + * without constructors, with and without SLAB_TYPESAFE_BY_RCU. + */ +static int __init test_kmemcache(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, flags, size; + bool ctor, rcu, zero; + + for (i = 0; i < 10; i++) { + size = 8 << i; + for (flags = 0; flags < 8; flags++) { + ctor = flags & 1; + rcu = flags & 2; + zero = flags & 4; + if (ctor & zero) + continue; + num_tests += do_kmem_cache_size(size, ctor, rcu, zero, + &failures); + } + } + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +/* Test the behavior of SLAB_TYPESAFE_BY_RCU caches of different sizes. */ +static int __init test_rcu_persistent(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 10; i++) { + size = 8 << i; + num_tests += do_kmem_cache_rcu_persistent(size, &failures); + } + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +/* + * Run the tests. Each test function returns the number of executed tests and + * updates |failures| with the number of failed tests. + */ +static int __init test_meminit_init(void) +{ + int failures = 0, num_tests = 0; + + num_tests += test_pages(&failures); + num_tests += test_kvmalloc(&failures); + num_tests += test_kmemcache(&failures); + num_tests += test_rcu_persistent(&failures); + + if (failures == 0) + pr_info("all %d tests passed!\n", num_tests); + else + pr_info("failures: %d out of %d\n", failures, num_tests); + + return failures ? -EINVAL : 0; +} +module_init(test_meminit_init); + +MODULE_LICENSE("GPL");