From patchwork Wed Jun 26 12:19:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 11017571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA0B41575 for ; Wed, 26 Jun 2019 12:19:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D00782877B for ; Wed, 26 Jun 2019 12:19:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C2EBC28780; Wed, 26 Jun 2019 12:19:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 521CE2877F for ; Wed, 26 Jun 2019 12:19:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3BDEF6B0006; Wed, 26 Jun 2019 08:19:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 347118E0005; Wed, 26 Jun 2019 08:19:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E83B8E0002; Wed, 26 Jun 2019 08:19:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f70.google.com (mail-yw1-f70.google.com [209.85.161.70]) by kanga.kvack.org (Postfix) with ESMTP id E867D6B0006 for ; Wed, 26 Jun 2019 08:19:53 -0400 (EDT) Received: by mail-yw1-f70.google.com with SMTP id p18so4522128ywe.17 for ; Wed, 26 Jun 2019 05:19:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=MF9i9Awz3dO0JlSO7N89WFnII9sqT4fve14p5wg/k3E=; b=W9MH49HE41ImZAHol+GenJL9gQmM01PORHn0BHX5nhm17fFvHRaHr0yPXPqNUqC50c nQxFpJEx52t8ksozUHY535yzEDWmW+e949V29Enf96j3oU2/ArOEDwkRFLk3GyqGtRDT pqqqqavlcvQYmBNKmqwsXhTK3akzSr4lyVkaw1ShUE6oShvHHWx+Krzs5fqzSby3QSbc srDzsgwNdfDULdcLCinvjxW2LKh5FN8O1HYdtOXZis9FNII66V4RXwjmzK4H3VEuCpJk mBspEkrM45ZoYkg21ADJzNxcxBuav/fcn2MKkVUVxIlyqd/B+pBzc9kXALsrmHbzyp39 mfnw== X-Gm-Message-State: APjAAAX4j1oubJAr1eYcI0JY4j6lA6CUJ7L4pHFXwYJmHWcm/HWZz7rz KvbrKSt2gH61nYRToKvKa3P/shxMSSWJbShNnN8GvGf5BkpLMfvFIV1OHeB7sfFXvUNbLp8p5UZ W5aWSNz1fpBGuBVJ6JQlYUM0usJA99+MWQegWk+7EyJXCHdokFlcAkqyH4WNsk+hs8g== X-Received: by 2002:a25:7056:: with SMTP id l83mr2385090ybc.426.1561551593661; Wed, 26 Jun 2019 05:19:53 -0700 (PDT) X-Received: by 2002:a25:7056:: with SMTP id l83mr2385031ybc.426.1561551592703; Wed, 26 Jun 2019 05:19:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561551592; cv=none; d=google.com; s=arc-20160816; b=HUqEvK1PfXMvrrqvyDEna7nhhUD4nXU4+kyaZ1j+Qk8eq8aLuwvpWbz70VoAtatcUc JLkBWB5yo9E7t+hyBeAfkrP0gDJvfxh4ITC7f+F06Han7Iid9aRSqXBmT0m3LRo7ducd ZcPXFLCsV+pm9uWf6LYmU19YD430FV9g7uVUnI4VeWwed5zeWwK7qeDwr0FuxI3rzjMh c4tdXsxje57k7pQm80wKytFtXGjnSqHbViL5jUjE/LCyD99ChaCIRt3XjfHBsrRJOxnF d75HjmCD5Vx/zaInu16QTur14JL6Cp+5U0Jjj2yxTIXs1FEGuDctMPqZV/Dt7yQ/EJvp 98aQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=MF9i9Awz3dO0JlSO7N89WFnII9sqT4fve14p5wg/k3E=; b=EDxSVx+9ISOZgrlGkSPmucJa8bdJI/MF9j1y69kMEaIBAdhmjtHcQxBiAa18t5/o5a 2GUpJBXiH7oel9Tv0weQVSAYRPUuuswIdzHvxeSU1qg+xcTKxEeffpipSb5OT27d4Cel L2kGuNFigx7jxC3e20q34gc7uiKp6QaUMQD9Kl36knkrv2VBKaS+iU752LkDN20oL7gC PR4LhIJz4OY8AeF/qkJlJ7IOXGVIcg5fFGT1g7oHWwHaaJ/Ow9TD0gD7GsjlR30b3V3F dUf//eIG4IdGIJhUs39pzaLni3VaC5BzvsgB5CMhFtncxysfJeUGelxWdHiZ0DoRnp54 XIUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=JyMLL8yZ; spf=pass (google.com: domain of 36gitxqykcokrwtopcrzzrwp.nzxwtyfi-xxvglnv.zcr@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=36GITXQYKCOkRWTOPcRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id p130sor9274670ywg.139.2019.06.26.05.19.52 for (Google Transport Security); Wed, 26 Jun 2019 05:19:52 -0700 (PDT) Received-SPF: pass (google.com: domain of 36gitxqykcokrwtopcrzzrwp.nzxwtyfi-xxvglnv.zcr@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=JyMLL8yZ; spf=pass (google.com: domain of 36gitxqykcokrwtopcrzzrwp.nzxwtyfi-xxvglnv.zcr@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=36GITXQYKCOkRWTOPcRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MF9i9Awz3dO0JlSO7N89WFnII9sqT4fve14p5wg/k3E=; b=JyMLL8yZJd+9QC3dwgbLicSCvUx4nZJskahQibcyfLaASQSwU2AKE+msgtYP459no4 b0ZnxAlJHMdF2cx4tZ9o1HDwrZcRMBgPtuvIzSWE6vFc9eMF9TohH5w0kzhEEdZgHeAh J5KX56QybuJcX6v2r51Kb9bDlwk85Eml3ZvLlDCBxmU0wBfz5vyCg0sjx1gK7rziQS27 YzOYbJ5aqYAVa8Ia1Re1bPgg3U7QSTgaEz281kIEcS0G3SZ9PhO+8m42DcSl1Ctjnda8 KH3AWEpGRvQspFeGUusPZvV8WP5149WyDWy+hZIwqsXgNjxxtzmVvIUlJbFvbEG8VTUe 70mA== X-Google-Smtp-Source: APXvYqxQcnXyCeDt4XcKLpETLck9j99wwKpYJgjH0Qda6MgRCkGISit9BWzNtFAzZ27T8st539NTCLK7rZM= X-Received: by 2002:a0d:edc5:: with SMTP id w188mr2462262ywe.17.1561551592200; Wed, 26 Jun 2019 05:19:52 -0700 (PDT) Date: Wed, 26 Jun 2019 14:19:42 +0200 In-Reply-To: <20190626121943.131390-1-glider@google.com> Message-Id: <20190626121943.131390-2-glider@google.com> Mime-Version: 1.0 References: <20190626121943.131390-1-glider@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options From: Alexander Potapenko To: Andrew Morton , Christoph Lameter , Kees Cook Cc: Alexander Potapenko , Masahiro Yamada , Michal Hocko , James Morris , "Serge E. Hallyn" , Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Randy Dunlap , Jann Horn , Mark Rutland , Marco Elver , Qian Cai , linux-mm@kvack.org, linux-security-module@vger.kernel.org, kernel-hardening@lists.openwall.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic. This is expected to be on-by-default on Android and Chrome OS. And it gives the opportunity for anyone else to use it under distros too via the boot args. (The init_on_free feature is regularly requested by folks where memory forensics is included in their threat models.) init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses. Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and SLAB_TYPESAFE_BY_RCU flag. Those are never zero-initialized to preserve their semantics. Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON. If either SLUB poisoning or page poisoning is enabled, we disable init_on_alloc and init_on_free so that initialization doesn't interfere with debugging. Slowdown for the new features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%) Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both on_alloc and on_free hooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization. Although init_on_free is rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructure for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series. Signed-off-by: Alexander Potapenko Acked-by: Kees Cook To: Andrew Morton To: Christoph Lameter To: Kees Cook Cc: Masahiro Yamada Cc: Michal Hocko Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: Marco Elver Cc: Qian Cai Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com Signed-off-by: Andrew Morton --- v2: - unconditionally initialize pages in kernel_init_free_pages() - comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening v3: - don't call kernel_init_free_pages() from memblock_free_pages() - adopted some Kees' comments for the patch description v4: - use NULL instead of 0 in slab_alloc_node() (found by kbuild test robot) - don't write to NULL object in slab_alloc_node() (found by Android testing) v5: - adjusted documentation wording as suggested by Kees - disable SLAB_POISON if auto-initialization is on - don't wipe RCU cache allocations made without __GFP_ZERO - dropped SLOB support v7: - rebase the patch, added the Acked-by: tag v8: - addressed comments by Michal Hocko: revert kernel/kexec_core.c and apply initialization in dma_pool_free() - disable init_on_alloc/init_on_free if slab poisoning or page poisoning are enabled, as requested by Qian Cai - skip the redzone when initializing a freed heap object, as requested by Qian Cai and Kees Cook - use s->offset to address the freeptr (suggested by Kees Cook) - updated the patch description, added Signed-off-by: tag --- .../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ mm/dmapool.c | 4 +- mm/page_alloc.c | 71 +++++++++++++++++-- mm/slab.c | 16 ++++- mm/slab.h | 19 +++++ mm/slub.c | 43 +++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 ++++++++ 10 files changed, 199 insertions(+), 18 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 138f6664b2e2..84ee1121a2b9 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1673,6 +1673,15 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can diff --git a/drivers/infiniband/core/uverbs_ioctl.c b/drivers/infiniband/core/uverbs_ioctl.c index 829b0c6944d8..61758201d9b2 100644 --- a/drivers/infiniband/core/uverbs_ioctl.c +++ b/drivers/infiniband/core/uverbs_ioctl.c @@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used = ALIGN(new_used, sizeof(*pbundle->internal_buffer)); - if (flags & __GFP_ZERO) + if (want_init_on_alloc(flags)) memset(res, 0, size); return res; } diff --git a/include/linux/mm.h b/include/linux/mm.h index dd0b5f4e1e45..96be2604f313 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2696,6 +2696,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE(init_on_alloc); +#endif +static inline bool want_init_on_alloc(gfp_t flags) +{ + if (static_branch_unlikely(&init_on_alloc)) + return true; + return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_free); +#else +DECLARE_STATIC_KEY_FALSE(init_on_free); +#endif +static inline bool want_init_on_free(void) +{ + return static_branch_unlikely(&init_on_free); +} + extern bool _debug_pagealloc_enabled; static inline bool debug_pagealloc_enabled(void) diff --git a/mm/dmapool.c b/mm/dmapool.c index 8c94c89a6f7e..fe5d33060415 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -378,7 +378,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, #endif spin_unlock_irqrestore(&pool->lock, flags); - if (mem_flags & __GFP_ZERO) + if (want_init_on_alloc(mem_flags)) memset(retval, 0, pool->size); return retval; @@ -428,6 +428,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) } offset = vaddr - page->vaddr; + if (want_init_on_free()) + memset(vaddr, 0, pool->size); #ifdef DMAPOOL_DEBUG if ((dma - page->dma) != offset) { spin_unlock_irqrestore(&pool->lock, flags); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d66bc8abe0af..991d23e8a9e2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -136,6 +136,56 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_alloc); +#else +DEFINE_STATIC_KEY_FALSE(init_on_alloc); +#endif +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_free); +#else +DEFINE_STATIC_KEY_FALSE(init_on_free); +#endif + +static int __init early_init_on_alloc(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result && IS_ENABLED(CONFIG_PAGE_POISONING)) { + pr_warn("mem auto-init: Disabling init_on_alloc: CONFIG_PAGE_POISONING is on\n"); + bool_result = false; + } + if (bool_result) + static_branch_enable(&init_on_alloc); + else + static_branch_disable(&init_on_alloc); + return ret; +} +early_param("init_on_alloc", early_init_on_alloc); + +static int __init early_init_on_free(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result && IS_ENABLED(CONFIG_PAGE_POISONING)) { + pr_warn("mem auto-init: Disabling init_on_free: CONFIG_PAGE_POISONING is on\n"); + bool_result = false; + } + if (bool_result) + static_branch_enable(&init_on_free); + else + static_branch_disable(&init_on_free); + return ret; +} +early_param("init_on_free", early_init_on_free); /* * A cached value of the page's pageblock's migratetype, used when the page is @@ -1090,6 +1140,14 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) return ret; } +static void kernel_init_free_pages(struct page *page, int numpages) +{ + int i; + + for (i = 0; i < numpages; i++) + clear_highpage(page + i); +} + static __always_inline bool free_pages_prepare(struct page *page, unsigned int order, bool check_free) { @@ -1142,6 +1200,8 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); + if (want_init_on_free()) + kernel_init_free_pages(page, 1 << order); if (debug_pagealloc_enabled()) kernel_map_pages(page, 1 << order, 0); @@ -2020,8 +2080,8 @@ static inline int check_new_page(struct page *page) static inline bool free_pages_prezeroed(void) { - return IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && - page_poisoning_enabled(); + return (IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && + page_poisoning_enabled()) || want_init_on_free(); } #ifdef CONFIG_DEBUG_VM @@ -2075,13 +2135,10 @@ inline void post_alloc_hook(struct page *page, unsigned int order, static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags) { - int i; - post_alloc_hook(page, order, gfp_flags); - if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO)) - for (i = 0; i < (1 << order); i++) - clear_highpage(page + i); + if (!free_pages_prezeroed() && want_init_on_alloc(gfp_flags)) + kernel_init_free_pages(page, 1 << order); if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); diff --git a/mm/slab.c b/mm/slab.c index f7117ad9b3a3..98a89d7c922d 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1830,6 +1830,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num = 0; + /* + * If slab auto-initialization on free is enabled, store the freelist + * off-slab, so that its contents don't end up in one of the allocated + * objects. + */ + if (unlikely(slab_want_init_on_free(cachep))) + return false; + if (cachep->ctor || flags & SLAB_TYPESAFE_BY_RCU) return false; @@ -3263,7 +3271,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, local_irq_restore(save_flags); ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller); - if (unlikely(flags & __GFP_ZERO) && ptr) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &ptr); @@ -3320,7 +3328,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller); prefetchw(objp); - if (unlikely(flags & __GFP_ZERO) && objp) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &objp); @@ -3441,6 +3449,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, struct array_cache *ac = cpu_cache_get(cachep); check_irq_off(); + if (unlikely(slab_want_init_on_free(cachep))) + memset(objp, 0, cachep->object_size); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); @@ -3528,7 +3538,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_); /* Clear memory outside IRQ disabled section */ - if (unlikely(flags & __GFP_ZERO)) + if (unlikely(slab_want_init_on_alloc(flags, s))) for (i = 0; i < size; i++) memset(p[i], 0, s->object_size); diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..31032d488b29 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -524,4 +524,23 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep, static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { } #endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc)) { + if (c->ctor) + return false; + if (c->flags & SLAB_TYPESAFE_BY_RCU) + return flags & __GFP_ZERO; + return true; + } + return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free(struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_free)) + return !(c->ctor || (c->flags & SLAB_TYPESAFE_BY_RCU)); + return false; +} + #endif /* MM_SLAB_H */ diff --git a/mm/slub.c b/mm/slub.c index cd04dbd2b5d0..d746e86c45a8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1279,6 +1279,13 @@ static int __init setup_slub_debug(char *str) if (*str == ',') slub_debug_slabs = str + 1; out: + if ((static_branch_unlikely(&init_on_alloc) || + static_branch_unlikely(&init_on_free)) && + (slub_debug & SLAB_POISON)) { + pr_warn("mem auto-init: Disabling init_on_alloc/init_on_free: can't be used together with SLAB_POISON\n"); + static_branch_disable(&init_on_alloc); + static_branch_disable(&init_on_free); + } return 1; } @@ -1424,6 +1431,28 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x) static inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail) { + + void *object; + void *next = *head; + void *old_tail = *tail ? *tail : *head; + int rsize; + + if (slab_want_init_on_free(s)) + do { + object = next; + next = get_freepointer(s, object); + /* + * Clear the object and the metadata, but don't touch + * the redzone. + */ + memset(object, 0, s->object_size); + rsize = (s->flags & SLAB_RED_ZONE) ? s->red_left_pad + : 0; + memset((char *)object + s->inuse, 0, + s->size - s->inuse - rsize); + set_freepointer(s, object, next); + } while (object != old_tail); + /* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here. @@ -1433,9 +1462,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, defined(CONFIG_DEBUG_OBJECTS_FREE) || \ defined(CONFIG_KASAN) - void *object; - void *next = *head; - void *old_tail = *tail ? *tail : *head; + next = *head; /* Head and tail of the reconstructed freelist */ *head = NULL; @@ -2741,8 +2768,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, prefetch_freepointer(s, next_object); stat(s, ALLOC_FASTPATH); } + /* + * If the object has been wiped upon free, make sure it's fully + * initialized by zeroing out freelist pointer. + */ + if (unlikely(slab_want_init_on_free(s)) && object) + memset(object + s->offset, 0, sizeof(void *)); - if (unlikely(gfpflags & __GFP_ZERO) && object) + if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object) memset(object, 0, s->object_size); slab_post_alloc_hook(s, gfpflags, 1, &object); @@ -3163,7 +3196,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, local_irq_enable(); /* Clear memory outside IRQ disabled fastpath loop */ - if (unlikely(flags & __GFP_ZERO)) { + if (unlikely(slab_want_init_on_alloc(flags, s))) { int j; for (j = 0; j < i; j++) diff --git a/net/core/sock.c b/net/core/sock.c index af09a23e4822..425e97f693ce 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1596,7 +1596,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO); if (!sk) return sk; - if (priority & __GFP_ZERO) + if (want_init_on_alloc(priority)) sk_prot_clear_nulls(sk, prot->obj_size); } else sk = kmalloc(prot->obj_size, priority); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index c6cb2d9b2905..a1ffe2eb4d5f 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -160,6 +160,35 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Enable heap memory zeroing on allocation by default" + help + This has the effect of setting "init_on_alloc=1" on the kernel + command line. This can be disabled with "init_on_alloc=0". + When "init_on_alloc" is enabled, all page allocator and slab + allocator memory will be zeroed when allocated, eliminating + many kinds of "uninitialized heap memory" flaws, especially + heap content exposures. The performance impact varies by + workload, but most cases see <1% impact. Some synthetic + workloads have measured as high as 7%. + +config INIT_ON_FREE_DEFAULT_ON + bool "Enable heap memory zeroing on free by default" + help + This has the effect of setting "init_on_free=1" on the kernel + command line. This can be disabled with "init_on_free=0". + Similar to "init_on_alloc", when "init_on_free" is enabled, + all page allocator and slab allocator memory will be zeroed + when freed, eliminating many kinds of "uninitialized heap memory" + flaws, especially heap content exposures. The primary difference + with "init_on_free" is that data lifetime in memory is reduced, + as anything freed is wiped immediately, making live forensics or + cold boot memory attacks unable to recover freed memory contents. + The performance impact varies by workload, but is more expensive + than "init_on_alloc" due to the negative cache effects of + touching "cold" memory areas. Most cases see 3-5% impact. Some + synthetic workloads have measured as high as 8%. + endmenu endmenu From patchwork Wed Jun 26 12:19:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 11017575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6A8C813AF for ; Wed, 26 Jun 2019 12:20:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5867E2877F for ; Wed, 26 Jun 2019 12:20:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4A4B628780; Wed, 26 Jun 2019 12:20:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0416328783 for ; Wed, 26 Jun 2019 12:20:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C21D56B0007; Wed, 26 Jun 2019 08:19:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BAAF78E0005; Wed, 26 Jun 2019 08:19:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4C268E0002; Wed, 26 Jun 2019 08:19:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 840C46B0007 for ; Wed, 26 Jun 2019 08:19:58 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id p34so2644100qtp.1 for ; Wed, 26 Jun 2019 05:19:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=OTM/dBWKP5pXAZwQDJU1YbjuxxUPU+6Vr1IAcbjuDVo=; b=VW9n7OkRHo3+bUwppd7bcdF1H+NTc7S+7XK1xZL3Jxs9v68FBBwuIcnF7WOpV0sp9W hfu9OTaIOj7Tz6xRmMqu65aIBlkmxBEQrSEzbRyytOIaK5pUhKLQ+um5ZN/uNg0Zy5it +/gA5/Rl5j9crPQ86TJ5nZ8WnkTJPHdmiOXjEnMmTYn3+XAHdDWk9b0ruf7we0pFZkSr 5xIqU4pjVj8qUQVpbgEx1l5IVmDhL0O891Qj+fEBG87dTp7pChjZ0YAsIr2U4pR7zhUu MhXJwCeqB4LweA/RUz8ALDoR9ADo1sf21bNMjVWglKDj3VlCf3XdyoSi2vwX8jQvd7Qv 72BA== X-Gm-Message-State: APjAAAW1CRsUU3k9yMUslNkSurfguc7gYNH1dnSH5Cg4zwTqYwNQlTfc f4zQpjgOL3xoqFP+2BNEEprfwSks0VfFMhr7TLhGounLpCxmVP7qAye9EvybM1YD76JbchYPgJq 2WZqH+2lgvp9mpOXw2AKFzgteUepfOqQvKG5yj5oeD+KrN67HhZwDSjWQEnc23VT3hg== X-Received: by 2002:a0c:fde3:: with SMTP id m3mr3174935qvu.205.1561551598304; Wed, 26 Jun 2019 05:19:58 -0700 (PDT) X-Received: by 2002:a0c:fde3:: with SMTP id m3mr3174884qvu.205.1561551597661; Wed, 26 Jun 2019 05:19:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561551597; cv=none; d=google.com; s=arc-20160816; b=X+CUARGhUIjYjUDbk2Cu3YdU64SqW8OqsWp+Hn5ulNXxhB38S5Wzf7kk7mKmL8Kh7x xwsbgtD1mOUrO6Ru5aW7nAvT9Y8mOw2re5LnOJvrrw94TfHyWrVGfxoWtFvfQEDnMHJ7 gxbU/NI055BJy8QBCjQqec5Snvp5Qh20InUMNTmx+F7tvG89RsHMqIEec7YXh5FVJDnO fcGcdL4hYrF0j0Vhzx6ZT+lJ42ocl8tljEVo+T6/sdG2Wir9v7kpcrc6QMne7vlac1AC LAO/FN390LRJxZV0J6XD7Fuk5TPdfRHfADUhS9NW/9JkaGRM35Smyvtv7wYiVbQNnYRN uHaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=OTM/dBWKP5pXAZwQDJU1YbjuxxUPU+6Vr1IAcbjuDVo=; b=xQhJTZhCwHjtc8vITRWHTsDbHcbdyL/wg8Jm2eM3utQ5+pDrw4W5N0iOhPRtM66PTW mk872R0/BRqv78KiS7dwJYc9AJSwqVZiS5AfP8PES6mCXWKS2wiP+xH7zch1L03tB30g PhmYr5pa70J6gpOwK4f5r/blvXjwmkb8bYHml4dNg7oV8TNCIQR14LGTZAj2025HUYm3 AWnDVi+lEwc49a+AH9brLULEPVa2rsGIAKf3IzeXKw29uxyBlNc9Lx/JNFTiz9spy06n h9lCjduFu70/7oWm00TjLbkYe4Ue2TnrlZ3fINl4UMzB1ExuFVXbWV2vmkiCdP6cDh0y B02w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=syWBPxEF; spf=pass (google.com: domain of 37witxqykco4wbytuhweewbu.secbydkn-ccalqsa.ehw@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=37WITXQYKCO4WbYTUhWeeWbU.SecbYdkn-ccalQSa.ehW@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id c19sor24257244qta.34.2019.06.26.05.19.57 for (Google Transport Security); Wed, 26 Jun 2019 05:19:57 -0700 (PDT) Received-SPF: pass (google.com: domain of 37witxqykco4wbytuhweewbu.secbydkn-ccalqsa.ehw@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=syWBPxEF; spf=pass (google.com: domain of 37witxqykco4wbytuhweewbu.secbydkn-ccalqsa.ehw@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=37WITXQYKCO4WbYTUhWeeWbU.SecbYdkn-ccalQSa.ehW@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OTM/dBWKP5pXAZwQDJU1YbjuxxUPU+6Vr1IAcbjuDVo=; b=syWBPxEF/AbUhjRgTsNM2BYGj8YN2ayZPfWd2w0N+baqMKNfmYDs3psuZ0fJjnmvMK 8B80Kl55p65EioVsUup+/eJYiZoERn8ga+Ns6c8t+e2zM7fUtewfGAHKbcaepfj3mAsr QDYudk4+4uSvXCjPULthVBmohV475Bm8+ZhVje7oM5PmZGgjo7DE2HsqOMPOW4xrVkTr suPz43xTls29ETSlHXpdKVUnGVPjARGFKxRIeJ1NGVFoH/UeP6mtPjuDCmrixyWKwNuX dNHHr1RlVkLXy0tuKRZLLgBhBdhseNxqCUhMQXCsW1fuymDJCTA4whMM/PyKBDoZy05/ Qsdw== X-Google-Smtp-Source: APXvYqwJ6l1H8lkFtTKpfdzkClk7tEj5d/8h25zPgByAql9o2QQ6HmyKzhgUAkYCGmo80dn12sSTVp6qisQ= X-Received: by 2002:aed:39e5:: with SMTP id m92mr3381054qte.135.1561551597192; Wed, 26 Jun 2019 05:19:57 -0700 (PDT) Date: Wed, 26 Jun 2019 14:19:43 +0200 In-Reply-To: <20190626121943.131390-1-glider@google.com> Message-Id: <20190626121943.131390-3-glider@google.com> Mime-Version: 1.0 References: <20190626121943.131390-1-glider@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v8 2/2] mm: init: report memory auto-initialization features at boot time From: Alexander Potapenko To: Andrew Morton , Christoph Lameter Cc: Alexander Potapenko , Kees Cook , Dmitry Vyukov , James Morris , Jann Horn , Kostya Serebryany , Laura Abbott , Mark Rutland , Masahiro Yamada , Matthew Wilcox , Nick Desaulniers , Randy Dunlap , Sandeep Patil , "Serge E. Hallyn" , Souptick Joarder , Marco Elver , Kaiwan N Billimoria , kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-security-module@vger.kernel.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Print the currently enabled stack and heap initialization modes. Stack initialization is enabled by a config flag, while heap initialization is configured at boot time with defaults being set in the config. It's more convenient for the user to have all information about these hardening measures in one place at boot, so the user can reason about the expected behavior of the running system. The possible options for stack are: - "all" for CONFIG_INIT_STACK_ALL; - "byref_all" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL; - "byref" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF; - "__user" for CONFIG_GCC_PLUGIN_STRUCTLEAK_USER; - "off" otherwise. Depending on the values of init_on_alloc and init_on_free boottime options we also report "heap alloc" and "heap free" as "on"/"off". In the init_on_free mode initializing pages at boot time may take a while, so print a notice about that as well. This depends on how much memory is installed, the memory bandwidth, etc. On a relatively modern x86 system, it takes about 0.75s/GB to wipe all memory: [ 0.418722] mem auto-init: stack:byref_all, heap alloc:off, heap free:on [ 0.419765] mem auto-init: clearing system memory may take some time... [ 12.376605] Memory: 16408564K/16776672K available (14339K kernel code, 1397K rwdata, 3756K rodata, 1636K init, 11460K bss, 368108K reserved, 0K cma-reserved) Signed-off-by: Alexander Potapenko Suggested-by: Kees Cook Acked-by: Kees Cook To: Andrew Morton To: Christoph Lameter Cc: Dmitry Vyukov Cc: James Morris Cc: Jann Horn Cc: Kostya Serebryany Cc: Laura Abbott Cc: Mark Rutland Cc: Masahiro Yamada Cc: Matthew Wilcox Cc: Nick Desaulniers Cc: Randy Dunlap Cc: Sandeep Patil Cc: "Serge E. Hallyn" Cc: Souptick Joarder Cc: Marco Elver Cc: Kaiwan N Billimoria Cc: kernel-hardening@lists.openwall.com Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Signed-off-by: Andrew Morton --- v6: - update patch description, fixed message about clearing memory v7: - rebase the patch, add the Acked-by: tag; - more description updates as suggested by Kees; - make report_meminit() static. v8: - added the Signed-off-by: tag --- init/main.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/init/main.c b/init/main.c index 66a196c5e4c3..ff5803b0841c 100644 --- a/init/main.c +++ b/init/main.c @@ -520,6 +520,29 @@ static inline void initcall_debug_enable(void) } #endif +/* Report memory auto-initialization states for this boot. */ +static void __init report_meminit(void) +{ + const char *stack; + + if (IS_ENABLED(CONFIG_INIT_STACK_ALL)) + stack = "all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)) + stack = "byref_all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF)) + stack = "byref"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER)) + stack = "__user"; + else + stack = "off"; + + pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", + stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", + want_init_on_free() ? "on" : "off"); + if (want_init_on_free()) + pr_info("mem auto-init: clearing system memory may take some time...\n"); +} + /* * Set up kernel memory allocators */ @@ -530,6 +553,7 @@ static void __init mm_init(void) * bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem(); + report_meminit(); mem_init(); kmem_cache_init(); pgtable_init();