From patchwork Tue May 14 14:35:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10943087 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B877513AD for ; Tue, 14 May 2019 14:36:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A606A286DD for ; Tue, 14 May 2019 14:36:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9738D286E4; Tue, 14 May 2019 14:36:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34C36286DD for ; Tue, 14 May 2019 14:36:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CAAB6B0005; Tue, 14 May 2019 10:36:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 17BC66B0006; Tue, 14 May 2019 10:36:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06A716B0007; Tue, 14 May 2019 10:36:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id D3F016B0005 for ; Tue, 14 May 2019 10:36:02 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id n190so31629076ywf.4 for ; Tue, 14 May 2019 07:36:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=tNK6c3aJODeVfKVlftZ1eTaADkEG96XAn6GGaXTgxaQ=; b=aYYi6JHbaIpGE0M1p091HyNVKQrAPyf9sBRJGAWMFC1yiPEVaf80EMPTTqb3zEPSDG hhC7U9XhVCaGRZHhfgIY2H4xjjK7XzQmvGXKqr5DozGXYpOGail7Vfpje/Rhbh7VY9bK qI1JCQ9XkCpah/y2m7SPwBTXh/DuZwuW35xeXSw9h9bNfSOwfvF+2v4kO1jnD20g0uqt rWieKabyJdmqaDQRJa+b1sTI6LJ+9QrRDDVeom6KedrZNyDYmYAeq8zEwUIARtv6n0iN ysSOheijrlVobI3kVNgazdg+d5gG0TFnTHzZ4E1BmGr/7u1zReLSRdh3c4hI4rKdav5r MF4w== X-Gm-Message-State: APjAAAXVoG81TeCimqSiZBp9JqIyj+PT78RBR4kcmHFZdhCLihRoqSDL lSowA6WEF6NxNrh02igEMYs38+Wx6yGxuM8jTd+NZxf2JtB2h+Pjgcw3FA9BvDl4ju6joJsPUri 0cV7GnzSxpdUrGFj4fXsBz4CkPkVvnP4mTzmnHr0pwrkRRo6tZmuz+hrqXdKYV2+bPA== X-Received: by 2002:a25:cbc6:: with SMTP id b189mr17592182ybg.284.1557844562383; Tue, 14 May 2019 07:36:02 -0700 (PDT) X-Received: by 2002:a25:cbc6:: with SMTP id b189mr17592033ybg.284.1557844559833; Tue, 14 May 2019 07:35:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557844559; cv=none; d=google.com; s=arc-20160816; b=IWOfJN9hGdwyAVfXDj2maqQWyAaAzqVeQ2zZt34ZRQ1zpxKZ/u/3ouHrXxYjhjXGqO vlYq+PaX6e0mrnjZL8elB5RKOidOAxvufZ4FRDezWkZvyX7roABk4w8xcr57+ZUq+Rg6 azrEu5swLdBIYvny13Oz3SrRmp0W7YjJr8AtwtxFSGUemdqdkExbn3bPxEHLpYJclhSk ybXxyKiALGd8qONEUvyUszv5nNz38v5aSTyBnSew58ApuhKSARZC1XItZSIHPbrdWbwy PUCu+mSK2Hzd3BsB88KZmirxf+i4lduvsBq51FVQHl9hAz8AxkNcjaN3AJkaDaQR4+mP OTpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=tNK6c3aJODeVfKVlftZ1eTaADkEG96XAn6GGaXTgxaQ=; b=AA3ISSCVK1ovfDn7haLbkhenk9vh6CWWPTseDBNqv2Q8Zd4jvoD67GA5/LTuQ6LCsm 7w+L+ru5ZEV5uBj2S2TTR8kMboeHe/uHRfRvTp2dZCpYDIMexNyNkrMxsda6ZwhIl+nq udbldZ46/G+TLPYixhV7cK+z0ntxYeWD4LQCEPuCtQSQdbjcbB8hA/NDxZlrqhtREf9y tOwwFAPDxM3d5Uyr0dJnUQl9lFdoIPInKZjtQ9T80yXbg4uKz8n7kJ+lsZi78Kn8l8D8 qYJ8jHT9otG1B4nUE9CpWQuu/OFyEBRg9pSacsnbrhiNjSh8L+F5ZbqKNR8tw01GZoWt L7Vg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=vG9gQLI2; spf=pass (google.com: domain of 3ttlaxaykcestyvqr4t11tyr.p1zyv07a-zzx8npx.14t@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3TtLaXAYKCEstyvqr4t11tyr.p1zyv07A-zzx8npx.14t@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id h28sor8687710ybj.41.2019.05.14.07.35.59 for (Google Transport Security); Tue, 14 May 2019 07:35:59 -0700 (PDT) Received-SPF: pass (google.com: domain of 3ttlaxaykcestyvqr4t11tyr.p1zyv07a-zzx8npx.14t@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=vG9gQLI2; spf=pass (google.com: domain of 3ttlaxaykcestyvqr4t11tyr.p1zyv07a-zzx8npx.14t@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3TtLaXAYKCEstyvqr4t11tyr.p1zyv07A-zzx8npx.14t@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tNK6c3aJODeVfKVlftZ1eTaADkEG96XAn6GGaXTgxaQ=; b=vG9gQLI2lXywaOvrrIEFCVZp6zeKS60jUXH/kBcVGz4gs8DO2PORTBGYpvSQBqvwWR fkOlbwzJy7Z08kV3LwjO99kMveD5Cy5THpaGjt3sTwpiFXFLbrLVg2NXZqnaiLIZHpmP KPRuZezJaQjfA/N7+4Em7IwQZvb9JOV2QLIbopbM7qSGJn3hNYAdKygawWJx5fZIh9ev tj1lRwxVa5o88EmITP+jzWnzn0l5WpVlNZyyqCzVlUS4XaYk3YKUYDzrVooZSDo2eJyd tLomXqk1yF5oAyBZUbTqTkhIG7pivvHVNWMo6p+6mqi/s3UnN1UiBZlXO1VsyykVE08l V9HA== X-Google-Smtp-Source: APXvYqx0Gci0O8TuOUC0PcZn/6jZdfvXrJBwxOf4Cb5x2emHjiYnnBvX91urZmqQe0/D0DVQD0O/m9MyJnw= X-Received: by 2002:a25:70c3:: with SMTP id l186mr17482948ybc.54.1557844558228; Tue, 14 May 2019 07:35:58 -0700 (PDT) Date: Tue, 14 May 2019 16:35:34 +0200 In-Reply-To: <20190514143537.10435-1-glider@google.com> Message-Id: <20190514143537.10435-2-glider@google.com> Mime-Version: 1.0 References: <20190514143537.10435-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org Cc: kernel-hardening@lists.openwall.com, Masahiro Yamada , James Morris , "Serge E. Hallyn" , Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Randy Dunlap , Jann Horn , Mark Rutland , linux-mm@kvack.org, linux-security-module@vger.kernel.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic. init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses. Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The only exception is slab caches with constructors. Those are never zero-initialized to preserve their semantics. For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior, i.e. objects are zeroed at both allocation and deallocation time. This is done because SLOB may otherwise return multiple freelist pointers in the allocated object. For SLAB and SLUB enabling either init_on_alloc or init_on_free leads to one-time initialization of the object. Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON. Slowdown for the new features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%) Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. Signed-off-by: Alexander Potapenko To: Andrew Morton To: Christoph Lameter To: Kees Cook Cc: Masahiro Yamada Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- v2: - unconditionally initialize pages in kernel_init_free_pages() - comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening --- .../admin-guide/kernel-parameters.txt | 8 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 68 ++++++++++++++++--- mm/slab.c | 16 ++++- mm/slab.h | 16 +++++ mm/slob.c | 22 +++++- mm/slub.c | 27 ++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 14 ++++ 12 files changed, 178 insertions(+), 23 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 08df58805703..cece9a56ddb1 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1673,6 +1673,14 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can diff --git a/drivers/infiniband/core/uverbs_ioctl.c b/drivers/infiniband/core/uverbs_ioctl.c index 829b0c6944d8..61758201d9b2 100644 --- a/drivers/infiniband/core/uverbs_ioctl.c +++ b/drivers/infiniband/core/uverbs_ioctl.c @@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used = ALIGN(new_used, sizeof(*pbundle->internal_buffer)); - if (flags & __GFP_ZERO) + if (want_init_on_alloc(flags)) memset(res, 0, size); return res; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 083d7b4863ed..18d96f1d07c5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,6 +2610,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE(init_on_alloc); +#endif +static inline bool want_init_on_alloc(gfp_t flags) +{ + if (static_branch_unlikely(&init_on_alloc)) + return true; + return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_free); +#else +DECLARE_STATIC_KEY_FALSE(init_on_free); +#endif +static inline bool want_init_on_free(void) +{ + return static_branch_unlikely(&init_on_free); +} + extern bool _debug_pagealloc_enabled; static inline bool debug_pagealloc_enabled(void) diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index fd5c95ff9251..2f75dd0d0d81 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) arch_kexec_post_alloc_pages(page_address(pages), count, gfp_mask); - if (gfp_mask & __GFP_ZERO) + if (want_init_on_alloc(gfp_mask)) for (i = 0; i < count; i++) clear_highpage(pages + i); } diff --git a/mm/dmapool.c b/mm/dmapool.c index 76a160083506..493d151067cb 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, #endif spin_unlock_irqrestore(&pool->lock, flags); - if (mem_flags & __GFP_ZERO) + if (want_init_on_alloc(mem_flags)) memset(retval, 0, pool->size); return retval; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 59661106da16..463c681a3633 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -133,6 +133,48 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_alloc); +#else +DEFINE_STATIC_KEY_FALSE(init_on_alloc); +#endif +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_free); +#else +DEFINE_STATIC_KEY_FALSE(init_on_free); +#endif + +static int __init early_init_on_alloc(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_alloc); + else + static_branch_disable(&init_on_alloc); + return ret; +} +early_param("init_on_alloc", early_init_on_alloc); + +static int __init early_init_on_free(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_free); + else + static_branch_disable(&init_on_free); + return ret; +} +early_param("init_on_free", early_init_on_free); /* * A cached value of the page's pageblock's migratetype, used when the page is @@ -1092,6 +1134,14 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) return ret; } +static void kernel_init_free_pages(struct page *page, int numpages) +{ + int i; + + for (i = 0; i < numpages; i++) + clear_highpage(page + i); +} + static __always_inline bool free_pages_prepare(struct page *page, unsigned int order, bool check_free) { @@ -1144,9 +1194,10 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); + if (want_init_on_free()) + kernel_init_free_pages(page, 1 << order); if (debug_pagealloc_enabled()) kernel_map_pages(page, 1 << order, 0); - kasan_free_nondeferred_pages(page, order); return true; @@ -1452,8 +1503,10 @@ meminit_pfn_in_nid(unsigned long pfn, int node, void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (early_page_uninitialised(pfn)) + if (early_page_uninitialised(pfn)) { + kernel_init_free_pages(page, 1 << order); return; + } __free_pages_core(page, order); } @@ -1971,8 +2024,8 @@ static inline int check_new_page(struct page *page) static inline bool free_pages_prezeroed(void) { - return IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && - page_poisoning_enabled(); + return (IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && + page_poisoning_enabled()) || want_init_on_free(); } #ifdef CONFIG_DEBUG_VM @@ -2026,13 +2079,10 @@ inline void post_alloc_hook(struct page *page, unsigned int order, static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags) { - int i; - post_alloc_hook(page, order, gfp_flags); - if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO)) - for (i = 0; i < (1 << order); i++) - clear_highpage(page + i); + if (!free_pages_prezeroed() && want_init_on_alloc(gfp_flags)) + kernel_init_free_pages(page, 1 << order); if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); diff --git a/mm/slab.c b/mm/slab.c index 284ab737faee..d00e9de26a45 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1855,6 +1855,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num = 0; + /* + * If slab auto-initialization on free is enabled, store the freelist + * off-slab, so that its contents don't end up in one of the allocated + * objects. + */ + if (unlikely(slab_want_init_on_free(cachep))) + return false; + if (cachep->ctor || flags & SLAB_TYPESAFE_BY_RCU) return false; @@ -3294,7 +3302,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, local_irq_restore(save_flags); ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller); - if (unlikely(flags & __GFP_ZERO) && ptr) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &ptr); @@ -3351,7 +3359,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller); prefetchw(objp); - if (unlikely(flags & __GFP_ZERO) && objp) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &objp); @@ -3472,6 +3480,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, struct array_cache *ac = cpu_cache_get(cachep); check_irq_off(); + if (unlikely(slab_want_init_on_free(cachep))) + memset(objp, 0, cachep->object_size); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); @@ -3559,7 +3569,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_); /* Clear memory outside IRQ disabled section */ - if (unlikely(flags & __GFP_ZERO)) + if (unlikely(slab_want_init_on_alloc(flags, s))) for (i = 0; i < size; i++) memset(p[i], 0, s->object_size); diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..24ae887359b8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep, static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { } #endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc)) + return !(c->ctor); + else + return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free(struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_free)) + return !(c->ctor); + else + return false; +} + #endif /* MM_SLAB_H */ diff --git a/mm/slob.c b/mm/slob.c index 307c2c9feb44..351d3dfee000 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -212,6 +212,19 @@ static void slob_free_pages(void *b, int order) free_pages((unsigned long)b, order); } +/* + * init_on_free=1 also implies initialization at allocation time. + * This is because newly allocated objects may contain freelist pointers + * somewhere in the middle. + */ +static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc) || + static_branch_unlikely(&init_on_free)) + return c ? (!c->ctor) : true; + return flags & __GFP_ZERO; +} + /* * Allocate a slob block within a given slob_page sp. */ @@ -330,8 +343,6 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node) BUG_ON(!b); spin_unlock_irqrestore(&slob_lock, flags); } - if (unlikely(gfp & __GFP_ZERO)) - memset(b, 0, size); return b; } @@ -366,6 +377,9 @@ static void slob_free(void *block, int size) return; } + if (unlikely(want_init_on_free())) + memset(block, 0, size); + if (!slob_page_free(sp)) { /* This slob page is about to become partially free. Easy! */ sp->units = units; @@ -461,6 +475,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) } kmemleak_alloc(ret, size, 1, gfp); + if (unlikely(slob_want_init_on_alloc(gfp, 0))) + memset(ret, 0, size); return ret; } @@ -559,6 +575,8 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node) WARN_ON_ONCE(flags & __GFP_ZERO); c->ctor(b); } + if (unlikely(slob_want_init_on_alloc(flags, c))) + memset(b, 0, c->size); kmemleak_alloc_recursive(b, c->size, 1, c->flags, flags); return b; diff --git a/mm/slub.c b/mm/slub.c index 6b28cd2b5a58..01424e910800 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1423,6 +1423,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x) static inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail) { + + void *object; + void *next = *head; + void *old_tail = *tail ? *tail : *head; + + if (slab_want_init_on_free(s)) + do { + object = next; + next = get_freepointer(s, object); + memset(object, 0, s->size); + set_freepointer(s, object, next); + } while (object != old_tail); + /* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here. @@ -1432,9 +1445,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, defined(CONFIG_DEBUG_OBJECTS_FREE) || \ defined(CONFIG_KASAN) - void *object; - void *next = *head; - void *old_tail = *tail ? *tail : *head; + next = *head; /* Head and tail of the reconstructed freelist */ *head = NULL; @@ -2740,8 +2751,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, prefetch_freepointer(s, next_object); stat(s, ALLOC_FASTPATH); } + /* + * If the object has been wiped upon free, make sure it's fully + * initialized by zeroing out freelist pointer. + */ + if (slab_want_init_on_free(s)) + *(void **)object = 0; - if (unlikely(gfpflags & __GFP_ZERO) && object) + if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object) memset(object, 0, s->object_size); slab_post_alloc_hook(s, gfpflags, 1, &object); @@ -3163,7 +3180,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, local_irq_enable(); /* Clear memory outside IRQ disabled fastpath loop */ - if (unlikely(flags & __GFP_ZERO)) { + if (unlikely(slab_want_init_on_alloc(flags, s))) { int j; for (j = 0; j < i; j++) diff --git a/net/core/sock.c b/net/core/sock.c index 75b1c950b49f..9ceb90c875bc 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1602,7 +1602,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO); if (!sk) return sk; - if (priority & __GFP_ZERO) + if (want_init_on_alloc(priority)) sk_prot_clear_nulls(sk, prot->obj_size); } else sk = kmalloc(prot->obj_size, priority); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index 0a1d4ca314f4..87883e3e3c2a 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -159,6 +159,20 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Set init_on_alloc=1 by default" + help + Enable init_on_alloc=1 by default, making the kernel initialize every + page and heap allocation with zeroes. + init_on_alloc can be overridden via command line. + +config INIT_ON_FREE_DEFAULT_ON + bool "Set init_on_free=1 by default" + help + Enable init_on_free=1 by default, making the kernel initialize freed + pages and slab memory with zeroes. + init_on_free can be overridden via command line. + endmenu endmenu From patchwork Tue May 14 14:35:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10943091 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6BCDE13AD for ; Tue, 14 May 2019 14:36:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B93A286DD for ; Tue, 14 May 2019 14:36:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F6E4286E0; Tue, 14 May 2019 14:36:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8409E286DD for ; Tue, 14 May 2019 14:36:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28F1F6B0006; Tue, 14 May 2019 10:36:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1F1076B0007; Tue, 14 May 2019 10:36:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DF196B0008; Tue, 14 May 2019 10:36:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id DF1BA6B0006 for ; Tue, 14 May 2019 10:36:05 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id v127so31297297ywb.20 for ; Tue, 14 May 2019 07:36:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=B56NDtOL5xZxqPmoBf4XKPM0hIbi/G3vgoZpQOkU934=; b=O1OIHISuG5yqkJiuxH3A+HEDwJS0xeZs9IVyqCt9uArbFeoqPFZVjf0/z9Zm5QS41N trBjeZGT9iWXqbjR9EtLk/0jbnM7vYknDrClGD65VjOdFM1yEeVfuqpqFuMTjDDfNDYp RrIVvpM6gm+ZwO5HrM735yPcvoQIe5bLx5j7LBm8IAU+QQMSSMJIA11Syvcfha7FdORk zEd3blORQ9J6DSZrQLgUV2mp8ke/01ysqMa/8WDFoA9Ah+BkWE1Q/mHLV/aB8iuE7yBb nSEmVBZnwiB6ijv7yvHi/I4WU39Cs54sv3l8aOwxV7VbqbXbjQqjlyw+oxZBZ20l4PL0 elhQ== X-Gm-Message-State: APjAAAVeKmeyI44TjjQ0bU1UPPTN+0ypRGv3tke2+RTCnl9Pn84FvFoR tuGWhOlKO9mQTEVuqe9xO8P5C48hQ/+AAwydfYlE9Ppwntm2ftonLp9kHoIusnIuzbr9pEef+Pn rXJsVYJYog35TcdSSnEmyhrcvtSOBG278fpqFxZb0JAyUJtLKFlGPSHOh56f7ieLSIQ== X-Received: by 2002:a81:5214:: with SMTP id g20mr16976447ywb.365.1557844565625; Tue, 14 May 2019 07:36:05 -0700 (PDT) X-Received: by 2002:a81:5214:: with SMTP id g20mr16976396ywb.365.1557844564894; Tue, 14 May 2019 07:36:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557844564; cv=none; d=google.com; s=arc-20160816; b=pT2zHg+MiiQ3Y3Yme2vldZDKiqQK0LsX8m/lWrvqWa3VwWgh2joOnYj/3Uu7vmklYt wJmUUOFrIg6kvXNtkHmzoVBHbYX7L265tCW3f5fKs9Jo76ZQ2JxSNit1Wb/NErz9ZASu BN3T4btXstfF0U5R4nx8i5evT/8t7cghHhhZJRluDYwPhfYBLcMvzYLNtn1InKNXTYZ7 2+EUuuhNEk+za+49H5s3BaQry2PuKSS1MOXiay5nYpHobbzl6Z9vwo3jJNicHQPuUOlF 392pt6RB6TT/aNRA5gRLWCDmHleyussj97vWdOyddmPps8nr3j9Gk6pLI7hWy8qxriRe kV4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=B56NDtOL5xZxqPmoBf4XKPM0hIbi/G3vgoZpQOkU934=; b=OTE1aN3+e3S2Qw6tUV5GRwkbYmILNqjgCoarDVPAYKWmhooLr7Wy5SoKxY4ppNe6BO nAr3CKQzNc7nWdm0qbOhNP03J/Rgz24OJPmWqBR/rWEfyMd+SmjRWHVcMFjBvWUJ49Ob C8pvHTmsCZI9WUf8yRQxevfI/JUrQuBAo8/1PTQoerp4Nn7VkfIl9nmFYju+j+CEv8LK tGr9nCd4reLesnVZt5SdIu7ArTG6w0fwZi5Z2vZ0CPZC4DoVAuH99u2+wD8hHogCWtNI oozgIwhAvh5FrLx4DFWcgiNAgHyoTuvqYpQKSDS0Vl7cUrFd6bMnT+qLVONmXzYhhj+R LAdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=FHR6usGJ; spf=pass (google.com: domain of 3vnlaxaykcfez41wxaz77z4x.v75416dg-553etv3.7az@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3VNLaXAYKCFEz41wxAz77z4x.v75416DG-553Etv3.7Az@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id 74sor8151621ywp.167.2019.05.14.07.36.04 for (Google Transport Security); Tue, 14 May 2019 07:36:04 -0700 (PDT) Received-SPF: pass (google.com: domain of 3vnlaxaykcfez41wxaz77z4x.v75416dg-553etv3.7az@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=FHR6usGJ; spf=pass (google.com: domain of 3vnlaxaykcfez41wxaz77z4x.v75416dg-553etv3.7az@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3VNLaXAYKCFEz41wxAz77z4x.v75416DG-553Etv3.7Az@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=B56NDtOL5xZxqPmoBf4XKPM0hIbi/G3vgoZpQOkU934=; b=FHR6usGJshdPyEV8Aak1BgcwnljqtriHJkgj5J2aYlmLYw4AidY6SxiAGKzPI6heGk M7s4k+7pcnSrJZaLLKZgghkGGT/JmXsDUr+smqm13C/NSTZ/qTLK7V/WoT+FYxSlhMmU h5INfaZnRIBMAtH7ivc/Ax9f1GJRcH+Ip7kZBMi+4qux53yuZc3BYyOVNFfnnoDvD7BP 5MJZ6n5EequqoW63YBpjaXn0du8xnOhQkd34Pyjz4i3iNz7iQfgQ8AgLqI81tuxaPGBi 6t8ZG+QZQZdizKlEiyFBZm32K+OiwvVqYEV8p+T6KSyE6w3DTDweXhdIyDkmLE5xF2ow cAxg== X-Google-Smtp-Source: APXvYqzaMJGf9ZwPtTAOK12Dhjk0gPbDIoj+2PJs7jr1TXz7qsRVrx7qy+Q3P9umvaEiKNqB4jAt38BOZ9c= X-Received: by 2002:a81:35cc:: with SMTP id c195mr16506221ywa.311.1557844564255; Tue, 14 May 2019 07:36:04 -0700 (PDT) Date: Tue, 14 May 2019 16:35:35 +0200 In-Reply-To: <20190514143537.10435-1-glider@google.com> Message-Id: <20190514143537.10435-3-glider@google.com> Mime-Version: 1.0 References: <20190514143537.10435-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH v2 2/4] lib: introduce test_meminit module From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org Cc: kernel-hardening@lists.openwall.com, Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Jann Horn , linux-mm@kvack.org, linux-security-module@vger.kernel.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add tests for heap and pagealloc initialization. These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization. Signed-off-by: Alexander Potapenko To: Kees Cook To: Andrew Morton To: Christoph Lameter Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Jann Horn Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com Reviewed-by: Kees Cook Tested-by: Kees Cook --- lib/Kconfig.debug | 8 ++ lib/Makefile | 1 + lib/test_meminit.c | 205 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 214 insertions(+) create mode 100644 lib/test_meminit.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index d695ec1477f3..6c3fc68a4a77 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2020,6 +2020,14 @@ config TEST_STACKINIT If unsure, say N. +config TEST_MEMINIT + tristate "Test level of heap/page initialization" + help + Test if the kernel is zero-initializing heap and page allocations. + This can be useful to test init_on_alloc and init_on_free features. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTEST diff --git a/lib/Makefile b/lib/Makefile index 83d7df2661ff..29c5afbe9882 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -91,6 +91,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o +obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/ diff --git a/lib/test_meminit.c b/lib/test_meminit.c new file mode 100644 index 000000000000..67d759498030 --- /dev/null +++ b/lib/test_meminit.c @@ -0,0 +1,205 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for SL[AOU]B/page initialization at alloc/free time. + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include + +#define GARBAGE_INT (0x09A7BA9E) +#define GARBAGE_BYTE (0x9E) + +#define REPORT_FAILURES_IN_FN() \ + do { \ + if (failures) \ + pr_info("%s failed %d out of %d times\n", \ + __func__, failures, num_tests); \ + else \ + pr_info("all %d tests in %s passed\n", \ + num_tests, __func__); \ + } while (0) + +/* Calculate the number of uninitialized bytes in the buffer. */ +static int count_nonzero_bytes(void *ptr, size_t size) +{ + int i, ret = 0; + unsigned char *p = (unsigned char *)ptr; + + for (i = 0; i < size; i++) + if (p[i]) + ret++; + return ret; +} + +static void fill_with_garbage(void *ptr, size_t size) +{ + unsigned int *p = (unsigned int *)ptr; + int i = 0; + + while (size >= sizeof(*p)) { + p[i] = GARBAGE_INT; + i++; + size -= sizeof(*p); + } + if (size) + memset(&p[i], GARBAGE_BYTE, size); +} + +static int __init do_alloc_pages_order(int order, int *total_failures) +{ + struct page *page; + void *buf; + size_t size = PAGE_SIZE << order; + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + fill_with_garbage(buf, size); + __free_pages(page, order); + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + __free_pages(page, order); + return 1; +} + +static int __init test_pages(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i; + + for (i = 0; i < 10; i++) + num_tests += do_alloc_pages_order(i, &failures); + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +static int __init do_kmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = kmalloc(size, GFP_KERNEL); + fill_with_garbage(buf, size); + kfree(buf); + + buf = kmalloc(size, GFP_KERNEL); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + kfree(buf); + return 1; +} + +static int __init do_vmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = vmalloc(size); + fill_with_garbage(buf, size); + vfree(buf); + + buf = vmalloc(size); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + vfree(buf); + return 1; +} + +static int __init test_kvmalloc(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 20; i++) { + size = 1 << i; + num_tests += do_kmalloc_size(size, &failures); + num_tests += do_vmalloc_size(size, &failures); + } + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +#define CTOR_BYTES 4 +/* Initialize the first 4 bytes of the object. */ +void some_ctor(void *obj) +{ + memset(obj, 'A', CTOR_BYTES); +} + +static int __init do_kmem_cache_size(size_t size, bool want_ctor, + int *total_failures) +{ + struct kmem_cache *c; + void *buf; + int iter, bytes = 0; + int fail = 0; + + c = kmem_cache_create("test_cache", size, 1, 0, + want_ctor ? some_ctor : NULL); + for (iter = 0; iter < 10; iter++) { + buf = kmem_cache_alloc(c, GFP_KERNEL); + if (!want_ctor || iter == 0) + bytes = count_nonzero_bytes(buf, size); + if (want_ctor) { + /* + * Newly initialized memory must be initialized using + * the constructor. + */ + if (iter == 0 && bytes < CTOR_BYTES) + fail = 1; + } else { + if (bytes) + fail = 1; + } + fill_with_garbage(buf, size); + kmem_cache_free(c, buf); + } + kmem_cache_destroy(c); + + *total_failures += fail; + return 1; +} + +static int __init test_kmemcache(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 10; i++) { + size = 4 << i; + num_tests += do_kmem_cache_size(size, false, &failures); + num_tests += do_kmem_cache_size(size, true, &failures); + } + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +static int __init test_meminit_init(void) +{ + int failures = 0, num_tests = 0; + + num_tests += test_pages(&failures); + num_tests += test_kvmalloc(&failures); + num_tests += test_kmemcache(&failures); + + if (failures == 0) + pr_info("all %d tests passed!\n", num_tests); + else + pr_info("failures: %d out of %d\n", failures, num_tests); + + return failures ? -EINVAL : 0; +} +module_init(test_meminit_init); From patchwork Tue May 14 14:35:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10943095 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF6EC933 for ; Tue, 14 May 2019 14:36:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE179286DE for ; Tue, 14 May 2019 14:36:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C2182286E4; Tue, 14 May 2019 14:36:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9F52286DE for ; Tue, 14 May 2019 14:36:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C72E6B0007; Tue, 14 May 2019 10:36:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 376DC6B0008; Tue, 14 May 2019 10:36:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28D326B000A; Tue, 14 May 2019 10:36:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by kanga.kvack.org (Postfix) with ESMTP id 006A46B0007 for ; Tue, 14 May 2019 10:36:09 -0400 (EDT) Received: by mail-oi1-f197.google.com with SMTP id c64so6134186oia.22 for ; Tue, 14 May 2019 07:36:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=lbdUPdHa+bdM+dxU1m+K2QEpeUH2T4hzEiJEzMD7K4o=; b=RzRxsobignDBenEYLYPzI2BQlwTR3fehbIOUQUXZJDakQQ+ju0412bu4DOQHH2wR+2 0B998cF/7CTELUiXAVKJo35zqpWxV3JeXXBAAIiD+z8UJqhuOUuEPuxvLp8KvMGyKQuX hPjWuCWy9bTHrH7eBIba0Uleu9VCFrRQrokq839ujh+JEk2XAZO8AbB9Jy6xXeJhs1hE XCIM97KIP370dvz8dImsUAf0oGA3nedeKv6aw0qaTSelzw4Ej4aKa9qQ70c6a0j2G+ZQ vHIoXDz65GBTHo844tBweDW+6MHxEsUG3mcZymV8O+CjAdm++5wVuZz1Dk4vQjWBwBfr xH+g== X-Gm-Message-State: APjAAAXdxOqz8jAnTLDHaX6zzLw8/IAN716c1ICNhusJokW+QFyQownS FOf0Wn4ivjDwgzZBwUnVTgDiOqefPFqsLEMeUCRFAStDZ6cnS7sDEwh2VWKFJJRlNewLkfzZ/4h WS5Y1too2hqxeEO/6/Js6AtwJ1Gc6svNhF4/+FVyzq7gQKSFLpzh5Jv1U+ewpZqJSLA== X-Received: by 2002:a9d:4047:: with SMTP id o7mr8645530oti.231.1557844568469; Tue, 14 May 2019 07:36:08 -0700 (PDT) X-Received: by 2002:a9d:4047:: with SMTP id o7mr8645474oti.231.1557844567635; Tue, 14 May 2019 07:36:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557844567; cv=none; d=google.com; s=arc-20160816; b=NxCjHxWTLXMAqan5J7lAU+9+o5P1c/jWFSURNdHbH7vYzUu4qTRZFODXw+urzKi0fA g6Qfj46e7XL6lMzYfn0VgfDsdJ5whAi5JwEY3wQTSeIX44S1WyJeiTizQhNYsY608AKU Lc5ea6z6YKiKwKZC/VYFDGpvftCSgCY7TOij5nN3LL241Rl8SJG/VSujK6bbzAsNHnTI JCGICLAx+YaJ8HZArGRGNftIfYP/h+dFrrDPEesI8zKNAnT0eyYhB6KZEg+AUgSN2Ulm 0lFnzpJAEy5wJ+NXD5bkNBBJJRJeeEZRBb4NJ9YNaSRBTHwK/0tuniWmPaA+uJeAzKVJ Y4qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=lbdUPdHa+bdM+dxU1m+K2QEpeUH2T4hzEiJEzMD7K4o=; b=ipZvRmsyacmOEuCO/86MlEYzC47XlxwDC9qG4poctRcoDgBsczbvZzvLjarYhofI5i h+PDUrbMJ42ZB/gEMHQQNaL/t0R3QevLLxpoTyDEXVRpV6mADykSp5GbyjVnGDcjnRoz nnA+1/INYuEZzv2bJ3qK24oibSrzTnn2ooorT0AQi5LOs68s0P2et3eCocZvwX8DGzSi kyjBfzP7nGfpp54/XlqPKVsknSAJzshPYMiuAe4YHU9Ma8n/6ewwWplqGw0IdvVDFWOX MWCAvWlruuXzcPDS9kV6HtNitvkDA4DN/GPQ5YcVQ4xszVi5MoZqB7rp1t8oMdSE6RGK 4sRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=HnfouE+v; spf=pass (google.com: domain of 3v9laxaykcfq274z0d2aa270.ya8749gj-886hwy6.ad2@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3V9LaXAYKCFQ274z0D2AA270.yA8749GJ-886Hwy6.AD2@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id p186sor2012276oig.128.2019.05.14.07.36.07 for (Google Transport Security); Tue, 14 May 2019 07:36:07 -0700 (PDT) Received-SPF: pass (google.com: domain of 3v9laxaykcfq274z0d2aa270.ya8749gj-886hwy6.ad2@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=HnfouE+v; spf=pass (google.com: domain of 3v9laxaykcfq274z0d2aa270.ya8749gj-886hwy6.ad2@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3V9LaXAYKCFQ274z0D2AA270.yA8749GJ-886Hwy6.AD2@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lbdUPdHa+bdM+dxU1m+K2QEpeUH2T4hzEiJEzMD7K4o=; b=HnfouE+vVHCSk/BQWNA2IoM8dz0DlJb1ySFeCJDuDts8JvsGv0c3ABDhxeZs5R1jQ5 fL3ARiq3DAbXqQTqJfJuZH+qPM0Rlxs+eQXmzqAwtxAnJNYEaSrvHiNcxGnelj3HmfgX SSeCx+dAjpm4EnYIiFU/I55GSIaka0an9xHoe2esmSu0PB5SH7by0fAzZu2mm0Mixvs1 WAgEPqulx2gTZH/2wtpBrBL+/yGwJvgPaaOhPEwkgOBXuSvFAk/GEC3w+P2hNj9dN06E 3f5VELZcrJ00AMfyT4YjpAwlQDk+F+t0Pa9AiePOGat0x6TjdKPGnk4TKNYrnaALiQhA jI1g== X-Google-Smtp-Source: APXvYqzwuuntM7HozbX9bxO0OlitRtQOtjK1eJ98Ru/9/M+wLdZyTT23D/aLqfOSqP4W7vywT6UYFvkqKIU= X-Received: by 2002:aca:5f84:: with SMTP id t126mr3137888oib.18.1557844567217; Tue, 14 May 2019 07:36:07 -0700 (PDT) Date: Tue, 14 May 2019 16:35:36 +0200 In-Reply-To: <20190514143537.10435-1-glider@google.com> Message-Id: <20190514143537.10435-4-glider@google.com> Mime-Version: 1.0 References: <20190514143537.10435-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org Cc: kernel-hardening@lists.openwall.com, Masahiro Yamada , James Morris , "Serge E. Hallyn" , Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Randy Dunlap , Jann Horn , Mark Rutland , Souptick Joarder , Matthew Wilcox , linux-mm@kvack.org, linux-security-module@vger.kernel.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away. __GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc. __GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution. This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers. Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +6.84% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.25% sys time (st.err 0.72%) Linux build with -j12, init_on_free=1: +8.52% wall time (st.err 0.42%) Linux build with -j12, init_on_free=1: +24.31% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.16% wall time (st.err 0.40%) Linux build with -j12, init_on_alloc=1: +1.24% sys time (st.err 0.39%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. Signed-off-by: Alexander Potapenko To: Andrew Morton To: Kees Cook To: Christoph Lameter Cc: Masahiro Yamada Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: Souptick Joarder Cc: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- v2: - renamed __GFP_NOINIT to __GFP_NO_AUTOINIT, updated patch name/description --- include/linux/gfp.h | 13 +++++++++---- include/linux/mm.h | 2 +- kernel/kexec_core.c | 3 ++- mm/slab.c | 2 +- mm/slob.c | 3 ++- mm/slub.c | 1 + 6 files changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index fdab7de7490d..e1a83bd0ca67 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -44,6 +44,7 @@ struct vm_area_struct; #else #define ___GFP_NOLOCKDEP 0 #endif +#define ___GFP_NO_AUTOINIT 0x1000000u /* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* @@ -208,16 +209,20 @@ struct vm_area_struct; * %__GFP_COMP address compound page metadata. * * %__GFP_ZERO returns a zeroed page on success. + * + * %__GFP_NO_AUTOINIT requests non-initialized memory from the underlying + * allocator. */ -#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) -#define __GFP_COMP ((__force gfp_t)___GFP_COMP) -#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) +#define __GFP_COMP ((__force gfp_t)___GFP_COMP) +#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +#define __GFP_NO_AUTOINIT ((__force gfp_t)___GFP_NO_AUTOINIT) /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (25) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/include/linux/mm.h b/include/linux/mm.h index 18d96f1d07c5..ce6c63396002 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2618,7 +2618,7 @@ DECLARE_STATIC_KEY_FALSE(init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { if (static_branch_unlikely(&init_on_alloc)) - return true; + return !(flags & __GFP_NO_AUTOINIT); return flags & __GFP_ZERO; } diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index 2f75dd0d0d81..7fc37bacac79 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -302,7 +302,8 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) { struct page *pages; - pages = alloc_pages(gfp_mask & ~__GFP_ZERO, order); + pages = alloc_pages((gfp_mask & ~__GFP_ZERO) | __GFP_NO_AUTOINIT, + order); if (pages) { unsigned int count, i; diff --git a/mm/slab.c b/mm/slab.c index d00e9de26a45..1089461fc22b 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, struct page *page; int nr_pages; - flags |= cachep->allocflags; + flags |= (cachep->allocflags | __GFP_NO_AUTOINIT); page = __alloc_pages_node(nodeid, flags, cachep->gfporder); if (!page) { diff --git a/mm/slob.c b/mm/slob.c index 351d3dfee000..d505f36aa398 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node) { void *page; + gfp |= __GFP_NO_AUTOINIT; #ifdef CONFIG_NUMA if (node != NUMA_NO_NODE) page = __alloc_pages_node(node, gfp, order); @@ -221,7 +222,7 @@ static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) { if (static_branch_unlikely(&init_on_alloc) || static_branch_unlikely(&init_on_free)) - return c ? (!c->ctor) : true; + return c ? (!c->ctor) : !(flags & __GFP_NO_AUTOINIT); return flags & __GFP_ZERO; } diff --git a/mm/slub.c b/mm/slub.c index 01424e910800..0aa306f5769a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1495,6 +1495,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, struct page *page; unsigned int order = oo_order(oo); + flags |= __GFP_NO_AUTOINIT; if (node == NUMA_NO_NODE) page = alloc_pages(flags, order); else From patchwork Tue May 14 14:35:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10943099 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC1F5933 for ; Tue, 14 May 2019 14:36:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAEB0286DE for ; Tue, 14 May 2019 14:36:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF686286E4; Tue, 14 May 2019 14:36:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 020F5286DE for ; Tue, 14 May 2019 14:36:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28F216B0008; Tue, 14 May 2019 10:36:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2408D6B000A; Tue, 14 May 2019 10:36:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 132606B000C; Tue, 14 May 2019 10:36:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by kanga.kvack.org (Postfix) with ESMTP id D983E6B0008 for ; Tue, 14 May 2019 10:36:11 -0400 (EDT) Received: by mail-vs1-f71.google.com with SMTP id x10so524902vsj.19 for ; Tue, 14 May 2019 07:36:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=Go4t+yeK4FmEV47RVV/K/R0RvYRio180dlBaNN/E/uc=; b=o8rmyJsQY92QWBDaoNTqtmHh5qfk+RmWAZ4696EQy2zNpMq3ulLqnztEhZBBTVfLh3 cOlnLfRPNm54ulSpQojQ5REdC7bzWaq4ljuPG7LB/bVW8hwiKB3z9JNGcadu54y3TCBB VBNsFd7gtDdNHvu+I7zC+owJjlQMF+cfWfbpVb4CWcrv1N5VlXss9SJchAy3Fa7JNkQe 3wbO6WhR/wP9Vlwtx03eBbYI+ND08++c/wzZ7NyfUOiO5+kiGdUaIou7XeHTAMwTOK0K N2vQ3jEWgdrE4oJY7qBUGnKeoG/hKoU2sUs/HhmSCl/lpm8DoBdT6fxc1eY78ULxRZcd EZVQ== X-Gm-Message-State: APjAAAXYW9KfI83sHaBKQDmFvs6Daio/krnB1TlZnA9/pQJ+H/AQANtb Cra5dYd5IZ51u3EIU0tFAn1ombN1Zk6scwF4w8D4sXFpsC46sSr0B5glx363aR5Fl/ZJ3RSy6s6 ShffB/VHeG/ZwVYTTBssu0BspYjeDLgyvy5krOfy1mET+Y0CM1kcfE9jpW51p11Y0JA== X-Received: by 2002:a1f:8991:: with SMTP id l139mr15593253vkd.49.1557844571455; Tue, 14 May 2019 07:36:11 -0700 (PDT) X-Received: by 2002:a1f:8991:: with SMTP id l139mr15593196vkd.49.1557844570611; Tue, 14 May 2019 07:36:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557844570; cv=none; d=google.com; s=arc-20160816; b=g2BzOCE73d0MLxBwMFzlmyUcG3WH4j/uw1zc+gwLv0PNjrfn+pQ6p7fj6+u/+/uZoE JRL+fWMZUXT8Umo6H3MLQGXEbX241A75gQ5ZS2o8iB2Q6GqFn2xGdriHKG+FjqsD1Ypo z2a0+Fxl8EfK016uwTQTf8PcVVDttXuBQ8c5UxFEaLvHj+dVlq8+SK8W0foCbsqFsdFr 3Ctc0j+ge5F60kMmLvpli+l+Oc8mJk8HtVw8LtkuvGjcMg+LdvDMZyh8p9Vb+1vlQRFM DRC9qeE3UcULs8EE7LzJ1lOGvmoAIc8KIbOJR6JXCt0Hrg5bOmi/tCPqPKvHBk38moqy FSdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=Go4t+yeK4FmEV47RVV/K/R0RvYRio180dlBaNN/E/uc=; b=bWw1BuAsY0lUKD+cL+WbjSjIJgJpMaRr+0Mv7wYn58RuVRLRVDuAq2fJqOk+ZxzODX 23ZWMBT7A9sZ6XBVhjfYYCqEAtkxyE2f2OY3bcIpjB9z4hauWssylrxj4O4uYasa2U+k 2UDn2n6vqWHguKIjVklSxv1TsuLT57mgE6Rur/V+YJ0M2UD0GASyu19tzJVts+UcR3nL fiyIqg+txO3vFExnMIHyrlUVdxlhAxmV34vtWAj474mj/oDMIoqyeckKDr0EgymyPO/E MAq136PKDuVPh9MJC7EIuHZKTVKsNiFDwqGDrM/1KeZFMwAcaTjAFPKmUKwRvpn5wKD1 jp2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=pikeD1jl; spf=pass (google.com: domain of 3wtlaxaykcfc5a723g5dd5a3.1dba7cjm-bb9kz19.dg5@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3WtLaXAYKCFc5A723G5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id x4sor8164313vsk.29.2019.05.14.07.36.10 for (Google Transport Security); Tue, 14 May 2019 07:36:10 -0700 (PDT) Received-SPF: pass (google.com: domain of 3wtlaxaykcfc5a723g5dd5a3.1dba7cjm-bb9kz19.dg5@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=pikeD1jl; spf=pass (google.com: domain of 3wtlaxaykcfc5a723g5dd5a3.1dba7cjm-bb9kz19.dg5@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3WtLaXAYKCFc5A723G5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Go4t+yeK4FmEV47RVV/K/R0RvYRio180dlBaNN/E/uc=; b=pikeD1jlvnTEk+uey8TbYun2V8C/qn4NetuazfEjFSiIKXfvKb/Y0U2HoOWYrcxEIV RE5qYPWIkYK3IPMiDScQeSxK/srM5ptI7dU2jw4oxcInPVP6DLkKA9s1SNWvB2og986s NIFLfXn07Iz5SdRS75NF4d20AWJ5nnn1P7j3lT5arszZfOIm7sh/AGz7F0MmvLClYAZn cIlQ/W4XGYhtJCeuCcbxJGVNh5q+guM3Rv3w1yAkzVnieYfruF0+p9N4wanKkn7/p/SW BszdBhtZ3MtpABxG0+mAR5ZuJsszQrRt2f6CbWb34BCMWiBptbrnvbZ6TunbLM8L5NWs jCsA== X-Google-Smtp-Source: APXvYqzAvmFIr0uQeT7SswT+cvcHP8YHOoi7bCs4Uurq0vvk2jm2KoyvFDp206pk1BoSOeKx0QNC4J5Rkl8= X-Received: by 2002:a67:ed11:: with SMTP id l17mr17817001vsp.154.1557844570196; Tue, 14 May 2019 07:36:10 -0700 (PDT) Date: Tue, 14 May 2019 16:35:37 +0200 In-Reply-To: <20190514143537.10435-1-glider@google.com> Message-Id: <20190514143537.10435-5-glider@google.com> Mime-Version: 1.0 References: <20190514143537.10435-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org Cc: kernel-hardening@lists.openwall.com, Masahiro Yamada , James Morris , "Serge E. Hallyn" , Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Randy Dunlap , Jann Horn , Mark Rutland , linux-mm@kvack.org, linux-security-module@vger.kernel.org X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%. Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%) Linux build with -j12, init_on_free=1: +8.34% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.13% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.04% wall time (st.err 0.46%) Linux build with -j12, init_on_alloc=1: +0.50% sys time (st.err 0.45%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. Signed-off-by: Alexander Potapenko To: Andrew Morton To: Christoph Lameter To: Kees Cook Cc: Masahiro Yamada Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- v2: - changed __GFP_NOINIT to __GFP_NO_AUTOINIT --- include/net/sock.h | 5 +++++ net/core/sock.c | 29 +++++++++++++++++++++++++---- net/unix/af_unix.c | 13 +++++++------ 3 files changed, 37 insertions(+), 10 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 4d208c0f9c14..0dcb90a0c14d 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1626,6 +1626,11 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, unsigned long data_len, int noblock, int *errcode, int max_page_order); +struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk, + unsigned long header_len, + unsigned long data_len, + int noblock, int *errcode, + int max_page_order); void *sock_kmalloc(struct sock *sk, int size, gfp_t priority); void sock_kfree_s(struct sock *sk, void *mem, int size); void sock_kzfree_s(struct sock *sk, void *mem, int size); diff --git a/net/core/sock.c b/net/core/sock.c index 9ceb90c875bc..7c24b70b7069 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2192,9 +2192,11 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo) * Generic send/receive buffer handlers */ -struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, - unsigned long data_len, int noblock, - int *errcode, int max_page_order) +struct sk_buff *sock_alloc_send_pskb_internal(struct sock *sk, + unsigned long header_len, + unsigned long data_len, + int noblock, int *errcode, + int max_page_order, gfp_t gfp) { struct sk_buff *skb; long timeo; @@ -2223,7 +2225,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, timeo = sock_wait_for_wmem(sk, timeo); } skb = alloc_skb_with_frags(header_len, data_len, max_page_order, - errcode, sk->sk_allocation); + errcode, sk->sk_allocation | gfp); if (skb) skb_set_owner_w(skb, sk); return skb; @@ -2234,8 +2236,27 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, *errcode = err; return NULL; } + +struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, + unsigned long data_len, int noblock, + int *errcode, int max_page_order) +{ + return sock_alloc_send_pskb_internal(sk, header_len, data_len, + noblock, errcode, max_page_order, /*gfp*/0); +} EXPORT_SYMBOL(sock_alloc_send_pskb); +struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk, + unsigned long header_len, + unsigned long data_len, + int noblock, int *errcode, + int max_page_order) +{ + return sock_alloc_send_pskb_internal(sk, header_len, data_len, + noblock, errcode, max_page_order, /*gfp*/__GFP_NO_AUTOINIT); +} +EXPORT_SYMBOL(sock_alloc_send_pskb_noinit); + struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, int noblock, int *errcode) { diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index e68d7454f2e3..a4c15620b66d 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1627,9 +1627,9 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg, BUILD_BUG_ON(SKB_MAX_ALLOC < PAGE_SIZE); } - skb = sock_alloc_send_pskb(sk, len - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - PAGE_ALLOC_COSTLY_ORDER); + skb = sock_alloc_send_pskb_noinit(sk, len - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + PAGE_ALLOC_COSTLY_ORDER); if (skb == NULL) goto out; @@ -1824,9 +1824,10 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb_noinit(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, + &err, + get_order(UNIX_SKB_FRAGS_SZ)); if (!skb) goto out_err;