From patchwork Fri Mar 29 14:56:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: York Jasper Niebuhr X-Patchwork-Id: 13610687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACAEACD11DD for ; Fri, 29 Mar 2024 14:56:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 024586B007B; Fri, 29 Mar 2024 10:56:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3D7E6B0082; Fri, 29 Mar 2024 10:56:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E052E6B0083; Fri, 29 Mar 2024 10:56:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C33A06B007B for ; Fri, 29 Mar 2024 10:56:46 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 744B01203DC for ; Fri, 29 Mar 2024 14:56:46 +0000 (UTC) X-FDA: 81950378412.12.E86093D Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) by imf11.hostedemail.com (Postfix) with ESMTP id 70E044001A for ; Fri, 29 Mar 2024 14:56:44 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jZ2nwFyQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of yjnworkstation@gmail.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=yjnworkstation@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711724204; a=rsa-sha256; cv=none; b=Bp45uDP5Wqd80Kub+wFpHaYqhhpvb74EXjM1TCkA2jawIPYNjdbS34WAnjV8vO+DsyHZiL JULDsr9zk4pkYxrGi6ZfWrL7PQDLBfFQpWF4A5vsyvuX9P9TYwFu1zQ9Iye+o6D9zSaahv YpcGFpUTEjjvgBvxXuQPkKUVtFlPu58= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jZ2nwFyQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of yjnworkstation@gmail.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=yjnworkstation@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711724204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=KsUlkYzw442RI4qHF46wvm4R0rzxgWigVON0ueOGsCA=; b=v6OR+dlF2lzObAX90z+pqyoupnDDNX+puabQ92vFgxY9WnD8boE9/lXvwK6Hjewj7fqnKL QPhCXRlau92UOsmMMrGO9VpSfgzXbjpFdLCQZfx0T7TFzwSgYaDmIHZi9gECrVyBiqPrYz 7ZpyuYakkmP2N4e5W2P6NhTJcKv2r4I= Received: by mail-ed1-f46.google.com with SMTP id 4fb4d7f45d1cf-56c08d1e900so2371059a12.3 for ; Fri, 29 Mar 2024 07:56:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711724202; x=1712329002; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=KsUlkYzw442RI4qHF46wvm4R0rzxgWigVON0ueOGsCA=; b=jZ2nwFyQ0D0KMeojZ9u9we0RZcOi1ColAYD55tx+b0buGdq3Gv1LBH/GCReB1BDz6s ek9D1PD2x/zXpitqRB9tFkiQcb2kxrgOxPTsrtdSMYVZ3nkq48kvYa+/ftxKWeFE267J nnjN5pqopKnG8zvPIPOZ4x+kuWe1m9UXclppGm4jjqebe0A0MgyRROlPEyHXvCvhvogX 0LLtmdPUocz1EeGzq8OWnAGo+aoanoPc92cUhtUTYincmqGZXNoMk4mCKog4HrgffmnN n2oOy8hHtmvdcmWkzeTPdCz3Zu5FxnzDKl+sOO9vwZKwXIl3yGDNMhBVMhJG2GQR3Ncu 1r5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711724202; x=1712329002; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KsUlkYzw442RI4qHF46wvm4R0rzxgWigVON0ueOGsCA=; b=lpqp/KuFFaHtvdTuQngXiIvYc7MMUbhWtDkpXxTP97iizu1RpCra855YrDSORylVT0 Da4tautAJYUwJddF0Z37tygofcKqqaLleQAqhigKPFux1D8d2BCEJW4SP3uc3vTzndnu CGIPZFfK2XCDtwysytoCe8LPw/qKQ17f7mFJ68B0y6drUYc4f/U48bOiQmyNvLJHkfIb bs4ymcJ0SP7r7c3cvIpcOan2t12GZtxG9flctjNVgf9Sl+/fQgFaa29rkpFe0D5fIpvC 3hsFHkmkTd1HcSan3Fc52KypM6JptSyBi0W9PqNZ644LPLU3wA1ZmGdtUBVlZolp5Til lEGA== X-Forwarded-Encrypted: i=1; AJvYcCWMbHWz4foO5C3o9uIVgfDjcZM/Po5d8NyHoHKpq3AV0T3snxmbtIFecaxiMGqy96+mkUAP3T9k28kq40TteDIVdbg= X-Gm-Message-State: AOJu0YzvAZTLN3KTeZ9Q7KXPrMtvf/Y4L/SLWkx/x0ESgONWhQjXJn35 HOuPateT/KYDp2R6L576meBdMQe2ZYL+bdHe1L/XGB8udS1YWnvU X-Google-Smtp-Source: AGHT+IH5qjcVDhzFMNsiXL9Tn/9WkDZgTXqvMy4P+gEbuvCuEFLXgADIgfkPH1RvOpCout3gkWLM1Q== X-Received: by 2002:a17:907:868a:b0:a46:c11d:dd01 with SMTP id qa10-20020a170907868a00b00a46c11ddd01mr2124294ejc.50.1711724202314; Fri, 29 Mar 2024 07:56:42 -0700 (PDT) Received: from yjn-Lenovo-V14-G2-ITL.fritz.box (p200300dcbf196600f6ba6208ca0ed095.dip0.t-ipconnect.de. [2003:dc:bf19:6600:f6ba:6208:ca0e:d095]) by smtp.gmail.com with ESMTPSA id q16-20020a1709060e5000b00a47531764fdsm2011866eji.65.2024.03.29.07.56.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Mar 2024 07:56:41 -0700 (PDT) From: York Jasper Niebuhr To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, willy@infradead.org, linux-mm@kvack.org, York Jasper Niebuhr Subject: [PATCH] mm: init_mlocked_on_free_v3 Date: Fri, 29 Mar 2024 15:56:05 +0100 Message-Id: <20240329145605.149917-1-yjnworkstation@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 70E044001A X-Stat-Signature: tcp4uhwn4sd8heiqix9e4ztetk7gs61c X-HE-Tag: 1711724204-312288 X-HE-Meta: U2FsdGVkX1/S14Jj0gK4mJnVZyAQPqzFyKu21xzf+sfWtoSqJgGKci0sIlapSB3hQk1Y3rzGLjQokG5Xj7QIzMlB+NwMtnm09NcjOzNJ1bSyOZYHerGcjsdqiyRITZYmai4GA/uX94sAKUnXdktTxZmKQqgKAPEneXH2FG0YFtMFrp3bfPld2i+BcmRpHg1g81KDoqrpje88/wfktsVeIVsdCDsWsJa6o6q1jyd11r7MmEgJ+bL+nGSD8A1BRuBH43LdN+CBrx5E+B4Xc5C2dP0JiB68u0EyXmVPGU2YtlC4otTMyw/IZOCqhP/XYD7b0JDaWQFMcJ4Q9ulFGSGRvPP+K2OPFSA3Ybzrg1TI8EGfaFv4A5ZL5npCGuZGtNRBK8TQZuOzbym28gOzaiCJVFlT30wydEBQapnNMcaZYhAqH2FgKbfjrUiol1/8BxS2fO1ZjM2YgjZkeUeLQAx89eQqEUTq3V4ePV/DTScOY3M2TLF8c5jhV9SmhcuwzBYXuDICM7WXhE9mZz/4ctfHdqPYBLHdM0UbHWutdNroCaoQ4wEchzGHIelvxFntRNgHiNuOrBRCcmACqZm6G7EhTBE/h3Xhi5U7PRSX2esWGt/nJVwhuKME3jrxd3QbrvzJ9C0JXt1B2Hx/JQaz04AVHCSE9V6c1bxTQRXCrn/MlTOBJTevOfJ0pzo19LZY/hNqyFuqhPB4btf6y04kMJdrQ2O/h3CXAPSsgbtnPZnMX343U0Ja/2pmJ/OaSoYeBuDRhHtLkyAkfN4SAnbC86PXi2/+MgEcHnAF+ohIX9x/QAqdc/MWeYGSLvvz7wR4L0oqZ2hjAfEuUbeLQ4TzV/iZLwaRtJa8M6KoJRBbu2YT1tj5UusapRFEZu4ItdsLgt8eupUrFVfAVEBVE7gQQs6Mu2+6MNebT8UEgRgghkU2wzMQa5/ueBMHQToR0dxtEwU5J7/I19mwsIJxwd23cjk V5iNCqSP 9p07Wr3jrEuNgd3yvcmEvCIJCyTs44Gm5Jj/OX5Z56q8FJmD8Q722F/2/hNpqbDBp9Gl7mGLa4bblTF9xBckd+ndyNWIHKaLtvIVCAn7RlYZvDabsRv7HrDciesebn8P2b+A6DtOEE+hfqRl0FHC79rj9cpmFjdwpFXXev+Pc+PPYaTdwZV0irtkqm+pz5gY9aMnAhd2KihMSAN3TKPzutNUb8DbeGBl4vfZvMd83UjphsoAJu31SyDiCdjYLwxE1UkbxGEc22L5LDR0f2gw3QEPaHiWJOKF+pMnxs/94ho5jES/Seh6vIPNiaRucJisiccFVYGLrimZcqmQrQ1+SkcPaCBlxptmx+toLezmqD0Yaed8Y4/3kVkxKXcO4V7qRsaM2ktAjq1XoViKGTmiG/viLbn11GRxkqotGyHrgsrmjpgBsOhmleOZUSYS9cd89OSrHAQ7y9V5yfOKK7PfL1Fi1cXa22gyrvoM5kHPqEblgk9ikFTX4FslU6w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implements the "init_mlocked_on_free" boot option. When this boot option is enabled, any mlock'ed pages are zeroed on free. If the pages are munlock'ed beforehand, no initialization takes place. This boot option is meant to combat the performance hit of "init_on_free" as reported in commit 6471384af2a6 ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options"). With "init_mlocked_on_free=1" only relevant data is freed while everything else is left untouched by the kernel. Correspondingly, this patch introduces no performance hit for unmapping non-mlock'ed memory. The unmapping overhead for purely mlocked memory was measured to be approximately 13%. Realistically, most systems mlock only a fraction of the total memory so the real-world system overhead should be close to zero. Optimally, userspace programs clear any key material or other confidential memory before exit and munlock the according memory regions. If a program crashes, userspace key managers fail to do this job. Accordingly, no munlock operations are performed so the data is caught and zeroed by the kernel. Should the program not crash, all memory will ideally be munlocked so no overhead is caused. CONFIG_INIT_MLOCKED_ON_FREE_DEFAULT_ON can be set to enable "init_mlocked_on_free" by default. Signed-off-by: York Jasper Niebuhr --- .../admin-guide/kernel-parameters.txt | 6 +++ include/linux/mm.h | 9 +++- mm/internal.h | 1 + mm/memory.c | 6 +++ mm/mm_init.c | 43 ++++++++++++++++--- mm/page_alloc.c | 2 +- security/Kconfig.hardening | 15 +++++++ 7 files changed, 73 insertions(+), 9 deletions(-) base-commit: a6bd6c9333397f5a0e2667d4d82fef8c970108f2 diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index bb884c14b2f6..34bdbf29aaf9 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2148,6 +2148,12 @@ Format: 0 | 1 Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_mlocked_on_free= [MM] Fill freed userspace memory with zeroes if + it was mlock'ed and not explicitly munlock'ed + afterwards. + Format: 0 | 1 + Default set by CONFIG_INIT_MLOCKED_ON_FREE_DEFAULT_ON + init_pkru= [X86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can diff --git a/include/linux/mm.h b/include/linux/mm.h index 8eccaabe44ee..a216489aef2b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3751,7 +3751,14 @@ DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { return static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, - &init_on_free); + &init_on_free); +} + +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_MLOCKED_ON_FREE_DEFAULT_ON, init_mlocked_on_free); +static inline bool want_init_mlocked_on_free(void) +{ + return static_branch_maybe(CONFIG_INIT_MLOCKED_ON_FREE_DEFAULT_ON, + &init_mlocked_on_free); } extern bool _debug_pagealloc_enabled_early; diff --git a/mm/internal.h b/mm/internal.h index 8e11f7b2da21..8ec85b5e62e6 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -506,6 +506,7 @@ extern void __putback_isolated_page(struct page *page, unsigned int order, extern void memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order); extern void __free_pages_core(struct page *page, unsigned int order); +extern void kernel_init_pages(struct page *page, int numpages); /* * This will have no effect, other than possibly generating a warning, if the diff --git a/mm/memory.c b/mm/memory.c index 36191a9c799c..706361b9d1af 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1506,6 +1506,12 @@ static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } + + if (want_init_mlocked_on_free() && folio_test_mlocked(folio) && + !delay_rmap && folio_test_anon(folio)) { + kernel_init_pages(page, folio_nr_pages(folio)); + } + if (unlikely(__tlb_remove_folio_pages(tlb, page, nr, delay_rmap))) { *force_flush = true; *force_break = true; diff --git a/mm/mm_init.c b/mm/mm_init.c index e3450f12d79a..bede28e8d7cc 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2515,6 +2515,9 @@ EXPORT_SYMBOL(init_on_alloc); DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_MLOCKED_ON_FREE_DEFAULT_ON, init_mlocked_on_free); +EXPORT_SYMBOL(init_mlocked_on_free); + static bool _init_on_alloc_enabled_early __read_mostly = IS_ENABLED(CONFIG_INIT_ON_ALLOC_DEFAULT_ON); static int __init early_init_on_alloc(char *buf) @@ -2532,6 +2535,14 @@ static int __init early_init_on_free(char *buf) } early_param("init_on_free", early_init_on_free); +static bool _init_mlocked_on_free_enabled_early __read_mostly + = IS_ENABLED(CONFIG_INIT_MLOCKED_ON_FREE_DEFAULT_ON); +static int __init early_init_mlocked_on_free(char *buf) +{ + return kstrtobool(buf, &_init_mlocked_on_free_enabled_early); +} +early_param("init_mlocked_on_free", early_init_mlocked_on_free); + DEFINE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); /* @@ -2559,12 +2570,21 @@ static void __init mem_debugging_and_hardening_init(void) } #endif - if ((_init_on_alloc_enabled_early || _init_on_free_enabled_early) && + if ((_init_on_alloc_enabled_early || _init_on_free_enabled_early || + _init_mlocked_on_free_enabled_early) && page_poisoning_requested) { pr_info("mem auto-init: CONFIG_PAGE_POISONING is on, " - "will take precedence over init_on_alloc and init_on_free\n"); + "will take precedence over init_on_alloc, init_on_free " + "and init_mlocked_on_free\n"); _init_on_alloc_enabled_early = false; _init_on_free_enabled_early = false; + _init_mlocked_on_free_enabled_early = false; + } + + if (_init_mlocked_on_free_enabled_early && _init_on_free_enabled_early) { + pr_info("mem auto-init: init_on_free is on, " + "will take precedence over init_mlocked_on_free\n"); + _init_mlocked_on_free_enabled_early = false; } if (_init_on_alloc_enabled_early) { @@ -2581,9 +2601,17 @@ static void __init mem_debugging_and_hardening_init(void) static_branch_disable(&init_on_free); } - if (IS_ENABLED(CONFIG_KMSAN) && - (_init_on_alloc_enabled_early || _init_on_free_enabled_early)) - pr_info("mem auto-init: please make sure init_on_alloc and init_on_free are disabled when running KMSAN\n"); + if (_init_mlocked_on_free_enabled_early) { + want_check_pages = true; + static_branch_enable(&init_mlocked_on_free); + } else { + static_branch_disable(&init_mlocked_on_free); + } + + if (IS_ENABLED(CONFIG_KMSAN) && (_init_on_alloc_enabled_early || + _init_on_free_enabled_early || _init_mlocked_on_free_enabled_early)) + pr_info("mem auto-init: please make sure init_on_alloc, init_on_free and " + "init_mlocked_on_free are disabled when running KMSAN\n"); #ifdef CONFIG_DEBUG_PAGEALLOC if (debug_pagealloc_enabled()) { @@ -2622,9 +2650,10 @@ static void __init report_meminit(void) else stack = "off"; - pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", + pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s, mlocked free:%s\n", stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", - want_init_on_free() ? "on" : "off"); + want_init_on_free() ? "on" : "off", + want_init_mlocked_on_free() ? "on" : "off"); if (want_init_on_free()) pr_info("mem auto-init: clearing system memory may take some time...\n"); } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5b8ab1bfb9d5..8a8ca555b9d6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1038,7 +1038,7 @@ static inline bool should_skip_kasan_poison(struct page *page) return page_kasan_tag(page) == KASAN_TAG_KERNEL; } -static void kernel_init_pages(struct page *page, int numpages) +void kernel_init_pages(struct page *page, int numpages) { int i; diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index 2cff851ebfd7..effbf5982be1 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -255,6 +255,21 @@ config INIT_ON_FREE_DEFAULT_ON touching "cold" memory areas. Most cases see 3-5% impact. Some synthetic workloads have measured as high as 8%. +config INIT_MLOCKED_ON_FREE_DEFAULT_ON + bool "Enable mlocked memory zeroing on free" + depends on !KMSAN + help + This config has the effect of setting "init_mlocked_on_free=1" + on the kernel command line. If it is enabled, all mlocked process + memory is zeroed when freed. This restriction to mlocked memory + improves performance over "init_on_free" but can still be used to + protect confidential data like key material from content exposures + to other processes, as well as live forensics and cold boot attacks. + Any non-mlocked memory is not cleared before it is reassigned. This + configuration can be overwritten by setting "init_mlocked_on_free=0" + on the command line. The "init_on_free" boot option takes + precedence over "init_mlocked_on_free". + config CC_HAS_ZERO_CALL_USED_REGS def_bool $(cc-option,-fzero-call-used-regs=used-gpr) # https://github.com/ClangBuiltLinux/linux/issues/1766