From patchwork Mon Apr 22 18:57:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 262901708 for ; Mon, 22 Apr 2019 18:58:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1854D28759 for ; Mon, 22 Apr 2019 18:58:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0C8E828764; Mon, 22 Apr 2019 18:58:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DDEAE28759 for ; Mon, 22 Apr 2019 18:58:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 386EF6B0003; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 30CC76B0007; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C2946B000D; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id CB5BF6B0003 for ; Mon, 22 Apr 2019 14:58:43 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id o8so8479682pgq.5 for ; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=2uQh+p4niM1XazCyX0W8SClLNopGSLFeOTKZKD97XRE=; b=n4Iwd7t0W2867DU6O4oY3WWOIQx2wBP93LnxYFdXpPKcfR51Bs6IFW0NcECLlMrRrL EGOH+ot+K7jK/2xlCXD724m+BKNXMehFL/ss8MX13UEBqZV9Ijc8B44v+wX+GzOqxuXw KdqYGLxlp3HCFbnHhkbAaStRobS+gafMD+9KM9cXyoUIy4n4+DAYDSZCX+GDnm4kaQP7 psxitxEJHNoPTNEEExoOegkmu/a0gieGsm69WpgjbTKMzJwk45H0H6J5vGThEiWmkWYC BxOrVjyCWYfS707/7PCb2wElbsxzSkbLBnxEnzioGh7lwLm5aeJ7OsbAt4MQvm9zeDn7 FJqQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXmjlnlvnz7YioPjLHOGjgILa7lpeuy9roqEXQsppn1IIVacS9d M2z/56cgYgIeNXi6wyNAae6ymCOQzqIE6skVd2E4sobVExeicaamsScQN98dkset7lUPxtiteyI U0rUJYTVNeP8JJejvQQQp/zBSXMqhv1L+Dwhc2SFS4iTY0eZVh8uh2xsd3vIk+x06Aw== X-Received: by 2002:a63:6fcf:: with SMTP id k198mr20149697pgc.158.1555959523193; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqym7m/sJHOJ5H106O5w6z5v1U3smcyroZ/93ELbtQXKOaCclXHp6Cs3cHMNJLJXbfUoedeJ X-Received: by 2002:a63:6fcf:: with SMTP id k198mr20149644pgc.158.1555959522203; Mon, 22 Apr 2019 11:58:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959522; cv=none; d=google.com; s=arc-20160816; b=J+sRKXeEyrbbHnMQgLYSko/oGz++d5QGHUIKuQviTHbCCvXrHdhf95eoWYg8N9aklu fKuCKmcw/p0qZy4aKvodc6xZKyIUhM4HoVOjODzQETlJCHIxWO7Vb/hqSCnNlT2ul6bF oPh6A5IDJiql03qzuTb5P0fQya3FldE4qGXUzpQ5Vswrql+F+fpwPK2PtjtsFEO4LwPQ oCdP3XEuLR+iFFFN842QmMDbBCKlhW8P7foYb0EEdcqdTR9pe4X/3QbH/J9KJV3LVkFP tFa8STnnwq7A8vqAEBulUKl+Ec1DWOxZax3JOFJobeoWtjQ0Oa5g0V3eQE7O6uosv17X mmdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=2uQh+p4niM1XazCyX0W8SClLNopGSLFeOTKZKD97XRE=; b=QPJcZt7evCAtr0SaUVicXQcgv6oMbYvDwwQqKf8j4Cexh29GrlrAY2IFIPxoO/pzBm 0owClqPBPQ5r33kPURtwvrLxk/Ptj2z1VV9GitPAtH3VH86powmS+TbMEBehkMvrotOz twmn/lbsmeltC2zmb42Ldx67mgy7M1vy4GVr6MtxnejjA8z/iUSKum9tWzBZG+8EWmM+ 9aaClu3vqDQH9Z9aKZjKGrxtERfLMOzwd1e4yQ1l+kZ5wMyL/Lk967Q6akh0tAUSMn9v lUPoSmAsIDLBiNQgM5JTsPw6atgdI3IcvOE5cd8nDrunJvp/cMW197GcZKWUQY2C62jB 9KlA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:42 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417120" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:40 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH v4 01/23] Fix "x86/alternatives: Lockdep-enforce text_mutex in text_poke*()" Date: Mon, 22 Apr 2019 11:57:43 -0700 Message-Id: <20190422185805.1169-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_mutex is currently expected to be held before text_poke() is called, but kgdb does not take the mutex, and instead *supposedly* ensures the lock is not taken and will not be acquired by any other core while text_poke() is running. The reason for the "supposedly" comment is that it is not entirely clear that this would be the case if gdb_do_roundup is zero. Create two wrapper functions, text_poke() and text_poke_kgdb(), which do or do not run the lockdep assertion respectively. While we are at it, change the return code of text_poke() to something meaningful. One day, callers might actually respect it and the existing BUG_ON() when patching fails could be removed. For kgdb, the return value can actually be used. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Fixes: 9222f606506c ("x86/alternatives: Lockdep-enforce text_mutex in text_poke*()") Suggested-by: Peter Zijlstra Acked-by: Jiri Kosina Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 52 ++++++++++++++++++++-------- arch/x86/kernel/kgdb.c | 11 +++--- 3 files changed, 45 insertions(+), 19 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index e85ff65c43c3..f8fc8e86cf01 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -35,6 +35,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); * inconsistent instruction while you patch. */ extern void *text_poke(void *addr, const void *opcode, size_t len); +extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 9a79c7808f9c..0a814d73547a 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -679,18 +679,7 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } -/** - * text_poke - Update instructions on a live kernel - * @addr: address to modify - * @opcode: source of the copy - * @len: length to copy - * - * Only atomic text poke/set should be allowed when not doing early patching. - * It means the size must be writable atomically and the address must be aligned - * in a way that permits an atomic write. It also makes sure we fit on a single - * page. - */ -void *text_poke(void *addr, const void *opcode, size_t len) +static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; char *vaddr; @@ -703,8 +692,6 @@ void *text_poke(void *addr, const void *opcode, size_t len) */ BUG_ON(!after_bootmem); - lockdep_assert_held(&text_mutex); - if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); pages[1] = vmalloc_to_page(addr + PAGE_SIZE); @@ -733,6 +720,43 @@ void *text_poke(void *addr, const void *opcode, size_t len) return addr; } +/** + * text_poke - Update instructions on a live kernel + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + */ +void *text_poke(void *addr, const void *opcode, size_t len) +{ + lockdep_assert_held(&text_mutex); + + return __text_poke(addr, opcode, len); +} + +/** + * text_poke_kgdb - Update instructions on a live kernel by kgdb + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + * + * Context: should only be used by kgdb, which ensures no other core is running, + * despite the fact it does not hold the text_mutex. + */ +void *text_poke_kgdb(void *addr, const void *opcode, size_t len) +{ + return __text_poke(addr, opcode, len); +} + static void do_sync_core(void *info) { sync_core(); diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 4ff6b4cdb941..2b203ee5b879 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -759,13 +759,13 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) if (!err) return err; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) return -EBUSY; - text_poke((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, - BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err) return err; @@ -784,12 +784,13 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) goto knl_write; - text_poke((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) goto knl_write; From patchwork Mon Apr 22 18:57:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61BD51515 for ; Mon, 22 Apr 2019 18:58:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55CEC28759 for ; Mon, 22 Apr 2019 18:58:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 49B4D28764; Mon, 22 Apr 2019 18:58:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA3CB28759 for ; Mon, 22 Apr 2019 18:58:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 117256B0006; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 002CF6B026B; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5F1A6B0006; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 0C3C36B0006 for ; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id e14so8196988pgg.12 for ; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Y4ax2hOd4Buhijx/oNSuXPhWpslJbmlh2LJYha9eHFk=; b=cOHSf8SDOjOCCn0xUf5xre6sOmrV2u06McI6rxqRJQT4/Kv1jwX4TK+AX2tbEIYgNq AEA2bYA+7TurL2d42jzIdOobsrmsEFDS5p/DGT3TOBpMqZ7kRdrzGn36e3DcbS3qUUWC aDQ/M28n7S5/K//uKmUKC5zAvQ35d+HrdqfBalSa8Dh200bRcYJ8DcNS6T+FIrdH3sRP SrMgsVa1I3n2FY4EoKLfpH3Mf/bpkGxCwB5ydb+Ll3hxmko7ZORPlum3dmSDV7kNfPZQ eg8N/XoNp3xpdnMGd23omCRSh3SGb+ZU9Z01vuO9Pz0Y8XCjbVipZCDQGoABmmB6v07B /oLA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAX9HeyiyZp1hk+wkbo6QMHjgwdWW7rs/MD+PtxRTGD2WAgKFh+d pt/MYhE7ZPXMGmhdQQdAjlrAuh04M//peCIm17dzXsGyS1EY4TdJECa0uqmB9cDHf1sG+yA6OXf +iCGpEJYczR0wqSKFdxJ8AUxuhjtFzyDfEeD569NpYYAsds8dSOuAcMxsl/O5/a7mYw== X-Received: by 2002:a63:fd49:: with SMTP id m9mr20238026pgj.16.1555959523481; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqwhbjsUZ6D5GgJQ14xg07i4WhP6ruHS3XoNeDnaVh/2kiHf4l36uWcfoQG27DtuKsnjr3Jn X-Received: by 2002:a63:fd49:: with SMTP id m9mr20237976pgj.16.1555959522571; Mon, 22 Apr 2019 11:58:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959522; cv=none; d=google.com; s=arc-20160816; b=FxQxErtYL1nhWG3UATS5FB7BKyYyYpoSqTQ+cjfFaxeU1hIyJZsl3UkZopT4j7Q7zG nQlmAwAm6qBq0QywqOwhe+UAuBVzSJ7S+p7zMKjLAQFL2N2gr0Gc+F29/nsNQYBP2Den M3kR4IBbXM04C8Nipz0gh/59QLSbhlCmhrvDxyZG4WKV/kwObwtsP6rNJDJbjhTdj10N e6glP93c+8O7KWN7qjdzTRATJE7DuzFuzjEgiiLFe+VeWp3DBtEhSklo838hclM8DodR xRwl9ZLr0IsJ1dCfws2kCG5unDR5bILNxlhHwevFFRF+SSyGkoJzZD8uXYgvSM4rOegy s9lQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Y4ax2hOd4Buhijx/oNSuXPhWpslJbmlh2LJYha9eHFk=; b=N+QAZKYaQlM86OogavV30KiFsEd5hSfD3AJ8vd2zU0yHHD3gI1AxlYlp1XVzN8XY/Q svYz1N6WJMDB6WmD6jwxHV38RAYa5ZxAhbf9xERfMRw8OMNuC0OlAQ1b6NfKtKpmCa24 Pnw0g51+AFLUQvnTBlUn8WNjxFDzoQHWu45q4qMOYJc3YVauYiMQ7xLCwk3+b9px4ILY NTbXytQ1oTeX5NGZiL4v75ZRMsfAFe3ll+FkLcpeOjmhqlgJ5rO8OAsmSXGwHx3o64j3 Y3cBF5uakPgmYuZ2Xb8Fh5amOc7n4BtAwEzQLgtu9Vi4Hv8lc4WOmvCg+tI/mjbzCUsE Hy8Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:42 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417122" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH v4 02/23] x86/jump_label: Use text_poke_early() during early init Date: Mon, 22 Apr 2019 11:57:44 -0700 Message-Id: <20190422185805.1169-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit There is no apparent reason not to use text_poke_early() during early-init, since no patching of code that might be on the stack is done and only a single core is running. This is required for the next patches that would set a temporary mm for text poking, and this mm is only initialized after some static-keys are enabled/disabled. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index f99bd26bd3f1..e7d8c636b228 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -50,7 +50,12 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - if (early_boot_irqs_disabled) + /* + * As long as only a single processor is running and the code is still + * not marked as RO, text_poke_early() can be used; Checking that + * system_state is SYSTEM_BOOTING guarantees it. + */ + if (system_state == SYSTEM_BOOTING) poker = text_poke_early; if (type == JUMP_LABEL_JMP) { From patchwork Mon Apr 22 18:57:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 273BC1575 for ; Mon, 22 Apr 2019 18:58:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C9A628759 for ; Mon, 22 Apr 2019 18:58:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 10D5028764; Mon, 22 Apr 2019 18:58:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9247728759 for ; Mon, 22 Apr 2019 18:58:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1AEF6B0007; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9C07A6B000D; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 813346B0008; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 104AA6B000A for ; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id u2so8459535pgi.10 for ; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=TDi+ZaxaGFwMJnbi/KBZqpgNpblOvtV7DEanhYCdmjY=; b=K640UsPHOTzMPTlgl4a/1Kl4kuLc9ZamKZ9feGQZWGT/p6gwLFKrfcRlN43SwtIYyu taLqt4zsuCWDbDKjc3Au4Jn6RUf3V3ld/qwBFSe3VHnk4jduSli1xETTgEudx+9Vv9N/ 6oa+Nhwg573CBHBHjyrdR1hADucpAf1R6/NT74XpQvgbH6MtxZdiCMahPtMRVKh/qRoO jUzNRk/TyD5IYjCEYw97m9qWpFjBl2G6oSsyYByQmAEU/QG3uirDHune/Mf2nc9Tg97y zY2Gz+ksc1Bmqygwu6kROTrJ1RcbpkZ8OOwrDxsI+Tb2fe9iEyLzQyKxkvgp0tRghrci FGJQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWK7ntJeANDSMbR9lR2y3ElVD0c4brmsnxSn0OV4FTG/bf6W4tC d46afvWnxaYPFmjmLlpKuk+HCqrq9q8LNvV7YO4rTTEBgB40eIt9CVxa4ViLxu93bDPV9Te3fSA xpgRtfxqatPawqlO4RV63WvqvW7jq7tB94EVau4YduUqclfHmQ9lYlu7xkoDoGgTHMA== X-Received: by 2002:a17:902:28ab:: with SMTP id f40mr4435678plb.297.1555959523544; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqxArDSB4Jn30qhacmD7H5ZmFtydVFqmQG/MM8BhjT2Am0swLevXMahgeTHiZApSAeKmnsI/ X-Received: by 2002:a17:902:28ab:: with SMTP id f40mr4435590plb.297.1555959521973; Mon, 22 Apr 2019 11:58:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959521; cv=none; d=google.com; s=arc-20160816; b=SpRZD1EmgK3HNv/+Tvo8l6+/K10tPuWTj3i4CtYWK/zaTkVDqTr4zIy9U4iFN+Uexd mWecqBuMvEIyBoAsjR5gFy24zkZHN+YJav0wV77iUaYbhvLSEalw6WSkaku74gtIh6T9 djoOAy4ZbmQx9bbe8U4/DaysZu49BZw3GBydsRqAPUpf0zH41M9pS9hmK+L4zsJ6fqh5 6b5UHnS6/a9d+ZsgsS8u8NbLk+ASRgyQAL5D8e5jsmiHyFIPC8dWg7H4tAk1sElun3qh kHslVoLGoQpuu4/VtzKHsmfBQYPu7Q4qybaMYbJ/7vA+yV+XJk0sHZ/u/s+ZOuysiqHV 4e3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=TDi+ZaxaGFwMJnbi/KBZqpgNpblOvtV7DEanhYCdmjY=; b=D+pp+pCitZ4BSWq3nPuc48fNRIoLnQUl0Ln+YifLo0yo6bkiyzJcvScic3jBIDs2HX Ls1gWq5QCbUxvxaHC1+C/IdQO5sEER9ZXIvST9bIY3OWtK4BsK9b59rnFecM+hL6hIhW vAzZvRa+YpcP1+E3LW2cdvpCVAblO6vYoZOOg88ATF7bnfT9thmNZUbsWdAtepIZ+tDe 4Wy+/hloHSx4puPFObRZb6bGcb7uocIiQXZlpb9G1VOURjji2r7ZZwdEB2sVld6Pncaz GF6AREu3Qa8feFyPcSirjF2ghEj4o+Uz7tlywy97rX9xrFcneLPT1OSqKDF8XBo1oYVd rwKA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:41 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417125" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Kees Cook , Dave Hansen , Nadav Amit , Rick Edgecombe Subject: [PATCH v4 03/23] x86/mm: Introduce temporary mm structs Date: Mon, 22 Apr 2019 11:57:45 -0700 Message-Id: <20190422185805.1169-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andy Lutomirski Using a dedicated page-table for temporary PTEs prevents other cores from using - even speculatively - these PTEs, thereby providing two benefits: (1) Security hardening: an attacker that gains kernel memory writing abilities cannot easily overwrite sensitive data. (2) Avoiding TLB shootdowns: the PTEs do not need to be flushed in remote page-tables. To do so a temporary mm_struct can be used. Mappings which are private for this mm can be set in the userspace part of the address-space. During the whole time in which the temporary mm is loaded, interrupts must be disabled. The first use-case for temporary mm struct, which will follow, is for poking the kernel text. [ Commit message was written by Nadav Amit ] Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/mmu_context.h | 33 ++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 19d18fae6ec6..d684b954f3c0 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -356,4 +356,37 @@ static inline unsigned long __get_current_cr3_fast(void) return cr3; } +typedef struct { + struct mm_struct *prev; +} temp_mm_state_t; + +/* + * Using a temporary mm allows to set temporary mappings that are not accessible + * by other cores. Such mappings are needed to perform sensitive memory writes + * that override the kernel memory protections (e.g., W^X), without exposing the + * temporary page-table mappings that are required for these write operations to + * other cores. Using temporary mm also allows to avoid TLB shootdowns when the + * mapping is torn down. + * + * Context: The temporary mm needs to be used exclusively by a single core. To + * harden security IRQs must be disabled while the temporary mm is + * loaded, thereby preventing interrupt handler bugs from overriding + * the kernel memory protection. + */ +static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) +{ + temp_mm_state_t state; + + lockdep_assert_irqs_disabled(); + state.prev = this_cpu_read(cpu_tlbstate.loaded_mm); + switch_mm_irqs_off(NULL, mm, current); + return state; +} + +static inline void unuse_temporary_mm(temp_mm_state_t prev) +{ + lockdep_assert_irqs_disabled(); + switch_mm_irqs_off(NULL, prev.prev, current); +} + #endif /* _ASM_X86_MMU_CONTEXT_H */ From patchwork Mon Apr 22 18:57:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911395 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFFBB1515 for ; Mon, 22 Apr 2019 18:58:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E34B028759 for ; Mon, 22 Apr 2019 18:58:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D6F6728764; Mon, 22 Apr 2019 18:58:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60EFE28759 for ; Mon, 22 Apr 2019 18:58:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BBF56B000C; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 386496B0266; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFA1A6B0010; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 69E5A6B0007 for ; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id j1so8140264pff.1 for ; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=EpX7PmmGjCsoe95ecfJDQVEnFby3S4g+dchtH/l+eVw=; b=D+R5Hb8WgvlRCwpAKkQFm63Q+dYdVQv/DyW/JTx86FMxIcBuDjH583RBL7+la4lV0+ gK89ZRJ3tMjD7yLwgicosvWYRBd1JgxcjBzaeqkUohGufTZVFMzXUQ4J/9E07WM87hk2 +ij/Chj+1Rf2OlYg3EWje85jRNhsvjfMfiFvbO2H9GbkLPqfSNudHHElzXJaawK+LhjK zInqT3K/UEW9/19BgBmYGVIEun4E2LZ1glRlyorXLmmeP8oEUj0RoHtCdqw0KkYN8bcw /pD0lsjnQTCQom/3W32eZfnDMA5ekpnFNSyUBlYzBVE6fzXh+YtOAi1PW3n5Hq2Uqcx3 0Nag== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUzMB+vKp4Xzrtc+Lfz69we/Myx7yZJJadwJXtVBqMNYtEGIOVY PZyIn6FixGpSirR1kOXkjT3w5+sWm6iQz1qQmorNp7QwU/xJvPA999SZA8U/4BeOmGkE229BqZ6 BEA/RR/kgZa2BlNSfEMS/dt6kuxARc6rfon3wHnXMObHEwsXQ7mRrWkHkeHWT6bCVkQ== X-Received: by 2002:a65:524a:: with SMTP id q10mr19741338pgp.224.1555959524039; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqyzY1F+/tQEbrVRcXdxG71FSOSZ5itmH4sINA/uVENlkFaH6XUIQw3B1VAowhqy13eoDhNR X-Received: by 2002:a65:524a:: with SMTP id q10mr19741273pgp.224.1555959522814; Mon, 22 Apr 2019 11:58:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959522; cv=none; d=google.com; s=arc-20160816; b=X33vT83UW+49Vh5mWDlEZGqwZ6QjkuhXxcGdHD9ABapcMG5VzCp4T2DyMm+9r4Z6EF 6fRQuPEUOw49F6htp+DcdlRPZOPCVc/DDDMNpFFapeio69OuQja8RdsdIcA+bR014SFn 9tfpD52TEwsosAR5b6st77E4H2Z7nZFUrHCDc7QNaPEhldyg1puRqir8hKC1o2OV6KV1 udefRjonZfFtk3s/ObLebZM8U818Oh1ZMsVAUFax8fJn0E5QzWilHj71EG9HmJXJrHr+ om5HaDC8FrjJImCi5VbJVQBSV5rdliGtMTkvVWtICTBJJI+LHJoX7ZEfCJlmXVCTWMf7 J1jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=EpX7PmmGjCsoe95ecfJDQVEnFby3S4g+dchtH/l+eVw=; b=cxVQ7+mcx6EJOEMR88XkgI18qvXPS6pefxsYyojRdTFjskJEzYFgiwI5yKRw8H3BeT zXq/ZhQr7ueld86VlnvBJ2EFSzsRwDFBJpBzLzpHXzAwCTJLP2qD8MmXB6ya5UH/T+Iy MIYwt5++zCSgPpF8l986arqRH5gw42y1FGPx3fBc/4Wmknr/3kgz3AMw96FLuLjWOfOR JXq+vOYTbrE5ycqasPltKWXb7IocshszIof2SaqNLMMNHV/xbAsozsTZNrkIPtpggZqC 5iQ+s4MmEdMD5/lww25l7LbliZfK5XXKYe+1QGNLcmpY3DzH6wabjCcvSfwtdKku2K9N oUjQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:42 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417128" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Rick Edgecombe Subject: [PATCH v4 04/23] x86/mm: Save DRs when loading a temporary mm Date: Mon, 22 Apr 2019 11:57:46 -0700 Message-Id: <20190422185805.1169-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Prevent user watchpoints from mistakenly firing while the temporary mm is being used. As the addresses that of the temporary mm might overlap those of the user-process, this is necessary to prevent wrong signals or worse things from happening. Cc: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe Acked-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/mmu_context.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index d684b954f3c0..81861862038a 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -380,6 +381,21 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) lockdep_assert_irqs_disabled(); state.prev = this_cpu_read(cpu_tlbstate.loaded_mm); switch_mm_irqs_off(NULL, mm, current); + + /* + * If breakpoints are enabled, disable them while the temporary mm is + * used. Userspace might set up watchpoints on addresses that are used + * in the temporary mm, which would lead to wrong signals being sent or + * crashes. + * + * Note that breakpoints are not disabled selectively, which also causes + * kernel breakpoints (e.g., perf's) to be disabled. This might be + * undesirable, but still seems reasonable as the code that runs in the + * temporary mm should be short. + */ + if (hw_breakpoint_active()) + hw_breakpoint_disable(); + return state; } @@ -387,6 +403,13 @@ static inline void unuse_temporary_mm(temp_mm_state_t prev) { lockdep_assert_irqs_disabled(); switch_mm_irqs_off(NULL, prev.prev, current); + + /* + * Restore the breakpoints if they were disabled before the temporary mm + * was loaded. + */ + if (hw_breakpoint_active()) + hw_breakpoint_restore(); } #endif /* _ASM_X86_MMU_CONTEXT_H */ From patchwork Mon Apr 22 18:57:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911387 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF4861575 for ; Mon, 22 Apr 2019 18:58:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C315528759 for ; Mon, 22 Apr 2019 18:58:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B6AD328764; Mon, 22 Apr 2019 18:58:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31FD828759 for ; Mon, 22 Apr 2019 18:58:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6DEC6B000D; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CA8456B0008; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 888EA6B000E; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 13B3F6B000C for ; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id o1so8457710pgv.15 for ; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=+opaO1awQtWXl9ApsaZ/rgEyOZ23bi+ZXWhS2cx66gg=; b=VLacFO/eRYx3xU+8yGgqaapuX1dBkkTpOKfIsqcAdS3CBx8MqeYz8h1EOhmd/TbaC6 wATSwEtWvhUIGWqEbTH7EFLZU375Ec2GgfggGTUveyeIgrQ/i1aMqbm5IWEC0YXZZH3v t3dI8PsTA8Fzk9sXl2jXAc/P2OA7aqIGs/GcT2Gid18WGjAdFHNQpucQWDoXsob5saem tFkiIesFbCU2tHFiW5ysdaY3EpZk23j7GFKq6JrW2fvujviZQbataowQjrDVKAnD4fJL KuyIgmlz49iZn7drVPoUdtTDtu6llFAvGk7ObF5wTekpcXvRfrtqSePIezRybQF8NR24 QYHg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWOOlsLZBSU3SBXT+pRS/ZOTVw6DkmY+7W5NEr00pP9+o3Qy4dB z6Met1eL1/kAVek6BHW/4wHJphSlJSvURYjlgOe4+eRxTfSBucieDgBrbkVTrJrZbFEcJ3C85Cp 8xqp0k1r+PxB0Dg8Qlh9/oibi9OgiqEaqu9SgATli03BH6CWTy5LNO9GgJQ7Sc4hNYA== X-Received: by 2002:a65:4802:: with SMTP id h2mr18937043pgs.98.1555959523726; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqzKJo+cK5HWrFUB2d3Y5F0DvfXrbxgp96JQUE5a8z1blVUH6mZsH6AASeHpcbRxytVhBRP6 X-Received: by 2002:a65:4802:: with SMTP id h2mr18936993pgs.98.1555959522802; Mon, 22 Apr 2019 11:58:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959522; cv=none; d=google.com; s=arc-20160816; b=SKZzqELvL+K1Abb2dArXvVRcJwDqtyF9gQMjmT0sx4nR7xKsNfZNwcpDA9EMPeIVLX IeYZxiH8Ui3a7nfU16jsor7Ge+x/USbQvBi6OAtq6oHW6PF9pstK4OVhsaYCZTJBeDZr iLROlrqgqOIdUY5hihtmIxuNbNFTVY+ZLEotdB9q0cFwB8JVv0oFH6kUvVs+0UkMIDjE 3NPs2JLdS2NOzMUvV5t1HfjPU5Gu2YucB4iEhLVFigioSKjoZxEkqTyB86UBV189adJw BhULmicBZ+D06QNyh6NxRH6VrPatbdNSRPjIS7MW0zfMgAinI2Rm4In4y/xWtwAjYWvu T+ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=+opaO1awQtWXl9ApsaZ/rgEyOZ23bi+ZXWhS2cx66gg=; b=H7WUJPHGSJxxZbo6Abp+/lM6hzrt4Vu8j9Bk7d1mbR7WMZh7MTGy6jygjUDsbKiqcT g4IjK5Wyxlpb+6Nj2uWDEUe2tJc74dTGVeQG9PLTohAU11baK8lYOhzRAQCutMb5ll+Z gfY+1FA8LV+JWRo0PZGrXy40uGHXo86cO3y+1bZsl4eOBFYnmNm23jzURiGvaJVsvnk2 NSLDvGf6NfGmUQ7XcL3rUNU1YBTbS+rc9XVgAiuNNeMAVHAKnvSDEapN/RfUeMTbrKgY vLNuZIggnoZr4EpsQ8csPOIlFzQuLhhEQ4476RVvQmbLLtkjrN2l2BAAJe0naFMnJMrr KJNQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a20si5314305pgb.421.2019.04.22.11.58.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:42 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417131" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Rick Edgecombe Subject: [PATCH v4 05/23] fork: Provide a function for copying init_mm Date: Mon, 22 Apr 2019 11:57:47 -0700 Message-Id: <20190422185805.1169-6-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Provide a function for copying init_mm. This function will be later used for setting a temporary mm. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- include/linux/sched/task.h | 1 + kernel/fork.c | 24 ++++++++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h index 2e97a2227045..f1227f2c38a4 100644 --- a/include/linux/sched/task.h +++ b/include/linux/sched/task.h @@ -76,6 +76,7 @@ extern void exit_itimers(struct signal_struct *); extern long _do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *, unsigned long); extern long do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *); struct task_struct *fork_idle(int); +struct mm_struct *copy_init_mm(void); extern pid_t kernel_thread(int (*fn)(void *), void *arg, unsigned long flags); extern long kernel_wait4(pid_t, int __user *, int, struct rusage *); diff --git a/kernel/fork.c b/kernel/fork.c index 9dcd18aa210b..099cca8f701c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1298,13 +1298,20 @@ void mm_release(struct task_struct *tsk, struct mm_struct *mm) complete_vfork_done(tsk); } -/* - * Allocate a new mm structure and copy contents from the - * mm structure of the passed in task structure. +/** + * dup_mm() - duplicates an existing mm structure + * @tsk: the task_struct with which the new mm will be associated. + * @oldmm: the mm to duplicate. + * + * Allocates a new mm structure and duplicates the provided @oldmm structure + * content into it. + * + * Return: the duplicated mm or NULL on failure. */ -static struct mm_struct *dup_mm(struct task_struct *tsk) +static struct mm_struct *dup_mm(struct task_struct *tsk, + struct mm_struct *oldmm) { - struct mm_struct *mm, *oldmm = current->mm; + struct mm_struct *mm; int err; mm = allocate_mm(); @@ -1371,7 +1378,7 @@ static int copy_mm(unsigned long clone_flags, struct task_struct *tsk) } retval = -ENOMEM; - mm = dup_mm(tsk); + mm = dup_mm(tsk, current->mm); if (!mm) goto fail_nomem; @@ -2186,6 +2193,11 @@ struct task_struct *fork_idle(int cpu) return task; } +struct mm_struct *copy_init_mm(void) +{ + return dup_mm(NULL, &init_mm); +} + /* * Ok, this is the main fork-routine. * From patchwork Mon Apr 22 18:57:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911407 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D94B41575 for ; Mon, 22 Apr 2019 18:59:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC16B28759 for ; Mon, 22 Apr 2019 18:59:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C070A28764; Mon, 22 Apr 2019 18:59:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C35528759 for ; Mon, 22 Apr 2019 18:59:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13CBB6B026D; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0F2326B0271; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7E8E6B0270; Mon, 22 Apr 2019 14:58:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id B09F56B026D for ; Mon, 22 Apr 2019 14:58:54 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id o8so8479848pgq.5 for ; Mon, 22 Apr 2019 11:58:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=u6wlgWiwWVaBEbsh01acCfZKxBAGv1NeORm+6C+eZcs=; b=J1fvC/5g06QT9RD0qDyMxV/COfdsmer73wVRDFCSCwrYjsgUGjVKpa19u/mH4S5RMQ yrOrBUGipNw8KF3GLqTtDc/lOPCB/HsZsQI4PC0jjKlf4lJTJALLz70HmppGzFMYKkpr WBQGzOHbHyt0yDzFE3t7UqQjnXqnQfV4rAa8wQMAHJRbE/vPe3HHwdl6VchnZWGGOMLM lyZEL/RaYWWt5ovG6M26RSRRpoBaHH08rQ0l/4uw7Z+tnC+ivsf5wlXhV2wNVwoCy/XP s7jUPjbSTzpy/K9m54UBjvHc9Vtcu4sCwH4kuoFJ1WWyiU8j9IpblhsV8TEL5+VzWkGR vFeQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWxZMm1ES07vHoJq7eGmndygMDqoPNYB78FH0s6I26CujGA4NQi qChLEKkPYmsK8I8xG8dH51cie9xes+mf5HgeuU7fpE+Sb34aS4jDGjpW2+buSo1BSNIbEBc09ua 0p5t2WIwk79DhYetZHdNSkHYiAlpjbQwD0YniAw1C0D3lvEgLtDR7VJQdrt+Zv9yqJg== X-Received: by 2002:a17:902:f215:: with SMTP id gn21mr11673917plb.146.1555959534311; Mon, 22 Apr 2019 11:58:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqy/TMDilYNCfR7/YsZ7Ol4OlN3HWT0ZR3IaEn+/cG3vbb4M9v6r44UJYbNaEUSnbPaPgBE8 X-Received: by 2002:a17:902:f215:: with SMTP id gn21mr11673295plb.146.1555959523027; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=sj5Tv/K6wm5TcGPqMZoTmR99y94kVe/6UrXrNApKGExU665fTPtS3RhM2PqERbitER njYKHmV7zbYjcelrhoXBNeJ5NB8T3NuHizzyiz9i4F6WGmHhgVTukEXa8kGNYU/j10O4 wmNSaxQ64Rw3E0aX437aHOHdWMhUjNeBVoAcovj8AoHX2RJ20KAIudSKBiSPJn26cWJX 1CAM24yOJi3nkYVyqnTiNI4JokjgVbVVwMSmd8+eXwj3Z00Fau1OUTs0PylPq28nN5dl X8T8UvkEA4JZVNcl2izlzO5Tfvibw0MtvkiqFwWxQM28ny9mO1iAXiOgmY4Ob6nHYreq zIUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=u6wlgWiwWVaBEbsh01acCfZKxBAGv1NeORm+6C+eZcs=; b=gEXeQXy+hMd6oPrOxbStd0KuaGtQK/FmtUlfeXWBn0J/ZGzQctLjB3MoyuK3T4L83a 1mTrfHhet8Ck3uX0tnV0QoVC/7dCRqa1wO9xel8MaGidoADj52foU6Y0JfniVqu5wisz Vl84z1Flj5s+GqUWCJmeNX5hM0h/4BHKTNuh+QoPbZCavLbNZMExFsoUPwCh/PY0nLnl viOqPrl/GQrcucZxDQ9b+iLT38mE78HaLGZ5PG1GVYKSW1UXr2apahW9wgiD7wcBc2Tt GtxyxFYNa0qGEVdMdsYsKISXyqDJZqzFU6FdY/c0cvD2Etyzpx2uUMCMiAydwYw/Dv6v sMdA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417134" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Rick Edgecombe Subject: [PATCH v4 06/23] x86/alternative: Initialize temporary mm for patching Date: Mon, 22 Apr 2019 11:57:48 -0700 Message-Id: <20190422185805.1169-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit To prevent improper use of the PTEs that are used for text patching, the next patches will use a temporary mm struct. Initailize it by copying the init mm. The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory. Finally, randomize the address of the PTEs to harden against exploits that use these PTEs. Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 5cfbbb6d458d..6b6bfdfe83aa 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1038,6 +1038,9 @@ static inline void __meminit init_trampoline_default(void) /* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)]; } + +void __init poking_init(void); + # ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline(void); # else diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index f8fc8e86cf01..a75eed841eed 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr; #endif /* _ASM_X86_TEXT_PATCHING_H */ diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 0a814d73547a..11d5c710a94f 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -679,6 +679,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; + static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index bccff68e3267..125c8c48aa24 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -53,6 +53,7 @@ #include #include #include +#include #include "mm_internal.h" @@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void) return memory_block_size_probed; } +/* + * Initialize an mm_struct to be used during poking and a pointer to be used + * during patching. + */ +void __init poking_init(void) +{ + spinlock_t *ptl; + pte_t *ptep; + + poking_mm = copy_init_mm(); + BUG_ON(!poking_mm); + + /* + * Randomize the poking address, but make sure that the following page + * will be mapped at the same PMD. We need 2 pages, so find space for 3, + * and adjust the address if the PMD ends after the first one. + */ + poking_addr = TASK_UNMAPPED_BASE; + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + poking_addr += (kaslr_get_random_long("Poking") & PAGE_MASK) % + (TASK_SIZE - TASK_UNMAPPED_BASE - 3 * PAGE_SIZE); + + if (((poking_addr + PAGE_SIZE) & ~PMD_MASK) == 0) + poking_addr += PAGE_SIZE; + + /* + * We need to trigger the allocation of the page-tables that will be + * needed for poking now. Later, poking may be performed in an atomic + * section, which might cause allocation to fail. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + BUG_ON(!ptep); + pte_unmap_unlock(ptep, ptl); +} + #ifdef CONFIG_SPARSEMEM_VMEMMAP /* * Initialise the sparsemem vmemmap using huge-pages at the PMD level. diff --git a/init/main.c b/init/main.c index 598e278b46f7..949eed8015ec 100644 --- a/init/main.c +++ b/init/main.c @@ -504,6 +504,8 @@ void __init __weak thread_stack_cache_init(void) void __init __weak mem_encrypt_init(void) { } +void __init __weak poking_init(void) { } + bool initcall_debug; core_param(initcall_debug, initcall_debug, bool, 0644); @@ -737,6 +739,7 @@ asmlinkage __visible void __init start_kernel(void) taskstats_init_early(); delayacct_init(); + poking_init(); check_bugs(); acpi_subsystem_init(); From patchwork Mon Apr 22 18:57:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911397 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B81A11515 for ; Mon, 22 Apr 2019 18:59:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A96CB28759 for ; Mon, 22 Apr 2019 18:59:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9DA0028764; Mon, 22 Apr 2019 18:59:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD4E628759 for ; Mon, 22 Apr 2019 18:59:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 903D76B0008; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F43B6B0269; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 073646B0008; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 934206B000C for ; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id y2so8124222pfn.13 for ; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=uR9dojmxOmmxTLzrNkqbIEwKeNq3pUquy4Aauxui5YA=; b=beiFvAnxZ75uOI52zwBjQtycjJIn33eBDyBTXarMYp0Uj/NJJ6HM5vGfMRVg0fSt3Q OT/+DO5ERb+aKqEeXUMldmNUIv0bZ38X4unU0IztM8q9x8T7gKvTT3M8ASwB32c5kQjo IrRJ/VdcjbO6U9TVq8oxS8NA9IAJxY1EYpZyU03+WSsHCU+E5sm6B1t9q96jKUUjOpaE IUp1D3l7DVdUBlcmbOZa4jiTLuaW/sMJVtqqE95bhtdFqT99Oq8KpWtM/Q1rlr+i3hvG Jhqt5Fn9MWz1USQxaxwuYcpo+9CDHs8UCVVOzf78F4LHMK9sU4ZO8B2VZL/vD+Qrcdj3 FBKQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUmfLh+9nKq/2ZZg5RDeBt35G8CPSwpW/G2D5emE+ig6moP7QFV ESckn7BrTxtf1zr7pkt02JvRF6nQgixphsPiYQQH1FjVECbzH4RS7w6A1rNU37AiLlwVLdW9zuA ZDiCtSDISySWtkIWeW89jrY4mxml4c69BSg3xhIq6w4zwvQDWDI0dToSD/8kMSzABLw== X-Received: by 2002:a65:4108:: with SMTP id w8mr20379800pgp.236.1555959524169; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqwl5xHI72qwpU5pl0NGUYw/lgZjHFtjxBKzVsm0jnDm5EKbyzHrCQYZf5ivUaXGtq2VnZbH X-Received: by 2002:a65:4108:: with SMTP id w8mr20379740pgp.236.1555959523069; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=pCEoUwziZSZVFXgNK+D4YTsbxmjglHeq0xUX17gVuFVhGN+VJkUIfUqA3fh80czjeG aTOh+VpKamAOYXo8rD+ZnA63t0i8Z+CgsHtZhYqC+9NgY5/4ZwZ/cHe6EBC1bxjU/TJN x+aTlC6bYLGzSrnEnpQ7ijE2tmj/foS+Dzgr6CpRVL5mZtTUS8Tjw8p0uu5uNkcnDJDr /Kw49eXH/mdMFvlVJVXX3nuvgJUclEidQmdFU7jopJXsnzPZzkegx+D9jKgGKttLp81p 9TzAp/CJjQ0q1U9FM4+gSww2RK+hu1guEVD9MsSPL/YUciEapwZvt2JFrCGC7TZRTcgD IW/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=uR9dojmxOmmxTLzrNkqbIEwKeNq3pUquy4Aauxui5YA=; b=O6qJMvsO5jDp3U1MkP4BiRiUX9nacwSmZFx+GEe0A/4dTS88UrMJC0NZyOHfAh15h1 MzqbCBf6vWsKbSip7/EsEbp0wULSRLPBdXTUz3N0IhPX2cHTVnpYA+isKIcmG6C4kJFB iB76RMDV+yfItSyz7sHS0S1VUX5Xc6VSvza15uuXVwFz8T7WRu7+TeqTqg+J07PjPrAf /SJ4IwRHTIqmHi02kv8X8G+asFHJr4z/HEYvfGkNsEUq2rWMfyRM8q0CrRxS4p2x5lIf 53aSkauhr9idp5srbQVrkrFO+euaXR3s9jNNf5AOJAk4VVZtK6ZEMuW2XNYYNOSg049A GAEw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a20si5314305pgb.421.2019.04.22.11.58.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417137" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH v4 07/23] x86/alternative: Use temporary mm for text poking Date: Mon, 22 Apr 2019 11:57:49 -0700 Message-Id: <20190422185805.1169-8-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_poke() can potentially compromise security as it sets temporary PTEs in the fixmap. These PTEs might be used to rewrite the kernel code from other cores accidentally or maliciously, if an attacker gains the ability to write onto kernel memory. Moreover, since remote TLBs are not flushed after the temporary PTEs are removed, the time-window in which the code is writable is not limited if the fixmap PTEs - maliciously or accidentally - are cached in the TLB. To address these potential security hazards, use a temporary mm for patching the code. Finally, text_poke() is also not conservative enough when mapping pages, as it always tries to map 2 pages, even when a single one is sufficient. So try to be more conservative, and do not map more than needed. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/fixmap.h | 2 - arch/x86/kernel/alternative.c | 108 +++++++++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 2 - 3 files changed, 86 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h index 50ba74a34a37..9da8cccdf3fb 100644 --- a/arch/x86/include/asm/fixmap.h +++ b/arch/x86/include/asm/fixmap.h @@ -103,8 +103,6 @@ enum fixed_addresses { #ifdef CONFIG_PARAVIRT FIX_PARAVIRT_BOOTMAP, #endif - FIX_TEXT_POKE1, /* reserve 2 pages for text_poke() */ - FIX_TEXT_POKE0, /* first page is last, because allocation is backward */ #ifdef CONFIG_X86_INTEL_MID FIX_LNW_VRTC, #endif diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 11d5c710a94f..599203876c32 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -684,41 +685,104 @@ __ro_after_init unsigned long poking_addr; static void *__text_poke(void *addr, const void *opcode, size_t len) { + bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE; + struct page *pages[2] = {NULL}; + temp_mm_state_t prev; unsigned long flags; - char *vaddr; - struct page *pages[2]; - int i; + pte_t pte, *ptep; + spinlock_t *ptl; + pgprot_t pgprot; /* - * While boot memory allocator is runnig we cannot use struct - * pages as they are not yet initialized. + * While boot memory allocator is running we cannot use struct pages as + * they are not yet initialized. There is no way to recover. */ BUG_ON(!after_bootmem); if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); - pages[1] = vmalloc_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = vmalloc_to_page(addr + PAGE_SIZE); } else { pages[0] = virt_to_page(addr); WARN_ON(!PageReserved(pages[0])); - pages[1] = virt_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = virt_to_page(addr + PAGE_SIZE); } - BUG_ON(!pages[0]); + /* + * If something went wrong, crash and burn since recovery paths are not + * implemented. + */ + BUG_ON(!pages[0] || (cross_page_boundary && !pages[1])); + local_irq_save(flags); - set_fixmap(FIX_TEXT_POKE0, page_to_phys(pages[0])); - if (pages[1]) - set_fixmap(FIX_TEXT_POKE1, page_to_phys(pages[1])); - vaddr = (char *)fix_to_virt(FIX_TEXT_POKE0); - memcpy(&vaddr[(unsigned long)addr & ~PAGE_MASK], opcode, len); - clear_fixmap(FIX_TEXT_POKE0); - if (pages[1]) - clear_fixmap(FIX_TEXT_POKE1); - local_flush_tlb(); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ - for (i = 0; i < len; i++) - BUG_ON(((char *)addr)[i] != ((char *)opcode)[i]); + + /* + * Map the page without the global bit, as TLB flushing is done with + * flush_tlb_mm_range(), which is intended for non-global PTEs. + */ + pgprot = __pgprot(pgprot_val(PAGE_KERNEL) & ~_PAGE_GLOBAL); + + /* + * The lock is not really needed, but this allows to avoid open-coding. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + + /* + * This must not fail; preallocated in poking_init(). + */ + VM_BUG_ON(!ptep); + + pte = mk_pte(pages[0], pgprot); + set_pte_at(poking_mm, poking_addr, ptep, pte); + + if (cross_page_boundary) { + pte = mk_pte(pages[1], pgprot); + set_pte_at(poking_mm, poking_addr + PAGE_SIZE, ptep + 1, pte); + } + + /* + * Loading the temporary mm behaves as a compiler barrier, which + * guarantees that the PTE will be set at the time memcpy() is done. + */ + prev = use_temporary_mm(poking_mm); + + kasan_disable_current(); + memcpy((u8 *)poking_addr + offset_in_page(addr), opcode, len); + kasan_enable_current(); + + /* + * Ensure that the PTE is only cleared after the instructions of memcpy + * were issued by using a compiler barrier. + */ + barrier(); + + pte_clear(poking_mm, poking_addr, ptep); + if (cross_page_boundary) + pte_clear(poking_mm, poking_addr + PAGE_SIZE, ptep + 1); + + /* + * Loading the previous page-table hierarchy requires a serializing + * instruction that already allows the core to see the updated version. + * Xen-PV is assumed to serialize execution in a similar manner. + */ + unuse_temporary_mm(prev); + + /* + * Flushing the TLB might involve IPIs, which would require enabled + * IRQs, but not if the mm is not used, as it is in this point. + */ + flush_tlb_mm_range(poking_mm, poking_addr, poking_addr + + (cross_page_boundary ? 2 : 1) * PAGE_SIZE, + PAGE_SHIFT, false); + + /* + * If the text does not match what we just wrote then something is + * fundamentally screwy; there's nothing we can really do about that. + */ + BUG_ON(memcmp(addr, opcode, len)); + + pte_unmap_unlock(ptep, ptl); local_irq_restore(flags); return addr; } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index a21e1734fc1f..beb44e22afdf 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2318,8 +2318,6 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot) #elif defined(CONFIG_X86_VSYSCALL_EMULATION) case VSYSCALL_PAGE: #endif - case FIX_TEXT_POKE0: - case FIX_TEXT_POKE1: /* All local page mappings */ pte = pfn_pte(phys, prot); break; From patchwork Mon Apr 22 18:57:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911417 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E72281708 for ; Mon, 22 Apr 2019 18:59:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAA3028759 for ; Mon, 22 Apr 2019 18:59:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE98E28764; Mon, 22 Apr 2019 18:59:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5337A28759 for ; Mon, 22 Apr 2019 18:59:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86CA86B0271; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 75B216B026E; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E36C6B0271; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id DDEFE6B026E for ; Mon, 22 Apr 2019 14:58:54 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id 132so8436601pgc.18 for ; Mon, 22 Apr 2019 11:58:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=3mFI7V2IpbwOhuCcp+g9CO2dE2meFXsvyyAeb18NWyo=; b=fnzdWan1Z5sBJvBF44Yi6VVgB1VudELxIMBtHaQOvnu5WeJUBrFjTPmAL16/CJDDWW gI7TyCI/nv49kzLVCWZlYS2EJDzlW5ZjpI/Zn+UR0huwlcR3+7Rt/wY3JXGyAR7gepsk qjuTf/6i3sJYyOi/p3v4a0F/7Pi5jXTFdrKfIPgCBXeitOFLu9wI+QVYuqYVIeF9QmyW MZ7V6ZXCzd7s/e/Ua4n62uMIxIxF73vctdotWhJMrxsGsnQ6jpFWndds6QB0f/Afc9YR UtoJe3N+v8nqglPZZkMRJ5ZA6MdLzpHVGw4N8szYuG9HYCNrNsgKMv37HO0ulHg1qlwU cvTQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVeuzOofa6d6XE5xMlzhfLtRqjt6ooQaIy/bzeFtbuRQaRKvAvn 8jhouz3hBAfDGVnSpSC4if6m9pB+inVQ76a7kAczKr6tIbzA1zUSiplLF5Sxby1jYod+4gZvFN1 samxh6BrnikDgsryxdq9lKQgio5vyM9ayhlXx6+dEjUhLCF55L9HB2hOMhpSXKTS8Hw== X-Received: by 2002:a63:3281:: with SMTP id y123mr20452289pgy.272.1555959534517; Mon, 22 Apr 2019 11:58:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqygQWToNVYgEOHPy2ZS9ereJqkOFwCvtNhHZ3h+9+8saUbuz+HKI2reMQVulZ3SgMWB8h4A X-Received: by 2002:a63:3281:: with SMTP id y123mr20451661pgy.272.1555959523359; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=k/BgZ+PD/RpyuAd6ziaISb7wk7hzWL/ttc8KwLGlyu0H3qpFvVkfw2/ADrUGIXpBlE e+efm1LVx2EwhM+azY/LGEE7VZmK5qlTggHM8bUaxMBXVpw8waMSauTAkFDNmFNbAZKN 6emqfw3UfkTIE+eF38x7w10edH3/IUla1nEIHhkZ95s6KVSZPBfcV2s98I0TYlCuxBZ5 CGe9QUyegqk6+iI1x5gA84melXn14WDkMj7EVV+p6y4H7igQnBcFireKVNd/Fo67iBxL C9ErWgvEXG1IKKwz+yTcorBlOZZQVgIfaRIGg31ElHHzK4Z32gONrvSPqtxkvp1spmvu DVAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=3mFI7V2IpbwOhuCcp+g9CO2dE2meFXsvyyAeb18NWyo=; b=TJbIhnwQTpc3a+wbTeNixsIhlPM+RbYjke6hAX7/YAhpVhhVmGmwAbJaqWEC3FSc2G Xl0Utwwox2y7HOK4PDW1Luw3p76ckLR0TihC8LWMJfK0e9FljIuYhyy3VKDaEYl26ihK uWxXA8LoLwJY1H+I2SC0fSw7VuWIBfTM5G+7MvlWZlnodmoQjajmzxNHDVONXHvfBWaa 7ogbh1xrT0b87AErJDbHpmaDcFhSPIs/cP4y0xcHQKpasrnRPYHjQ6uly/toENLZgJAu G5n6hiQo7mDm2vjGl+DhPjLmZr0ywmSLRK3c/4sxwP8i/05gJ22OzoS4r9volYmk4o8A 5U5g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a20si5314305pgb.421.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417140" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Rick Edgecombe Subject: [PATCH v4 08/23] x86/kgdb: Avoid redundant comparison of patched code Date: Mon, 22 Apr 2019 11:57:50 -0700 Message-Id: <20190422185805.1169-9-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_poke() already ensures that the written value is the correct one and fails if that is not the case. There is no need for an additional comparison. Remove it. Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kgdb.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 2b203ee5b879..13b13311b792 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -747,7 +747,6 @@ void kgdb_arch_set_pc(struct pt_regs *regs, unsigned long ip) int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) { int err; - char opc[BREAK_INSTR_SIZE]; bpt->type = BP_BREAKPOINT; err = probe_kernel_read(bpt->saved_instr, (char *)bpt->bpt_addr, @@ -766,11 +765,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) return -EBUSY; text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err) - return err; - if (memcmp(opc, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE)) - return -EINVAL; bpt->type = BP_POKE_BREAKPOINT; return err; @@ -778,9 +772,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) { - int err; - char opc[BREAK_INSTR_SIZE]; - if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* @@ -791,10 +782,7 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) goto knl_write; text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) - goto knl_write; - return err; + return 0; knl_write: return probe_kernel_write((char *)bpt->bpt_addr, From patchwork Mon Apr 22 18:57:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911399 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC2A91575 for ; Mon, 22 Apr 2019 18:59:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE8FE28759 for ; Mon, 22 Apr 2019 18:59:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C259028764; Mon, 22 Apr 2019 18:59:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47C9828759 for ; Mon, 22 Apr 2019 18:59:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B79116B0266; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6917B6B0010; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 257AF6B0010; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id B39A86B0269 for ; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id n5so8474522pgk.9 for ; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=CLKl6NZzJuOK2yM4IgvyVy0XzBWA5gDBiZYSHfLNQ00=; b=Ruja2QPURe+6IsigPi7rLyBcR7+r8uh+jczLnLnviBCfHeJq5aMP9CW8+JiS4glaeU iXZkAQb2gwYvqLUD4kth30Ai2KLGbds7n1CTeevEsYI/1ST4lqNLU+Wo77bux25AQCTl QqID8h8KJ4pvyrSlKkQnrRYhJoDEUPJBmsp8Vs21RkiJ0NXL9ojAjHoHEBsZLVN4Rszo zSUwxlgjC8psIk7iZBVfjavJYGx/gTjIA0X/O9xqh25uU45oGY51dP6LHxgFfJSLvyMr VyRlgyyt42xQ/jsg6D9JgTNf+MEE9CZDz/AJ5ER30Hgh5uvPBcW35rYl7HyUxw0Xa9rR 1eCw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVVuZfi4YcrsONBtdXgLCV0Ie6VdNSmu8uOsI2kRj0xcLHm8Ft/ W8mWRaXAhWPDVC/cl1ossjHuJuRyTxpS3G81GWQT1FONJpM4qL6ti6sPRCGimH9XJTtCCyz1mDa XUSXKhXE1RiyAUGts+JyJYCd9D+GdeZwcDa9GdSq7dPoyUyU5pP/pgr24grCfX72qfA== X-Received: by 2002:a17:902:20c6:: with SMTP id v6mr20749444plg.276.1555959524395; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqxFd/C5ODM2my4PUmNlkfyHBw5qK1S+aeT6PcS951bxAhA8IdWfEaSs2Vu8K8PladKM9yQW X-Received: by 2002:a17:902:20c6:: with SMTP id v6mr20749394plg.276.1555959523454; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=FkHmAXXRcwwuXgtdr1oS8gOqMMknf+ZXYEJIzYC81XQ2OIerTcMQs+71hB3N0emirB Q5k4OdbJczY3adc81oIy31lvRAOsOJ2u3G7lWEManAfxxltJpvyLNbNpbnMZTYelWwLo Ez/LI4vty2giKn3w2TIEiBp7Xd1HXlcb3DIPDxdxQBD2Gv7yICX1AIhYJT1SLjvw9UcG EHG3cZpDmS93F2awqxcnUH/hpIR76JflHQCXkD85kSmNjxxOOqqSoQQDxcC5aKLiCOLj MIsQD5H0DOCq6/KjrgEYXsS1sarKSFB8BUFpaq8L3Bf+60OfJEcTONpMtH9FiZT/OEta 5NGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=CLKl6NZzJuOK2yM4IgvyVy0XzBWA5gDBiZYSHfLNQ00=; b=zsnA61SB2KFN8zQRhS5F5kJ3rNYCJ1c2kfOn0ANpG5PbGJFzBtk8auPM2o+4T7ag14 fVHBcohcZso/Q1MeatI+5pr/97jN8MbBQR9QAEmQFkpd/6F3MEvkCZaSZrwox22pAWbF 08Mb3Eq+okHJGqmGw7k1nmV5GPSwmkIjNNo4GLRAapnbWaZKTMt57DICs8nFPd75fFa3 kFBkhyPCZ02PMuWzpDpZkfpC4hDAKqCYcie35083Rh8G7To6d401hMAVzMtf+VoEkEr2 BjMjyTK9uHMpeMttmK3hU6ACbA1CXgVMSSrNKQ0yz4yeQPlyJDBQ46yXxVVruQLKbhfV MKTA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417143" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Rick Edgecombe Subject: [PATCH v4 09/23] x86/ftrace: Set trampoline pages as executable Date: Mon, 22 Apr 2019 11:57:51 -0700 Message-Id: <20190422185805.1169-10-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Since alloc_module() will not set the pages as executable soon, set ftrace trampoline pages as executable after they are allocated. For the time being, do not change ftrace to use the text_poke() interface. As a result, ftrace still breaks W^X. Reviewed-by: Steven Rostedt (VMware) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/ftrace.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index ef49517f6bb2..53ba1aa3a01f 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -730,6 +730,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) unsigned long end_offset; unsigned long op_offset; unsigned long offset; + unsigned long npages; unsigned long size; unsigned long retq; unsigned long *ptr; @@ -762,6 +763,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) return 0; *tramp_size = size + RET_SIZE + sizeof(void *); + npages = DIV_ROUND_UP(*tramp_size, PAGE_SIZE); /* Copy ftrace_caller onto the trampoline memory */ ret = probe_kernel_read(trampoline, (void *)start_offset, size); @@ -806,6 +808,12 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + /* + * Module allocation needs to be completed by making the page + * executable. The page is still writable, which is a security hazard, + * but anyhow ftrace breaks W^X completely. + */ + set_memory_x((unsigned long)trampoline, npages); return (unsigned long)trampoline; fail: tramp_free(trampoline, *tramp_size); From patchwork Mon Apr 22 18:57:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911419 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F35E41575 for ; Mon, 22 Apr 2019 18:59:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E640C28759 for ; Mon, 22 Apr 2019 18:59:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DA87B28764; Mon, 22 Apr 2019 18:59:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 67C6C28759 for ; Mon, 22 Apr 2019 18:59:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B72EE6B026E; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 86A606B0270; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 695C86B0272; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 074AD6B026F for ; Mon, 22 Apr 2019 14:58:55 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id b12so8132071pfj.5 for ; Mon, 22 Apr 2019 11:58:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=yNv4uuWY37+QDIo+8eJ5Cs1RQKFygWuhu/mUpTFc+50=; b=sOmvkPM0hjPLNVKbD/nO4NRCPqU+sRwrgTYx2nW/zeoeu0HabIFSTKjAJOr9AMz1We 8Hm4mPkKu/aqw6vT0uhP/u2Nt4z+Bc8GmHgprkzFffJIzUlyahwX4TUMth7MV+pg8xkN Elzvv9UaHoTIPvXx0clm/Z6CL2PVj+nT9ya/K1ytKd6OVdO80tbqK94klEucfopuMuR+ DEJ5/9qMEyqMqNCR+l/c2SMBBLiWv1A4fjlxZIllKsxagnfPtAEoi49zr8siRr7SKNvS Tuelp6JqCPzDApW5HxgfIQYXCesbnO6nuqAnmLPa2HdoySSwdEEd/c2JBo9fVAxM2kSy PEpA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXSou8xQ8bgdD/A4/dwPWcfOXr9uOrOgIEtDSDcSnjVZ9NsHVvJ 4puaoTu5mru1UbkDA4w2Eb/Rl1vcYmiYiNnjs5o4sda14f/222mjuMiLQ38rCmzAvktFfkCfVSC kiKeU+xOU5kf5QsoRPDX29CPPnY/6Kk8jmYQQ35XDf4FoS2AIRZciVnzxjk5EAFpUFg== X-Received: by 2002:a62:6086:: with SMTP id u128mr22585385pfb.148.1555959534698; Mon, 22 Apr 2019 11:58:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqxtJvIHxx/i2Ts3cNM7XFD4I1LfROwkoBTclyOfRewUUm2q1+kxow7IgGo9jgKt5ObsRqsq X-Received: by 2002:a62:6086:: with SMTP id u128mr22584615pfb.148.1555959523584; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=H/KOPbNa2iOWnGgMDAJAGdvUb4n3FIrNRJfMsqjIKrHgM3l6QKG0Rbc6rgU71z+nTo HZhMygM3y/hJHYgi1mEmHbpxT91Mgjvn77jjKgnTPWsc+UVzL1J2N5ZD/1EMJ9zQkcBr CrLrUGuD5Wl08hu7+uPKi0nzO/KcT4wk7d+Xna/ZzvPxxyI981SFi2xW7NzQlFF44sAa gS03Bd1NNdNFYumpJsROlpG/Ygk4iNNaIl/55yLBiL7RmPRYt/+6+FAebVbGGCjZhXwm Gb6VHhw57DW1pFEZzlUJfeMFmxdp/d4LZVIqWfMbIgxWaHkdICt11sQ4BEG49hsLoZvX L/pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=yNv4uuWY37+QDIo+8eJ5Cs1RQKFygWuhu/mUpTFc+50=; b=cHuuHOArXWDvlW+OPQqp13NMQgTzrTQtk/vX9uJ2Dc8xeOZNt3WI/gFkeDVE/+TkMM spBQDtKrh6M3wJBTKYGcu/cHiOgk7+S5kcqgoECg5bgwEO0Hp0cyoORFBjYSnyjXro0m cyKyXOgjOliOP9dBlTE3iVQelaZQFCEezO4ze9Nfm8wY90YRYN28NjZ8B43PZqLCJV/w KwfI+ritT10UzobF7cLFSeGUF5NaBQpRuW2kNph8at+6gwguj6cNGYiNp39XHFNsE396 YuBcQsMCUMWqM8AIDtGMf9EBL6vldXrp1EYlmbPI3wXjqHDMmK46l7XfpB4jbmkj0nAe K5fg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a20si5314305pgb.421.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417146" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Rick Edgecombe Subject: [PATCH v4 10/23] x86/kprobes: Set instruction page as executable Date: Mon, 22 Apr 2019 11:57:52 -0700 Message-Id: <20190422185805.1169-11-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Set the page as executable after allocation. This patch is a preparatory patch for a following patch that makes module allocated pages non-executable. While at it, do some small cleanup of what appears to be unnecessary masking. Acked-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kprobes/core.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index a034cb808e7e..1591852d3ac4 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -431,8 +431,20 @@ void *alloc_insn_page(void) void *page; page = module_alloc(PAGE_SIZE); - if (page) - set_memory_ro((unsigned long)page & PAGE_MASK, 1); + if (!page) + return NULL; + + /* + * First make the page read-only, and only then make it executable to + * prevent it from being W+X in between. + */ + set_memory_ro((unsigned long)page, 1); + + /* + * TODO: Once additional kernel code protection mechanisms are set, ensure + * that the page was not maliciously altered and it is still zeroed. + */ + set_memory_x((unsigned long)page, 1); return page; } @@ -440,8 +452,12 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - set_memory_nx((unsigned long)page & PAGE_MASK, 1); - set_memory_rw((unsigned long)page & PAGE_MASK, 1); + /* + * First make the page non-executable, and only then make it writable to + * prevent it from being W+X in between. + */ + set_memory_nx((unsigned long)page, 1); + set_memory_rw((unsigned long)page, 1); module_memfree(page); } From patchwork Mon Apr 22 18:57:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911461 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48A331575 for ; Mon, 22 Apr 2019 19:00:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3DFFC28673 for ; Mon, 22 Apr 2019 19:00:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 31AA728680; Mon, 22 Apr 2019 19:00:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C0BB28673 for ; Mon, 22 Apr 2019 19:00:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9504A6B0003; Mon, 22 Apr 2019 15:00:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8FEAE6B027A; Mon, 22 Apr 2019 15:00:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 777CF6B027B; Mon, 22 Apr 2019 15:00:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 3C2CD6B0003 for ; Mon, 22 Apr 2019 15:00:05 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id v9so8452535pgg.8 for ; Mon, 22 Apr 2019 12:00:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=0/++kCIdz9NinK/+MFp07AkuSYwP6lr30tD8Xi+P1Ns=; b=jJ9KLHQYCX0dOGsRtkQX8WXDzGj0AdvSLEqf/Z5rErYDuUyDWnHsJwHmlYdBYrmmqM AlRA7rguEYwLusQml7r+CZD7xcsGM/K+oROSxV5N+DL/QPX22XPoBRgcUtdpejV6MzHf vHl3p7cbTmEaNwE3AK6y0lfyyk0NpcRzHABvNQgL+4WGyLdIa57mXxT5lchHeiGR4c4R knOCtXyyVYTAMS8Zd3pMOkPZFMz1Z4gUibmtO57NCGz2RLasD3qYL+vGUn2qsXQX2USc ouw5NZv2nNhT84OVZ+EL0aK37Egnrjfrxx+IV1gbViu3YaDOqEsUmRyN0n8yEfL9joKd Rjvg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXEfI1vwA6MBsW0Qgopvc+IB2DtYa/Do47yw94yB9JcoFbZ3IHM nxlWOC0LOCFZ0fKoYEVsFAOqcYKXfbYli+wuuu/eLkNdLZSPLJJQi2thua8PlsbbD6bNCeTQe6j LPrgAF4MIJNeYHszG3TfL6e1VmbsN06XYu3ITe0VXVydE8tBr+RmiGOImKGxDS9+wdg== X-Received: by 2002:a62:6402:: with SMTP id y2mr22864591pfb.194.1555959604888; Mon, 22 Apr 2019 12:00:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqxR5rbISH/ZjIU4RQFwpY6Sh1blp/ORIuJ8MRcb51/0jIVmtxYVtkYPlbaleA89/lB5SZGy X-Received: by 2002:a62:6402:: with SMTP id y2mr22858825pfb.194.1555959523682; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=CprnHrxlgi17BrxRkNaMeraS9rxzQeJaWpem5KZEFqhWF7Z12eaD2K77/9Mdo9lU2C bp+LWCsn0Vzzg8oLdKLL/bhNjOaTEH3+MdftL356KvexpKNMUJHj1d4dflYlGqVS7lYW 6/wnBDQ2kMDO5H537rLrj3qes2zx0wxBXms7vfyItoxVrdieu6cNuQzXBOp4b0izxcCx yFplFcjLZADn8PYCCL2uho/YOznEXDxqCuo74J9haudiIHGNFmOOFRfAiSYsLeu/RaJF K5g+VLmfVxyQTq8AwaB1XHi0Ec/WWD2vUhibmw/uZcBQ6UKS+9WwQEYWnqJwx4KlgFRV zYKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=0/++kCIdz9NinK/+MFp07AkuSYwP6lr30tD8Xi+P1Ns=; b=VIIEG8UUQGCJcTUGAyXV8O7UwNd/J8S43dMYtE/8UYV5az35RMGPZBzYZjQhK4h/ZK rVCxAmBFTbQYV0HSLR7sTi5bP8W0J1eaPUejW+QeE8iW+b1X32VJ1DKvwRruSTjzomS2 2zraFQp43GNbjm3ry/pvVcxQzhpOZw+glPYTF4ozQxAaBVfMkszK/qhql0G+PUyBRgKX Ff6q20fgcDtmKCNbncbTWtGEG2PcuzUyAd+NM40ZcHS3AR1DRqxvULg36oabzdRIlDl6 u4BjE/Bo7wqlqnI04TdmlLrA4U3RxoR8Q8PyvmrvCThdK/yiRzZdiMQLEx2mxUO3EtFH InZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id w15si615875pga.591.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417149" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Jessica Yu , Rick Edgecombe Subject: [PATCH v4 11/23] x86/module: Avoid breaking W^X while loading modules Date: Mon, 22 Apr 2019 11:57:53 -0700 Message-Id: <20190422185805.1169-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit When modules and BPF filters are loaded, there is a time window in which some memory is both writable and executable. An attacker that has already found another vulnerability (e.g., a dangling pointer) might be able to exploit this behavior to overwrite kernel code. Prevent having writable executable PTEs in this stage. In addition, avoiding having W+X mappings can also slightly simplify the patching of modules code on initialization (e.g., by alternatives and static-key), as would be done in the next patch. This was actually the main motivation for this patch. To avoid having W+X mappings, set them initially as RW (NX) and after they are set as RO set them as X as well. Setting them as executable is done as a separate step to avoid one core in which the old PTE is cached (hence writable), and another which sees the updated PTE (executable), which would break the W^X protection. Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Cc: Masami Hiramatsu Cc: Jessica Yu Suggested-by: Thomas Gleixner Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/alternative.c | 28 +++++++++++++++++++++------- arch/x86/kernel/module.c | 2 +- include/linux/filter.h | 1 + kernel/module.c | 5 +++++ 4 files changed, 28 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 599203876c32..3d2b6b6fb20c 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -668,15 +668,29 @@ void __init alternative_instructions(void) * handlers seeing an inconsistent instruction while you patch. */ void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) + size_t len) { unsigned long flags; - local_irq_save(flags); - memcpy(addr, opcode, len); - local_irq_restore(flags); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ + + if (boot_cpu_has(X86_FEATURE_NX) && + is_module_text_address((unsigned long)addr)) { + /* + * Modules text is marked initially as non-executable, so the + * code cannot be running and speculative code-fetches are + * prevented. Just change the code. + */ + memcpy(addr, opcode, len); + } else { + local_irq_save(flags); + memcpy(addr, opcode, len); + local_irq_restore(flags); + sync_core(); + + /* + * Could also do a CLFLUSH here to speed up CPU recovery; but + * that causes hangs on some VIA CPUs. + */ + } return addr; } diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b052e883dd8c..cfa3106faee4 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -87,7 +87,7 @@ void *module_alloc(unsigned long size) p = __vmalloc_node_range(size, MODULE_ALIGN, MODULES_VADDR + get_module_load_offset(), MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, + PAGE_KERNEL, 0, NUMA_NO_NODE, __builtin_return_address(0)); if (p && (kasan_module_alloc(p, size) < 0)) { vfree(p); diff --git a/include/linux/filter.h b/include/linux/filter.h index 6074aa064b54..14ec3bdad9a9 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -746,6 +746,7 @@ static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { set_memory_ro((unsigned long)hdr, hdr->pages); + set_memory_x((unsigned long)hdr, hdr->pages); } static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) diff --git a/kernel/module.c b/kernel/module.c index 0b9aa8ab89f0..2b2845ae983e 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -1950,8 +1950,13 @@ void module_enable_ro(const struct module *mod, bool after_init) return; frob_text(&mod->core_layout, set_memory_ro); + frob_text(&mod->core_layout, set_memory_x); + frob_rodata(&mod->core_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_x); + frob_rodata(&mod->init_layout, set_memory_ro); if (after_init) From patchwork Mon Apr 22 18:57:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911401 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 623AC1575 for ; Mon, 22 Apr 2019 18:59:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 515EF28759 for ; Mon, 22 Apr 2019 18:59:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4129C28764; Mon, 22 Apr 2019 18:59:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC00828759 for ; Mon, 22 Apr 2019 18:59:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2B976B0010; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 756796B000E; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B86C6B0008; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id D737D6B000E for ; Mon, 22 Apr 2019 14:58:44 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id s19so8829718plp.6 for ; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=g3nO4xFM3yb0X+OIqTEtEp6LbffMj61RqStKCNeSSiM=; b=HcoQJ0mc7iK73K1XktxPfMTMGnlNQhja5Dc0xRvYH/L4CjWHXU1yTylFS7jkcJylmi L7cWW48NND8MuW8JMA1zvbfnCzMBQ/quYf4KE14ZWh6AaHPBC/g0P4Z9D6++fSm3HLwt b/f4tFzTGPYqqyzI8MuVhJrhxAsHc08XGGfTIgWpnWditG8QMHlh4XqjBhjrSXlusfGC vstpAgShNZPODEmQs7FsRShN6BzLP85f+msZ2VfobS1MMNjPSRrbA2titrPMFsvvjWNz l7EoEqigqOCG2cwN5CUljU7/gv+g4TD5mKSK03nrN4KWAXF1k3sAXM7qK6GbL4QgoX8d 4ZjQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXoTgzEfiI1ZWOGFdW4xhQUKC589xd5qmgYicxqHmmJdeTE4/f6 ocLHHIRo8KjNOogXrmziRGVlQ+vBQsHD/TBCnwprLhoSIJMNsrLlUkkhlT5uDYrL/HhqHW5LSsQ dhdzZjFWb9sy/ddMq0jMVRiC9M0BWfUs1TDjd1RdyBKrr/koPStWPFYqDiry1Q1wrNw== X-Received: by 2002:a17:902:70c8:: with SMTP id l8mr22150509plt.177.1555959524529; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqweBJHlpFdyv6pmg3+0kLdzjylHl9yi+W4MfJcHSKm2iPxhPz6OKqoJfwB0HEDBtPNElzlh X-Received: by 2002:a17:902:70c8:: with SMTP id l8mr22150454plt.177.1555959523666; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=luKyJ+uYhdSl+YmonvGd/wADQiCVckVJS2N3Jz75Xy1We7EiMWxkObZLpQTOa+JTvq fU96pKXj0nELGLYE7Uh+nFs21wWIIxO/QIpfAqkNb+tlybzXufRXA6hWcGU4ETgho7D5 qT33bDMo520IzPH6Ym6vBsOm0LwwckMYkZAVkbI1feOlVIjNqwzDpC1QkEQbQeNrutE2 AyJoZIFLM71m9RsoETwNbfVjoKNhJEUqpnDBPqqZus88SFb1cmxJCEqu5/CsIzC8gdOZ mjcDWxqykvzn3YeWf0jWZ3XA3jH86A0VDvGpH+Z9amL6xx2L5zAApOQdxrnD7QSBiZ48 Ts1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=g3nO4xFM3yb0X+OIqTEtEp6LbffMj61RqStKCNeSSiM=; b=mlEkCcGoaHMwWGSYW0Z9EsrgwHEAr9ylzdz8wY4qWcWWR0cCRznayxnf74FkylQCZq Hd/tLzAbTE6HYVH05vkIQTTtK8f26famPpOnewlHEU/iJc9/zypWtU5SdNR6Hu1gkTu6 R2VgQWFrrPulSck9onsVwTEt5tvnjJKqakmMAdSDG551TvPmSVohP45WZAZOjXuUGhWZ YIj/KDSl80SaLGOSQeOzvFNM/FhGhXbQY6ccW/tiyW/TXXTKsjISwDLDIBhSgjPo3bfh AI3PiJyr32Jr5H0IH9AOFlsAnwFZVXB6nBfuz2tAR/j+is6ao+BCfpeFK2YcLH0XqIqt njuQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417152" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH v4 12/23] x86/jump-label: Remove support for custom poker Date: Mon, 22 Apr 2019 11:57:54 -0700 Message-Id: <20190422185805.1169-13-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit There are only two types of poking: early and breakpoint based. The use of a function pointer to perform poking complicates the code and is probably inefficient due to the use of indirect branches. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e7d8c636b228..e631c358f7f4 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -37,7 +37,6 @@ static void bug_at(unsigned char *ip, int line) static void __ref __jump_label_transform(struct jump_entry *entry, enum jump_label_type type, - void *(*poker)(void *, const void *, size_t), int init) { union jump_code_union jmp; @@ -50,14 +49,6 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - /* - * As long as only a single processor is running and the code is still - * not marked as RO, text_poke_early() can be used; Checking that - * system_state is SYSTEM_BOOTING guarantees it. - */ - if (system_state == SYSTEM_BOOTING) - poker = text_poke_early; - if (type == JUMP_LABEL_JMP) { if (init) { expect = default_nop; line = __LINE__; @@ -80,16 +71,19 @@ static void __ref __jump_label_transform(struct jump_entry *entry, bug_at((void *)jump_entry_code(entry), line); /* - * Make text_poke_bp() a default fallback poker. + * As long as only a single processor is running and the code is still + * not marked as RO, text_poke_early() can be used; Checking that + * system_state is SYSTEM_BOOTING guarantees it. It will be set to + * SYSTEM_SCHEDULING before other cores are awaken and before the + * code is write-protected. * * At the time the change is being done, just ignore whether we * are doing nop -> jump or jump -> nop transition, and assume * always nop being the 'currently valid' instruction - * */ - if (poker) { - (*poker)((void *)jump_entry_code(entry), code, - JUMP_LABEL_NOP_SIZE); + if (init || system_state == SYSTEM_BOOTING) { + text_poke_early((void *)jump_entry_code(entry), code, + JUMP_LABEL_NOP_SIZE); return; } @@ -101,7 +95,7 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { mutex_lock(&text_mutex); - __jump_label_transform(entry, type, NULL, 0); + __jump_label_transform(entry, type, 0); mutex_unlock(&text_mutex); } @@ -131,5 +125,5 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, jlstate = JL_STATE_NO_UPDATE; } if (jlstate == JL_STATE_UPDATE) - __jump_label_transform(entry, type, text_poke_early, 1); + __jump_label_transform(entry, type, 1); } From patchwork Mon Apr 22 18:57:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911473 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2A6351575 for ; Mon, 22 Apr 2019 19:00:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1FC18286B0 for ; Mon, 22 Apr 2019 19:00:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 13381287A6; Mon, 22 Apr 2019 19:00:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8976B286B0 for ; Mon, 22 Apr 2019 19:00:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B5EC6B027E; Mon, 22 Apr 2019 15:00:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 68C736B0280; Mon, 22 Apr 2019 15:00:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 557E06B0281; Mon, 22 Apr 2019 15:00:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 1BF166B027E for ; Mon, 22 Apr 2019 15:00:25 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id l13so8487575pgp.3 for ; Mon, 22 Apr 2019 12:00:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=RkWBxdSztj8rR+UGwRQ8u46DAnXrPhDoGJsRNKYizGU=; b=VdASqhthpj4VS6NLcapUoEyp6tgrMpSthZKYjHgccMx1zs9VxlLXx+ZtyZfErWCiPZ Zj9CIoQlim6lkEIZTkb5HCMfleMIqRdlKc9eBsp2D3kuZ1/fIw4EPGK/XDX97i8ibRf1 ZIe+Pw3H/HxPehkDQCBurgJl8yasfNP94Y8dH0Gnu7qQx6ttPUZBBIezk3zwIYChNQb8 Apve4+60o0M9E9247DIcpj7HFPaGtobU6YcvMRxEU2yZJ1cmTCjsAx40yem4JJkvZG6i 1qPn94kTlFKrIMDdZCFThdL0Rrxbhrp7OK18VYmJwF/lw9Odt+Zde9b0KEYctKc2HodN AL+A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWZBURu8KrZuhueSTwezR7ECwE4BpdFKZDeUU3kqv8Fl9ROlvCB GSDbOfMTX/y4J7PXvV15I6YRKvpe5O8uE4lF6bnm+sxdFjlU/Rp5jlDDy/9/o93fu8vlkO9HCet 8xTIknHF7/4H3b+UOxXeXfdAnDePrcnl1zez93ambshftJv9zwx336W2jTLEwTdhrHA== X-Received: by 2002:a63:5c43:: with SMTP id n3mr20180971pgm.163.1555959624770; Mon, 22 Apr 2019 12:00:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqyq8i+iaBrpWDs/gLIxeVTK0hYKvXj0Xhz7MZttV//COV7qyNeh56KMj/Wx0yyU4gX+Xnv2 X-Received: by 2002:a63:5c43:: with SMTP id n3mr20174252pgm.163.1555959523841; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=bv+/fk5bAlNhlK/W930uUylm+AgOV8X+5x7p+fHGO06/5erbxtEG6vufdHSS2pcpeT Q/wGFhlhIdMNPdpezQdQ4V0n4RNdilkiDKEyowqyw8QTDSEDsBf8LWDBfpe17ozuC74B E8wAV0/X3vhJnG1YzYxUNHCfe/rMy6V8FiqZdxCF9wXjJHFZL4+BEqV7SM9Z0iKtwMOi PfQx+B5lcZcpShfFOf21arltXBhicgs0f4rbp7Sd/wIowwTKGOd33jtN3ggClmw1r/RM tFyF8v/s5x6CcIDYwQd/3CrkkNNPNZKlmnm3D0zoJ+4ogKN3ybeZOquzVsri9VkkLsDt +pfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=RkWBxdSztj8rR+UGwRQ8u46DAnXrPhDoGJsRNKYizGU=; b=TVSEstu7BVfrf3xqhJDJhG7L0Q5E9CSNKaXl90jUedpMog2Mze/q4UdTd2OW05UgWL RRaGq76qWImsem3ZHNGpZMDMk/s7p4bwJnHz7nrJZtrZPFZQV84tHYGvr96Pf+nFwIuA f/SWew0vnmAVJqHZxk0ziAX905CDMbquK4yOzu09v6QBpRzLgn+VCNqLEfP4ild+dK2t oLW69Mv+3MhmW9MW13VDCDX5avy+1qU2Xq3PKPUjl2b8gb82xkF8vi3OgfvFskstC4p1 +/ZnSlwQxXg/DGScnaV1aV/No1/T2RI3/m+eV4At6ZuBbksxs7o+IMGWcWeAMfbwhLpT Xekw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a20si5314305pgb.421.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417155" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:41 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH v4 13/23] x86/alternative: Remove the return value of text_poke_*() Date: Mon, 22 Apr 2019 11:57:55 -0700 Message-Id: <20190422185805.1169-14-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit The return value of text_poke_early() and text_poke_bp() is useless. Remove it. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/text-patching.h | 4 ++-- arch/x86/kernel/alternative.c | 11 ++++------- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index a75eed841eed..c90678fd391a 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -18,7 +18,7 @@ static inline void apply_paravirt(struct paravirt_patch_site *start, #define __parainstructions_end NULL #endif -extern void *text_poke_early(void *addr, const void *opcode, size_t len); +extern void text_poke_early(void *addr, const void *opcode, size_t len); /* * Clear and restore the kernel write-protection flag on the local CPU. @@ -37,7 +37,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); extern void *text_poke(void *addr, const void *opcode, size_t len); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); -extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); +extern void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; extern __ro_after_init struct mm_struct *poking_mm; extern __ro_after_init unsigned long poking_addr; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 3d2b6b6fb20c..18f959975ea0 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -265,7 +265,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len) extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; -void *text_poke_early(void *addr, const void *opcode, size_t len); +void text_poke_early(void *addr, const void *opcode, size_t len); /* * Are we looking at a near JMP with a 1 or 4-byte displacement. @@ -667,8 +667,8 @@ void __init alternative_instructions(void) * instructions. And on the local CPU you need to be protected again NMI or MCE * handlers seeing an inconsistent instruction while you patch. */ -void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) +void __init_or_module text_poke_early(void *addr, const void *opcode, + size_t len) { unsigned long flags; @@ -691,7 +691,6 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, * that causes hangs on some VIA CPUs. */ } - return addr; } __ro_after_init struct mm_struct *poking_mm; @@ -893,7 +892,7 @@ NOKPROBE_SYMBOL(poke_int3_handler); * replacing opcode * - sync cores */ -void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) +void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) { unsigned char int3 = 0xcc; @@ -935,7 +934,5 @@ void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) * the writing of the new instruction. */ bp_patching_in_progress = false; - - return addr; } From patchwork Mon Apr 22 18:57:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911405 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1AF81575 for ; Mon, 22 Apr 2019 18:59:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9708428759 for ; Mon, 22 Apr 2019 18:59:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B12928764; Mon, 22 Apr 2019 18:59:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00C0628759 for ; Mon, 22 Apr 2019 18:59:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9F906B000E; Mon, 22 Apr 2019 14:58:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B03CD6B026A; Mon, 22 Apr 2019 14:58:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F0E56B026B; Mon, 22 Apr 2019 14:58:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 2F44F6B000E for ; Mon, 22 Apr 2019 14:58:46 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id x2so8448365pge.16 for ; Mon, 22 Apr 2019 11:58:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=WY0+L0pIgfyvCUmzFdIy1LTc5Xw09u0bbyF0A1uVjso=; b=YTpz0NgTceX30YxmCZIcrkr8j0A2vSLMrkX3NzkCH9nNra8h91vWIafkbXKx2prJVj cM8FzKquaEVATZsZsWQfSXQQd+mxIa/72nzJfDArk9Bs67AeGlw9/shFTxG3sKUk8no0 GoYCs5lrVZFGrjgcHn8pLbPJSbhSz1G57STNov2UXnSFnOFTUyABzzkE+H2vnwAfjbru AyRw7AufAe/uUY2wWlDMNILuaVGzysaoOrarb9c6ByoFZ26oDMcwVgfm5fJEHK4wmjLA kbDCqlN1wPuZ8U0rTRDdezgfjd6NF6E10PaSpH2Yw6SBERkT1PQqncQWrwBHJyvbohpk tudw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWj/ojfaNaPiSup6wj833VbUdkgMMR0AQ4BwhAimSGX5LzDTJi+ JN7wrRAkdI8A0S9jAZds2R67KFlEuzJtA6l8r/42KwgK/Z6T96JW1jm1yI+dJBMaDzphMQ3ncEg GHYQWXiMBMk3EaPYVexxVrv0lynjdBYLY4ntmYl1AovAuIT3gi6L1dy+PeR0MOVu3IA== X-Received: by 2002:a65:5206:: with SMTP id o6mr4088765pgp.341.1555959525792; Mon, 22 Apr 2019 11:58:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqyi1UFK2Go08wsVw0XGCnfVfVSDpGj8548I8nC8RZvFhiK9bSmqRlAXfhRXNbM7EWXE5MCc X-Received: by 2002:a65:5206:: with SMTP id o6mr4088661pgp.341.1555959523923; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=COeBenfkR2Stq0kLiZH3VPoI7PZuZNZv+Ver0koeMAiov41TxzESVq/e9adCQFLuPU tn67GNxv9I/FiYlnmEm0qn9TQlUDCb8n6AyQUC5mI7xRNi3LCEXsM5F+yWqwzxyFUOEM 5Asv2iB1Ojpcs3Ii1qKCMJr1y27mjrfZX/GnqJGOdb6Vcz/Vdf30pkkfriaZ81080UkD 6FnYV8RuM0LX87LOECUQbVaniXzCplrO68+Xc8whzbE0561Hn0AbW6KH7rjU/fnG67vm Lw+7iIvNSlPwn8Ueoc9Vto+Ovv16lWCyPTMuyulfHN2y216Is0aG+lcCxPY5Z0Q9kzJS G98A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=WY0+L0pIgfyvCUmzFdIy1LTc5Xw09u0bbyF0A1uVjso=; b=xe48gWxzX+Cc2dpjhvUbUm3xGQUhQrAWCh6yI3+2Hu5yvVOYiAp/wEiMjhdmviUh5E LfvpNS9yHbyf15zQqK4/5/H2Eh4BOyr9ed8GIZFVutpNMKrj29XqCCD/VCXIJIOSFhp4 AXXRKZ7XEc9gP06xdPy3or52suw/kyHHwypPTQScoOjdIJZf9siV3P6q0gnnhGtNLlLZ UnDiTuqcNY9gnNxhx1wswvSngYNmdOQUIlRNl/7PVAHEB2s0mq+27h4DwC+TAUp4H1DU qYyC0qymJ+0qyVBpYcVtELYxk5Z+4NUrbsOKk9Mwh6YG1mnIJO1WEtnqIO6BUkcLhVfu 4Ptw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417158" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v4 14/23] x86/mm/cpa: Add set_direct_map_ functions Date: Mon, 22 Apr 2019 11:57:56 -0700 Message-Id: <20190422185805.1169-15-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add two new functions set_direct_map_default_noflush() and set_direct_map_invalid_noflush() for setting the direct map alias for the page to its default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions do not flush the TLB. Note, __kernel_map_pages() does something similar but flushes the TLB and doesn't reset the permission bits to default on all architectures. Also add an ARCH config ARCH_HAS_SET_DIRECT_MAP for specifying whether these have an actual implementation or a default empty one. Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Signed-off-by: Rick Edgecombe --- arch/Kconfig | 4 ++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/set_memory.h | 3 +++ arch/x86/mm/pageattr.c | 14 +++++++++++--- include/linux/set_memory.h | 11 +++++++++++ 5 files changed, 30 insertions(+), 3 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 3ab446bd12ef..5e43fcbad4ca 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -249,6 +249,10 @@ config ARCH_HAS_FORTIFY_SOURCE config ARCH_HAS_SET_MEMORY bool +# Select if arch has all set_direct_map_invalid/default() functions +config ARCH_HAS_SET_DIRECT_MAP + bool + # Select if arch init_task must go in the __init_task_data section config ARCH_TASK_STRUCT_ON_STACK bool diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2ec5e850b807..45d788354376 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -66,6 +66,7 @@ config X86 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_UACCESS_MCSAFE if X86_64 && X86_MCE select ARCH_HAS_SET_MEMORY + select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h index 07a25753e85c..ae7b909dc242 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -85,6 +85,9 @@ int set_pages_nx(struct page *page, int numpages); int set_pages_ro(struct page *page, int numpages); int set_pages_rw(struct page *page, int numpages); +int set_direct_map_invalid_noflush(struct page *page); +int set_direct_map_default_noflush(struct page *page); + extern int kernel_set_to_readonly; void set_kernel_text_rw(void); void set_kernel_text_ro(void); diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 4c570612e24e..3574550192c6 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2209,8 +2209,6 @@ int set_pages_rw(struct page *page, int numpages) return set_memory_rw(addr, numpages); } -#ifdef CONFIG_DEBUG_PAGEALLOC - static int __set_pages_p(struct page *page, int numpages) { unsigned long tempaddr = (unsigned long) page_address(page); @@ -2249,6 +2247,17 @@ static int __set_pages_np(struct page *page, int numpages) return __change_page_attr_set_clr(&cpa, 0); } +int set_direct_map_invalid_noflush(struct page *page) +{ + return __set_pages_np(page, 1); +} + +int set_direct_map_default_noflush(struct page *page) +{ + return __set_pages_p(page, 1); +} + +#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2282,7 +2291,6 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) } #ifdef CONFIG_HIBERNATION - bool kernel_page_present(struct page *page) { unsigned int level; diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 2a986d282a97..b5071497b8cb 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -17,6 +17,17 @@ static inline int set_memory_x(unsigned long addr, int numpages) { return 0; } static inline int set_memory_nx(unsigned long addr, int numpages) { return 0; } #endif +#ifndef CONFIG_ARCH_HAS_SET_DIRECT_MAP +static inline int set_direct_map_invalid_noflush(struct page *page) +{ + return 0; +} +static inline int set_direct_map_default_noflush(struct page *page) +{ + return 0; +} +#endif + #ifndef set_mce_nospec static inline int set_mce_nospec(unsigned long pfn) { From patchwork Mon Apr 22 18:57:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911495 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 00BA81708 for ; Mon, 22 Apr 2019 19:00:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB596286B0 for ; Mon, 22 Apr 2019 19:00:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DF0AD287C6; Mon, 22 Apr 2019 19:00:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CE4328793 for ; Mon, 22 Apr 2019 19:00:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA4CB6B0285; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E558A6B0288; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D45BF6B0289; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 98D396B0285 for ; Mon, 22 Apr 2019 15:00:39 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id ba11so674527plb.21 for ; Mon, 22 Apr 2019 12:00:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=6pDLV47YuMug94kghJNMRU0wCJe8grVkxn5O7bN8YZE=; b=e2eD5iQ0Iej3aIeIrPGIAuc5WKeiF9vkziyQLkjGMhviLeChhKqWXD2zcYfe0ctgwm dL9aJ5hgIg0K8pzdFADqmhWRjUHuSKVIhN3xVGJxSNhbj2il7uPZYVrG/MzFkzc0Zykr KeJt/zS0FuV/fcfcCUQCs0kJWfgALq8n0YcfRRf5GwVYaGFCDu5QY3DYwKYgu14ep4Cm 9Qs08RmmN5T/CJsEbRoQzsYglEjstZYyvFw7x74w8zHB1plOFWXO97EZL+55gW7y0muc e4bDETjAwGpThjViTa3Dj4PuObYE0JzMBtbHRVE3TfiiLJBNg3GS4K2fCghGWCt5VdIj q1cg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWL+6xT+W5XANFSQg5JWcdTmXmaqHRYzngHltmb/n0t/Cwb+I1V a2MoWbE+sADDUX8GX6c2Iia2/KtNm1yMx/UyQ+6Nq1gALVIL31rpy6MitYJPTr60jkJjViYrifC sYq9KYNs36be/iBF2fKKAT/6X4wV00a2vIt9sPPF3yQfACGDopAR9VsH6pDZ77D1JdA== X-Received: by 2002:a62:b418:: with SMTP id h24mr21753604pfn.145.1555959639273; Mon, 22 Apr 2019 12:00:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqxjMSUn8Ju0Q5n6PWgVvWOnQ+sNQS1MeDYEf1boOgg6PaWuGsbtsSs8/ytRMv7ZlLjPXCs9 X-Received: by 2002:a62:b418:: with SMTP id h24mr21744436pfn.145.1555959523908; Mon, 22 Apr 2019 11:58:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959523; cv=none; d=google.com; s=arc-20160816; b=Gr4S5GLUtIdlh0G8UcbUzEMhpeJgDmwIK0O1GqxtT2lTCnMDm0UqEOon2a5sNi2rau ZJkAkW+PM2ksh/t4cYjUe77YPRnNY7e5XVC08SYN1doMloMdFG87WB3vFA6ImdyzKIAw SD5eoi7zRLF01lfu1bkLpV/ZqRKubmshOtm3J/G9wJ06dmi+mvqZpyxjGkCPGOQ7WTtK Q60GWmYZ5IVheuEuZKY7egulb0VTUMQm7jXep2nWFERi+/6FcU0NqyMUXdFRMScz0U/k /wHLuSPOK3WJyr3bmy27/ZLvFKsjYnWFYxumZoiZikx82f1TY/fcItAE5cDnjJDd417T +gjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=6pDLV47YuMug94kghJNMRU0wCJe8grVkxn5O7bN8YZE=; b=FoUQrYC836EEPYkDL42AiE9S9DEUUN9b3yUDCoVa5gK4jn3zUFmuL8yi2Vblyojwnt cVnYn1cucw2p/5+2iENfoKa2ThDFuwTbXjQ1N/CP37lKE/uQFam0kLARc8DguLFR/OJt nJKAZj6VRzZQY/zFAzekeMfVcis8gDEBZG2Qj3qc4CLUQOGLea1bkPqw7m8wHrLzSUMv L8SH0AO+nu0oAADEQUYfHLB1rM68vqHKccr47pLiAp1jJIKxqx60uUkaaeDbs14u7fBX eQTccvKNFFVIK5cD5Jl8NymKIMdiF0NjxS5Ew/+IDjAiLw10dvhx3Se3E208dIfI1kqy YihA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id w15si615875pga.591.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:43 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417161" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , "Rafael J. Wysocki" , Pavel Machek Subject: [PATCH v4 15/23] mm: Make hibernate handle unmapped pages Date: Mon, 22 Apr 2019 11:57:57 -0700 Message-Id: <20190422185805.1169-16-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Make hibernate handle unmapped pages on the direct map when CONFIG_ARCH_HAS_SET_ALIAS is set. These functions allow for setting pages to invalid configurations, so now hibernate should check if the pages have valid mappings and handle if they are unmapped when doing a hibernate save operation. Previously this checking was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact. The speed of the saving operation before this change was measured as 819.02 MB/s, and after was measured at 813.32 MB/s. Before: [ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s) After: [ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s) Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Borislav Petkov Acked-by: Pavel Machek Signed-off-by: Rick Edgecombe --- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ kernel/power/snapshot.c | 5 +++-- mm/page_alloc.c | 7 +++++-- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 3574550192c6..daf4d645e537 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2257,7 +2257,6 @@ int set_direct_map_default_noflush(struct page *page) return __set_pages_p(page, 1); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte = lookup_address((unsigned long)page_address(page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } - #endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ - int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, unsigned numpages, unsigned long page_flags) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 6b10c21630f5..083d7b4863ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,37 +2610,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages(struct page *page, int numpages, int enable); static inline bool debug_pagealloc_enabled(void) { - return _debug_pagealloc_enabled; + return IS_ENABLED(CONFIG_DEBUG_PAGEALLOC) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_ARCH_HAS_SET_DIRECT_MAP) +extern void __kernel_map_pages(struct page *page, int numpages, int enable); + static inline void kernel_map_pages(struct page *page, int numpages, int enable) { - if (!debug_pagealloc_enabled()) - return; - __kernel_map_pages(page, numpages, enable); } #ifdef CONFIG_HIBERNATION extern bool kernel_page_present(struct page *page); #endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ static inline void kernel_map_pages(struct page *page, int numpages, int enable) {} #ifdef CONFIG_HIBERNATION static inline bool kernel_page_present(struct page *page) { return true; } #endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled(void) -{ - return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ #ifdef __HAVE_ARCH_GATE_AREA extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index f08a1e4ee1d4..bc9558ab1e5b 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1342,8 +1342,9 @@ static inline void do_copy_page(long *dst, long *src) * safe_copy_page - Copy a page in a safe way. * * Check if the page we are going to copy is marked as present in the kernel - * page tables (this always is the case if CONFIG_DEBUG_PAGEALLOC is not set - * and in that case kernel_page_present() always returns 'true'). + * page tables. This always is the case if CONFIG_DEBUG_PAGEALLOC or + * CONFIG_ARCH_HAS_SET_DIRECT_MAP is not set. In that case kernel_page_present() + * always returns 'true'. */ static void safe_copy_page(void *dst, struct page *s_page) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d96ca5bc555b..34a70681a4af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1131,7 +1131,9 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); - kernel_map_pages(page, 1 << order, 0); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 0); + kasan_free_nondeferred_pages(page, order); return true; @@ -2001,7 +2003,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_refcounted(page); arch_alloc_page(page, order); - kernel_map_pages(page, 1 << order, 1); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); kernel_poison_pages(page, 1 << order, 1); set_page_owner(page, order, gfp_flags); From patchwork Mon Apr 22 18:57:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911501 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 513D1161F for ; Mon, 22 Apr 2019 19:00:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 45D7B286B0 for ; Mon, 22 Apr 2019 19:00:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39E98287A6; Mon, 22 Apr 2019 19:00:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6FE78286B0 for ; Mon, 22 Apr 2019 19:00:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 382B36B0288; Mon, 22 Apr 2019 15:00:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 35F076B028A; Mon, 22 Apr 2019 15:00:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 110876B028B; Mon, 22 Apr 2019 15:00:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id BB9346B0288 for ; Mon, 22 Apr 2019 15:00:40 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id u191so4240130pgc.0 for ; Mon, 22 Apr 2019 12:00:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=J/MHWg9P2bFxy449D6LfpA34veClVhAHFrMEACKVEKQ=; b=cGKkOfr4k6sIDmNnVF1cWv8V036u+FHyWprn8pyrVPM0XAkc6Fjsq8u2da3nItmiem snuTxUUjy2Gl7Xu2E5uf8zEC5M+aVjfSUkb8hMEeW1odvo4Tu8M1j0vIp25RmuCpRszV NYCWf7D9W7EuBETVZGQNRV+pY0q2eND77tFFw39bfcm5h4HAak+1ZYy/mAbz05zwoyp5 qj1rdv+36xStJ1aVcr8ACt1OajVi+c120xnmLuX/PiuHbjcbgeITozjJtcrbdJc6ZvR0 4PmugR6Qk0q8MWGuiK2VLEVthmSNNOAUO7bL2oiHm+q17VReo/SIC/pd67O7WefUCJ2H hNHw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVUE4yyvKLyMN8Ck6iVM237KK+sU31lguhhNftTStRLrBfBW5L0 Wbue5CRDJEGVOcDjr1JFwyCNGNLlsGFS9lqpZ7oYYQXnTOayfo/Vp83EIZD3YNyinewohonLjdw Uksu7vNDcMXNFfTu8Lp736iIa4HRasxeYSj9Neg1b4WJLndentyIoNyRp03Vq3rJnHw== X-Received: by 2002:a63:d04b:: with SMTP id s11mr543375pgi.137.1555959640403; Mon, 22 Apr 2019 12:00:40 -0700 (PDT) X-Google-Smtp-Source: APXvYqxxx0HAXuIu+BGxo+dbnTjKKleanyuE3SSyLzpk8Rjpw2tf5gzkN7+ECE2zge0CYgu3jeeQ X-Received: by 2002:a63:d04b:: with SMTP id s11mr535019pgi.137.1555959524152; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=1HtYURjTvetwS9dWPNZ51iR8WRU/Ju7Vuqv4xbCfmngD54tmabX8VxLftX+FD7uNER xUkgbpMMyCZL2CcG9beoie+V6nFotCFHFhpbFmCpv+G8cW8B/IblTTK2JYwhsaBwtYLJ 87tfIVxHVz0n6Zx8PWZr3+e2jKRjmnoKGuI1CyeFCR2m0wGAs2f503lspgYtkt1SdzsJ 8MAea5SmWtu0ZWgSKzgXRwkq4lCde1DVfDYu39F3eIKP7t4Gnl1XrJ2ybklJHQISyHbl g2CDGUYGPZKgsJdeu2yjLM7gDu2iwuxnhlEKBIscc/aF5aImW0C5prkK2METoRohZb7c 5iMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=J/MHWg9P2bFxy449D6LfpA34veClVhAHFrMEACKVEKQ=; b=YZ20R2XnpasoD4b5tPOYXx8Oux8vnTmm5gVHToyPMuPgIapxZMPKSIOK9gR+rDuJCT ZMnRDXkIcoJSoNC8cg/CtlCb+o9agg8G3mnXiiHPZVFy1d8xPljj7G12JvjlcVZ798DK qbFVUtc0UGVnJuf+UAbrzGWJqrERr4HgAvYRCSM829Om2okE5F44y3NJ2MAR1GGC/Ysx nm/1LeRjcA8OKnnDt8CFGC0bqdDq0RPzE5vWywK8Syoz9+aCZeDNwz+nVKzXE9H2LYl4 tl7hxQX33wuDgVTf/wTCJDal9SGxA+8OIq1PRTCtSuA8veEZofdzdYkyoDvS8BC7HjvQ 8tog== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a20si5314305pgb.421.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417165" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v4 16/23] vmalloc: Add flag for free of special permsissions Date: Mon, 22 Apr 2019 11:57:58 -0700 Message-Id: <20190422185805.1169-17-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add a new flag VM_FLUSH_RESET_PERMS, for enabling vfree operations to immediately clear executable TLB entries before freeing pages, and handle resetting permissions on the directmap. This flag is useful for any kind of memory with elevated permissions, or where there can be related permissions changes on the directmap. Today this is RO+X and RO memory. Although this enables directly vfreeing non-writeable memory now, non-writable memory cannot be freed in an interrupt because the allocation itself is used as a node on deferred free list. So when RO memory needs to be freed in an interrupt the code doing the vfree needs to have its own work queue, as was the case before the deferred vfree list was added to vmalloc. For architectures with set_direct_map_ implementations this whole operation can be done with one TLB flush when centralized like this. For others with directmap permissions, currently only arm64, a backup method using set_memory functions is used to reset the directmap. When arm64 adds set_direct_map_ functions, this backup can be removed. When the TLB is flushed to both remove TLB entries for the vmalloc range mapping and the direct map permissions, the lazy purge operation could be done to try to save a TLB flush later. However today vm_unmap_aliases could flush a TLB range that does not include the directmap. So a helper is added with extra parameters that can allow both the vmalloc address and the direct mapping to be flushed during this operation. The behavior of the normal vm_unmap_aliases function is unchanged. Cc: Borislav Petkov Suggested-by: Dave Hansen Suggested-by: Andy Lutomirski Suggested-by: Will Deacon Signed-off-by: Rick Edgecombe --- include/linux/vmalloc.h | 15 ++++++ mm/vmalloc.c | 113 +++++++++++++++++++++++++++++++++------- 2 files changed, 109 insertions(+), 19 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 398e9c95cd61..c6eebb839552 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -21,6 +21,11 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +/* + * Memory with VM_FLUSH_RESET_PERMS cannot be freed in an interrupt or with + * vfree_atomic(). + */ +#define VM_FLUSH_RESET_PERMS 0x00000100 /* Reset direct map and flush TLB on unmap */ /* bits [20..32] reserved for arch specific ioremap internals */ /* @@ -142,6 +147,13 @@ extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); extern void unmap_kernel_range_noflush(unsigned long addr, unsigned long size); extern void unmap_kernel_range(unsigned long addr, unsigned long size); +static inline void set_vm_flush_reset_perms(void *addr) +{ + struct vm_struct *vm = find_vm_area(addr); + + if (vm) + vm->flags |= VM_FLUSH_RESET_PERMS; +} #else static inline int map_kernel_range_noflush(unsigned long start, unsigned long size, @@ -157,6 +169,9 @@ static inline void unmap_kernel_range(unsigned long addr, unsigned long size) { } +static inline void set_vm_flush_reset_perms(void *addr) +{ +} #endif /* Allocate/destroy a 'vmalloc' VM area. */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e86ba6e74b50..e5e9e1fcac01 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -1059,24 +1060,9 @@ static void vb_free(const void *addr, unsigned long size) spin_unlock(&vb->lock); } -/** - * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer - * - * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily - * to amortize TLB flushing overheads. What this means is that any page you - * have now, may, in a former life, have been mapped into kernel virtual - * address by the vmap layer and so there might be some CPUs with TLB entries - * still referencing that page (additional to the regular 1:1 kernel mapping). - * - * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can - * be sure that none of the pages we have control over will have any aliases - * from the vmap layer. - */ -void vm_unmap_aliases(void) +static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) { - unsigned long start = ULONG_MAX, end = 0; int cpu; - int flush = 0; if (unlikely(!vmap_initialized)) return; @@ -1113,6 +1099,27 @@ void vm_unmap_aliases(void) flush_tlb_kernel_range(start, end); mutex_unlock(&vmap_purge_lock); } + +/** + * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer + * + * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily + * to amortize TLB flushing overheads. What this means is that any page you + * have now, may, in a former life, have been mapped into kernel virtual + * address by the vmap layer and so there might be some CPUs with TLB entries + * still referencing that page (additional to the regular 1:1 kernel mapping). + * + * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can + * be sure that none of the pages we have control over will have any aliases + * from the vmap layer. + */ +void vm_unmap_aliases(void) +{ + unsigned long start = ULONG_MAX, end = 0; + int flush = 0; + + _vm_unmap_aliases(start, end, flush); +} EXPORT_SYMBOL_GPL(vm_unmap_aliases); /** @@ -1505,6 +1512,72 @@ struct vm_struct *remove_vm_area(const void *addr) return NULL; } +static inline void set_area_direct_map(const struct vm_struct *area, + int (*set_direct_map)(struct page *page)) +{ + int i; + + for (i = 0; i < area->nr_pages; i++) + if (page_address(area->pages[i])) + set_direct_map(area->pages[i]); +} + +/* Handle removing and resetting vm mappings related to the vm_struct. */ +static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) +{ + unsigned long addr = (unsigned long)area->addr; + unsigned long start = ULONG_MAX, end = 0; + int flush_reset = area->flags & VM_FLUSH_RESET_PERMS; + int i; + + /* + * The below block can be removed when all architectures that have + * direct map permissions also have set_direct_map_() implementations. + * This is concerned with resetting the direct map any an vm alias with + * execute permissions, without leaving a RW+X window. + */ + if (flush_reset && !IS_ENABLED(CONFIG_ARCH_HAS_SET_DIRECT_MAP)) { + set_memory_nx(addr, area->nr_pages); + set_memory_rw(addr, area->nr_pages); + } + + remove_vm_area(area->addr); + + /* If this is not VM_FLUSH_RESET_PERMS memory, no need for the below. */ + if (!flush_reset) + return; + + /* + * If not deallocating pages, just do the flush of the VM area and + * return. + */ + if (!deallocate_pages) { + vm_unmap_aliases(); + return; + } + + /* + * If execution gets here, flush the vm mapping and reset the direct + * map. Find the start and end range of the direct mappings to make sure + * the vm_unmap_aliases() flush includes the direct map. + */ + for (i = 0; i < area->nr_pages; i++) { + if (page_address(area->pages[i])) { + start = min(addr, start); + end = max(addr, end); + } + } + + /* + * Set direct map to something invalid so that it won't be cached if + * there are any accesses after the TLB flush, then flush the TLB and + * reset the direct map permissions to the default. + */ + set_area_direct_map(area, set_direct_map_invalid_noflush); + _vm_unmap_aliases(start, end, 1); + set_area_direct_map(area, set_direct_map_default_noflush); +} + static void __vunmap(const void *addr, int deallocate_pages) { struct vm_struct *area; @@ -1526,7 +1599,8 @@ static void __vunmap(const void *addr, int deallocate_pages) debug_check_no_locks_freed(area->addr, get_vm_area_size(area)); debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); - remove_vm_area(addr); + vm_remove_mappings(area, deallocate_pages); + if (deallocate_pages) { int i; @@ -1961,8 +2035,9 @@ EXPORT_SYMBOL(vzalloc_node); */ void *vmalloc_exec(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL_EXEC, - NUMA_NO_NODE, __builtin_return_address(0)); + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) From patchwork Mon Apr 22 18:57:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911475 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D2DB1575 for ; Mon, 22 Apr 2019 19:00:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80EA1286B0 for ; Mon, 22 Apr 2019 19:00:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 74EA3287A6; Mon, 22 Apr 2019 19:00:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4780F286B0 for ; Mon, 22 Apr 2019 19:00:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 879136B0280; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7C5376B0282; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6662E6B0284; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 28F786B0280 for ; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id x9so8851688pln.0 for ; Mon, 22 Apr 2019 12:00:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Ad8lsaTX3gMSj0SQ10uKFMNKwQGvntfT8IOy2gn/yIE=; b=rSk0W5dnrPmcAUEVnMIxIR2+FJIKgVKQr0UsXCuZT/0/qk9sO+Xpzx9hElAqOCXPkT EC+ez161yRBMjM0NM20eDorzrtT///KvSYraZnJ7apH4kzvvpp+3x2nm81GClM0SyemJ Hb2wqzs57GBja4DFAJ3PLT9T894PKN+FT+0zr57kqxoGrwqy60gEhfMV1+Gi3ipqh4d4 lB1KWkm8RXLvJ/x2TYasUWO4Oe5DU7LEleTJ5CFQELCLkU34TqLHNRA+tKQ+LeO8TDVp q+RDGOobl45rUpTcDwE6bSEhwxbJ8iNRm4QWz78SCF3BWGlRqWIdkr6T63qrnhN7zZP8 zGdQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXG3Q8+Svz7r1NHIOc5P0ZxiCr8VZZ46OUBbqWvDaWNGaXhIp11 OKjtZRwtbvQvdSUQKYLqKz8prB7rLFEu2gQ3AucMPtgxiQqV1Ks+X3IwqvMgrnCiz2N4xB6d+oa RQsDgKaPUaig5rdotfWliI0f96u5ryfliuGXqMORk9WFDXsKucxpCiIDO0Emc2zyzGw== X-Received: by 2002:aa7:91d5:: with SMTP id z21mr22217009pfa.222.1555959625811; Mon, 22 Apr 2019 12:00:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqyC5fdFY/sB3ZFIIWPVPAkiL4KYThr7otguJ1leQO7QKLMOjdnQG7JKlBO9uk1TtXUj0DW7 X-Received: by 2002:aa7:91d5:: with SMTP id z21mr22209255pfa.222.1555959524150; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=aOdC1Wz2XXaMD8wXbu/w5rhcipEshCjk/ILUHYSBAx80Lgq5fNmo2sU2P+3D7cyF2k l3lqlyUsQUxyy5ZeyhZuSAD7a7ye0wv2LaFWhFHGC7ON90ygHSUByTOcyazuU2kYK6xE yY0bId+e5F/2TqnI+pkb3wTD19j+0Yy4/yxwuZM9eCJocdK+9U79X0OhzWDSE+3B/knY hBMujZQeLLOSRlk6luqHhOUu8U4SG1A5UqH3LxiQgLb8K9NCYS2/AKzTVWQAxzWZymby IOa4r1GkthbNgvxY/EgfjKMS5iBCGHx3fNwJTBvyI+HQAafe+A1utqruq4uxt4YGnLp7 k5sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Ad8lsaTX3gMSj0SQ10uKFMNKwQGvntfT8IOy2gn/yIE=; b=Te++ohD+Z7JdNi2ftZkOVL1aHio6hDvI3vnYigrpusPII2bXgkvdAzSAzP/iWkXQ7N sx5BrfqbbiGstjDP4AKZBQY6wfg5WZHwmhBF15x4q0LzeBnXFKdtA4nLXT+75a46ZSXZ 34f0+E8mtNTdGgxsj9ejKPOAU2LUeKoQcU0on1aPikD0PX90to9MijZI60f8UA9wFNDu rtpdddIdQFW/euooBeSP2lrtpBmPacgb7FgYIex0fN6l+kZYw/dtOMIKHBjZNaAGPyzP BY3mP0qx6brqOu8tCMhr7syumacL2jyIaHZxl1PCbGlWFW3fgOULZFdqSh2mnaaEsuGz xe2g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id w15si615875pga.591.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417168" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Jessica Yu , Steven Rostedt Subject: [PATCH v4 17/23] modules: Use vmalloc special flag Date: Mon, 22 Apr 2019 11:57:59 -0700 Message-Id: <20190422185805.1169-18-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use new flag for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Since freeing of VM_FLUSH_RESET_PERMS memory is not supported in an interrupt by vmalloc, the freeing of init sections is moved to a work queue. Instead of call_rcu it now uses synchronize_rcu() in the work queue. Lastly, there is now a WARN_ON in module_memfree since it should not be called in an interrupt with special memory as is required for VM_FLUSH_RESET_PERMS. Cc: Jessica Yu Cc: Steven Rostedt Signed-off-by: Rick Edgecombe --- kernel/module.c | 77 +++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 38 deletions(-) diff --git a/kernel/module.c b/kernel/module.c index 2b2845ae983e..a9020bdd4cf6 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -98,6 +98,10 @@ DEFINE_MUTEX(module_mutex); EXPORT_SYMBOL_GPL(module_mutex); static LIST_HEAD(modules); +/* Work queue for freeing init sections in success case */ +static struct work_struct init_free_wq; +static struct llist_head init_free_list; + #ifdef CONFIG_MODULES_TREE_LOOKUP /* @@ -1949,6 +1953,8 @@ void module_enable_ro(const struct module *mod, bool after_init) if (!rodata_enabled) return; + set_vm_flush_reset_perms(mod->core_layout.base); + set_vm_flush_reset_perms(mod->init_layout.base); frob_text(&mod->core_layout, set_memory_ro); frob_text(&mod->core_layout, set_memory_x); @@ -1972,15 +1978,6 @@ static void module_enable_nx(const struct module *mod) frob_writable_data(&mod->init_layout, set_memory_nx); } -static void module_disable_nx(const struct module *mod) -{ - frob_rodata(&mod->core_layout, set_memory_x); - frob_ro_after_init(&mod->core_layout, set_memory_x); - frob_writable_data(&mod->core_layout, set_memory_x); - frob_rodata(&mod->init_layout, set_memory_x); - frob_writable_data(&mod->init_layout, set_memory_x); -} - /* Iterate through all modules and set each module's text as RW */ void set_all_modules_text_rw(void) { @@ -2024,23 +2021,8 @@ void set_all_modules_text_ro(void) } mutex_unlock(&module_mutex); } - -static void disable_ro_nx(const struct module_layout *layout) -{ - if (rodata_enabled) { - frob_text(layout, set_memory_rw); - frob_rodata(layout, set_memory_rw); - frob_ro_after_init(layout, set_memory_rw); - } - frob_rodata(layout, set_memory_x); - frob_ro_after_init(layout, set_memory_x); - frob_writable_data(layout, set_memory_x); -} - #else -static void disable_ro_nx(const struct module_layout *layout) { } static void module_enable_nx(const struct module *mod) { } -static void module_disable_nx(const struct module *mod) { } #endif #ifdef CONFIG_LIVEPATCH @@ -2120,6 +2102,11 @@ static void free_module_elf(struct module *mod) void __weak module_memfree(void *module_region) { + /* + * This memory may be RO, and freeing RO memory in an interrupt is not + * supported by vmalloc. + */ + WARN_ON(in_interrupt()); vfree(module_region); } @@ -2171,7 +2158,6 @@ static void free_module(struct module *mod) mutex_unlock(&module_mutex); /* This may be empty, but that's OK */ - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); module_memfree(mod->init_layout.base); kfree(mod->args); @@ -2181,7 +2167,6 @@ static void free_module(struct module *mod) lockdep_free_key_range(mod->core_layout.base, mod->core_layout.size); /* Finally, free the core (containing the module structure) */ - disable_ro_nx(&mod->core_layout); module_memfree(mod->core_layout.base); } @@ -3420,17 +3405,34 @@ static void do_mod_ctors(struct module *mod) /* For freeing module_init on success, in case kallsyms traversing */ struct mod_initfree { - struct rcu_head rcu; + struct llist_node node; void *module_init; }; -static void do_free_init(struct rcu_head *head) +static void do_free_init(struct work_struct *w) { - struct mod_initfree *m = container_of(head, struct mod_initfree, rcu); - module_memfree(m->module_init); - kfree(m); + struct llist_node *pos, *n, *list; + struct mod_initfree *initfree; + + list = llist_del_all(&init_free_list); + + synchronize_rcu(); + + llist_for_each_safe(pos, n, list) { + initfree = container_of(pos, struct mod_initfree, node); + module_memfree(initfree->module_init); + kfree(initfree); + } } +static int __init modules_wq_init(void) +{ + INIT_WORK(&init_free_wq, do_free_init); + init_llist_head(&init_free_list); + return 0; +} +module_init(modules_wq_init); + /* * This is where the real work happens. * @@ -3507,7 +3509,6 @@ static noinline int do_init_module(struct module *mod) #endif module_enable_ro(mod, true); mod_tree_remove_init(mod); - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); mod->init_layout.base = NULL; mod->init_layout.size = 0; @@ -3518,14 +3519,18 @@ static noinline int do_init_module(struct module *mod) * We want to free module_init, but be aware that kallsyms may be * walking this with preempt disabled. In all the failure paths, we * call synchronize_rcu(), but we don't want to slow down the success - * path, so use actual RCU here. + * path. module_memfree() cannot be called in an interrupt, so do the + * work and call synchronize_rcu() in a work queue. + * * Note that module_alloc() on most architectures creates W+X page * mappings which won't be cleaned up until do_free_init() runs. Any * code such as mark_rodata_ro() which depends on those mappings to * be cleaned up needs to sync with the queued work - ie * rcu_barrier() */ - call_rcu(&freeinit->rcu, do_free_init); + if (llist_add(&freeinit->node, &init_free_list)) + schedule_work(&init_free_wq); + mutex_unlock(&module_mutex); wake_up_all(&module_wq); @@ -3822,10 +3827,6 @@ static int load_module(struct load_info *info, const char __user *uargs, module_bug_cleanup(mod); mutex_unlock(&module_mutex); - /* we can't deallocate the module until we clear memory protection */ - module_disable_ro(mod); - module_disable_nx(mod); - ddebug_cleanup: ftrace_release_mod(mod); dynamic_debug_remove(mod, info->debug); From patchwork Mon Apr 22 18:58:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911467 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B68F61708 for ; Mon, 22 Apr 2019 19:00:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A93B128673 for ; Mon, 22 Apr 2019 19:00:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D6E028688; Mon, 22 Apr 2019 19:00:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 074D02867C for ; Mon, 22 Apr 2019 19:00:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1873B6B027A; Mon, 22 Apr 2019 15:00:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 15DFB6B027C; Mon, 22 Apr 2019 15:00:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04D1D6B027D; Mon, 22 Apr 2019 15:00:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id BDE446B027A for ; Mon, 22 Apr 2019 15:00:12 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id x2so8450626pge.16 for ; Mon, 22 Apr 2019 12:00:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=wig8MszCtUvg2yw20GAeD+70ac75PEX/7kXXzCB+iko=; b=cuY6OGIdKniqQzmIGwbcQDR1//vTQnDRlKKF293nFwy3TuIra9fQ8cEU1vWGkUF0I8 TcQstcTktaAAx9w8Bw2sV8tYvod3WjuFc30UtW8QPdmnY2Ttgi4POlWg5ByUWKleDcns qD+GFp6EdR71EN8kAptw03X2bCihVhJKWjgHp4G2fNzbknx8u0YA68yEDYnfuS9hP6iH HomedUVE2MnLOUoUfSawwzQ4HWzcTxj6kxtq6E0VLMmcrhuMaZhvd8OZaEK0j47+KAV9 in8x+y9H9uFQyRVgNTXOmQ/FQpXhXhnAimTr3/LQ4bOvefFuhjOv2zBqtmvnkRC7coGW XfdA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXGTdxtxPeejXL+B82YvVPLAD7sdPMxQsh0GqBp0T8bDUscoEwn 8+qaXVG685Xsml8V1gNc2ElT8dDpuyeZAkViy2Pg6DF56vGdsZxC5/9/TuK1X2wOr7KOtwHpAtr Xwl3Wh5ajgO6XNS+hY/Q8fVhDVT1WJbAggbR/KCbu2wEWeK33V3Q7hbO95Qxo9/RsRg== X-Received: by 2002:aa7:8289:: with SMTP id s9mr22141075pfm.208.1555959612443; Mon, 22 Apr 2019 12:00:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqzvV0GCTjO37DMsOGs/KK8hCp8tSVa+HJj1Dd0DIso1ckmD4yvuMq9YHf2BJFoMxsAlvLGA X-Received: by 2002:aa7:8289:: with SMTP id s9mr22134794pfm.208.1555959524147; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=CQWZV2KNeTMSl2u+qs1IvlJQ1pOeupxhS5YoOkjQWTDoZbZ1/majz6uFUlqb8fp8vb EaNk3p+DD7JDMupHevE9ksBKtRyhtaAV5zyeKUVBq0DTLIOG17XiP4/8T/eJ2uQvu2z7 lv+We4pn+sLF/iN23eVZbRuGQd2N7ueQKpM9P2w7FZLR4NJA+d428B+hoBxYd9FBWZ5J PG1XvGnoy/iHCm5Aw2ipU7T5SDi4qDctZGSRegsNzTYP8/KVEHcxCB+l7INaXT9hM/iJ 1Yhqu+PI3uEGUtuZBE121O0lFmUbW3ztoc/lAnn1xSdUYBNtmMLFJvqZTgxZeAauyd0k y9Og== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=wig8MszCtUvg2yw20GAeD+70ac75PEX/7kXXzCB+iko=; b=uuSN1X10nQYd1fYUJXt0iYiIyPHCczIs2dgQJMUjhOqCzB3QmDuCMUt7dKUMoWmkI7 eQzIMdKKCTrKJrz/rDGQdrvJQqt8FeHq4dL3EjW62r4EtE9Dtt68Trbrz0ZMmyppVjkl 95oBPbJ4JlGC6E+Hu5TYZy3vViYKMNy1HrOA51QNSh4QiVrA87B/pWnbv/eZ+IKGN+ke zUT2ealmUHN3VxCix/UW07nmQ7KdZU347afs5n673cmhCF1+bKqmrSzkbF54FQhoLVWF PEc2RG54gGtGMnpqwTFJEDjjd9k7qr5Eiyw1RbJH6G49mfs1g4N8M9g1IYqeoDOOz527 ugXQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417170" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Daniel Borkmann , Alexei Starovoitov Subject: [PATCH v4 18/23] bpf: Use vmalloc special flag Date: Mon, 22 Apr 2019 11:58:00 -0700 Message-Id: <20190422185805.1169-19-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Don't track if the memory is RO anymore because it is now tracked in vmalloc. Cc: Daniel Borkmann Cc: Alexei Starovoitov Signed-off-by: Rick Edgecombe --- include/linux/filter.h | 17 +++-------------- kernel/bpf/core.c | 1 - 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 14ec3bdad9a9..7d3abde3f183 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -20,6 +20,7 @@ #include #include #include +#include #include @@ -503,7 +504,6 @@ struct bpf_prog { u16 pages; /* Number of allocated pages */ u16 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ - undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ @@ -733,27 +733,17 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default) static inline void bpf_prog_lock_ro(struct bpf_prog *fp) { - fp->undo_set_mem = 1; + set_vm_flush_reset_perms(fp); set_memory_ro((unsigned long)fp, fp->pages); } -static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) -{ - if (fp->undo_set_mem) - set_memory_rw((unsigned long)fp, fp->pages); -} - static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { + set_vm_flush_reset_perms(hdr); set_memory_ro((unsigned long)hdr, hdr->pages); set_memory_x((unsigned long)hdr, hdr->pages); } -static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) -{ - set_memory_rw((unsigned long)hdr, hdr->pages); -} - static inline struct bpf_binary_header * bpf_jit_binary_hdr(const struct bpf_prog *fp) { @@ -789,7 +779,6 @@ void __bpf_prog_free(struct bpf_prog *fp); static inline void bpf_prog_unlock_free(struct bpf_prog *fp) { - bpf_prog_unlock_ro(fp); __bpf_prog_free(fp); } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ff09d32a8a1b..c605397c79f0 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -848,7 +848,6 @@ void __weak bpf_jit_free(struct bpf_prog *fp) if (fp->jited) { struct bpf_binary_header *hdr = bpf_jit_binary_hdr(fp); - bpf_jit_binary_unlock_ro(hdr); bpf_jit_binary_free(hdr); WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(fp)); From patchwork Mon Apr 22 18:58:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911489 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83C7C1708 for ; Mon, 22 Apr 2019 19:00:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7951828793 for ; Mon, 22 Apr 2019 19:00:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6D988287AE; Mon, 22 Apr 2019 19:00:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2C6B28793 for ; Mon, 22 Apr 2019 19:00:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 57FA96B0284; Mon, 22 Apr 2019 15:00:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 52F126B0285; Mon, 22 Apr 2019 15:00:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 334E06B0287; Mon, 22 Apr 2019 15:00:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id EB2B16B0284 for ; Mon, 22 Apr 2019 15:00:30 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id a17so3811622plm.5 for ; Mon, 22 Apr 2019 12:00:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=3ZPWotgC+OZrWZLU7OmIKSzSgOEFrjgXAzYtiypTDWI=; b=PgYDNLsV7e3jVLtszwkwqylou5h3IAqnKTleAO0GMMXjlEB7KVeqMsqKkslvdmMz8r QxBTLbT98EtrVVX1DeE5M1WZesbbm2XNcUXhtInyvOc6v2ql1hla4c/Lws1V0Zh4qyv8 o9jI972VlUadapzw9OKu66wpL+GeIkzrYsgFqFDnzb2itLn7KpR1qfYlnhRE9eqBtN2K h5ABrPc60BsZwRJpFBY0DvLL5ApjzyU+gIf46FWVKdcI1TDXDYTjxNoFVqpUqXKgFxYu TUAjHvbSfrrs/v00T6mjpTfz27Nt+6K9nThRqyA/CpiGR2bpGuXyhQgLhsOKZLCTum/U lHBQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXuQYPDkiKfA9wvINPldtYByXEOYEt4vjkTYPisiWQ+PklxXj9S 5D1NZeY8vmvhN9WSHdeqFvjk9vn59DHIT0S59T0j9uhaIfGm2RU3voY0Y/RzvcLFmD+8zy5c6W3 GvSEwterd+FTl1E0K+982b4zOM4WIRMxk/yLHyVtF/FOMaoQbZEtriTMdxLH5xFwJSA== X-Received: by 2002:a63:5405:: with SMTP id i5mr20178301pgb.212.1555959630586; Mon, 22 Apr 2019 12:00:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqxGge6BqRIG0ADGr7plZgYlLlIEBQn5O65n0EXpw7MBRu95GgyDSExtk86GxQJQJBSX4s2U X-Received: by 2002:a63:5405:: with SMTP id i5mr20171045pgb.212.1555959524343; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=iKebrQewscfEtcjMUVbTnBY0yFZShuk3shsRu87VGjSToXSban4FhVBmpIo3T8HDeO /HS0TyDr1tmXLshlHv6zTgKojwXKgo0qB15xbJaq9aBOgvlWUZMzsZp2/3hj3Cg3pAJl 0g/5YHEfKMVDRAQPyIUqFn2tGXngdIy6XvB9j9cnjFiXt7gCmMFze1lcBGRkbjR8FvTT t1fyKheVQKbdGimQjOI3eFeeAf4bfAX2Y6J6TXxFhXa1dmrMkCryf1BVcSApTiRDDYfw /ujFT4wOb3jDd8ZkgS+WHA7w0MiauIHMguMEuIi4razG1+Mlc5IjO8FuJdlFYWGYCBzU JEWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=3ZPWotgC+OZrWZLU7OmIKSzSgOEFrjgXAzYtiypTDWI=; b=hfsvLIZ+hT0hMIS2U6X+MJL9M5awYEGlSUbzQXA5xK+JAcftd/cwQJM6e7THRdzCo/ TVHc2QRpaygFUgM/JoWQydneJGDwLwGBroyUuEqY6e1Xxvgerr+xNRjO26imkAl+n1rD a49+gtZGP3mxiuN1LAoHquKWS+Zu/2buNvV5O62wAkv7IQaPLye49pSHarje2joNtlhi Si1o9wQb157YSfjX6edJeZ5AZYLGZ0Wm/eyOA7wzn/jD+qjk6dqnfVtA623zD9oEoY8R AusJLm6QxqSFGj9TBnhEbnvE6iPup93Anb6AoPpDH/WhPhPCzuF/zrd7IrR+jHGeaqtp 4g7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417173" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Steven Rostedt Subject: [PATCH v4 19/23] x86/ftrace: Use vmalloc special flag Date: Mon, 22 Apr 2019 11:58:01 -0700 Message-Id: <20190422185805.1169-20-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed. Cc: Steven Rostedt Acked-by: Steven Rostedt (VMware) Signed-off-by: Rick Edgecombe Tested-by: Steven Rostedt (VMware) Acked-by: Steven Rostedt (VMware) --- arch/x86/kernel/ftrace.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 53ba1aa3a01f..0caf8122d680 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -678,12 +678,8 @@ static inline void *alloc_tramp(unsigned long size) { return module_alloc(size); } -static inline void tramp_free(void *tramp, int size) +static inline void tramp_free(void *tramp) { - int npages = PAGE_ALIGN(size) >> PAGE_SHIFT; - - set_memory_nx((unsigned long)tramp, npages); - set_memory_rw((unsigned long)tramp, npages); module_memfree(tramp); } #else @@ -692,7 +688,7 @@ static inline void *alloc_tramp(unsigned long size) { return NULL; } -static inline void tramp_free(void *tramp, int size) { } +static inline void tramp_free(void *tramp) { } #endif /* Defined as markers to the end of the ftrace default trampolines */ @@ -808,6 +804,8 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + set_vm_flush_reset_perms(trampoline); + /* * Module allocation needs to be completed by making the page * executable. The page is still writable, which is a security hazard, @@ -816,7 +814,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) set_memory_x((unsigned long)trampoline, npages); return (unsigned long)trampoline; fail: - tramp_free(trampoline, *tramp_size); + tramp_free(trampoline); return 0; } @@ -947,7 +945,7 @@ void arch_ftrace_trampoline_free(struct ftrace_ops *ops) if (!ops || !(ops->flags & FTRACE_OPS_FL_ALLOC_TRAMP)) return; - tramp_free((void *)ops->trampoline, ops->trampoline_size); + tramp_free((void *)ops->trampoline); ops->trampoline = 0; } From patchwork Mon Apr 22 18:58:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911477 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 576481575 for ; Mon, 22 Apr 2019 19:00:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4BB9A286B0 for ; Mon, 22 Apr 2019 19:00:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3F2FE287A6; Mon, 22 Apr 2019 19:00:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3314B286B0 for ; Mon, 22 Apr 2019 19:00:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C02B06B0282; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AD5616B0285; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DB0C6B0283; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 40E6D6B0282 for ; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id c64so8126975pfb.6 for ; Mon, 22 Apr 2019 12:00:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=vrkYcNOiqWdKT5EnvOrVHGGI0BGyhk1VWyHkq9CBPjA=; b=Tr44IoPH4/5xR2lWA4QMViBebDUoo7bQE58m63MkKJqcFm+jYefWWwKwHfYy+wCdJJ wEonUqREBx05sN8nApYlxjUhsnAmIo3f7YqiH9JxgpUAVqkeY2xO6Bh25QeKi91b9lAC smZbSv89tg0UUfWU0mcxpb5qnLCka3YvGq9lptVsguMzVPb6ld8ubah1Rr11gp+nX3AD 5NCqEf2PLXIBTBQSwS9/sanag/0ZmrPbwREbKP0dcnO+02yZbESsvI1+bJvIDVIOG7Pa VzDx8HW+6vr6VVvTw7L9Lh4WYO247GmdcU9g9+Woe1S4ah56ebZeHbaftwNTsGk1FKRf zmSQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAU9XNVfcgfubsXDX25DCi/re0fE4whaan8TdO0A0GErUD1J4JmR dCd+oAxLKdI0fsLcCAByhrnUICwQNka6/qg0DFhtSNTbO7kB+eSHnNArlDbW7fPbsfH/xFQnOEs tn+XD60SOHD5GhPgkGr/n141fF3Bqs1FRBmouONdSBPODD5hmX464ZdzZWuWN+UmwwQ== X-Received: by 2002:a63:4b21:: with SMTP id y33mr20933293pga.37.1555959625934; Mon, 22 Apr 2019 12:00:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqzYe+7nx5RX/Yp9g5O/G5o4nzZzhgco8JVsrt7cupWlIJ5x4AHJ6HzTr7P81laXYMZREH/6 X-Received: by 2002:a63:4b21:: with SMTP id y33mr20926173pga.37.1555959524391; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=bDK1VL8TEfCf6CdiK/g13iZfeBLHHoYx9LXqB6Afk2ANbcjscuS2mEYqk8z6/T32N4 J5N7xqEtSYNOwaN0aPJpK8FkGgfWDUCJpU+HpKSnw+PzH3opLngfKvYUFhCynVB4ETd8 GelC+Tm6hNKvgK+chWf7lAYfps+b3nCDLiCTkeSjxXXBN63H9s9eF0J8bNOkMzkCx8Dz LszLiUiwzDlTv4g68EYI6Tu2lqTlkNbVNQRzDk4sUQNeafqXv4Q8zqwHTdbg0BlQEk6p yHocknzYe9Gho+jf1Bt9ewaObKjQqraeJVpOMIE7HauRVAYBxL0tqraMTr613en/4qDy tbRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=vrkYcNOiqWdKT5EnvOrVHGGI0BGyhk1VWyHkq9CBPjA=; b=Aim0LC3XqKJ2bloKH3h3a696616dPXaciISWjuiNUljua0yqB/4G06Rx/R97gHzAqk 9FsSxpyb3mZCzQSPdA8doyfx43ipi/iF11d9J/JbVlxxAwk8m6kmUdh9Snxi4uxsiKh8 SBZtHSzWqm/n+VP8ZIG1UqM49ortx//goX2I6k5zPBhbbIK/Td78BxXeKmriguEBhgOy 6e5vTb3hbtg/t3L2RMNhJhupzqvL7ZogJNBNTbJFqDIgGAd/dlS6h1OKC7h6JER6ybM5 P0bwDCSWUwVhRXzHa6xjnFZhGqT6enBbpZX5BrLtLjE07vjUxFsYfwJDqVWYCY11x9ro Zxrg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id w15si615875pga.591.2019.04.22.11.58.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417176" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Masami Hiramatsu Subject: [PATCH v4 20/23] x86/kprobes: Use vmalloc special flag Date: Mon, 22 Apr 2019 11:58:02 -0700 Message-Id: <20190422185805.1169-21-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed. Cc: Masami Hiramatsu Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kprobes/core.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 1591852d3ac4..136695e4434a 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -434,6 +434,7 @@ void *alloc_insn_page(void) if (!page) return NULL; + set_vm_flush_reset_perms(page); /* * First make the page read-only, and only then make it executable to * prevent it from being W+X in between. @@ -452,12 +453,6 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - /* - * First make the page non-executable, and only then make it writable to - * prevent it from being W+X in between. - */ - set_memory_nx((unsigned long)page, 1); - set_memory_rw((unsigned long)page, 1); module_memfree(page); } From patchwork Mon Apr 22 18:58:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911503 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69C47161F for ; Mon, 22 Apr 2019 19:00:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E457286B0 for ; Mon, 22 Apr 2019 19:00:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5235E287A6; Mon, 22 Apr 2019 19:00:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2FDF286B0 for ; Mon, 22 Apr 2019 19:00:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99D7E6B028A; Mon, 22 Apr 2019 15:00:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 94B376B028C; Mon, 22 Apr 2019 15:00:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8633E6B028D; Mon, 22 Apr 2019 15:00:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 4FCE06B028A for ; Mon, 22 Apr 2019 15:00:48 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id m9so4027596pge.7 for ; Mon, 22 Apr 2019 12:00:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Mq726et3FUsm8L5SG93p+KUhd7ShOgthVM8qUN29ytY=; b=tsVgT7d+AwarlsAZ8YIzHSO6sB8Ofsi/bDgekv99XcZRHTf2HSsoIFRhAVoxFNkJ0i NGRhg1mIPWtrbeiKHgTtw92YG5vaSH2yEGSl39tDEbyZQotZi3Jnf17CpC31abHKq/Wu CQkKUolKtWwEA/5nLs19mXn5fqfLz4OVlhYxrEGhgwacJjLJU9EmF1oV2Pf8X8QQ46GB 3EmB21sN/spJQVejTi4xN/upmm5VY67ii/vJ48ZrMKb4SEdCeu54tvVadHnfjJ02BpHU 88j8D66AzxW8FWiWKcpjEZvHtUx85Z7q/vQTB1OWUk4uFP1yTwdr3wj+1jLARYhzSXEG B4hA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAW0hM5G2Q9oY3SG0fToH/Yse+xvgqNJzaSj3WgDSTsDu2Oclqrm jkJUHYAswCmVUf0mM0IM29hG+TequN+9GVbcHoTP2isAmco0cdutwW0OogsIR1jg9y1wwvH4M4N tAho0zbGaMr9g65oUX3/bIJAQjEBctNimsJxItL+D31CqrK1q7iv1W6CXuHMxv/W95w== X-Received: by 2002:aa7:9ae9:: with SMTP id y9mr15824728pfp.111.1555959647985; Mon, 22 Apr 2019 12:00:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqzzpDPDN71sXTXpfhpNUytktmHFKk/Zckmevj3IodAYa0Q1E7aP1ypO2iREiAgzlitSCPcj X-Received: by 2002:aa7:9ae9:: with SMTP id y9mr15814097pfp.111.1555959524408; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=sn9syfbaUUlSKdLmsf26iJP7asdZ3BrUb0G8oa8XDkwBwXX3UJQ2jIT5Z8b89I4CpI BZhkSdptpAr0ARUO7Zv5kNVgHDPu6gvP5EAdN9swV8NUD1I3LAojPea1K+NI/wjbedZ3 +lsahbJoStDjXAWpcM1+lTOYafFwfT4ql1IMHJMCE6NSi6vDKEgglBQRddvFc01JLEeR ZkN3ukVMZAwysiyo71Lij0U1+w5MUoGIb6WAjfgBAZc5VxCvaezpEHaDs6GUvf8x08cA 5bx15nAFKaRkhWhGwLS2PZuDoyIJkJbLh1MQK29o5fPVZnyjpl1jdx/NgfqOR32EMsta d5hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Mq726et3FUsm8L5SG93p+KUhd7ShOgthVM8qUN29ytY=; b=Eof6E8dhYOityICyufq45gPCEWJVJkbPikHiVgA0yx5iC1WTE4tkgycrNB4CxxzDaW LGFAwdGMd434UyaTWuJmr4yrrjscto2WUC/w2C724wijYYRXCPrfGFGn712tbCiAPT5d gzXgP4R6WTEKjrYnh6HE0o0DEC2X5k3I9oRnbfSA3bgR66B+ogkpMjnYjZXH2qR5diZA J8REDOHDSMgA9W5MRBrk4wZTCU/5EZdhwEXicpSMlU9gASjHiyVoqnQaitWKKlyWC6mj n9Ttq/ueIxcNZk7HcepwSXmHEbEddiZX8HWAXbS6PIGYj+uOkEgeqaq+myGJqO7BfLVi hTTA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a20si5314305pgb.421.2019.04.22.11.58.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417179" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH v4 21/23] x86/alternative: Comment about module removal races Date: Mon, 22 Apr 2019 11:58:03 -0700 Message-Id: <20190422185805.1169-22-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Add a comment to clarify that users of text_poke() must ensure that no races with module removal take place. Cc: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/alternative.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 18f959975ea0..7b9b49dfc05a 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -810,6 +810,11 @@ static void *__text_poke(void *addr, const void *opcode, size_t len) * It means the size must be writable atomically and the address must be aligned * in a way that permits an atomic write. It also makes sure we fit on a single * page. + * + * Note that the caller must ensure that if the modified code is part of a + * module, the module would not be removed during poking. This can be achieved + * by registering a module notifier, and ordering module removal and patching + * trough a mutex. */ void *text_poke(void *addr, const void *opcode, size_t len) { From patchwork Mon Apr 22 18:58:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911403 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C98281515 for ; Mon, 22 Apr 2019 18:59:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BD9A928759 for ; Mon, 22 Apr 2019 18:59:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AE99B28764; Mon, 22 Apr 2019 18:59:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43D5928759 for ; Mon, 22 Apr 2019 18:59:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AEA66B0269; Mon, 22 Apr 2019 14:58:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 358C66B026B; Mon, 22 Apr 2019 14:58:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 223536B026A; Mon, 22 Apr 2019 14:58:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id CCA2C6B0269 for ; Mon, 22 Apr 2019 14:58:45 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id x2so8448357pge.16 for ; Mon, 22 Apr 2019 11:58:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=p6XVxxmeId6Fyrw7m3B0ifgQzW6nhwdk5H1SAhfRKS0=; b=BNI3LEBLVaMVJXoFqtKWg2Fw5YCft0+Q6hp/sA3soM1NE2BKstay8DxGolYaWm7eJr vh2LubBuIAlJDJm4XFS5/Aq++5r0KvVdiC1wAseoRmfr/QvgU1AS1rYtsCXtTC5qZqaU HYNj4CaXFWSp1yVLWlObRIBLLEiXclhFN81RGMRcJYRxBfZyzqFvGl5dg0M0DuyKM8l2 G611CwuT3Iy9bzr+iOOL4SpUkOlDHujdXHVqtM5JpBL7Zv53v65g7o5cvQTSbVEbliJf 8Eh4AHrrJw9SvsMGEoYHD9AvbA1kRXUftIPPAgV23iIgwRS1JyTILNzvKfKYVOGbC6JL nWBg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUxY5Y5fEYUrK2qgFVq5nHgnnAgrOnyBilYAidYIqJliifSt/zM AxxEn8xQO7jQoduZzy4Gh6rvsrY6HWsTBt0EHsp/sX9l23BVbzUXX+v4YnO/ULzbgz1B7P3Gxml 7pcbUFwv+AuVYeCOl7uAtcStC+L4KgkrfcRzFMH7Y9Wv//riyun0XRKPzwJxCvf62Pg== X-Received: by 2002:a63:5c43:: with SMTP id n3mr20174334pgm.163.1555959525406; Mon, 22 Apr 2019 11:58:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqx9e1eaEuPY13QE4437Qli9bBZ9CzkZoXTrXqwsRrMDm/xZoa8rx5UuJCpJHy4EulnkfbLL X-Received: by 2002:a63:5c43:: with SMTP id n3mr20174297pgm.163.1555959524616; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=o/FlyCMekM/QBtxJogXiNshf4fORTf5jV5u/1GzfmemjiUJMChIZcwV1HgfXdomsBv QHmJhb07klkRXPu5U23vU+dcwF8jpYXY8x+rwa4SpAW76KGgwI20ht8rSuMdsiMfy72i 9nbnc/phGCTQyyeLbDtP5hWSbP+ieWYBCTL0gcIq3qFr/vV50POrlCKEZ1BThjyyag38 S4o9q4KFczfRY3eqnQyZ+rHFLSX4SwfgQTfr+IeGzzV4xbojeNQbaYsp8aynVJCn0NBL 92dCndsHX8XkYabVWzdQ/zVs5iMH68tGmklMs/Mg61Ha+3sEsR5mNiqiKTcn7GByi7of De+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=p6XVxxmeId6Fyrw7m3B0ifgQzW6nhwdk5H1SAhfRKS0=; b=PpudQdPyWtKVa+9uYVoo2qxGYOuICKJSX/+lMG3/bL+JODnO1c+J1W+mf+RFemXZYd 1FQ2Yd/Vx09BdURKf0LArMeNGNz0WcLBZypfxNnvnwo4emrW363vmDZ0rwaH3TBfld/l d0HHyGqN6bGFKNI8hPX1UURQBQr4f2P0Mq/eBvdRqhDIz2XzphsuJtuQngEaQO59iFf7 da3NvU2Yo9BqDB3qUg2SNp8wIG/cXLUp0bechrG9b7inL/RriMb7WLB1u9RKeHu5MrGe d01E6nhGeSuBmz+BrCEfVZyXhMrkPpoG7R1pBo+68/VZ5rB4nOY7nUmndZjvocz/YWVv kEmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id a2si12975117pgn.530.2019.04.22.11.58.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417182" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Rick Edgecombe Subject: [PATCH v4 22/23] tlb: provide default nmi_uaccess_okay() Date: Mon, 22 Apr 2019 11:58:04 -0700 Message-Id: <20190422185805.1169-23-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit x86 has an nmi_uaccess_okay(), but other architectures do not. Arch-independent code might need to know whether access to user addresses is ok in an NMI context or in other code whose execution context is unknown. Specifically, this function is needed for bpf_probe_write_user(). Add a default implementation of nmi_uaccess_okay() for architectures that do not have such a function. Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/tlbflush.h | 2 ++ include/asm-generic/tlb.h | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 90926e8dd1f8..dee375831962 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -274,6 +274,8 @@ static inline bool nmi_uaccess_okay(void) return true; } +#define nmi_uaccess_okay nmi_uaccess_okay + /* Initialize cr4 shadow for this CPU. */ static inline void cr4_init_shadow(void) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b9edc7608d90..480e5b2a5748 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -21,6 +21,15 @@ #include #include +/* + * Blindly accessing user memory from NMI context can be dangerous + * if we're in the middle of switching the current user task or switching + * the loaded mm. + */ +#ifndef nmi_uaccess_okay +# define nmi_uaccess_okay() true +#endif + #ifdef CONFIG_MMU /* From patchwork Mon Apr 22 18:58:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911471 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5E321575 for ; Mon, 22 Apr 2019 19:00:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7007286B0 for ; Mon, 22 Apr 2019 19:00:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B6BFB287A6; Mon, 22 Apr 2019 19:00:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D044286B0 for ; Mon, 22 Apr 2019 19:00:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 383176B027C; Mon, 22 Apr 2019 15:00:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 331656B027E; Mon, 22 Apr 2019 15:00:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F8F96B027F; Mon, 22 Apr 2019 15:00:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id D93406B027C for ; Mon, 22 Apr 2019 15:00:18 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id e20so8111191pfn.8 for ; Mon, 22 Apr 2019 12:00:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=TdxUBqKvjD04gvlVW2fqTrr9rgpG7ttN99ifXgy4wJ0=; b=LA7FUm0Bf7Ix8i1chPcq7KDAmrTG+LZLC6aPWNmEbR9ffwJ7ki04ATCBq2sAS6g8Fr L/aMOi49/r6HXXNYxVoMxsoyAkcKeQTJUQ+psT41RMAS09dlPGQ0XiZe6sO83aClrd9n chXC6Q0W6gtWMdlceI3Jbd4bZ4AMNMSw64z3b6cQ91OlfjHN32whtdKy7daNb+9p8J7U +ljrf5ATFxDP1n1ZK8tkqoIzFOQn22UiP5HE6vk437ZZYlaT7YJSS9NmoXwVXWPVN8WN SGvIg7NjJmcpTzVmCPgv36rL6/lCp+mpirFdm/sd2Dz6q9r/X4cbd7l561wi+FwJXIuR sYYQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAURAXeRQXiLdYylK9HLtfhzWNXtFuvlxRCuD39E56XvjcB39Bs3 hjfs1pxhLZchWKM2uVTEweHlS1wJDIpGvEY+WSqryddQccD301mE2XBUE5sNdqNExGB8414PZlA UqMuOMjrLdaOvoLRoV4uJlcB11CpdU3tzqWF9CAlYdn5Mxyoigv6gUL6zjMOzPWu0tQ== X-Received: by 2002:a62:8381:: with SMTP id h123mr21594925pfe.226.1555959618536; Mon, 22 Apr 2019 12:00:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqybgitSf8nYF6r0XiOi/2fRNRM2IGuVliv4NY+Q5+zGy7K3LBH9EkBSpr9zPEez3ZNC9V9s X-Received: by 2002:a62:8381:: with SMTP id h123mr21588233pfe.226.1555959524629; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=otacb2kMBFuYAx0qKOlXG6dqFKXmQ40HYoxiSNu+GNIui+FArlK1qoRMRfwZhxEjiY 6ULRtWdBGQ2jdX/CsywUdUN+y+YieGP5fR7Hr9+dAoH6yO2JfsPLpdGzpYwu20H7vm3h t3Zf3VwH730QLeXKi2fbQbx/xPEav8oZwAvu+0VhOOEt9bZMMFMD3c72D163TV8IsvXW p7djFiXzTvhWFFsqOhKh6NKl+Ybdj18njA67XIQRTcfEN9PoqneEXXOnA7Ja72Y/DPHp SusoC1+U2iALlKBAbZXk/bvw7oAKGyPc5ApCnxskvppLviBttfHet71K7BxwnPFjT1Ip r+3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=TdxUBqKvjD04gvlVW2fqTrr9rgpG7ttN99ifXgy4wJ0=; b=wgTOUpq3gX+dbLIEPzPceNdAL5xfp4mS8BWmYXKNmZHV5yG+EC/wmlxQvhLonKg5U6 nOLq/va5483IL/CU+zfTPci+H11Fwju1X7TujN+oaYbWk85J5eh31EokvjV0PoI9sjOH xDRBwzjOuTGPvKDVKToA4AbxKBzFQdl8FutCbvnmUVRUY/IEFy0HE/20G6CNRm83XdOJ GEHkUT4+YMu6wsCCXrNJ8lhFIx9/pj/Y8uNsY9a/SUYg2GXfAFov+1Xg3CiBSbLHRlxJ Y1Mz7r6gCutrEvWCDlQvwOZuueGeip+aNLHoxIrntYHoDuGnzuVbj7byd+2aNGbTWUC0 6d1A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id w15si615875pga.591.2019.04.22.11.58.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417185" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Daniel Borkmann , Alexei Starovoitov , Rick Edgecombe Subject: [PATCH v4 23/23] bpf: Fail bpf_probe_write_user() while mm is switched Date: Mon, 22 Apr 2019 11:58:05 -0700 Message-Id: <20190422185805.1169-24-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit When using a temporary mm, bpf_probe_write_user() should not be able to write to user memory, since user memory addresses may be used to map kernel memory. Detect these cases and fail bpf_probe_write_user() in such cases. Cc: Daniel Borkmann Cc: Alexei Starovoitov Reported-by: Jann Horn Suggested-by: Jann Horn Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- kernel/trace/bpf_trace.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index d64c00afceb5..94b0e37d90ef 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -14,6 +14,8 @@ #include #include +#include + #include "trace_probe.h" #include "trace.h" @@ -163,6 +165,10 @@ BPF_CALL_3(bpf_probe_write_user, void *, unsafe_ptr, const void *, src, * access_ok() should prevent writing to non-user memory, but in * some situations (nommu, temporary switch, etc) access_ok() does * not provide enough validation, hence the check on KERNEL_DS. + * + * nmi_uaccess_okay() ensures the probe is not run in an interim + * state, when the task or mm are switched. This is specifically + * required to prevent the use of temporary mm. */ if (unlikely(in_interrupt() || @@ -170,6 +176,8 @@ BPF_CALL_3(bpf_probe_write_user, void *, unsafe_ptr, const void *, src, return -EPERM; if (unlikely(uaccess_kernel())) return -EPERM; + if (unlikely(!nmi_uaccess_okay())) + return -EPERM; if (!access_ok(unsafe_ptr, size)) return -EPERM;