From patchwork Tue Dec 4 12:18:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10711665 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB3F814BD for ; Tue, 4 Dec 2018 12:18:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DC5872A0E0 for ; Tue, 4 Dec 2018 12:18:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CF0A42A0E7; Tue, 4 Dec 2018 12:18:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8EBB2A0E0 for ; Tue, 4 Dec 2018 12:18:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 552576B6EA4; Tue, 4 Dec 2018 07:18:46 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4D8FB6B6EA5; Tue, 4 Dec 2018 07:18:46 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A1EA6B6EA6; Tue, 4 Dec 2018 07:18:46 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by kanga.kvack.org (Postfix) with ESMTP id BAEBF6B6EA4 for ; Tue, 4 Dec 2018 07:18:45 -0500 (EST) Received: by mail-lf1-f72.google.com with SMTP id z25so1865813lfi.18 for ; Tue, 04 Dec 2018 04:18:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:reply-to:mime-version :content-transfer-encoding; bh=MKhXHv/I+suoXSvWY4nOkDQg0yGt8f18bNJLOYzoxHQ=; b=Zscr3LN10CwwOPztmcHYEIOdqKEQ21tlJeb1Nvop3jUmgXmKmL30SzVz7AdQnRUbKg BYY6KTlnUl06oIRVSMlTPZFqOeleWLW/CpGHwO1NTbKJTniMS39ujOen7rEyGVXDoqK6 nXJcjxr5qnUzq1qKyeap4JttX16k6V6zjZcfmzKJQ06SSBXj6mj68n1gVUbvrAA1BQCt 3hiekRSNgTXVzoPYvZNHoHRE5z0mb2DLIC/6hi43b0PmLHvbsW945zCju2GaZ/dGO+65 9M8C7jeV0W7A0pscOs/hFQC9v90pesYSsgjM1A0txYH06hw8odIb6m4+XI17CywWnjT1 MUjA== X-Gm-Message-State: AA+aEWaKXr6aeRIYnyz4amHEayIZb7ZHVI3m93kPjq63bWDBQplnLS12 gWPenkGuD2vNxSMnaKAvnpQ2ITi6Eu+PBzjO25bKxRaxkTsV7cMgvWNdOLE6u2lOvFQYp5aMSKP KRuisya0vs1QuhKlQHTAxbQ76ZKzOW4V0fe0WMhLAVWmxgn5EleT7FOsMOJfcU8NEHXu7YNlX2k eHm80akHSH63Wn4UKhAK1GY1/ajeSVCKJ+D7YfLuSjXE6ICj1eJGBrkpaMLTSsjw0FpVUMh7JzO ncQUgUTAHpARh0U2srhVltiujVBr8R7oCAydKCII0F2Lu+pmh6N5C1b+KXRQj9Ib+e9Tr6cFhex OJQgc+RjRsHsjsRSrwcqgrXdEwCOpuue+MlixgcqIcxQF46llBFLg5w27BxhsvHkog5a+yCK/wD A X-Received: by 2002:a2e:20f:: with SMTP id 15-v6mr12859297ljc.172.1543925925014; Tue, 04 Dec 2018 04:18:45 -0800 (PST) X-Received: by 2002:a2e:20f:: with SMTP id 15-v6mr12859215ljc.172.1543925923072; Tue, 04 Dec 2018 04:18:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543925923; cv=none; d=google.com; s=arc-20160816; b=xxsOgn8oE/mjLzdW6MnOaq9Fld2yRwNnn9Amr5lVIgsIJobWJX+50lFWMmeZ26F6Mx 62EdanznkTzZs9VDUWjFdHfgzKNFrpuPsG5KTYOcUgqZtq0mFXCgkRipXVvNVkAolW5G i+Y3fMDwxvbtMvcgvw4xnFwDcvB6bi1Bf1AkRBqKZTqs74j1QE3U7LrLYnc2k2MdIe/M xehNvvzVuuyOgxq0EAZ9Q7dCK7Nb6jAarIhlric3J3lnYRdPTomFXOPqrbFSav1EZerz yhoZwnlYyGL2Qxl7vLnxsUF0pe442kq7pF/htHslinip1mF1DVK5Nc6fb4lhoWodLlNm jGuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature; bh=MKhXHv/I+suoXSvWY4nOkDQg0yGt8f18bNJLOYzoxHQ=; b=biDqRB6FHxFvL59gafiOn1mCJywi0lq6VgdXyEALTrJKx/PaF7d4zUzHjfJ5EtR787 aIrC/PWJg6HS72Niicq8OO+f8en8ljPqtAllEanWFPhXjnXWJNm6QMGxC2UVvgnLqXMi jCLdYPqCOkzMYlOacefzdDOQPbkSrVeoy+H8Frko0wcalXQ51GLkwT+60Feb64Wgp8Z+ 0lfiCeM3coNUKTxahR2JhI9NcdMbXZXCKRC2M3//v2pjIgNYPBW1IeiG5opFNiC+zjyg XF8yXj725a+IaMKheyXIK+v7kJ6p7g42+DAlswdjixKTa4IMPdUNht026OHgkm0/aYWa 45nA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gfnQgWeQ; spf=pass (google.com: domain of igor.stoppa@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=igor.stoppa@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s74-v6sor9793333lje.7.2018.12.04.04.18.42 for (Google Transport Security); Tue, 04 Dec 2018 04:18:43 -0800 (PST) Received-SPF: pass (google.com: domain of igor.stoppa@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gfnQgWeQ; spf=pass (google.com: domain of igor.stoppa@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=igor.stoppa@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=MKhXHv/I+suoXSvWY4nOkDQg0yGt8f18bNJLOYzoxHQ=; b=gfnQgWeQffX3wVdRPVcjmboL00JGC8t7TbC+RAMpKpuzEMI+2s/f8LK1imQ05mYwT/ xcZLvzDYnNbaxT+JydGhBz58/N/6lbErp5QaiSfpLPs72sqz3Yv9UaHeyz2OlijZsumy dm1JMso9OG00MasdYPVwBFor6ktklkuQlSzYutp2DuNYA8+DIrvg4WC7etgHnXvUAa4f HopTED6G7o74rDwIPEvyW9qPC3D7wRyd9dNfxyDU2+8GU61BQNgpvnSDb+LfKXm0t1fJ kYS+RDYeDQW48BCz2BseM+MmIUDB52S/kihoje7fOf4VxfTtqJQbN0PsbeGLxD4JKGWK NFGw== X-Google-Smtp-Source: AFSGD/X5Ld4X34L6vNoHWDuvQhMxb5FqI8KA/+UtlMLx0Bgb9exvu91YMLJ7FP/CibfWnxue3VJ6fA== X-Received: by 2002:a2e:2c02:: with SMTP id s2-v6mr12920743ljs.118.1543925922527; Tue, 04 Dec 2018 04:18:42 -0800 (PST) Received: from localhost.localdomain (91-156-179-117.elisa-laajakaista.fi. [91.156.179.117]) by smtp.gmail.com with ESMTPSA id h3sm2899653lfj.25.2018.12.04.04.18.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Dec 2018 04:18:41 -0800 (PST) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Andy Lutomirski , Kees Cook , Matthew Wilcox Cc: igor.stoppa@huawei.com, Nadav Amit , Peter Zijlstra , Dave Hansen , linux-integrity@vger.kernel.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/6] __wr_after_init: write rare for static allocation Date: Tue, 4 Dec 2018 14:18:01 +0200 Message-Id: <20181204121805.4621-3-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181204121805.4621-1-igor.stoppa@huawei.com> References: <20181204121805.4621-1-igor.stoppa@huawei.com> Reply-To: Igor Stoppa MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Implementation of write rare for statically allocated data, located in a specific memory section through the use of the __write_rare label. The basic functions are: - wr_memset(): write rare counterpart of memset() - wr_memcpy(): write rare counterpart of memcpy() - wr_assign(): write rare counterpart of the assignment ('=') operator - wr_rcu_assign_pointer(): write rare counterpart of rcu_assign_pointer() The implementation is based on code from Andy Lutomirski and Nadav Amit for patching the text on x86 [here goes reference to commits, once merged] The modification of write protected data is done through an alternate mapping of the same pages, as writable. This mapping is local to each core and is active only for the duration of each write operation. Local interrupts are disabled, while the alternate mapping is active. In theory, it could introduce a non-predictable delay, in a preemptible system, however the amount of data to be altered is likely to be far smaller than a page. Signed-off-by: Igor Stoppa CC: Andy Lutomirski CC: Nadav Amit CC: Matthew Wilcox CC: Peter Zijlstra CC: Kees Cook CC: Dave Hansen CC: linux-integrity@vger.kernel.org CC: kernel-hardening@lists.openwall.com CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- include/linux/prmem.h | 133 ++++++++++++++++++++++++++++++++++++++++++ init/main.c | 2 + mm/Kconfig | 4 ++ mm/Makefile | 1 + mm/prmem.c | 124 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 264 insertions(+) create mode 100644 include/linux/prmem.h create mode 100644 mm/prmem.c diff --git a/include/linux/prmem.h b/include/linux/prmem.h new file mode 100644 index 000000000000..b0131c1f5dc0 --- /dev/null +++ b/include/linux/prmem.h @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * prmem.h: Header for memory protection library + * + * (C) Copyright 2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + * + * Support for: + * - statically allocated write rare data + */ + +#ifndef _LINUX_PRMEM_H +#define _LINUX_PRMEM_H + +#include +#include +#include +#include +#include +#include +#include +#include + +/** + * memtst() - test n bytes of the source to match the c value + * @p: beginning of the memory to test + * @c: byte to compare against + * @len: amount of bytes to test + * + * Returns 0 on success, non-zero otherwise. + */ +static inline int memtst(void *p, int c, __kernel_size_t len) +{ + __kernel_size_t i; + + for (i = 0; i < len; i++) { + u8 d = *(i + (u8 *)p) - (u8)c; + + if (unlikely(d)) + return d; + } + return 0; +} + + +#ifndef CONFIG_PRMEM + +static inline void *wr_memset(void *p, int c, __kernel_size_t len) +{ + return memset(p, c, len); +} + +static inline void *wr_memcpy(void *p, const void *q, __kernel_size_t size) +{ + return memcpy(p, q, size); +} + +#define wr_assign(var, val) ((var) = (val)) + +#define wr_rcu_assign_pointer(p, v) \ + rcu_assign_pointer(p, v) + +#else + +enum wr_op_type { + WR_MEMCPY, + WR_MEMSET, + WR_RCU_ASSIGN_PTR, + WR_OPS_NUMBER, +}; + +void *__wr_op(unsigned long dst, unsigned long src, __kernel_size_t len, + enum wr_op_type op); + +/** + * wr_memset() - sets n bytes of the destination to the c value + * @p: beginning of the memory to write to + * @c: byte to replicate + * @len: amount of bytes to copy + * + * Returns true on success, false otherwise. + */ +static inline void *wr_memset(void *p, int c, __kernel_size_t len) +{ + return __wr_op((unsigned long)p, (unsigned long)c, len, WR_MEMSET); +} + +/** + * wr_memcpy() - copyes n bytes from source to destination + * @dst: beginning of the memory to write to + * @src: beginning of the memory to read from + * @n_bytes: amount of bytes to copy + * + * Returns pointer to the destination + */ +static inline void *wr_memcpy(void *p, const void *q, __kernel_size_t size) +{ + return __wr_op((unsigned long)p, (unsigned long)q, size, WR_MEMCPY); +} + +/** + * wr_assign() - sets a write-rare variable to a specified value + * @var: the variable to set + * @val: the new value + * + * Returns: the variable + * + * Note: it might be possible to optimize this, to use wr_memset in some + * cases (maybe with NULL?). + */ + +#define wr_assign(var, val) ({ \ + typeof(var) tmp = (typeof(var))val; \ + \ + wr_memcpy(&var, &tmp, sizeof(var)); \ + var; \ +}) + +/** + * wr_rcu_assign_pointer() - initialize a pointer in rcu mode + * @p: the rcu pointer + * @v: the new value + * + * Returns the value assigned to the rcu pointer. + * + * It is provided as macro, to match rcu_assign_pointer() + */ +#define wr_rcu_assign_pointer(p, v) ({ \ + __wr_op((unsigned long)&p, v, sizeof(p), WR_RCU_ASSIGN_PTR); \ + p; \ +}) +#endif +#endif diff --git a/init/main.c b/init/main.c index a461150adfb1..a36f2e54f937 100644 --- a/init/main.c +++ b/init/main.c @@ -498,6 +498,7 @@ void __init __weak thread_stack_cache_init(void) void __init __weak mem_encrypt_init(void) { } void __init __weak poking_init(void) { } +void __init __weak wr_poking_init(void) { } bool initcall_debug; core_param(initcall_debug, initcall_debug, bool, 0644); @@ -734,6 +735,7 @@ asmlinkage __visible void __init start_kernel(void) delayacct_init(); poking_init(); + wr_poking_init(); check_bugs(); acpi_subsystem_init(); diff --git a/mm/Kconfig b/mm/Kconfig index d85e39da47ae..9b09339c027f 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -142,6 +142,10 @@ config ARCH_DISCARD_MEMBLOCK config MEMORY_ISOLATION bool +config PRMEM + def_bool n + depends on STRICT_KERNEL_RWX && X86_64 + # # Only be set on architectures that have completely implemented memory hotplug # feature. If you are not sure, don't touch it. diff --git a/mm/Makefile b/mm/Makefile index d210cc9d6f80..ef3867c16ce0 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -58,6 +58,7 @@ obj-$(CONFIG_SPARSEMEM) += sparse.o obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o obj-$(CONFIG_SLOB) += slob.o obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o +obj-$(CONFIG_PRMEM) += prmem.o obj-$(CONFIG_KSM) += ksm.o obj-$(CONFIG_PAGE_POISONING) += page_poison.o obj-$(CONFIG_SLAB) += slab.o diff --git a/mm/prmem.c b/mm/prmem.c new file mode 100644 index 000000000000..e8ab76701831 --- /dev/null +++ b/mm/prmem.c @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * prmem.c: Memory Protection Library + * + * (C) Copyright 2017-2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + */ + +#include +#include +#include +#include +#include +#include +#include + +static __ro_after_init bool wr_ready; +static __ro_after_init struct mm_struct *wr_poking_mm; +static __ro_after_init unsigned long wr_poking_base; + +/* + * The following two variables are statically allocated by the linker + * script at the the boundaries of the memory region (rounded up to + * multiples of PAGE_SIZE) reserved for __wr_after_init. + */ +extern long __start_wr_after_init; +extern long __end_wr_after_init; + +static inline bool is_wr_after_init(unsigned long ptr, __kernel_size_t size) +{ + unsigned long start = (unsigned long)&__start_wr_after_init; + unsigned long end = (unsigned long)&__end_wr_after_init; + unsigned long low = ptr; + unsigned long high = ptr + size; + + return likely(start <= low && low <= high && high <= end); +} + + +void *__wr_op(unsigned long dst, unsigned long src, __kernel_size_t len, + enum wr_op_type op) +{ + temporary_mm_state_t prev; + unsigned long flags; + unsigned long offset; + unsigned long wr_poking_addr; + + /* Confirm that the writable mapping exists. */ + BUG_ON(!wr_ready); + + if (WARN_ONCE(op >= WR_OPS_NUMBER, "Invalid WR operation.") || + WARN_ONCE(!is_wr_after_init(dst, len), "Invalid WR range.")) + return (void *)dst; + + offset = dst - (unsigned long)&__start_wr_after_init; + wr_poking_addr = wr_poking_base + offset; + local_irq_save(flags); + prev = use_temporary_mm(wr_poking_mm); + + kasan_disable_current(); + if (op == WR_MEMCPY) + memcpy((void *)wr_poking_addr, (void *)src, len); + else if (op == WR_MEMSET) + memset((u8 *)wr_poking_addr, (u8)src, len); + else if (op == WR_RCU_ASSIGN_PTR) + /* generic version of rcu_assign_pointer */ + smp_store_release((void **)wr_poking_addr, + RCU_INITIALIZER((void **)src)); + kasan_enable_current(); + + barrier(); /* XXX redundant? */ + + unuse_temporary_mm(prev); + /* XXX make the verification optional? */ + if (op == WR_MEMCPY) + BUG_ON(memcmp((void *)dst, (void *)src, len)); + else if (op == WR_MEMSET) + BUG_ON(memtst((void *)dst, (u8)src, len)); + else if (op == WR_RCU_ASSIGN_PTR) + BUG_ON(*(unsigned long *)dst != src); + local_irq_restore(flags); + return (void *)dst; +} + +struct mm_struct *copy_init_mm(void); +void __init wr_poking_init(void) +{ + unsigned long start = (unsigned long)&__start_wr_after_init; + unsigned long end = (unsigned long)&__end_wr_after_init; + unsigned long i; + unsigned long wr_range; + + wr_poking_mm = copy_init_mm(); + BUG_ON(!wr_poking_mm); + + /* XXX What if it's too large to fit in the task unmapped mem? */ + wr_range = round_up(end - start, PAGE_SIZE); + + /* Randomize the poking address base*/ + wr_poking_base = TASK_UNMAPPED_BASE + + (kaslr_get_random_long("Write Rare Poking") & PAGE_MASK) % + (TASK_SIZE - (TASK_UNMAPPED_BASE + wr_range)); + + /* Create alternate mapping for the entire wr_after_init range. */ + for (i = start; i < end; i += PAGE_SIZE) { + struct page *page; + spinlock_t *ptl; + pte_t pte; + pte_t *ptep; + unsigned long wr_poking_addr; + + BUG_ON(!(page = virt_to_page(i))); + wr_poking_addr = i - start + wr_poking_base; + + /* The lock is not needed, but avoids open-coding. */ + ptep = get_locked_pte(wr_poking_mm, wr_poking_addr, &ptl); + VM_BUG_ON(!ptep); + + pte = mk_pte(page, PAGE_KERNEL); + set_pte_at(wr_poking_mm, wr_poking_addr, ptep, pte); + spin_unlock(ptl); + } + wr_ready = true; +}