From patchwork Tue Oct 23 21:34:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653711 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE8DB13A4 for ; Tue, 23 Oct 2018 21:36:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA1022A3CD for ; Tue, 23 Oct 2018 21:36:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 999912A3D2; Tue, 23 Oct 2018 21:36:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id A37672A3CD for ; Tue, 23 Oct 2018 21:36:29 +0000 (UTC) Received: (qmail 13427 invoked by uid 550); 23 Oct 2018 21:36:11 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 12241 invoked from network); 23 Oct 2018 21:36:10 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=zu6YpzsO3OPxEg4tClPcjQ9+ceDNkNmVTbJmyduztxw=; b=IYU37E3Goc1gepM+z7hog9RH6DIyAowZ3kap0+WnwznOQYEPQmswmAxSGL4JWRQEyS CVrhUhp6EhYzB/yceyWg6J016IiEWahMGlmmJJMR8SB9NsCkUaysR15IMkJiU9XU5A4x MYbPNGz5r322XE/6vXeNUMLtZBrdp5hnW7p6Gu6ioi4MnKP5t0mRajj05ORcpQyv+vrY Z6ynoOqJzxtr0/3c7DGuoeWYCgLGwyy1UvRU4MZvg/8f07ITGGyeJcCceomGyRisd/Gm odqfJ5zXkKMEu4gInslFJueixysz3Std4Mztz2zYPYKLbOawgJ0QCO9GrluxawhCBR6S qfuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=zu6YpzsO3OPxEg4tClPcjQ9+ceDNkNmVTbJmyduztxw=; b=lPtlV8GADFY2EistCXuj7iJ0gf5c4ROWvY3K9k1uoHQYsNz3v7uBokv0I3PCdkaqpE acX1lQZnpmjYSrZctFuSGjhcos/eFGFcg2hypuOBinnsC1iSpZCRqZ8DhW9KNDHPBKWu kjeqf9OdTI5w3EHarE6JpppXdXGznF8lwFRoqe3KNvJ0RIf2IiO/p0TeoflHzaT+LVJk 7ONl/YrcNThJnmShTbZE5LHb3EyjXS4AHFJPEYbN0jBKSzsYxevcv/x1Ml9cCwPhhezq weV3KZrAuvB0XP05Z1B1Rvdp4C07GTu5qafq7eKjdzLYs4FG3mZznaM+zYwmqTkE1qvR dx7w== X-Gm-Message-State: ABuFfohuY69vZS5cI3UUfwQCkVDTo5M7/kLt9O3mBlo0lEg7E0Ic+Jyg pZe64lBxJZlSB3Lq/J06DpU= X-Google-Smtp-Source: ACcGV60Pq7KtfnwIP2lKFjHQF4lR+JAvWyiJuqcEsOhPvZE+obfCKGmgt17NFjnFkz3993bVkhoNew== X-Received: by 2002:a19:690d:: with SMTP id e13mr9045976lfc.84.1540330558395; Tue, 23 Oct 2018 14:35:58 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Arnd Bergmann , Thomas Gleixner , Kate Stewart , Greg Kroah-Hartman , Philippe Ombredanne , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/17] prmem: linker section for static write rare Date: Wed, 24 Oct 2018 00:34:48 +0300 Message-Id: <20181023213504.28905-2-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Introduce a section and a label for statically allocated write rare data. The label is named "__write_rare_after_init". As the name implies, after the init phase is completed, this section will be modifiable only by invoking write rare functions. NOTE: this needs rework, because the current write-rare mechanism works only on x86_64 and not arm64, due to arm64 mappings. Signed-off-by: Igor Stoppa CC: Arnd Bergmann CC: Thomas Gleixner CC: Kate Stewart CC: Greg Kroah-Hartman CC: Philippe Ombredanne CC: linux-arch@vger.kernel.org CC: linux-kernel@vger.kernel.org --- include/asm-generic/vmlinux.lds.h | 20 ++++++++++++++++++++ include/linux/cache.h | 17 +++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index d7701d466b60..fd40a15e3b24 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -300,6 +300,25 @@ . = __start_init_task + THREAD_SIZE; \ __end_init_task = .; +/* + * Allow architectures to handle wr_after_init data on their + * own by defining an empty WR_AFTER_INIT_DATA. + * However, it's important that pages containing WR_RARE data do not + * hold anything else, to avoid both accidentally unprotecting something + * that is supposed to stay read-only all the time and also to protect + * something else that is supposed to be writeable all the time. + */ +#ifndef WR_AFTER_INIT_DATA +#define WR_AFTER_INIT_DATA(align) \ + . = ALIGN(PAGE_SIZE); \ + __start_wr_after_init = .; \ + . = ALIGN(align); \ + *(.data..wr_after_init) \ + . = ALIGN(PAGE_SIZE); \ + __end_wr_after_init = .; \ + . = ALIGN(align); +#endif + /* * Allow architectures to handle ro_after_init data on their * own by defining an empty RO_AFTER_INIT_DATA. @@ -320,6 +339,7 @@ __start_rodata = .; \ *(.rodata) *(.rodata.*) \ RO_AFTER_INIT_DATA /* Read only after init */ \ + WR_AFTER_INIT_DATA(align) /* wr after init */ \ KEEP(*(__vermagic)) /* Kernel version magic */ \ . = ALIGN(8); \ __start___tracepoints_ptrs = .; \ diff --git a/include/linux/cache.h b/include/linux/cache.h index 750621e41d1c..9a7e7134b887 100644 --- a/include/linux/cache.h +++ b/include/linux/cache.h @@ -31,6 +31,23 @@ #define __ro_after_init __attribute__((__section__(".data..ro_after_init"))) #endif +/* + * __wr_after_init is used to mark objects that cannot be modified + * directly after init (i.e. after mark_rodata_ro() has been called). + * These objects become effectively read-only, from the perspective of + * performing a direct write, like a variable assignment. + * However, they can be altered through a dedicated function. + * It is intended for those objects which are occasionally modified after + * init, however they are modified so seldomly, that the extra cost from + * the indirect modification is either negligible or worth paying, for the + * sake of the protection gained. + */ +#ifndef __wr_after_init +#define __wr_after_init \ + __attribute__((__section__(".data..wr_after_init"))) +#endif + + #ifndef ____cacheline_aligned #define ____cacheline_aligned __attribute__((__aligned__(SMP_CACHE_BYTES))) #endif From patchwork Tue Oct 23 21:34:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653723 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 30EFC13A4 for ; Tue, 23 Oct 2018 21:36:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D00B2A3CD for ; Tue, 23 Oct 2018 21:36:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 113962A3D3; Tue, 23 Oct 2018 21:36:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 80F222A3CD for ; Tue, 23 Oct 2018 21:36:38 +0000 (UTC) Received: (qmail 13555 invoked by uid 550); 23 Oct 2018 21:36:12 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13498 invoked from network); 23 Oct 2018 21:36:11 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=lWr5dc2eNvTcJb+c+uRTUJwhKcPYmmWV6t14BAAM0+g=; b=K5HssUtWyqhs3CTyKNXHpefeYDtWkJbXaAX1XLxON4iECRSRB6QjMc5RCzYk0y3yfd 3c1iOvdGFcidkqsyYIElslGkPelZCRE/JLn4hSqqkZH2OH3h57NZu/YFoNNNAjmaYHMg 36urDvGuUDvpjx00WQrlW9VtHoMHr8CdHjsfQij3WvBYK3waVusZpw/gnqTE7YAg+T8z SO7hquz3XCCU2aYUVW1L7jBQGaZliOW0tEQjX97ZIN9jZwLllZqgweeeNE0Lm7lP2AZN mpclQP5Np4932IIJTLzPM1PMBsppoxoRMBjfYYZLQ0qprQKK8NxeH80SI6RaTIOohnqU wQQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=lWr5dc2eNvTcJb+c+uRTUJwhKcPYmmWV6t14BAAM0+g=; b=P7Oqufm8FdK92O95G2j7Rf+11lQ2IAdPGuUEt07e3dtIqKuLv5YrxlvfYZ2t5eT06q o6bUvoIb4UiwkVY3lt5HniUIr28tWGVV5GWf9QOY5oGa1Bi+OW484Qcje1Yynyip9lhd /0mm9WiiK7MFlmARm4Jocx03k5HJ0sjRGs1F41y1G7G6fLkfPCoIR7gTl6iBLzRxA3wt DASyNLlB1DHOoWxe9JXT2Dc0T1saz+ZK3/Xo+7CO8/+zmKAbrnwuO6mXooFGjcNxtNJu utXf71iAGFRmgoUSG4A0NcZS/u/GS8eZ7kINeA1SbF2TymxNnh88J+VxTQfP869hFvCV vXuQ== X-Gm-Message-State: ABuFfoh8rnNDERwrr5zjLo8DQxG1U3NEyEvTOU4LtMS+9QuPgD6TnQ06 ba5qnwgjjT8v6XWLAeBO6qA= X-Google-Smtp-Source: ACcGV63HspKtXzSwNphXjTw4U5KnYFl/LeBjRKri4bDR8FgHFnbdnJMdM8PgkfyEiZhsocjgUN4K2w== X-Received: by 2002:a19:2648:: with SMTP id m69-v6mr12821527lfm.78.1540330559911; Tue, 23 Oct 2018 14:35:59 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Pavel Tatashin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/17] prmem: write rare for static allocation Date: Wed, 24 Oct 2018 00:34:49 +0300 Message-Id: <20181023213504.28905-3-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Implementation of write rare for statically allocated data, located in a specific memory section through the use of the __write_rare label. The basic functions are wr_memcpy() and wr_memset(): the write rare counterparts of memcpy() and memset() respectively. To minimize chances of attacks, this implementation does not unprotect existing memory pages. Instead, it remaps them, one by one, at random free locations, as writable. Each page is mapped as writable strictly for the time needed to perform changes in said page. While a page is remapped, interrupts are disabled on the core performing the write rare operation, to avoid being frozen mid-air by an attack using interrupts for stretching the duration of the alternate mapping. OTOH, to avoid introducing unpredictable delays, the interrupts are re-enabled inbetween page remapping, when write operations are either completed or not yet started, and there is not alternate, writable mapping to exploit. Signed-off-by: Igor Stoppa CC: Michal Hocko CC: Vlastimil Babka CC: "Kirill A. Shutemov" CC: Andrew Morton CC: Pavel Tatashin CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- MAINTAINERS | 7 ++ include/linux/prmem.h | 213 ++++++++++++++++++++++++++++++++++++++++++ mm/Makefile | 1 + mm/prmem.c | 10 ++ 4 files changed, 231 insertions(+) create mode 100644 include/linux/prmem.h create mode 100644 mm/prmem.c diff --git a/MAINTAINERS b/MAINTAINERS index b2f710eee67a..e566c5d09faf 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9454,6 +9454,13 @@ F: kernel/sched/membarrier.c F: include/uapi/linux/membarrier.h F: arch/powerpc/include/asm/membarrier.h +MEMORY HARDENING +M: Igor Stoppa +L: kernel-hardening@lists.openwall.com +S: Maintained +F: include/linux/prmem.h +F: mm/prmem.c + MEMORY MANAGEMENT L: linux-mm@kvack.org W: http://www.linux-mm.org diff --git a/include/linux/prmem.h b/include/linux/prmem.h new file mode 100644 index 000000000000..3ba41d76a582 --- /dev/null +++ b/include/linux/prmem.h @@ -0,0 +1,213 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * prmem.h: Header for memory protection library + * + * (C) Copyright 2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + * + * Support for: + * - statically allocated write rare data + */ + +#ifndef _LINUX_PRMEM_H +#define _LINUX_PRMEM_H + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* ============================ Write Rare ============================ */ + +extern const char WR_ERR_RANGE_MSG[]; +extern const char WR_ERR_PAGE_MSG[]; + +/* + * The following two variables are statically allocated by the linker + * script at the the boundaries of the memory region (rounded up to + * multiples of PAGE_SIZE) reserved for __wr_after_init. + */ +extern long __start_wr_after_init; +extern long __end_wr_after_init; + +static __always_inline bool __is_wr_after_init(const void *ptr, size_t size) +{ + size_t start = (size_t)&__start_wr_after_init; + size_t end = (size_t)&__end_wr_after_init; + size_t low = (size_t)ptr; + size_t high = (size_t)ptr + size; + + return likely(start <= low && low < high && high <= end); +} + +/** + * wr_memset() - sets n bytes of the destination to the c value + * @dst: beginning of the memory to write to + * @c: byte to replicate + * @size: amount of bytes to copy + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_memset(const void *dst, const int c, size_t n_bytes) +{ + size_t size; + unsigned long flags; + uintptr_t d = (uintptr_t)dst; + + if (WARN(!__is_wr_after_init(dst, n_bytes), WR_ERR_RANGE_MSG)) + return false; + while (n_bytes) { + struct page *page; + uintptr_t base; + uintptr_t offset; + uintptr_t offset_complement; + + local_irq_save(flags); + page = virt_to_page(d); + offset = d & ~PAGE_MASK; + offset_complement = PAGE_SIZE - offset; + size = min(n_bytes, offset_complement); + base = (uintptr_t)vmap(&page, 1, VM_MAP, PAGE_KERNEL); + if (WARN(!base, WR_ERR_PAGE_MSG)) { + local_irq_restore(flags); + return false; + } + memset((void *)(base + offset), c, size); + vunmap((void *)base); + d += size; + n_bytes -= size; + local_irq_restore(flags); + } + return true; +} + +/** + * wr_memcpy() - copyes n bytes from source to destination + * @dst: beginning of the memory to write to + * @src: beginning of the memory to read from + * @n_bytes: amount of bytes to copy + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_memcpy(const void *dst, const void *src, size_t n_bytes) +{ + size_t size; + unsigned long flags; + uintptr_t d = (uintptr_t)dst; + uintptr_t s = (uintptr_t)src; + + if (WARN(!__is_wr_after_init(dst, n_bytes), WR_ERR_RANGE_MSG)) + return false; + while (n_bytes) { + struct page *page; + uintptr_t base; + uintptr_t offset; + uintptr_t offset_complement; + + local_irq_save(flags); + page = virt_to_page(d); + offset = d & ~PAGE_MASK; + offset_complement = PAGE_SIZE - offset; + size = (size_t)min(n_bytes, offset_complement); + base = (uintptr_t)vmap(&page, 1, VM_MAP, PAGE_KERNEL); + if (WARN(!base, WR_ERR_PAGE_MSG)) { + local_irq_restore(flags); + return false; + } + __write_once_size((void *)(base + offset), (void *)s, size); + vunmap((void *)base); + d += size; + s += size; + n_bytes -= size; + local_irq_restore(flags); + } + return true; +} + +/* + * rcu_assign_pointer is a macro, which takes advantage of being able to + * take the address of the destination parameter "p", so that it can be + * passed to WRITE_ONCE(), which is called in one of the branches of + * rcu_assign_pointer() and also, being a macro, can rely on the + * preprocessor for taking the address of its parameter. + * For the sake of staying compatible with the API, also + * wr_rcu_assign_pointer() is a macro that accepts a pointer as parameter, + * instead of the address of said pointer. + * However it is simply a wrapper to __wr_rcu_ptr(), which receives the + * address of the pointer. + */ +static __always_inline +uintptr_t __wr_rcu_ptr(const void *dst_p_p, const void *src_p) +{ + unsigned long flags; + struct page *page; + void *base; + uintptr_t offset; + const size_t size = sizeof(void *); + + if (WARN(!__is_wr_after_init(dst_p_p, size), WR_ERR_RANGE_MSG)) + return (uintptr_t)NULL; + local_irq_save(flags); + page = virt_to_page(dst_p_p); + offset = (uintptr_t)dst_p_p & ~PAGE_MASK; + base = vmap(&page, 1, VM_MAP, PAGE_KERNEL); + if (WARN(!base, WR_ERR_PAGE_MSG)) { + local_irq_restore(flags); + return (uintptr_t)NULL; + } + rcu_assign_pointer((*(void **)(offset + (uintptr_t)base)), src_p); + vunmap(base); + local_irq_restore(flags); + return (uintptr_t)src_p; +} + +#define wr_rcu_assign_pointer(p, v) __wr_rcu_ptr(&p, v) + +#define __wr_simple(dst_ptr, src_ptr) \ + wr_memcpy(dst_ptr, src_ptr, sizeof(*(src_ptr))) + +#define __wr_safe(dst_ptr, src_ptr, \ + unique_dst_ptr, unique_src_ptr) \ +({ \ + typeof(dst_ptr) unique_dst_ptr = (dst_ptr); \ + typeof(src_ptr) unique_src_ptr = (src_ptr); \ + \ + wr_memcpy(unique_dst_ptr, unique_src_ptr, \ + sizeof(*(unique_src_ptr))); \ +}) + +#define __safe_ops(dst, src) \ + (__typecheck(dst, src) && __no_side_effects(dst, src)) + +/** + * wr - copies an object over another of same type and size + * @dst_ptr: address of the destination object + * @src_ptr: address of the source object + */ +#define wr(dst_ptr, src_ptr) \ + __builtin_choose_expr(__safe_ops(dst_ptr, src_ptr), \ + __wr_simple(dst_ptr, src_ptr), \ + __wr_safe(dst_ptr, src_ptr, \ + __UNIQUE_ID(__dst_ptr), \ + __UNIQUE_ID(__src_ptr))) + +/** + * wr_ptr() - alters a pointer in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_ptr(const void *dst, const void *val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} +#endif diff --git a/mm/Makefile b/mm/Makefile index 26ef77a3883b..215c6a6d7304 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -64,6 +64,7 @@ obj-$(CONFIG_SPARSEMEM) += sparse.o obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o obj-$(CONFIG_SLOB) += slob.o obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o +obj-$(CONFIG_PRMEM) += prmem.o obj-$(CONFIG_KSM) += ksm.o obj-$(CONFIG_PAGE_POISONING) += page_poison.o obj-$(CONFIG_SLAB) += slab.o diff --git a/mm/prmem.c b/mm/prmem.c new file mode 100644 index 000000000000..de9258f5f29a --- /dev/null +++ b/mm/prmem.c @@ -0,0 +1,10 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * prmem.c: Memory Protection Library + * + * (C) Copyright 2017-2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + */ + +const char WR_ERR_RANGE_MSG[] = "Write rare on invalid memory range."; +const char WR_ERR_PAGE_MSG[] = "Failed to remap write rare page."; From patchwork Tue Oct 23 21:34:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653731 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 44CB313A4 for ; Tue, 23 Oct 2018 21:36:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EB662A3D2 for ; Tue, 23 Oct 2018 21:36:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1E1612A3D3; Tue, 23 Oct 2018 21:36:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 1DB372A3D1 for ; Tue, 23 Oct 2018 21:36:47 +0000 (UTC) Received: (qmail 13702 invoked by uid 550); 23 Oct 2018 21:36:13 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13575 invoked from network); 23 Oct 2018 21:36:13 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=6jXsMcNOASyEUd+AGjVZam8sdvDi65Nq+ICQnyI7NC8=; b=D2P3EntG9KSCG0qbpHMKa1F9RkSy7Fi1vmdHYMZHlxzpBrRFHHM9NmSDNJ6UUFNIPF XSi1KAxTa0aD5TdT4IY4q5HajQnHuMo2x/tOfbQmk3e8Gv/273K7kgKcTKKN8ICPJDC8 4nfmvBBbpIlF4xCgqU3MJDPqYXXkrivUSNIamTfQjKKdnAPxoAywQP5MZOy3u2PJdNbN 2NQqXXVPhvdWOQYVBqXNNymOa3SgIAbrVfxW3CzlsnFmaD5CaeVIfl4C900BUKgXU5an FofC3Bb9uPmIhTPS+tO5MFQl0G6FTHn7fQfWj2uSai/ifDGqQQf+uh5p7/Lk5iQgg19Q Wf2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=6jXsMcNOASyEUd+AGjVZam8sdvDi65Nq+ICQnyI7NC8=; b=NW2lFBZ0C8xEiRi4SqA6PpxQeZUfUwfVVR/ZNno0X65zDwvrpwFNXWE3kD7kn12d9q ZivBtdb8CRId7qgcOdxUs5HGot8P287FkGjD+sLVuWS3AVBz4XhnU3Yxxhep4Vkb/QI6 CcwCL4UVdkNpNYFcXheVY7watbSPdoJwVx6xuHFgCnH3RGCnL5hwPyFs8K2WUBDOOuni e3jPd0eYyhRZ9o0GviTaYBx2WHQ+cvMjRW4PtdsQnan8euCfWoGhPcmZLWsDABGQ1TAd +WZ2TRAmYbJh3GJQ49rE1zyRj1BXDLboLQIgn8H0TyCas4/mEtELA+9FIENsmGTlomrf bf/w== X-Gm-Message-State: ABuFfoh0+opQAJktKlOO8bzE0KLMv8F6FSQaciASGfkGmq2bdPR9FFvC QD8aGvvT8C/+dzx+ijIZLns= X-Google-Smtp-Source: ACcGV602U2IyIsctESkBUXllpEO17589f4KnIPcsRRLcANMrqKwA6sberqeguWGCzYnkS5FtCx9hDw== X-Received: by 2002:a19:c954:: with SMTP id z81mr8626755lff.150.1540330561406; Tue, 23 Oct 2018 14:36:01 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Andrew Morton , Chintan Pandya , Joe Perches , "Luis R. Rodriguez" , Thomas Gleixner , Kate Stewart , Greg Kroah-Hartman , Philippe Ombredanne , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/17] prmem: vmalloc support for dynamic allocation Date: Wed, 24 Oct 2018 00:34:50 +0300 Message-Id: <20181023213504.28905-4-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Prepare vmalloc for: - tagging areas used for dynamic allocation of protected memory - supporting various tags, related to the property that an area might have - extrapolating the pool containing a given area - chaining the areas in each pool - extrapolating the area containing a given memory address NOTE: Since there is a list_head structure that is used only when disposing of the allocation (the field purge_list), there are two pointers for the take, before it comes the time of freeing the allocation. To avoid increasing the size of the vmap_area structure, instead of using a standard doubly linked list for tracking the chain of vmap_areas, only one pointer is spent for this purpose, in a single linked list, while the other is used to provide a direct connection to the parent pool. Signed-off-by: Igor Stoppa CC: Michal Hocko CC: Andrew Morton CC: Chintan Pandya CC: Joe Perches CC: "Luis R. Rodriguez" CC: Thomas Gleixner CC: Kate Stewart CC: Greg Kroah-Hartman CC: Philippe Ombredanne CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- include/linux/vmalloc.h | 12 +++++++++++- mm/vmalloc.c | 2 +- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 398e9c95cd61..4d14a3b8089e 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -21,6 +21,9 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +#define VM_PMALLOC 0x00000100 /* pmalloc area - see docs */ +#define VM_PMALLOC_WR 0x00000200 /* pmalloc write rare area */ +#define VM_PMALLOC_PROTECTED 0x00000400 /* pmalloc protected area */ /* bits [20..32] reserved for arch specific ioremap internals */ /* @@ -48,7 +51,13 @@ struct vmap_area { unsigned long flags; struct rb_node rb_node; /* address sorted rbtree */ struct list_head list; /* address sorted list */ - struct llist_node purge_list; /* "lazy purge" list */ + union { + struct llist_node purge_list; /* "lazy purge" list */ + struct { + struct vmap_area *next; + struct pmalloc_pool *pool; + }; + }; struct vm_struct *vm; struct rcu_head rcu_head; }; @@ -134,6 +143,7 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size, const void *caller); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); +extern struct vmap_area *find_vmap_area(unsigned long addr); extern int map_vm_area(struct vm_struct *area, pgprot_t prot, struct page **pages); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a728fc492557..15850005fea5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -742,7 +742,7 @@ static void free_unmap_vmap_area(struct vmap_area *va) free_vmap_area_noflush(va); } -static struct vmap_area *find_vmap_area(unsigned long addr) +struct vmap_area *find_vmap_area(unsigned long addr) { struct vmap_area *va; From patchwork Tue Oct 23 21:34:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 124A513A4 for ; Tue, 23 Oct 2018 21:37:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F12182A3D3 for ; Tue, 23 Oct 2018 21:36:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E50A12A3DF; Tue, 23 Oct 2018 21:36:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id BE7822A417 for ; Tue, 23 Oct 2018 21:36:57 +0000 (UTC) Received: (qmail 13896 invoked by uid 550); 23 Oct 2018 21:36:16 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13715 invoked from network); 23 Oct 2018 21:36:14 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=BPrmQGSS7+P/PZf2emVi7OsnzNgO+iHidISXWBRSfPE=; b=DoiC6wQisGGrVv26XuC+Qp7H0Yhkdxn1dP0Di9HH+10Paxyrei26cSg84c2OlChW29 f4CCkE/aAEgQp6PICZOCMeCNE+XeH6QzCAqn7ALyI692P4es+ybgSHXV/ommvBp6GMHy rB3Fr1TRRL3bnoz22rFuzfWql2t+gJ24jTQNZL626O1b4IIvcfS0OHwQ8PBhaHTtG0Q0 rZZk4NVLgVpSSRkW25Q8oVf+w4tq7S9WvG0fZdHST1mAPnB228hfKhou36oVEcAezrlq ZPQkTPL+m4giqx/10RhfN5XKTo3aJOkPp1X0/WGvBR5FT8cBLcc+bC5az7VTOZ3wAPAr 4lOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=BPrmQGSS7+P/PZf2emVi7OsnzNgO+iHidISXWBRSfPE=; b=bghcV8iC+r+eTd3x6yIKyE6ahz3ajpwx4W6O/uJkHz9z9Ybsdmh2lPjISCHABVlzsC 7GKcrIbqxm4TXuqgjJhZNanZM46FG5jL89uxXOmpQgtRD6dYq2rqBIbqIKTFliEFCZpw kRn+CtW4Fhc6PJ/fWiddtQFYNwsx+vkownAGqe+LSAGzx2sgCalbsuRZsttgo1Dbh0F8 WXgJjtRxgaJpyu/Qd9XAxFMV95BPskzJecIcaOtS2tszes2qjkbqqlJv2cYC9w/C+TxH pjVIF/Yx/ECbxiq5Z7o8fIIwVVRE5P9GLhFM1HQOJyHKBYJYp85haLCcbs4wSOhz2cwv jbeQ== X-Gm-Message-State: ABuFfoi348PoadpcRGcqtn0mYdjnyx1WHOCEI70Es/79u5DbddplcKuI nU8WfW9kOc0+qyIq/bZbSJ8= X-Google-Smtp-Source: AJdET5c4Z8zxOfgi/VC2WlGmCkyv8DTNHUz5R8wbf90K3dA5F8NEVcO3Hk9pPIj9oC3Tc8GWx9/mKA== X-Received: by 2002:a2e:5c2:: with SMTP id 185-v6mr3789201ljf.127.1540330563026; Tue, 23 Oct 2018 14:36:03 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Pavel Tatashin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/17] prmem: dynamic allocation Date: Wed, 24 Oct 2018 00:34:51 +0300 Message-Id: <20181023213504.28905-5-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Extension of protected memory to dynamic allocations. Allocations are performed from "pools". A pool is a list of virtual memory areas, in various state of protection. Supported cases =============== Read Only Pool -------------- Memory is allocated from the pool, in writable state. Then it gets written and the content of the pool is write protected and it cannot be altered anymore. It is only possible to destroy the pool. Auto Read Only Pool ------------------- Same as the plain read only, but every time a memory area is full and phased out, it is automatically marked as read only. Write Rare Pool --------------- Memory is allocated from the pool, in writable state. Then it gets written and the content of the pool is write protected and it can be altered only by invoking special write rare functions. Auto Write Rare Pool -------------------- Same as the plain write rare, but every time a memory area is full and phased out, it is automatically marked as write rare. Start Write Rare Pool --------------------- The memory handed out is already in write rare mode and the only way to alter it is to use write rare functions. When a pool is destroyed, all the memory that was obtained from it is automatically freed. This is the only way to release protected memory. Signed-off-by: Igor Stoppa CC: Michal Hocko CC: Vlastimil Babka CC: "Kirill A. Shutemov" CC: Andrew Morton CC: Pavel Tatashin CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- include/linux/prmem.h | 220 +++++++++++++++++++++++++++++++++-- mm/Kconfig | 6 + mm/prmem.c | 263 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 482 insertions(+), 7 deletions(-) diff --git a/include/linux/prmem.h b/include/linux/prmem.h index 3ba41d76a582..26fd48410d97 100644 --- a/include/linux/prmem.h +++ b/include/linux/prmem.h @@ -7,6 +7,8 @@ * * Support for: * - statically allocated write rare data + * - dynamically allocated read only data + * - dynamically allocated write rare data */ #ifndef _LINUX_PRMEM_H @@ -22,6 +24,11 @@ #include #include +#define VM_PMALLOC_MASK \ + (VM_PMALLOC | VM_PMALLOC_WR | VM_PMALLOC_PROTECTED) +#define VM_PMALLOC_WR_MASK (VM_PMALLOC | VM_PMALLOC_WR) +#define VM_PMALLOC_PROTECTED_MASK (VM_PMALLOC | VM_PMALLOC_PROTECTED) + /* ============================ Write Rare ============================ */ extern const char WR_ERR_RANGE_MSG[]; @@ -45,11 +52,23 @@ static __always_inline bool __is_wr_after_init(const void *ptr, size_t size) return likely(start <= low && low < high && high <= end); } +static __always_inline bool __is_wr_pool(const void *ptr, size_t size) +{ + struct vmap_area *area; + + if (!is_vmalloc_addr(ptr)) + return false; + area = find_vmap_area((unsigned long)ptr); + return area && area->vm && (area->vm->size >= size) && + ((area->vm->flags & (VM_PMALLOC | VM_PMALLOC_WR)) == + (VM_PMALLOC | VM_PMALLOC_WR)); +} + /** * wr_memset() - sets n bytes of the destination to the c value * @dst: beginning of the memory to write to * @c: byte to replicate - * @size: amount of bytes to copy + * @n_bytes: amount of bytes to copy * * Returns true on success, false otherwise. */ @@ -59,8 +78,10 @@ bool wr_memset(const void *dst, const int c, size_t n_bytes) size_t size; unsigned long flags; uintptr_t d = (uintptr_t)dst; + bool is_virt = __is_wr_after_init(dst, n_bytes); - if (WARN(!__is_wr_after_init(dst, n_bytes), WR_ERR_RANGE_MSG)) + if (WARN(!(is_virt || likely(__is_wr_pool(dst, n_bytes))), + WR_ERR_RANGE_MSG)) return false; while (n_bytes) { struct page *page; @@ -69,7 +90,10 @@ bool wr_memset(const void *dst, const int c, size_t n_bytes) uintptr_t offset_complement; local_irq_save(flags); - page = virt_to_page(d); + if (is_virt) + page = virt_to_page(d); + else + page = vmalloc_to_page((void *)d); offset = d & ~PAGE_MASK; offset_complement = PAGE_SIZE - offset; size = min(n_bytes, offset_complement); @@ -102,8 +126,10 @@ bool wr_memcpy(const void *dst, const void *src, size_t n_bytes) unsigned long flags; uintptr_t d = (uintptr_t)dst; uintptr_t s = (uintptr_t)src; + bool is_virt = __is_wr_after_init(dst, n_bytes); - if (WARN(!__is_wr_after_init(dst, n_bytes), WR_ERR_RANGE_MSG)) + if (WARN(!(is_virt || likely(__is_wr_pool(dst, n_bytes))), + WR_ERR_RANGE_MSG)) return false; while (n_bytes) { struct page *page; @@ -112,7 +138,10 @@ bool wr_memcpy(const void *dst, const void *src, size_t n_bytes) uintptr_t offset_complement; local_irq_save(flags); - page = virt_to_page(d); + if (is_virt) + page = virt_to_page(d); + else + page = vmalloc_to_page((void *)d); offset = d & ~PAGE_MASK; offset_complement = PAGE_SIZE - offset; size = (size_t)min(n_bytes, offset_complement); @@ -151,11 +180,13 @@ uintptr_t __wr_rcu_ptr(const void *dst_p_p, const void *src_p) void *base; uintptr_t offset; const size_t size = sizeof(void *); + bool is_virt = __is_wr_after_init(dst_p_p, size); - if (WARN(!__is_wr_after_init(dst_p_p, size), WR_ERR_RANGE_MSG)) + if (WARN(!(is_virt || likely(__is_wr_pool(dst_p_p, size))), + WR_ERR_RANGE_MSG)) return (uintptr_t)NULL; local_irq_save(flags); - page = virt_to_page(dst_p_p); + page = is_virt ? virt_to_page(dst_p_p) : vmalloc_to_page(dst_p_p); offset = (uintptr_t)dst_p_p & ~PAGE_MASK; base = vmap(&page, 1, VM_MAP, PAGE_KERNEL); if (WARN(!base, WR_ERR_PAGE_MSG)) { @@ -210,4 +241,179 @@ bool wr_ptr(const void *dst, const void *val) { return wr_memcpy(dst, &val, sizeof(val)); } + +/* ============================ Allocator ============================ */ + +#define PMALLOC_REFILL_DEFAULT (0) +#define PMALLOC_DEFAULT_REFILL_SIZE PAGE_SIZE +#define PMALLOC_ALIGN_ORDER_DEFAULT ilog2(ARCH_KMALLOC_MINALIGN) + +#define PMALLOC_RO 0x00 +#define PMALLOC_WR 0x01 +#define PMALLOC_AUTO 0x02 +#define PMALLOC_START 0x04 +#define PMALLOC_MASK (PMALLOC_WR | PMALLOC_AUTO | PMALLOC_START) + +#define PMALLOC_MODE_RO PMALLOC_RO +#define PMALLOC_MODE_WR PMALLOC_WR +#define PMALLOC_MODE_AUTO_RO (PMALLOC_RO | PMALLOC_AUTO) +#define PMALLOC_MODE_AUTO_WR (PMALLOC_WR | PMALLOC_AUTO) +#define PMALLOC_MODE_START_WR (PMALLOC_WR | PMALLOC_START) + +struct pmalloc_pool { + struct mutex mutex; + struct list_head pool_node; + struct vmap_area *area; + size_t align; + size_t refill; + size_t offset; + uint8_t mode; +}; + +/* + * The write rare functionality is fully implemented as __always_inline, + * to prevent having an internal function call that is capable of modifying + * write protected memory. + * Fully inlining the function allows the compiler to optimize away its + * interface, making it harder for an attacker to hijack it. + * This still leaves the door open to attacks that might try to reuse part + * of the code, by jumping in the middle of the function, however it can + * be mitigated by having a compiler plugin that enforces Control Flow + * Integrity (CFI). + * Any addition/modification to the write rare path must follow the same + * approach. + */ + +void pmalloc_init_custom_pool(struct pmalloc_pool *pool, size_t refill, + short align_order, uint8_t mode); + +struct pmalloc_pool *pmalloc_create_custom_pool(size_t refill, + short align_order, + uint8_t mode); + +/** + * pmalloc_create_pool() - create a protectable memory pool + * @mode: can the data be altered after protection + * + * Shorthand for pmalloc_create_custom_pool() with default argument: + * * refill is set to PMALLOC_REFILL_DEFAULT + * * align_order is set to PMALLOC_ALIGN_ORDER_DEFAULT + * + * Returns: + * * pointer to the new pool - success + * * NULL - error + */ +static inline struct pmalloc_pool *pmalloc_create_pool(uint8_t mode) +{ + return pmalloc_create_custom_pool(PMALLOC_REFILL_DEFAULT, + PMALLOC_ALIGN_ORDER_DEFAULT, + mode); +} + +void *pmalloc(struct pmalloc_pool *pool, size_t size); + +/** + * pzalloc() - zero-initialized version of pmalloc() + * @pool: handle to the pool to be used for memory allocation + * @size: amount of memory (in bytes) requested + * + * Executes pmalloc(), initializing the memory requested to 0, before + * returning its address. + * + * Return: + * * pointer to the memory requested - success + * * NULL - error + */ +static inline void *pzalloc(struct pmalloc_pool *pool, size_t size) +{ + void *ptr = pmalloc(pool, size); + + if (unlikely(!ptr)) + return ptr; + if ((pool->mode & PMALLOC_MODE_START_WR) == PMALLOC_MODE_START_WR) + wr_memset(ptr, 0, size); + else + memset(ptr, 0, size); + return ptr; +} + +/** + * pmalloc_array() - array version of pmalloc() + * @pool: handle to the pool to be used for memory allocation + * @n: number of elements in the array + * @size: amount of memory (in bytes) requested for each element + * + * Executes pmalloc(), on an array. + * + * Return: + * * the pmalloc result - success + * * NULL - error + */ + +static inline +void *pmalloc_array(struct pmalloc_pool *pool, size_t n, size_t size) +{ + size_t total_size = n * size; + + if (unlikely(!(n && (total_size / n == size)))) + return NULL; + return pmalloc(pool, n * size); +} + +/** + * pcalloc() - array version of pzalloc() + * @pool: handle to the pool to be used for memory allocation + * @n: number of elements in the array + * @size: amount of memory (in bytes) requested for each element + * + * Executes pzalloc(), on an array. + * + * Return: + * * the pmalloc result - success + * * NULL - error + */ +static inline +void *pcalloc(struct pmalloc_pool *pool, size_t n, size_t size) +{ + size_t total_size = n * size; + + if (unlikely(!(n && (total_size / n == size)))) + return NULL; + return pzalloc(pool, n * size); +} + +/** + * pstrdup() - duplicate a string, using pmalloc() + * @pool: handle to the pool to be used for memory allocation + * @s: string to duplicate + * + * Generates a copy of the given string, allocating sufficient memory + * from the given pmalloc pool. + * + * Return: + * * pointer to the replica - success + * * NULL - error + */ +static inline char *pstrdup(struct pmalloc_pool *pool, const char *s) +{ + size_t len; + char *buf; + + len = strlen(s) + 1; + buf = pmalloc(pool, len); + if (unlikely(!buf)) + return buf; + if ((pool->mode & PMALLOC_MODE_START_WR) == PMALLOC_MODE_START_WR) + wr_memcpy(buf, s, len); + else + strncpy(buf, s, len); + return buf; +} + + +void pmalloc_protect_pool(struct pmalloc_pool *pool); + +void pmalloc_make_pool_ro(struct pmalloc_pool *pool); + +void pmalloc_destroy_pool(struct pmalloc_pool *pool); #endif diff --git a/mm/Kconfig b/mm/Kconfig index de64ea658716..1885f5565cbc 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -764,4 +764,10 @@ config GUP_BENCHMARK config ARCH_HAS_PTE_SPECIAL bool +config PRMEM + bool + depends on MMU + depends on ARCH_HAS_SET_MEMORY + default y + endmenu diff --git a/mm/prmem.c b/mm/prmem.c index de9258f5f29a..7dd13ea43304 100644 --- a/mm/prmem.c +++ b/mm/prmem.c @@ -6,5 +6,268 @@ * Author: Igor Stoppa */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + const char WR_ERR_RANGE_MSG[] = "Write rare on invalid memory range."; const char WR_ERR_PAGE_MSG[] = "Failed to remap write rare page."; + +static LIST_HEAD(pools_list); +static DEFINE_MUTEX(pools_mutex); + +#define MAX_ALIGN_ORDER (ilog2(sizeof(void *))) + + +/* Various helper functions. Inlined, to reduce the attack surface. */ + +static __always_inline void protect_area(struct vmap_area *area) +{ + set_memory_ro(area->va_start, area->vm->nr_pages); + area->vm->flags |= VM_PMALLOC_PROTECTED_MASK; +} + +static __always_inline bool empty(struct pmalloc_pool *pool) +{ + return unlikely(!pool->area); +} + +/* Allocation from a protcted area is allowed only for a START_WR pool. */ +static __always_inline bool unwritable(struct pmalloc_pool *pool) +{ + return (pool->area->vm->flags & VM_PMALLOC_PROTECTED) && + !((pool->area->vm->flags & VM_PMALLOC_WR) && + (pool->mode & PMALLOC_START)); +} + +static __always_inline +bool exhausted(struct pmalloc_pool *pool, size_t size) +{ + size_t space_before; + size_t space_after; + + space_before = round_down(pool->offset, pool->align); + space_after = pool->offset - space_before; + return unlikely(space_after < size && space_before < size); +} + +static __always_inline +bool space_needed(struct pmalloc_pool *pool, size_t size) +{ + return empty(pool) || unwritable(pool) || exhausted(pool, size); +} + +/** + * pmalloc_init_custom_pool() - initialize a protectable memory pool + * @pool: the pointer to the struct pmalloc_pool to initialize + * @refill: the minimum size to allocate when in need of more memory. + * It will be rounded up to a multiple of PAGE_SIZE + * The value of 0 gives the default amount of PAGE_SIZE. + * @align_order: log2 of the alignment to use when allocating memory + * Negative values give ARCH_KMALLOC_MINALIGN + * @mode: is the data RO or RareWrite and should be provided already in + * protected mode. + * The value is one of: + * PMALLOC_MODE_RO, PMALLOC_MODE_WR, PMALLOC_MODE_AUTO_RO + * PMALLOC_MODE_AUTO_WR, PMALLOC_MODE_START_WR + * + * Initializes an empty memory pool, for allocation of protectable + * memory. Memory will be allocated upon request (through pmalloc). + */ +void pmalloc_init_custom_pool(struct pmalloc_pool *pool, size_t refill, + short align_order, uint8_t mode) +{ + mutex_init(&pool->mutex); + pool->area = NULL; + if (align_order < 0) + pool->align = ARCH_KMALLOC_MINALIGN; + else + pool->align = 1UL << align_order; + pool->refill = refill ? PAGE_ALIGN(refill) : + PMALLOC_DEFAULT_REFILL_SIZE; + mode &= PMALLOC_MASK; + if (mode & PMALLOC_START) + mode |= PMALLOC_WR; + pool->mode = mode & PMALLOC_MASK; + pool->offset = 0; + mutex_lock(&pools_mutex); + list_add(&pool->pool_node, &pools_list); + mutex_unlock(&pools_mutex); +} +EXPORT_SYMBOL(pmalloc_init_custom_pool); + +/** + * pmalloc_create_custom_pool() - create a new protectable memory pool + * @refill: the minimum size to allocate when in need of more memory. + * It will be rounded up to a multiple of PAGE_SIZE + * The value of 0 gives the default amount of PAGE_SIZE. + * @align_order: log2 of the alignment to use when allocating memory + * Negative values give ARCH_KMALLOC_MINALIGN + * @mode: can the data be altered after protection + * + * Creates a new (empty) memory pool for allocation of protectable + * memory. Memory will be allocated upon request (through pmalloc). + * + * Return: + * * pointer to the new pool - success + * * NULL - error + */ +struct pmalloc_pool *pmalloc_create_custom_pool(size_t refill, + short align_order, + uint8_t mode) +{ + struct pmalloc_pool *pool; + + pool = kmalloc(sizeof(struct pmalloc_pool), GFP_KERNEL); + if (WARN(!pool, "Could not allocate pool meta data.")) + return NULL; + pmalloc_init_custom_pool(pool, refill, align_order, mode); + return pool; +} +EXPORT_SYMBOL(pmalloc_create_custom_pool); + +static int grow(struct pmalloc_pool *pool, size_t min_size) +{ + void *addr; + struct vmap_area *new_area; + unsigned long size; + uint32_t tag_mask; + + size = (min_size > pool->refill) ? min_size : pool->refill; + addr = vmalloc(size); + if (WARN(!addr, "Failed to allocate %zd bytes", PAGE_ALIGN(size))) + return -ENOMEM; + + new_area = find_vmap_area((uintptr_t)addr); + tag_mask = VM_PMALLOC; + if (pool->mode & PMALLOC_WR) + tag_mask |= VM_PMALLOC_WR; + new_area->vm->flags |= (tag_mask & VM_PMALLOC_MASK); + new_area->pool = pool; + if (pool->mode & PMALLOC_START) + protect_area(new_area); + if (pool->mode & PMALLOC_AUTO && !empty(pool)) + protect_area(pool->area); + /* The area size backed by pages, without the canary bird. */ + pool->offset = new_area->vm->nr_pages * PAGE_SIZE; + new_area->next = pool->area; + pool->area = new_area; + return 0; +} + +/** + * pmalloc() - allocate protectable memory from a pool + * @pool: handle to the pool to be used for memory allocation + * @size: amount of memory (in bytes) requested + * + * Allocates memory from a pool. + * If needed, the pool will automatically allocate enough memory to + * either satisfy the request or meet the "refill" parameter received + * upon creation. + * New allocation can happen also if the current memory in the pool is + * already write protected. + * Allocation happens with a mutex locked, therefore it is assumed to have + * exclusive write access to both the pool structure and the list of + * vmap_areas, while inside the lock. + * + * Return: + * * pointer to the memory requested - success + * * NULL - error + */ +void *pmalloc(struct pmalloc_pool *pool, size_t size) +{ + void *retval = NULL; + + mutex_lock(&pool->mutex); + if (unlikely(space_needed(pool, size)) && + unlikely(grow(pool, size) != 0)) + goto error; + pool->offset = round_down(pool->offset - size, pool->align); + retval = (void *)(pool->area->va_start + pool->offset); +error: + mutex_unlock(&pool->mutex); + return retval; +} +EXPORT_SYMBOL(pmalloc); + +/** + * pmalloc_protect_pool() - write-protects the memory in the pool + * @pool: the pool associated to the memory to write-protect + * + * Write-protects all the memory areas currently assigned to the pool + * that are still unprotected. + * This does not prevent further allocation of additional memory, that + * can be initialized and protected. + * The catch is that protecting a pool will make unavailable whatever + * free memory it might still contain. + * Successive allocations will grab more free pages. + */ +void pmalloc_protect_pool(struct pmalloc_pool *pool) +{ + struct vmap_area *area; + + mutex_lock(&pool->mutex); + for (area = pool->area; area; area = area->next) + protect_area(area); + mutex_unlock(&pool->mutex); +} +EXPORT_SYMBOL(pmalloc_protect_pool); + + +/** + * pmalloc_make_pool_ro() - forces a pool to become read-only + * @pool: the pool associated to the memory to make ro + * + * Drops the possibility to perform controlled writes from both the pool + * metadata and all the vm_area structures associated to the pool. + * In case the pool was configured to automatically protect memory when + * allocating it, the configuration is dropped. + */ +void pmalloc_make_pool_ro(struct pmalloc_pool *pool) +{ + struct vmap_area *area; + + mutex_lock(&pool->mutex); + pool->mode &= ~(PMALLOC_WR | PMALLOC_START); + for (area = pool->area; area; area = area->next) + protect_area(area); + mutex_unlock(&pool->mutex); +} +EXPORT_SYMBOL(pmalloc_make_pool_ro); + +/** + * pmalloc_destroy_pool() - destroys a pool and all the associated memory + * @pool: the pool to destroy + * + * All the memory associated to the pool will be freed, including the + * metadata used for the pool. + */ +void pmalloc_destroy_pool(struct pmalloc_pool *pool) +{ + struct vmap_area *area; + + mutex_lock(&pools_mutex); + list_del(&pool->pool_node); + mutex_unlock(&pools_mutex); + while (pool->area) { + area = pool->area; + pool->area = area->next; + set_memory_rw(area->va_start, area->vm->nr_pages); + area->vm->flags &= ~VM_PMALLOC_MASK; + vfree((void *)area->va_start); + } + kfree(pool); +} +EXPORT_SYMBOL(pmalloc_destroy_pool); From patchwork Tue Oct 23 21:34:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653755 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CACE914BB for ; Tue, 23 Oct 2018 21:37:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B78BF2A3DF for ; Tue, 23 Oct 2018 21:37:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AB8052A417; Tue, 23 Oct 2018 21:37:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 78E502A3DF for ; Tue, 23 Oct 2018 21:37:10 +0000 (UTC) Received: (qmail 13936 invoked by uid 550); 23 Oct 2018 21:36:17 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13766 invoked from network); 23 Oct 2018 21:36:16 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=9TeBZENTG4lUlljtOFNGnausIJ9loexqClGDhlY00q0=; b=JtoyO3mZ13IGzAbCIgy6S5RYsmImtY6hfg34KQRQ2X3TR1VVri5yBZ/LUjBcD0TzXa 0s8L1kB/YwwSuVWbA3820PquvwCu3SkJCcO1geuaAaOGfUNh9JAvXkHX3kScSRePUVzs LpBgTT1JYuKJ1aT49DAJUFExgxQPahyLwVz4BPku8ogRZBoPGO9RswhALPMjDi8XbttO OjGxJeSt7f0J7GWI7q5EdlapOM+CLE2SPmDyplRRclUZ20Yaw/5nyVtaOA6eSk9/RtMb rtjjIO80ygFKzuE+VuAYK60mF+XSs2WkRVh4yBh7Fgr9V0xTBNIf9zkz+/Qr1wr90r6X Paxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=9TeBZENTG4lUlljtOFNGnausIJ9loexqClGDhlY00q0=; b=Jo9/lQ2hgr2NyQ7FQZEWz1YwsBJhUNe2kqd373Esb5rBDK8eWtyA7/JnYlul6ssxdc 5109BF+u8S7HBvm2tAsGFn/IFPYtFmiMmHEoiB1YsqDVjZIJCTcJSRYS+P5qsemhk5ci +kz6ji7viA34lnSOZxjAZHhqLFM+iomvRUxDXgjrYmwdlRimBguzelylKelHqwTg+cMw 3fhMttkwcjJY8O9KroY3n+4bRlKtlx4me3i3b2D6lAhCLBWSX5uJgRxHI7kyHuCR/xPE Yzz9M7nqPqJpqAuT7XkPeRnvTS2A94B8zsXtcvKPhYhmK8IKC1gZn0QdxRWgv3ehYhZj +aBA== X-Gm-Message-State: AGRZ1gKEsrejCN8VoUyZ6W6CGUhPC9i5wLSFCrib51rV6Uy19BTj3/sM qAdnJM2aEdKXbmN4VVq1Nak= X-Google-Smtp-Source: AJdET5fm/SfTwnUQw/H/6N0AMH3Waqo8igwVte/fLnkcWUycwtZsZmlgduLgWV39YuSoalp0uJ0pjw== X-Received: by 2002:a2e:84c5:: with SMTP id q5-v6mr5777858ljh.65.1540330564560; Tue, 23 Oct 2018 14:36:04 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Pavel Tatashin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/17] prmem: shorthands for write rare on common types Date: Wed, 24 Oct 2018 00:34:52 +0300 Message-Id: <20181023213504.28905-6-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Wrappers around the basic write rare functionality, addressing several common data types found in the kernel, allowing to specify the new values through immediates, like constants and defines. Note: The list is not complete and could be expanded. Signed-off-by: Igor Stoppa CC: Michal Hocko CC: Vlastimil Babka CC: "Kirill A. Shutemov" CC: Andrew Morton CC: Pavel Tatashin CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- MAINTAINERS | 1 + include/linux/prmemextra.h | 133 +++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+) create mode 100644 include/linux/prmemextra.h diff --git a/MAINTAINERS b/MAINTAINERS index e566c5d09faf..df7221eca160 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9459,6 +9459,7 @@ M: Igor Stoppa L: kernel-hardening@lists.openwall.com S: Maintained F: include/linux/prmem.h +F: include/linux/prmemextra.h F: mm/prmem.c MEMORY MANAGEMENT diff --git a/include/linux/prmemextra.h b/include/linux/prmemextra.h new file mode 100644 index 000000000000..36995717720e --- /dev/null +++ b/include/linux/prmemextra.h @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * prmemextra.h: Shorthands for write rare of basic data types + * + * (C) Copyright 2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + * + */ + +#ifndef _LINUX_PRMEMEXTRA_H +#define _LINUX_PRMEMEXTRA_H + +#include + +/** + * wr_char - alters a char in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_char(const char *dst, const char val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_short - alters a short in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_short(const short *dst, const short val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_ushort - alters an unsigned short in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_ushort(const unsigned short *dst, const unsigned short val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_int - alters an int in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_int(const int *dst, const int val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_uint - alters an unsigned int in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_uint(const unsigned int *dst, const unsigned int val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_long - alters a long in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_long(const long *dst, const long val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_ulong - alters an unsigned long in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_ulong(const unsigned long *dst, const unsigned long val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_longlong - alters a long long in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_longlong(const long long *dst, const long long val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +/** + * wr_ulonglong - alters an unsigned long long in write rare memory + * @dst: target for write + * @val: new value + * + * Returns true on success, false otherwise. + */ +static __always_inline +bool wr_ulonglong(const unsigned long long *dst, + const unsigned long long val) +{ + return wr_memcpy(dst, &val, sizeof(val)); +} + +#endif From patchwork Tue Oct 23 21:34:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653769 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7DE341709 for ; Tue, 23 Oct 2018 21:37:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 692F22A457 for ; Tue, 23 Oct 2018 21:37:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5C00E2A4F0; Tue, 23 Oct 2018 21:37:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id E4D002A457 for ; Tue, 23 Oct 2018 21:37:22 +0000 (UTC) Received: (qmail 13994 invoked by uid 550); 23 Oct 2018 21:36:18 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13939 invoked from network); 23 Oct 2018 21:36:17 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=+x3wjQbIZdeZGc6lP1F3P1cnjhuI1El1LgW0Njws/+0=; b=C4wdxqvKukh1hR2H021cjWjr1iH+HfKmKOvjaW9GflyS+FVznw8fUOCxP21W7DQVwF datBHa6jy+35tG+E5UZL2nvoqZCsTC8roGcDdK6M6oes+nrdJYpQK2qF21d/UM7QjeqH czZuiCwokScBpZpqEpVFCVl1KRBGTaltxH9vhbX5GRk42gPCBPcH//iKgmWz0l4EMFrg iyDL/XZmSzY0RMuF6i5w3sA/FjEDqfpwUHku015OFS2WW9SGgFkh8lHLOh2jDK+LUiJN 8RV3ST40MTbH0t92c5BFOtAm4SfafZTcNTOjZJeeDMjIPkNGShA0J8NOxCA6Y56uFx6R wdOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=+x3wjQbIZdeZGc6lP1F3P1cnjhuI1El1LgW0Njws/+0=; b=TmLBvuHE+DAip+y/lOGo1Qz2QaC4yIFPljDgyJRYUpyA3A8MMtLRIo4BQQ/bMYbJ5q gRSrI8oJysvMll6LbVPe7RnU3/C/6v1ZvfWSy4jcyWQQQXH5BEbdAXOR1wRJj3ypvnA4 01fCh6VWTBK2XoahwbjJbMPZf/rnYDhE+F4OIuhfKks2qZzMpaPqV9NTuPjlsxj5LlmW DxtVv63xyFxrBymUSr6MXN5BMRnlgcW94RSVOcE+k8APw9v6YqCVazFwwnLgpmoNQlR/ NRNmV4+8fNA+lSr/GH5/XJXRFVQL14JYsHABd5KVuYdLknyvCLhIyGC4QTWTRKEOrC+w rZXQ== X-Gm-Message-State: AGRZ1gL4j6GxZ0HXQzDyZczqPqk91ftP/52ewSDI+JTTRfh9dpoEA+b0 +wGPNKDbkc9zIcIGxDQdEw0= X-Google-Smtp-Source: AJdET5c3IN0lDdmdWIU8iP0QNMnvHqIvH9eoglVtGKf8blfOXP0cHcpfvNfKzfj4EPX/HHCVxbEsaA== X-Received: by 2002:a19:ab1a:: with SMTP id u26-v6mr501479lfe.103.1540330566192; Tue, 23 Oct 2018 14:36:06 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Pavel Tatashin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/17] prmem: test cases for memory protection Date: Wed, 24 Oct 2018 00:34:53 +0300 Message-Id: <20181023213504.28905-7-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP The test cases verify the various interfaces offered by both prmem.h and prmemextra.h The tests avoid triggering crashes, by not performing actions that would be treated as illegal. That part is handled in the lkdtm patch. Signed-off-by: Igor Stoppa CC: Michal Hocko CC: Vlastimil Babka CC: "Kirill A. Shutemov" CC: Andrew Morton CC: Pavel Tatashin CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- MAINTAINERS | 2 + mm/Kconfig.debug | 9 + mm/Makefile | 1 + mm/test_pmalloc.c | 633 +++++++++++++++++++++++++++++++++++++++++++ mm/test_write_rare.c | 236 ++++++++++++++++ 5 files changed, 881 insertions(+) create mode 100644 mm/test_pmalloc.c create mode 100644 mm/test_write_rare.c diff --git a/MAINTAINERS b/MAINTAINERS index df7221eca160..ea979a5a9ec9 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9461,6 +9461,8 @@ S: Maintained F: include/linux/prmem.h F: include/linux/prmemextra.h F: mm/prmem.c +F: mm/test_write_rare.c +F: mm/test_pmalloc.c MEMORY MANAGEMENT L: linux-mm@kvack.org diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 9a7b8b049d04..57de5b3c0bae 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -94,3 +94,12 @@ config DEBUG_RODATA_TEST depends on STRICT_KERNEL_RWX ---help--- This option enables a testcase for the setting rodata read-only. + +config DEBUG_PRMEM_TEST + tristate "Run self test for protected memory" + depends on STRICT_KERNEL_RWX + select PRMEM + default n + help + Tries to verify that the memory protection works correctly and that + the memory is effectively protected. diff --git a/mm/Makefile b/mm/Makefile index 215c6a6d7304..93b503d4659f 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -65,6 +65,7 @@ obj-$(CONFIG_SPARSEMEM_VMEMMAP) += sparse-vmemmap.o obj-$(CONFIG_SLOB) += slob.o obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o obj-$(CONFIG_PRMEM) += prmem.o +obj-$(CONFIG_DEBUG_PRMEM_TEST) += test_write_rare.o test_pmalloc.o obj-$(CONFIG_KSM) += ksm.o obj-$(CONFIG_PAGE_POISONING) += page_poison.o obj-$(CONFIG_SLAB) += slab.o diff --git a/mm/test_pmalloc.c b/mm/test_pmalloc.c new file mode 100644 index 000000000000..f9ee8fb29eea --- /dev/null +++ b/mm/test_pmalloc.c @@ -0,0 +1,633 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * test_pmalloc.c + * + * (C) Copyright 2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + */ + +#include +#include +#include +#include +#include +#include + +#ifdef pr_fmt +#undef pr_fmt +#endif + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#define SIZE_1 (PAGE_SIZE * 3) +#define SIZE_2 1000 + +static const char MSG_NO_POOL[] = "Cannot allocate memory for the pool."; +static const char MSG_NO_PMEM[] = "Cannot allocate memory from the pool."; + +#define pr_success(test_name) \ + pr_info(test_name " test passed") + +/* --------------- tests the basic life-cycle of a pool --------------- */ + +static bool is_address_protected(void *p) +{ + struct page *page; + struct vmap_area *area; + + if (unlikely(!is_vmalloc_addr(p))) + return false; + page = vmalloc_to_page(p); + if (unlikely(!page)) + return false; + wmb(); /* Flush changes to the page table - is it needed? */ + area = find_vmap_area((uintptr_t)p); + if (unlikely((!area) || (!area->vm) || + ((area->vm->flags & VM_PMALLOC_PROTECTED_MASK) != + VM_PMALLOC_PROTECTED_MASK))) + return false; + return true; +} + +static bool create_and_destroy_pool(void) +{ + static struct pmalloc_pool *pool; + + pool = pmalloc_create_pool(PMALLOC_MODE_RO); + if (WARN(!pool, MSG_NO_POOL)) + return false; + pmalloc_destroy_pool(pool); + pr_success("pool creation and destruction"); + return true; +} + +/* verifies that it's possible to allocate from the pool */ +static bool test_alloc(void) +{ + static struct pmalloc_pool *pool; + static void *p; + + pool = pmalloc_create_pool(PMALLOC_MODE_RO); + if (WARN(!pool, MSG_NO_POOL)) + return false; + p = pmalloc(pool, SIZE_1 - 1); + pmalloc_destroy_pool(pool); + if (WARN(!p, MSG_NO_PMEM)) + return false; + pr_success("allocation capability"); + return true; +} + +/* ----------------------- tests self protection ----------------------- */ + +static bool test_auto_ro(void) +{ + struct pmalloc_pool *pool; + int *first_chunk; + int *second_chunk; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_AUTO_RO); + if (WARN(!pool, MSG_NO_POOL)) + return false; + first_chunk = (int *)pmalloc(pool, PMALLOC_DEFAULT_REFILL_SIZE); + if (WARN(!first_chunk, MSG_NO_PMEM)) + goto error; + second_chunk = (int *)pmalloc(pool, PMALLOC_DEFAULT_REFILL_SIZE); + if (WARN(!second_chunk, MSG_NO_PMEM)) + goto error; + if (WARN(!is_address_protected(first_chunk), + "Failed to automatically write protect exhausted vmarea")) + goto error; + pr_success("AUTO_RO"); + retval = true; +error: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_auto_wr(void) +{ + struct pmalloc_pool *pool; + int *first_chunk; + int *second_chunk; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_AUTO_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + first_chunk = (int *)pmalloc(pool, PMALLOC_DEFAULT_REFILL_SIZE); + if (WARN(!first_chunk, MSG_NO_PMEM)) + goto error; + second_chunk = (int *)pmalloc(pool, PMALLOC_DEFAULT_REFILL_SIZE); + if (WARN(!second_chunk, MSG_NO_PMEM)) + goto error; + if (WARN(!is_address_protected(first_chunk), + "Failed to automatically write protect exhausted vmarea")) + goto error; + pr_success("AUTO_WR"); + retval = true; +error: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_start_wr(void) +{ + struct pmalloc_pool *pool; + int *chunks[2]; + bool retval = false; + int i; + + pool = pmalloc_create_pool(PMALLOC_MODE_START_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + for (i = 0; i < 2; i++) { + chunks[i] = (int *)pmalloc(pool, 1); + if (WARN(!chunks[i], MSG_NO_PMEM)) + goto error; + if (WARN(!is_address_protected(chunks[i]), + "vmarea was not protected from the start")) + goto error; + } + if (WARN(vmalloc_to_page(chunks[0]) != vmalloc_to_page(chunks[1]), + "START_WR: mostly empty vmap area not reused")) + goto error; + pr_success("START_WR"); + retval = true; +error: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_self_protection(void) +{ + if (WARN(!(test_auto_ro() && + test_auto_wr() && + test_start_wr()), + "self protection tests failed")) + return false; + pr_success("self protection"); + return true; +} + +/* ----------------- tests basic write rare functions ----------------- */ + +#define INSERT_OFFSET (PAGE_SIZE * 3 / 2) +#define INSERT_SIZE (PAGE_SIZE * 2) +#define REGION_SIZE (PAGE_SIZE * 5) +static bool test_wr_memset(void) +{ + struct pmalloc_pool *pool; + char *region; + unsigned int i; + int retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_START_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + region = pzalloc(pool, REGION_SIZE); + if (WARN(!region, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < REGION_SIZE; i++) + if (WARN(region[i], "Failed to memset wr memory")) + goto destroy_pool; + retval = !wr_memset(region + INSERT_OFFSET, 1, INSERT_SIZE); + if (WARN(retval, "wr_memset failed")) + goto destroy_pool; + for (i = 0; i < REGION_SIZE; i++) + if (i >= INSERT_OFFSET && + i < (INSERT_SIZE + INSERT_OFFSET)) { + if (WARN(!region[i], + "Failed to alter target area")) + goto destroy_pool; + } else { + if (WARN(region[i] != 0, + "Unexpected alteration outside region")) + goto destroy_pool; + } + retval = true; + pr_success("wr_memset"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_strdup(void) +{ + const char src[] = "Some text for testing pstrdup()"; + struct pmalloc_pool *pool; + char *dst; + int retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + dst = pstrdup(pool, src); + if (WARN(!dst || strcmp(src, dst), "pmalloc wr strdup failed")) + goto destroy_pool; + retval = true; + pr_success("pmalloc wr strdup"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +/* Verify write rare across multiple pages, unaligned to PAGE_SIZE. */ +static bool test_wr_copy(void) +{ + struct pmalloc_pool *pool; + char *region; + char *mod; + unsigned int i; + int retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + region = pzalloc(pool, REGION_SIZE); + if (WARN(!region, MSG_NO_PMEM)) + goto destroy_pool; + mod = vmalloc(INSERT_SIZE); + if (WARN(!mod, "Failed to allocate memory from vmalloc")) + goto destroy_pool; + memset(mod, 0xA5, INSERT_SIZE); + pmalloc_protect_pool(pool); + retval = !wr_memcpy(region + INSERT_OFFSET, mod, INSERT_SIZE); + if (WARN(retval, "wr_copy failed")) + goto free_mod; + + for (i = 0; i < REGION_SIZE; i++) + if (i >= INSERT_OFFSET && + i < (INSERT_SIZE + INSERT_OFFSET)) { + if (WARN(region[i] != (char)0xA5, + "Failed to alter target area")) + goto free_mod; + } else { + if (WARN(region[i] != 0, + "Unexpected alteration outside region")) + goto free_mod; + } + retval = true; + pr_success("wr_copy"); +free_mod: + vfree(mod); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +/* ----------------- tests specialized write actions ------------------- */ + +#define TEST_ARRAY_SIZE 5 +#define TEST_ARRAY_TARGET (TEST_ARRAY_SIZE / 2) + +static bool test_wr_char(void) +{ + struct pmalloc_pool *pool; + char *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(char) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = (char)0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_char(array + TEST_ARRAY_TARGET, (char)0x5A), + "Failed to alter char variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? + (char)0x5A : (char)0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_char"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_short(void) +{ + struct pmalloc_pool *pool; + short *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(short) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = (short)0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_short(array + TEST_ARRAY_TARGET, (short)0x5A), + "Failed to alter short variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? + (short)0x5A : (short)0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_short"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_ushort(void) +{ + struct pmalloc_pool *pool; + unsigned short *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(unsigned short) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = (unsigned short)0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_ushort(array + TEST_ARRAY_TARGET, + (unsigned short)0x5A), + "Failed to alter unsigned short variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? + (unsigned short)0x5A : + (unsigned short)0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_ushort"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_int(void) +{ + struct pmalloc_pool *pool; + int *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(int) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = 0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_int(array + TEST_ARRAY_TARGET, 0x5A), + "Failed to alter int variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? 0x5A : 0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_int"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_uint(void) +{ + struct pmalloc_pool *pool; + unsigned int *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(unsigned int) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = 0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_uint(array + TEST_ARRAY_TARGET, 0x5A), + "Failed to alter unsigned int variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? 0x5A : 0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_uint"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_long(void) +{ + struct pmalloc_pool *pool; + long *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(long) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = 0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_long(array + TEST_ARRAY_TARGET, 0x5A), + "Failed to alter long variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? 0x5A : 0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_long"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_ulong(void) +{ + struct pmalloc_pool *pool; + unsigned long *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(unsigned long) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = 0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_ulong(array + TEST_ARRAY_TARGET, 0x5A), + "Failed to alter unsigned long variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? 0x5A : 0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_ulong"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_longlong(void) +{ + struct pmalloc_pool *pool; + long long *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(long long) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = 0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_longlong(array + TEST_ARRAY_TARGET, 0x5A), + "Failed to alter long variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? 0x5A : 0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_longlong"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_ulonglong(void) +{ + struct pmalloc_pool *pool; + unsigned long long *array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(unsigned long long) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = 0xA5; + pmalloc_protect_pool(pool); + if (WARN(!wr_ulonglong(array + TEST_ARRAY_TARGET, 0x5A), + "Failed to alter unsigned long long variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? 0x5A : 0xA5), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_ulonglong"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_wr_ptr(void) +{ + struct pmalloc_pool *pool; + int **array; + unsigned int i; + bool retval = false; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return false; + array = pmalloc(pool, sizeof(int *) * TEST_ARRAY_SIZE); + if (WARN(!array, MSG_NO_PMEM)) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + array[i] = NULL; + pmalloc_protect_pool(pool); + if (WARN(!wr_ptr(array + TEST_ARRAY_TARGET, array), + "Failed to alter ptr variable")) + goto destroy_pool; + for (i = 0; i < TEST_ARRAY_SIZE; i++) + if (WARN(array[i] != (i == TEST_ARRAY_TARGET ? + (void *)array : NULL), + "Unexpected value in test array.")) + goto destroy_pool; + retval = true; + pr_success("wr_ptr"); +destroy_pool: + pmalloc_destroy_pool(pool); + return retval; + +} + +static bool test_specialized_wrs(void) +{ + if (WARN(!(test_wr_char() && + test_wr_short() && + test_wr_ushort() && + test_wr_int() && + test_wr_uint() && + test_wr_long() && + test_wr_ulong() && + test_wr_longlong() && + test_wr_ulonglong() && + test_wr_ptr()), + "specialized write rare failed")) + return false; + pr_success("specialized write rare"); + return true; + +} + +/* + * test_pmalloc() -main entry point for running the test cases + */ +static int __init test_pmalloc_init_module(void) +{ + if (WARN(!(create_and_destroy_pool() && + test_alloc() && + test_self_protection() && + test_wr_memset() && + test_wr_strdup() && + test_wr_copy() && + test_specialized_wrs()), + "protected memory allocator test failed")) + return -EFAULT; + pr_success("protected memory allocator"); + return 0; +} + +module_init(test_pmalloc_init_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Igor Stoppa "); +MODULE_DESCRIPTION("Test module for pmalloc."); diff --git a/mm/test_write_rare.c b/mm/test_write_rare.c new file mode 100644 index 000000000000..e19473bb319b --- /dev/null +++ b/mm/test_write_rare.c @@ -0,0 +1,236 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * test_write_rare.c + * + * (C) Copyright 2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + * + * Caveat: the tests which perform modifications are run *during* init, so + * the memory they use could be still altered through a direct write + * operation. But the purpose of these tests is to confirm that the + * modification through remapping works correctly. This doesn't depend on + * the read/write status of the original mapping. + */ + +#include +#include +#include +#include +#include + +#ifdef pr_fmt +#undef pr_fmt +#endif + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#define pr_success(test_name) \ + pr_info(test_name " test passed") + +static int scalar __wr_after_init = 0xA5A5; + +/* The section must occupy a non-zero number of whole pages */ +static bool test_alignment(void) +{ + size_t pstart = (size_t)&__start_wr_after_init; + size_t pend = (size_t)&__end_wr_after_init; + + if (WARN((pstart & ~PAGE_MASK) || (pend & ~PAGE_MASK) || + (pstart >= pend), "Boundaries test failed.")) + return false; + pr_success("Boundaries"); + return true; +} + +/* Alter a scalar value */ +static bool test_simple_write(void) +{ + int new_val = 0x5A5A; + + if (WARN(!__is_wr_after_init(&scalar, sizeof(scalar)), + "The __wr_after_init modifier did NOT work.")) + return false; + + if (WARN(!wr(&scalar, &new_val) || scalar != new_val, + "Scalar write rare test failed")) + return false; + + pr_success("Scalar write rare"); + return true; +} + +#define LARGE_SIZE (PAGE_SIZE * 5) +#define CHANGE_SIZE (PAGE_SIZE * 2) +#define CHANGE_OFFSET (PAGE_SIZE / 2) + +static char large[LARGE_SIZE] __wr_after_init; + + +/* Alter data across multiple pages */ +static bool test_cross_page_write(void) +{ + unsigned int i; + char *src; + bool check; + + src = vmalloc(PAGE_SIZE * 2); + if (WARN(!src, "could not allocate memory")) + return false; + + for (i = 0; i < LARGE_SIZE; i++) + large[i] = 0xA5; + + for (i = 0; i < CHANGE_SIZE; i++) + src[i] = 0x5A; + + check = wr_memcpy(large + CHANGE_OFFSET, src, CHANGE_SIZE); + vfree(src); + if (WARN(!check, "The wr_memcpy() failed")) + return false; + + for (i = CHANGE_OFFSET; i < CHANGE_OFFSET + CHANGE_SIZE; i++) + if (WARN(large[i] != 0x5A, + "Cross-page write rare test failed")) + return false; + + pr_success("Cross-page write rare"); + return true; +} + +static bool test_memsetting(void) +{ + unsigned int i; + + wr_memset(large, 0, LARGE_SIZE); + for (i = 0; i < LARGE_SIZE; i++) + if (WARN(large[i], "Failed to reset memory")) + return false; + wr_memset(large + CHANGE_OFFSET, 1, CHANGE_SIZE); + for (i = 0; i < CHANGE_OFFSET; i++) + if (WARN(large[i], "Failed to set memory")) + return false; + for (i = CHANGE_OFFSET; i < CHANGE_OFFSET + CHANGE_SIZE; i++) + if (WARN(!large[i], "Failed to set memory")) + return false; + for (i = CHANGE_OFFSET + CHANGE_SIZE; i < LARGE_SIZE; i++) + if (WARN(large[i], "Failed to set memory")) + return false; + pr_success("Memsetting"); + return true; +} + +#define INIT_VAL 1 +#define END_VAL 4 + +/* Various tests for the shorthands provided for standard types. */ +static char char_var __wr_after_init = INIT_VAL; +static bool test_char(void) +{ + return wr_char(&char_var, END_VAL) && char_var == END_VAL; +} + +static short short_var __wr_after_init = INIT_VAL; +static bool test_short(void) +{ + return wr_short(&short_var, END_VAL) && + short_var == END_VAL; +} + +static unsigned short ushort_var __wr_after_init = INIT_VAL; +static bool test_ushort(void) +{ + return wr_ushort(&ushort_var, END_VAL) && + ushort_var == END_VAL; +} + +static int int_var __wr_after_init = INIT_VAL; +static bool test_int(void) +{ + return wr_int(&int_var, END_VAL) && + int_var == END_VAL; +} + +static unsigned int uint_var __wr_after_init = INIT_VAL; +static bool test_uint(void) +{ + return wr_uint(&uint_var, END_VAL) && + uint_var == END_VAL; +} + +static long long_var __wr_after_init = INIT_VAL; +static bool test_long(void) +{ + return wr_long(&long_var, END_VAL) && + long_var == END_VAL; +} + +static unsigned long ulong_var __wr_after_init = INIT_VAL; +static bool test_ulong(void) +{ + return wr_ulong(&ulong_var, END_VAL) && + ulong_var == END_VAL; +} + +static long long longlong_var __wr_after_init = INIT_VAL; +static bool test_longlong(void) +{ + return wr_longlong(&longlong_var, END_VAL) && + longlong_var == END_VAL; +} + +static unsigned long long ulonglong_var __wr_after_init = INIT_VAL; +static bool test_ulonglong(void) +{ + return wr_ulonglong(&ulonglong_var, END_VAL) && + ulonglong_var == END_VAL; +} + +static int referred_value = INIT_VAL; +static int *reference __wr_after_init; +static bool test_ptr(void) +{ + return wr_ptr(&reference, &referred_value) && + reference == &referred_value; +} + +static int *rcu_ptr __wr_after_init __aligned(sizeof(void *)); +static bool test_rcu_ptr(void) +{ + uintptr_t addr = wr_rcu_assign_pointer(rcu_ptr, &referred_value); + + return (addr == (uintptr_t)&referred_value) && + referred_value == *(int *)addr; +} + +static bool test_specialized_write_rare(void) +{ + if (WARN(!(test_char() && test_short() && + test_ushort() && test_int() && + test_uint() && test_long() && test_ulong() && + test_long() && test_ulong() && + test_longlong() && test_ulonglong() && + test_ptr() && test_rcu_ptr()), + "Specialized write rare test failed")) + return false; + pr_success("Specialized write rare"); + return true; +} + +static int __init test_static_wr_init_module(void) +{ + if (WARN(!(test_alignment() && + test_simple_write() && + test_cross_page_write() && + test_memsetting() && + test_specialized_write_rare()), + "static rare-write test failed")) + return -EFAULT; + pr_success("static write_rare"); + return 0; +} + +module_init(test_static_wr_init_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Igor Stoppa "); +MODULE_DESCRIPTION("Test module for static write rare."); From patchwork Tue Oct 23 21:34:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653781 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE96413A4 for ; Tue, 23 Oct 2018 21:37:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DBD492A4CB for ; Tue, 23 Oct 2018 21:37:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D00C22A4F6; Tue, 23 Oct 2018 21:37:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 4103A2A4CB for ; Tue, 23 Oct 2018 21:37:37 +0000 (UTC) Received: (qmail 14050 invoked by uid 550); 23 Oct 2018 21:36:20 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 13996 invoked from network); 23 Oct 2018 21:36:19 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=ofVQveRr/vSHa6u0TxgV0RAZGAgUhuV9JZxdBSjU2ZM=; b=Wv2uLp4uPKpikUORpT3l7bYWHFZAct2HyG652gDWzb4nSC1QkupkM/88spRvCI8k4S fFUNuiV8eJNVAoDDf655DicCaSv8Nbxaqul9xhOjt9seSGMCFY3FmNo71pmegWZbMy2V T5gsDh4WyAcfchCB6DOBEISLERDAX77sfPW6oAz4gDLvyhlCe816RQaGzYv9LM72ewLb m+u/ORDfG+kAt91XX+/l8bqOLtzf4tKgTPA7/GqQZztCikxP4pirwub035+eTGM9whJj XCRC5W/0Zl05uZdta0o8YA19oIjVqpRxJH4hzKeDtmZKcqVncucHsMTwC/lroowxaaLI vK8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=ofVQveRr/vSHa6u0TxgV0RAZGAgUhuV9JZxdBSjU2ZM=; b=J4QCAdZTGjIBXDv2g7Dm2NyeucPxIyng20cRgD1idxJoYtkbnRkFOcm056goMYKdmH MDCieHD3lxGBAWqNbaZizyB8PBk0AReAh3Vaf8dSyTRP5C6D4ihYzG4pb8roTkvqjoMU xUNiTBTzhpAVF5LsaFusggqUDP4MVHF4J2WuyWCpWL9QDLb70EAz1fLtWe1FfmDsMlRu HUu8gOIhHQ94WJkl/OG7EwqcR1J8Xmhkrg5SADPC6w4Y5Gz6JrFe4MWbhnRB639pxant F3RU1DvRVVhjp9CJAadqas6GmlHxfNAUnUpRHRdkLPIALhJ7kL31oVZzMRqfbRjM7t28 MQTw== X-Gm-Message-State: AGRZ1gJkkIjZwUNYFMYDOwommPEG2+ldOaYgc8b9qfcS4t2Lz8nik+vk K3Yz0kaq46/ThDmlypgp5Dg= X-Google-Smtp-Source: AJdET5fmkgPaBzjVtM+g+uAFc7+Q9gtq4t6sS5HPoaEpOogvTZsM3hZSkX72zxz7KAwQEG68XMc0/A== X-Received: by 2002:a2e:8184:: with SMTP id e4-v6mr401642ljg.138.1540330567676; Tue, 23 Oct 2018 14:36:07 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Greg Kroah-Hartman , Arnd Bergmann , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/17] prmem: lkdtm tests for memory protection Date: Wed, 24 Oct 2018 00:34:54 +0300 Message-Id: <20181023213504.28905-8-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Various cases meant to verify that illegal operations on protected memory will either BUG() or WARN(). The test cases fall into 2 main categories: - trying to overwrite (directly) something that is write protected - trying to use write rare functions on something that is not write rare Signed-off-by: Igor Stoppa CC: Kees Cook CC: Greg Kroah-Hartman CC: Arnd Bergmann CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- drivers/misc/lkdtm/core.c | 13 ++ drivers/misc/lkdtm/lkdtm.h | 13 ++ drivers/misc/lkdtm/perms.c | 248 +++++++++++++++++++++++++++++++++++++ 3 files changed, 274 insertions(+) diff --git a/drivers/misc/lkdtm/core.c b/drivers/misc/lkdtm/core.c index 2154d1bfd18b..41a3ba16bc57 100644 --- a/drivers/misc/lkdtm/core.c +++ b/drivers/misc/lkdtm/core.c @@ -155,6 +155,19 @@ static const struct crashtype crashtypes[] = { CRASHTYPE(ACCESS_USERSPACE), CRASHTYPE(WRITE_RO), CRASHTYPE(WRITE_RO_AFTER_INIT), + CRASHTYPE(WRITE_WR_AFTER_INIT), + CRASHTYPE(WRITE_WR_AFTER_INIT_ON_RO_AFTER_INIT), + CRASHTYPE(WRITE_WR_AFTER_INIT_ON_CONST), +#ifdef CONFIG_PRMEM + CRASHTYPE(WRITE_RO_PMALLOC), + CRASHTYPE(WRITE_AUTO_RO_PMALLOC), + CRASHTYPE(WRITE_WR_PMALLOC), + CRASHTYPE(WRITE_AUTO_WR_PMALLOC), + CRASHTYPE(WRITE_START_WR_PMALLOC), + CRASHTYPE(WRITE_WR_PMALLOC_ON_RO_PMALLOC), + CRASHTYPE(WRITE_WR_PMALLOC_ON_CONST), + CRASHTYPE(WRITE_WR_PMALLOC_ON_RO_AFT_INIT), +#endif CRASHTYPE(WRITE_KERN), CRASHTYPE(REFCOUNT_INC_OVERFLOW), CRASHTYPE(REFCOUNT_ADD_OVERFLOW), diff --git a/drivers/misc/lkdtm/lkdtm.h b/drivers/misc/lkdtm/lkdtm.h index 9e513dcfd809..08368c4545f7 100644 --- a/drivers/misc/lkdtm/lkdtm.h +++ b/drivers/misc/lkdtm/lkdtm.h @@ -38,6 +38,19 @@ void lkdtm_READ_BUDDY_AFTER_FREE(void); void __init lkdtm_perms_init(void); void lkdtm_WRITE_RO(void); void lkdtm_WRITE_RO_AFTER_INIT(void); +void lkdtm_WRITE_WR_AFTER_INIT(void); +void lkdtm_WRITE_WR_AFTER_INIT_ON_RO_AFTER_INIT(void); +void lkdtm_WRITE_WR_AFTER_INIT_ON_CONST(void); +#ifdef CONFIG_PRMEM +void lkdtm_WRITE_RO_PMALLOC(void); +void lkdtm_WRITE_AUTO_RO_PMALLOC(void); +void lkdtm_WRITE_WR_PMALLOC(void); +void lkdtm_WRITE_AUTO_WR_PMALLOC(void); +void lkdtm_WRITE_START_WR_PMALLOC(void); +void lkdtm_WRITE_WR_PMALLOC_ON_RO_PMALLOC(void); +void lkdtm_WRITE_WR_PMALLOC_ON_CONST(void); +void lkdtm_WRITE_WR_PMALLOC_ON_RO_AFT_INIT(void); +#endif void lkdtm_WRITE_KERN(void); void lkdtm_EXEC_DATA(void); void lkdtm_EXEC_STACK(void); diff --git a/drivers/misc/lkdtm/perms.c b/drivers/misc/lkdtm/perms.c index 53b85c9d16b8..3c14fd4d90ac 100644 --- a/drivers/misc/lkdtm/perms.c +++ b/drivers/misc/lkdtm/perms.c @@ -9,6 +9,7 @@ #include #include #include +#include #include /* Whether or not to fill the target memory area with do_nothing(). */ @@ -27,6 +28,10 @@ static const unsigned long rodata = 0xAA55AA55; /* This is marked __ro_after_init, so it should ultimately be .rodata. */ static unsigned long ro_after_init __ro_after_init = 0x55AA5500; +/* This is marked __wr_after_init, so it should be in .rodata. */ +static +unsigned long wr_after_init __wr_after_init = 0x55AA5500; + /* * This just returns to the caller. It is designed to be copied into * non-executable memory regions. @@ -104,6 +109,247 @@ void lkdtm_WRITE_RO_AFTER_INIT(void) *ptr ^= 0xabcd1234; } +void lkdtm_WRITE_WR_AFTER_INIT(void) +{ + unsigned long *ptr = &wr_after_init; + + /* + * Verify we were written to during init. Since an Oops + * is considered a "success", a failure is to just skip the + * real test. + */ + if ((*ptr & 0xAA) != 0xAA) { + pr_info("%p was NOT written during init!?\n", ptr); + return; + } + + pr_info("attempting bad wr_after_init write at %p\n", ptr); + *ptr ^= 0xabcd1234; +} + +#define INIT_VAL 0x5A +#define END_VAL 0xA5 + +/* Verify that write rare will not work against read-only memory. */ +static int ro_after_init_data __ro_after_init = INIT_VAL; +void lkdtm_WRITE_WR_AFTER_INIT_ON_RO_AFTER_INIT(void) +{ + pr_info("attempting illegal write rare to __ro_after_init"); + if (wr_int(&ro_after_init_data, END_VAL) || + ro_after_init_data == END_VAL) + pr_info("Unexpected successful write to __ro_after_init"); +} + +/* + * "volatile" to force the compiler to not optimize away the reading back. + * Is there a better way to do it, than using volatile? + */ +static volatile const int const_data = INIT_VAL; +void lkdtm_WRITE_WR_AFTER_INIT_ON_CONST(void) +{ + pr_info("attempting illegal write rare to const data"); + if (wr_int((int *)&const_data, END_VAL) || const_data == END_VAL) + pr_info("Unexpected successful write to const memory"); +} + +#ifdef CONFIG_PRMEM + +#define MSG_NO_POOL "Cannot allocate memory for the pool." +#define MSG_NO_PMEM "Cannot allocate memory from the pool." + +void lkdtm_WRITE_RO_PMALLOC(void) +{ + struct pmalloc_pool *pool; + int *i; + + pool = pmalloc_create_pool(PMALLOC_MODE_RO); + if (!pool) { + pr_info(MSG_NO_POOL); + return; + } + i = pmalloc(pool, sizeof(int)); + if (!i) { + pr_info(MSG_NO_PMEM); + pmalloc_destroy_pool(pool); + return; + } + *i = INT_MAX; + pmalloc_protect_pool(pool); + pr_info("attempting bad pmalloc write at %p\n", i); + *i = 0; /* Note: this will crash and leak the pool memory. */ +} + +void lkdtm_WRITE_AUTO_RO_PMALLOC(void) +{ + struct pmalloc_pool *pool; + int *i; + + pool = pmalloc_create_pool(PMALLOC_MODE_AUTO_RO); + if (!pool) { + pr_info(MSG_NO_POOL); + return; + } + i = pmalloc(pool, sizeof(int)); + if (!i) { + pr_info(MSG_NO_PMEM); + pmalloc_destroy_pool(pool); + return; + } + *i = INT_MAX; + pmalloc(pool, PMALLOC_DEFAULT_REFILL_SIZE); + pr_info("attempting bad pmalloc write at %p\n", i); + *i = 0; /* Note: this will crash and leak the pool memory. */ +} + +void lkdtm_WRITE_WR_PMALLOC(void) +{ + struct pmalloc_pool *pool; + int *i; + + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (!pool) { + pr_info(MSG_NO_POOL); + return; + } + i = pmalloc(pool, sizeof(int)); + if (!i) { + pr_info(MSG_NO_PMEM); + pmalloc_destroy_pool(pool); + return; + } + *i = INT_MAX; + pmalloc_protect_pool(pool); + pr_info("attempting bad pmalloc write at %p\n", i); + *i = 0; /* Note: this will crash and leak the pool memory. */ +} + +void lkdtm_WRITE_AUTO_WR_PMALLOC(void) +{ + struct pmalloc_pool *pool; + int *i; + + pool = pmalloc_create_pool(PMALLOC_MODE_AUTO_WR); + if (!pool) { + pr_info(MSG_NO_POOL); + return; + } + i = pmalloc(pool, sizeof(int)); + if (!i) { + pr_info(MSG_NO_PMEM); + pmalloc_destroy_pool(pool); + return; + } + *i = INT_MAX; + pmalloc(pool, PMALLOC_DEFAULT_REFILL_SIZE); + pr_info("attempting bad pmalloc write at %p\n", i); + *i = 0; /* Note: this will crash and leak the pool memory. */ +} + +void lkdtm_WRITE_START_WR_PMALLOC(void) +{ + struct pmalloc_pool *pool; + int *i; + + pool = pmalloc_create_pool(PMALLOC_MODE_START_WR); + if (!pool) { + pr_info(MSG_NO_POOL); + return; + } + i = pmalloc(pool, sizeof(int)); + if (!i) { + pr_info(MSG_NO_PMEM); + pmalloc_destroy_pool(pool); + return; + } + *i = INT_MAX; + pr_info("attempting bad pmalloc write at %p\n", i); + *i = 0; /* Note: this will crash and leak the pool memory. */ +} + +void lkdtm_WRITE_WR_PMALLOC_ON_RO_PMALLOC(void) +{ + struct pmalloc_pool *pool; + int *var_ptr; + + pool = pmalloc_create_pool(PMALLOC_MODE_RO); + if (!pool) { + pr_info(MSG_NO_POOL); + return; + } + var_ptr = pmalloc(pool, sizeof(int)); + if (!var_ptr) { + pr_info(MSG_NO_PMEM); + pmalloc_destroy_pool(pool); + return; + } + *var_ptr = INIT_VAL; + pmalloc_protect_pool(pool); + pr_info("attempting illegal write rare to R/O pool"); + if (wr_int(var_ptr, END_VAL)) + pr_info("Unexpected successful write to R/O pool"); + pmalloc_destroy_pool(pool); +} + +void lkdtm_WRITE_WR_PMALLOC_ON_CONST(void) +{ + struct pmalloc_pool *pool; + int *dummy; + bool write_result; + + /* + * The pool operations are only meant to simulate an attacker + * using a random pool as parameter for the attack against the + * const. + */ + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (!pool) { + pr_info(MSG_NO_POOL); + return; + } + dummy = pmalloc(pool, sizeof(*dummy)); + if (!dummy) { + pr_info(MSG_NO_PMEM); + pmalloc_destroy_pool(pool); + return; + } + *dummy = 1; + pmalloc_protect_pool(pool); + pr_info("attempting illegal write rare to const data"); + write_result = wr_int((int *)&const_data, END_VAL); + pmalloc_destroy_pool(pool); + if (write_result || const_data != INIT_VAL) + pr_info("Unexpected successful write to const memory"); +} + +void lkdtm_WRITE_WR_PMALLOC_ON_RO_AFT_INIT(void) +{ + struct pmalloc_pool *pool; + int *dummy; + bool write_result; + + /* + * The pool operations are only meant to simulate an attacker + * using a random pool as parameter for the attack against the + * const. + */ + pool = pmalloc_create_pool(PMALLOC_MODE_WR); + if (WARN(!pool, MSG_NO_POOL)) + return; + dummy = pmalloc(pool, sizeof(*dummy)); + if (WARN(!dummy, MSG_NO_PMEM)) { + pmalloc_destroy_pool(pool); + return; + } + *dummy = 1; + pmalloc_protect_pool(pool); + pr_info("attempting illegal write rare to ro_after_init"); + write_result = wr_int(&ro_after_init_data, END_VAL); + pmalloc_destroy_pool(pool); + WARN(write_result || ro_after_init_data != INIT_VAL, + "Unexpected successful write to ro_after_init memory"); +} +#endif + void lkdtm_WRITE_KERN(void) { size_t size; @@ -200,4 +446,6 @@ void __init lkdtm_perms_init(void) /* Make sure we can write to __ro_after_init values during __init */ ro_after_init |= 0xAA; + /* Make sure we can write to __wr_after_init during __init */ + wr_after_init |= 0xAA; } From patchwork Tue Oct 23 21:34:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653791 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 40A5D13A4 for ; Tue, 23 Oct 2018 21:37:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D4B02A4F5 for ; Tue, 23 Oct 2018 21:37:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 204FE2A4F6; Tue, 23 Oct 2018 21:37:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id D0E9E2A4FD for ; Tue, 23 Oct 2018 21:37:50 +0000 (UTC) Received: (qmail 14103 invoked by uid 550); 23 Oct 2018 21:36:21 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 14053 invoked from network); 23 Oct 2018 21:36:20 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=MxJyklPQotWhYa2J56SVcBSG16bYJ6WnJInuOUKovF4=; b=PdlteOY4o40QUDNjAThlAoTsVYTFd7zs3m3+vkS0xTuwkOXqjHspfcVAbQ+Eoq0IJl v9yCNh2HFu+CuGbd89TL4zyyHJHsybDQ5YR4q3QnmnxG8byJu7Mnlr9H2+e2fDSqcJ20 poPB83K9CC9XjqrXwH9vNnu+MF/PRBAlbQhlDOtGu3zaiW8TwcRqVWjIQU8e4S2gnZoe skIGMSbLBUSAnikw86eGtMDLm7ZsZV11Su0egjC0UHX9TQDCuZYZpUVrTL/8aKvmuMD1 jbaXi1ySxuzr6JaOyAgNi0/p7XDQifApzUZnsPpFC7qUfRxurqopGqRggnBteHuHhzIj 27pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=MxJyklPQotWhYa2J56SVcBSG16bYJ6WnJInuOUKovF4=; b=FnbXsEpiiisoGJz6Pao2A3RXkf330Ne6tTftfBTT6sJPTWD9F1pnkhvqgLX0W5BB8f aLn12Tt5Vuco0xfGoltCaMnfOdsPL/BeDMKsud3RC3HO8TV3Cm7hOZ5+/7Q7ee0RDec1 6iJgxnlwLyBgtAjkDS81NdCM3RHB55amE+kYo65oC92fRQhFGpdqgjxvu8sZQk1r1JNX 1L7hPd9EVqByB8lv9535YkNc96hghYOrFqTECtKlq/uYcP+5LFEMw0wgfcFwNaVMeqB4 sw5nitrzrYeFjZC2SjP3lhB1e0+z7byxsaDaQZgCReGS0Npe8aPbjta6s5kcgUBEBUFL lxLw== X-Gm-Message-State: AGRZ1gIvZD2g2Z0Ja4rm7wwXP7Mae2G94aj2KkCEIyW8xB37afkauzyJ y7PvoeITm0i4Ppaoctxns7w= X-Google-Smtp-Source: AJdET5e0SCGJjOdZZvB3VL2+kCFX64ASEyEQtsS+KMf/Fg23oC2goGdqiz7dgHvR3CIhTem3KMXl1A== X-Received: by 2002:a19:9cd5:: with SMTP id f204-v6mr196018lfe.41.1540330568969; Tue, 23 Oct 2018 14:36:08 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Pavel Tatashin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/17] prmem: struct page: track vmap_area Date: Wed, 24 Oct 2018 00:34:55 +0300 Message-Id: <20181023213504.28905-9-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP When a page is used for virtual memory, it is often necessary to obtain a handler to the corresponding vmap_area, which refers to the virtually continuous area generated, when invoking vmalloc. The struct page has a "private" field, which can be re-used, to store a pointer to the parent area. Note: in practice a virtual memory area is characterized both by a struct vmap_area and a struct vm_struct. The reason for referring from a page to its vmap_area, rather than to the vm_struct, is that the vmap_area contains a struct vm_struct *vm field, which can be used to reach also the information stored in the corresponding vm_struct. This link, however, is unidirectional, and it's not possible to easily identify the corresponding vmap_area, given a reference to a vm_struct. Furthermore, the struct vmap_area contains a list head node which is normally used only when it's queued for free and can be put to some other use during normal operations. The connection between each page and its vmap_area avoids more expensive searches through the btree of vmap_areas. Therefore, it is possible for find_vamp_area to be again a static function, while the rest of the code will rely on the direct reference from struct page. Signed-off-by: Igor Stoppa CC: Michal Hocko CC: Vlastimil Babka CC: "Kirill A. Shutemov" CC: Andrew Morton CC: Pavel Tatashin CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- include/linux/mm_types.h | 25 ++++++++++++++++++------- include/linux/prmem.h | 13 ++++++++----- include/linux/vmalloc.h | 1 - mm/prmem.c | 2 +- mm/test_pmalloc.c | 12 ++++-------- mm/vmalloc.c | 9 ++++++++- 6 files changed, 39 insertions(+), 23 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5ed8f6292a53..8403bdd12d1f 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -87,13 +87,24 @@ struct page { /* See page-flags.h for PAGE_MAPPING_FLAGS */ struct address_space *mapping; pgoff_t index; /* Our offset within mapping. */ - /** - * @private: Mapping-private opaque data. - * Usually used for buffer_heads if PagePrivate. - * Used for swp_entry_t if PageSwapCache. - * Indicates order in the buddy system if PageBuddy. - */ - unsigned long private; + union { + /** + * @private: Mapping-private opaque data. + * Usually used for buffer_heads if + * PagePrivate. + * Used for swp_entry_t if PageSwapCache. + * Indicates order in the buddy system if + * PageBuddy. + */ + unsigned long private; + /** + * @area: reference to the containing area + * For pages that are mapped into a virtually + * contiguous area, avoids performing a more + * expensive lookup. + */ + struct vmap_area *area; + }; }; struct { /* slab, slob and slub */ union { diff --git a/include/linux/prmem.h b/include/linux/prmem.h index 26fd48410d97..cf713fc1c8bb 100644 --- a/include/linux/prmem.h +++ b/include/linux/prmem.h @@ -54,14 +54,17 @@ static __always_inline bool __is_wr_after_init(const void *ptr, size_t size) static __always_inline bool __is_wr_pool(const void *ptr, size_t size) { - struct vmap_area *area; + struct vm_struct *vm; + struct page *page; if (!is_vmalloc_addr(ptr)) return false; - area = find_vmap_area((unsigned long)ptr); - return area && area->vm && (area->vm->size >= size) && - ((area->vm->flags & (VM_PMALLOC | VM_PMALLOC_WR)) == - (VM_PMALLOC | VM_PMALLOC_WR)); + page = vmalloc_to_page(ptr); + if (!(page && page->area && page->area->vm)) + return false; + vm = page->area->vm; + return ((vm->size >= size) && + ((vm->flags & VM_PMALLOC_WR_MASK) == VM_PMALLOC_WR_MASK)); } /** diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 4d14a3b8089e..43a444f8b1e9 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -143,7 +143,6 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size, const void *caller); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); -extern struct vmap_area *find_vmap_area(unsigned long addr); extern int map_vm_area(struct vm_struct *area, pgprot_t prot, struct page **pages); diff --git a/mm/prmem.c b/mm/prmem.c index 7dd13ea43304..96abf04909e7 100644 --- a/mm/prmem.c +++ b/mm/prmem.c @@ -150,7 +150,7 @@ static int grow(struct pmalloc_pool *pool, size_t min_size) if (WARN(!addr, "Failed to allocate %zd bytes", PAGE_ALIGN(size))) return -ENOMEM; - new_area = find_vmap_area((uintptr_t)addr); + new_area = vmalloc_to_page(addr)->area; tag_mask = VM_PMALLOC; if (pool->mode & PMALLOC_WR) tag_mask |= VM_PMALLOC_WR; diff --git a/mm/test_pmalloc.c b/mm/test_pmalloc.c index f9ee8fb29eea..c64872ff05ea 100644 --- a/mm/test_pmalloc.c +++ b/mm/test_pmalloc.c @@ -38,15 +38,11 @@ static bool is_address_protected(void *p) if (unlikely(!is_vmalloc_addr(p))) return false; page = vmalloc_to_page(p); - if (unlikely(!page)) + if (unlikely(!(page && page->area && page->area->vm))) return false; - wmb(); /* Flush changes to the page table - is it needed? */ - area = find_vmap_area((uintptr_t)p); - if (unlikely((!area) || (!area->vm) || - ((area->vm->flags & VM_PMALLOC_PROTECTED_MASK) != - VM_PMALLOC_PROTECTED_MASK))) - return false; - return true; + area = page->area; + return (area->vm->flags & VM_PMALLOC_PROTECTED_MASK) == + VM_PMALLOC_PROTECTED_MASK; } static bool create_and_destroy_pool(void) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 15850005fea5..ffef705f0523 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -742,7 +742,7 @@ static void free_unmap_vmap_area(struct vmap_area *va) free_vmap_area_noflush(va); } -struct vmap_area *find_vmap_area(unsigned long addr) +static struct vmap_area *find_vmap_area(unsigned long addr) { struct vmap_area *va; @@ -1523,6 +1523,7 @@ static void __vunmap(const void *addr, int deallocate_pages) struct page *page = area->pages[i]; BUG_ON(!page); + page->area = NULL; __free_pages(page, 0); } @@ -1731,8 +1732,10 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, const void *caller) { struct vm_struct *area; + struct vmap_area *va; void *addr; unsigned long real_size = size; + unsigned int i; size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages) @@ -1747,6 +1750,10 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, if (!addr) return NULL; + va = __find_vmap_area((unsigned long)addr); + for (i = 0; i < va->vm->nr_pages; i++) + va->vm->pages[i]->area = va; + /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED * flag. It means that vm_struct is not fully initialized. From patchwork Tue Oct 23 21:34:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653793 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D38814BB for ; Tue, 23 Oct 2018 21:38:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A1772A4FA for ; Tue, 23 Oct 2018 21:38:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7E1722A4FD; Tue, 23 Oct 2018 21:38:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id A499C2A4FA for ; Tue, 23 Oct 2018 21:38:03 +0000 (UTC) Received: (qmail 14152 invoked by uid 550); 23 Oct 2018 21:36:22 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 14109 invoked from network); 23 Oct 2018 21:36:21 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=pGpymzbPbfMd0UY2O/PnMm8wYTcULxEjQ2M1QiLWUtQ=; b=etVVDgLocA4AjN+jxAXLRtXyFmyCKDXzTHMYFdYt+3XfgEEQNZehB6ZAubiCYjyvy0 C5GrkJFQrsxCI5EKgJi2Hh9syjIDh3WhQy6uGJipgVAL10O0CGT25+kfZsvBKU7+PtEs hAvdrW/iABvj+SaTIJ7FSEwZoqcJMNTTM7IyOYe6ITWgreOcX1fRhiAu4Aie0D3L+6Ah 0CL0xm+qqL4/H+nxwdF5UjsZq5aTf3xxTf9y0JGxopS2MHxBOZ9aDWIX2bUY5gqeHeBx G6gvyawpjFtqRLlCBActSQCZFme72y8lx8IlyO4u8ngtDN4fbCBBViLuXUbGABcQ4L4f JH8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=pGpymzbPbfMd0UY2O/PnMm8wYTcULxEjQ2M1QiLWUtQ=; b=U57GEP4AURbnpXcweYHb5qaEHW8fsAhUk7JR3owSHUZaknP0yzy3jtz4vyIfyJ5ue4 iYOiB4a44eUz+xYBa7RtsjRyR62yfQC8e0Ks3r8DTcPAhxhNncAfkZOa3eXSWtNSKoJv lra9ABd0DJwRdg94DQaBJmotp1qCYJsaCvKuqYNoM3osWx3igKWkdkKgvH8XxI8uqlss dyn5MXCqdoeqPn+9x8q5pQzHDLUjySdMPklEIJjfop1ryJijoh1VFySIjUhIFAIpl7vk dLNFP81yt30vRdcZSlthQUBbGdnv7FTz+Vtn/HUWcTKkP2epBKLsA/V+S1XzVdd0jMMG CIQw== X-Gm-Message-State: AGRZ1gKS8tr9bdLg447HGYNVTZMVrBpzL2RK3+kezUQQrQNu5o8eqJQS E8OnIMx8NzMChmwJxqqk33U= X-Google-Smtp-Source: AJdET5exRyQPy5ypGmiJjncDtEgR9KmCyJjS90PbVTRQUbcCgZs2pc0eSgkwJSERums9s+tyA2MWxw== X-Received: by 2002:a19:3809:: with SMTP id f9mr170930lfa.148.1540330570334; Tue, 23 Oct 2018 14:36:10 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Chris von Recklinghausen , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/17] prmem: hardened usercopy Date: Wed, 24 Oct 2018 00:34:56 +0300 Message-Id: <20181023213504.28905-10-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Prevent leaks of protected memory to userspace. The protection from overwrited from userspace is already available, once the memory is write protected. Signed-off-by: Igor Stoppa CC: Kees Cook CC: Chris von Recklinghausen CC: linux-mm@kvack.org CC: linux-kernel@vger.kernel.org --- include/linux/prmem.h | 24 ++++++++++++++++++++++++ mm/usercopy.c | 5 +++++ 2 files changed, 29 insertions(+) diff --git a/include/linux/prmem.h b/include/linux/prmem.h index cf713fc1c8bb..919d853ddc15 100644 --- a/include/linux/prmem.h +++ b/include/linux/prmem.h @@ -273,6 +273,30 @@ struct pmalloc_pool { uint8_t mode; }; +void __noreturn usercopy_abort(const char *name, const char *detail, + bool to_user, unsigned long offset, + unsigned long len); + +/** + * check_pmalloc_object() - helper for hardened usercopy + * @ptr: the beginning of the memory to check + * @n: the size of the memory to check + * @to_user: copy to userspace or from userspace + * + * If the check is ok, it will fall-through, otherwise it will abort. + * The function is inlined, to minimize the performance impact of the + * extra check that can end up on a hot path. + * Non-exhaustive micro benchmarking with QEMU x86_64 shows a reduction of + * the time spent in this fragment by 60%, when inlined. + */ +static inline +void check_pmalloc_object(const void *ptr, unsigned long n, bool to_user) +{ + if (unlikely(__is_wr_after_init(ptr, n) || __is_wr_pool(ptr, n))) + usercopy_abort("pmalloc", "accessing pmalloc obj", to_user, + (const unsigned long)ptr, n); +} + /* * The write rare functionality is fully implemented as __always_inline, * to prevent having an internal function call that is capable of modifying diff --git a/mm/usercopy.c b/mm/usercopy.c index 852eb4e53f06..a080dd37b684 100644 --- a/mm/usercopy.c +++ b/mm/usercopy.c @@ -22,8 +22,10 @@ #include #include #include +#include #include + /* * Checks if a given pointer and length is contained by the current * stack frame (if possible). @@ -284,6 +286,9 @@ void __check_object_size(const void *ptr, unsigned long n, bool to_user) /* Check for object in kernel to avoid text exposure. */ check_kernel_text_object((const unsigned long)ptr, n, to_user); + + /* Check if object is from a pmalloc chunk. */ + check_pmalloc_object(ptr, n, to_user); } EXPORT_SYMBOL(__check_object_size); From patchwork Tue Oct 23 21:34:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653795 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6708A13A4 for ; Tue, 23 Oct 2018 21:38:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 533A22A4FD for ; Tue, 23 Oct 2018 21:38:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 46BF82A506; Tue, 23 Oct 2018 21:38:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id BE6272A4FD for ; Tue, 23 Oct 2018 21:38:15 +0000 (UTC) Received: (qmail 14220 invoked by uid 550); 23 Oct 2018 21:36:24 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 14161 invoked from network); 23 Oct 2018 21:36:23 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=6qJTNFwbH1TFDI3gbUvOjFkxUhT0SMG0kbjMlNfa96U=; b=A0/h3p8FiatFWDDQDLLr5RaEKmvC2OBV5CByt0aO7U3tkHCDzbEKHcuvyKbqrM79at B3DmjiNeG77aasQEPLLhdRBHG9rpmRDrnVbE6D9rhTdkzgvnT7f6dY1oLcu077JMxoik oRTu7OWVsK8ogbADS7YVE4uEMP7Jx8pQXNI0XrGPOLmrSR9F9jYgv3/3p4O14ws5XB6X gVath+ld78AmukgiCr7IMQoqAAGvV63hUhU06m1gmhZ1As5SPVAUMdI4bAmQmxaGVFaU YKggE6rs4nc+lXTGeyf/x4UDHs0bWe/8c8zeqPEGH0Sf4MdWx3soJY8ZzG/wnD/5ZN6J fWfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=6qJTNFwbH1TFDI3gbUvOjFkxUhT0SMG0kbjMlNfa96U=; b=p5MkcNF7ukxw00mYApKFL/JPzo6zsa59Enpysoe0tXXhgU7ENjd88gbRJHRAlDuuRW fWcGongoJRddZEclqaf8IEddVbqyFvwJnvbfGD99p0yLOZ2m7B32+F+hxzBo4G8wtBGp PZa5LeDMrWa3Iza2XmCg+rejvA+MIohyFzUJIGa+4Vs3A9Abt5Lb8fWcRweqWqT6seMp CFAunysL6xLco/hGpbfqdPtg5unw2skF3VNjmEtA2uOPhohM5DybBxSkYim3QdFJO1n/ /sTDpYCLrJFjLyNmtcLbMelgNtLPt94ZJPT70hNNbfBhSugKg2iuo7HEvSPglHFgLdry gaqg== X-Gm-Message-State: ABuFfohWZafyrK1Q1f1+xIW5fV5zCEHa8v+evCnWQZeUqEa6ExcXhWu7 YAEC2lBF3WYkimF/zuHjhzs= X-Google-Smtp-Source: ACcGV60oU4v9dmfm2pngODIU85Xo5lMoQ72mDmXmBqMbWj/N/dPKow1IaFhEzTDAdhPWjJt0crJx8Q== X-Received: by 2002:a19:2102:: with SMTP id h2-v6mr12783512lfh.119.1540330571974; Tue, 23 Oct 2018 14:36:11 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Randy Dunlap , Mike Rapoport , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/17] prmem: documentation Date: Wed, 24 Oct 2018 00:34:57 +0300 Message-Id: <20181023213504.28905-11-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Documentation for protected memory. Topics covered: * static memory allocation * dynamic memory allocation * write-rare Signed-off-by: Igor Stoppa CC: Jonathan Corbet CC: Randy Dunlap CC: Mike Rapoport CC: linux-doc@vger.kernel.org CC: linux-kernel@vger.kernel.org --- Documentation/core-api/index.rst | 1 + Documentation/core-api/prmem.rst | 172 +++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 3 files changed, 174 insertions(+) create mode 100644 Documentation/core-api/prmem.rst diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst index 26b735cefb93..1a90fa878d8d 100644 --- a/Documentation/core-api/index.rst +++ b/Documentation/core-api/index.rst @@ -31,6 +31,7 @@ Core utilities gfp_mask-from-fs-io timekeeping boot-time-mm + prmem Interfaces for kernel debugging =============================== diff --git a/Documentation/core-api/prmem.rst b/Documentation/core-api/prmem.rst new file mode 100644 index 000000000000..16d7edfe327a --- /dev/null +++ b/Documentation/core-api/prmem.rst @@ -0,0 +1,172 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _prmem: + +Memory Protection +================= + +:Date: October 2018 +:Author: Igor Stoppa + +Foreword +-------- +- In a typical system using some sort of RAM as execution environment, + **all** memory is initially writable. + +- It must be initialized with the appropriate content, be it code or data. + +- Said content typically undergoes modifications, i.e. relocations or + relocation-induced changes. + +- The present document doesn't address such transient. + +- Kernel code is protected at system level and, unlike data, it doesn't + require special attention. + +Protection mechanism +-------------------- + +- When available, the MMU can write protect memory pages that would be + otherwise writable. + +- The protection has page-level granularity. + +- An attempt to overwrite a protected page will trigger an exception. +- **Write protected data must go exclusively to write protected pages** +- **Writable data must go exclusively to writable pages** + +Available protections for kernel data +------------------------------------- + +- **constant** + Labelled as **const**, the data is never supposed to be altered. + It is statically allocated - if it has any memory footprint at all. + The compiler can even optimize it away, where possible, by replacing + references to a **const** with its actual value. + +- **read only after init** + By tagging an otherwise ordinary statically allocated variable with + **__ro_after_init**, it is placed in a special segment that will + become write protected, at the end of the kernel init phase. + The compiler has no notion of this restriction and it will treat any + write operation on such variable as legal. However, assignments that + are attempted after the write protection is in place, will cause + exceptions. + +- **write rare after init** + This can be seen as variant of read only after init, which uses the + tag **__wr_after_init**. It is also limited to statically allocated + memory. It is still possible to alter this type of variables, after + the kernel init phase is complete, however it can be done exclusively + with special functions, instead of the assignment operator. Using the + assignment operator after conclusion of the init phase will still + trigger an exception. It is not possible to transition a certain + variable from __wr_ater_init to a permanent read-only status, at + runtime. + +- **dynamically allocated write-rare / read-only** + After defining a pool, memory can be obtained through it, primarily + through the **pmalloc()** allocator. The exact writability state of the + memory obtained from **pmalloc()** and friends can be configured when + creating the pool. At any point it is possible to transition to a less + permissive write status the memory currently associated to the pool. + Once memory has become read-only, it the only valid operation, beside + reading, is to released it, by destroying the pool it belongs to. + + +Protecting dynamically allocated memory +--------------------------------------- + +When dealing with dynamically allocated memory, three options are + available for configuring its writability state: + +- **Options selected when creating a pool** + When creating the pool, it is possible to choose one of the following: + - **PMALLOC_MODE_RO** + - Writability at allocation time: *WRITABLE* + - Writability at protection time: *NONE* + - **PMALLOC_MODE_WR** + - Writability at allocation time: *WRITABLE* + - Writability at protection time: *WRITE-RARE* + - **PMALLOC_MODE_AUTO_RO** + - Writability at allocation time: + - the latest allocation: *WRITABLE* + - every other allocation: *NONE* + - Writability at protection time: *NONE* + - **PMALLOC_MODE_AUTO_WR** + - Writability at allocation time: + - the latest allocation: *WRITABLE* + - every other allocation: *WRITE-RARE* + - Writability at protection time: *WRITE-RARE* + - **PMALLOC_MODE_START_WR** + - Writability at allocation time: *WRITE-RARE* + - Writability at protection time: *WRITE-RARE* + + **Remarks:** + - The "AUTO" modes perform automatic protection of the content, whenever + the current vmap_area is used up and a new one is allocated. + - At that point, the vmap_area being phased out is protected. + - The size of the vmap_area depends on various parameters. + - It might not be possible to know for sure *when* certain data will + be protected. + - The functionality is provided as tradeoff between hardening and speed. + - Its usefulness depends on the specific use case at hand + - The "START_WR" mode is the only one which provides immediate protection, at the cost of speed. + +- **Protecting the pool** + This is achieved with **pmalloc_protect_pool()** + - Any vmap_area currently in the pool is write-protected according to its initial configuration. + - Any residual space still available from the current vmap_area is lost, as the area is protected. + - **protecting a pool after every allocation will likely be very wasteful** + - Using PMALLOC_MODE_START_WR is likely a better choice. + +- **Upgrading the protection level** + This is achieved with **pmalloc_make_pool_ro()** + - it turns the present content of a write-rare pool into read-only + - can be useful when the content of the memory has settled + + +Caveats +------- +- Freeing of memory is not supported. Pages will be returned to the + system upon destruction of their memory pool. + +- The address range available for vmalloc (and thus for pmalloc too) is + limited, on 32-bit systems. However it shouldn't be an issue, since not + much data is expected to be dynamically allocated and turned into + write-protected. + +- Regarding SMP systems, changing state of pages and altering mappings + requires performing cross-processor synchronizations of page tables. + This is an additional reason for limiting the use of write rare. + +- Not only the pmalloc memory must be protected, but also any reference to + it that might become the target for an attack. The attack would replace + a reference to the protected memory with a reference to some other, + unprotected, memory. + +- The users of rare write must take care of ensuring the atomicity of the + action, respect to the way they use the data being altered; for example, + take a lock before making a copy of the value to modify (if it's + relevant), then alter it, issue the call to rare write and finally + release the lock. Some special scenario might be exempt from the need + for locking, but in general rare-write must be treated as an operation + that can incur into races. + +- pmalloc relies on virtual memory areas and will therefore use more + tlb entries. It still does a better job of it, compared to invoking + vmalloc for each allocation, but it is undeniably less optimized wrt to + TLB use than using the physmap directly, through kmalloc or similar. + + +Utilization +----------- + +**add examples here** + +API +--- + +.. kernel-doc:: include/linux/prmem.h +.. kernel-doc:: mm/prmem.c +.. kernel-doc:: include/linux/prmemextra.h diff --git a/MAINTAINERS b/MAINTAINERS index ea979a5a9ec9..246b1a1cc8bb 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9463,6 +9463,7 @@ F: include/linux/prmemextra.h F: mm/prmem.c F: mm/test_write_rare.c F: mm/test_pmalloc.c +F: Documentation/core-api/prmem.rst MEMORY MANAGEMENT L: linux-mm@kvack.org From patchwork Tue Oct 23 21:34:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653797 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 25BC814BB for ; Tue, 23 Oct 2018 21:38:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 131B22A14F for ; Tue, 23 Oct 2018 21:38:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 06FB22A1B5; Tue, 23 Oct 2018 21:38:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 358602A14F for ; Tue, 23 Oct 2018 21:38:29 +0000 (UTC) Received: (qmail 14259 invoked by uid 550); 23 Oct 2018 21:36:25 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 14222 invoked from network); 23 Oct 2018 21:36:24 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=wfR+fMPqGcMIiSbYRT7t8ZKG+Wth8+A7neWCKMW39bo=; b=NSTm8KZSg3mTcjVAHwXvMefH1L9qCAsnuF0SiZouKOL34cs6TBhJ9u+3dCXhe9tcjj NFfLiY6U3spOFYXhmO/FSGNo5ANtqmnSBnyS1g7A1mskX9K0SgD6zO5VnE6s2fr91B4J htrS0hVNIEptrkthCgGEV+IvfGNvnkUYOg9mFPJCkmz/yP2S2JuRM+fzfkVpuKFgij/u XqqTPMKDADSrHYlCm1aZ6pkBs6MPXWIlMRyCaF2ABF6Ot8Ft8QdU54KQHlM1muamJZK/ RKcz16TjeZRaZsBmYR1rqRzq4xo0ph1FhdQbqY/RqDegyjmxrIiwnhBtLwP8EoMqWHwK 188w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=wfR+fMPqGcMIiSbYRT7t8ZKG+Wth8+A7neWCKMW39bo=; b=j1PjQcUoc01lzR900K8USZSkrbbiOPffly8BOf6lJt8c9qzIzdKVRoHWwWkj8YhfMZ kzsmblgHukR1WanZqKVvPnIg3NZRejJqxiUIdsL5m3qonxXrgGgsTTZALTz+hEl3Y5xB D3oYuly0b35ROaC24EIwYCVtWkVYV+cC65+B3OuGK8ze4TPjbdZyodO3iRgVWBVjb/7z f4G15wJpkV9emlwT9r/NQRXxTRaTKuBnbV/9N0eDY8lxqS+GiJ4qKRCf5ZSY45OoJoPn vuy5awrCV5d0b4Hr0usLLLrfznNSji6GWwkSpGtvmo55RSgcuaF92W1aGFuU4UEXeaNn Hm8A== X-Gm-Message-State: AGRZ1gJpHvOl4JAhlB0FFM1df2cp2OV86VUdVqJwwaMsUm8Y4BU/5mGK NYedu36lUGddB0/A3ZhSOcc= X-Google-Smtp-Source: AJdET5dVOeRLRbNE9ihJ7d4qwyx504gyotgQkH3VlmhhESWejLxDFLXxKx2mzRRXa361IDY4EQUANw== X-Received: by 2002:a2e:970d:: with SMTP id r13-v6mr5241lji.30.1540330573423; Tue, 23 Oct 2018 14:36:13 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Kate Stewart , "David S. Miller" , Edward Cree , Philippe Ombredanne , Greg Kroah-Hartman , linux-kernel@vger.kernel.org Subject: [PATCH 11/17] prmem: llist: use designated initializer Date: Wed, 24 Oct 2018 00:34:58 +0300 Message-Id: <20181023213504.28905-12-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Using a list_head in an unnamed union poses a problem with the current implementation of the initializer, since it doesn't specify the names of the fields it is initializing. This patch makes it use designated initializers. Signed-off-by: Igor Stoppa CC: Kate Stewart CC: "David S. Miller" CC: Edward Cree CC: Philippe Ombredanne CC: Greg Kroah-Hartman CC: linux-kernel@vger.kernel.org --- include/linux/list.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/include/linux/list.h b/include/linux/list.h index de04cc5ed536..184a7b60436f 100644 --- a/include/linux/list.h +++ b/include/linux/list.h @@ -18,7 +18,10 @@ * using the generic single-entry routines. */ -#define LIST_HEAD_INIT(name) { &(name), &(name) } +#define LIST_HEAD_INIT(name) { \ + .next = &(name), \ + .prev = &(name), \ +} #define LIST_HEAD(name) \ struct list_head name = LIST_HEAD_INIT(name) From patchwork Tue Oct 23 21:34:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653799 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04BD913A4 for ; Tue, 23 Oct 2018 21:38:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E5B562A1B5 for ; Tue, 23 Oct 2018 21:38:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9AD42A2FF; Tue, 23 Oct 2018 21:38:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 14F602A1B5 for ; Tue, 23 Oct 2018 21:38:43 +0000 (UTC) Received: (qmail 15438 invoked by uid 550); 23 Oct 2018 21:36:27 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 15410 invoked from network); 23 Oct 2018 21:36:26 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=Fg4OaLOHQqrefdjonA6DL8gKc44WsS6YzuIT3EPXPyc=; b=C9E4LDFEqa5quLrdvuVRMmj43Vc5eOZNvcS2YZIGdDAwvs+LtTqg/8VaP6Fd6nfz00 6Y+Kc9Ygxx4Hsd3pPwMK8bzJDFnCw4vw2zNaOu+VZoue6LSPaLP8Pj9crYs88OlfR1q+ pgboQVBRzc+cTmFCLpSlyQunl71QSdH3MpNbzWIhwCKldHH79R1k1WZ0T6H66boZOdBV n7mvD9IH+lRIsAMMUZZajrAW5p8MC5rRpTADj8KpIgELHPkKjU6yxSVwgCIBwszpr8xl SUuArLW37lu7ZoI1jQzaAXoh/YeWiB2bfrEDO18NdKo8GXSQzCWIeSwUtgS4Hgnm59MO +k8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=Fg4OaLOHQqrefdjonA6DL8gKc44WsS6YzuIT3EPXPyc=; b=YuJRC8XNnsLlC49ztEOtTKt8WqfH4pWWtQAtwSxoKr2orAoL6J2n8SLFjEDtCeGFt9 TzQVS1XYwEwLQt/7u0A2rJkVcmsjKHYR6l/JLNxNvQeoeceK2wVeBpBpxztSR8RWuAAX TCKVYSAP5vIB/mF+swAyaUmKfB/SlqIqiVuDKBvzNO514PxUvxvrcyRznHiYGSCPi+6m SKdi17WqPtYD27d6AzLtwyWjrK8rGFJnvQzEoj0c6lu7IQnSiKW37NjKZmCnTCNm0U/U s5o4LCRAKBmIkxdthw+jsW18jL+qUh6/XMo3qBfaBAg1Uu/43Svy8LEF36EEQO76cS1J J8uw== X-Gm-Message-State: ABuFfog6S1IpaRbnM7vaC36AHyi9mfmy/wPc2i8P03vQgcbzOx1wf/jN YE8+LoqkUwcpdOCStJY4SdQ= X-Google-Smtp-Source: ACcGV611Z5tTnO5/5bvrHUOX3goc1TR758Ld/VtLL2Gme2c4YdgEJiORfSTHNppuLgxH+rxf/KtJWg== X-Received: by 2002:ac2:5082:: with SMTP id f2-v6mr12824852lfm.47.1540330574898; Tue, 23 Oct 2018 14:36:14 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Greg Kroah-Hartman , Andrew Morton , Masahiro Yamada , Alexey Dobriyan , Pekka Enberg , "Paul E. McKenney" , Lihao Liang , linux-kernel@vger.kernel.org Subject: [PATCH 12/17] prmem: linked list: set alignment Date: Wed, 24 Oct 2018 00:34:59 +0300 Message-Id: <20181023213504.28905-13-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP As preparation to using write rare on the nodes of various types of lists, specify that the fields in the basic data structures must be aligned to sizeof(void *) It is meant to ensure that any static allocation will not cross a page boundary, to allow pointers to be updated in one step. Signed-off-by: Igor Stoppa CC: Greg Kroah-Hartman CC: Andrew Morton CC: Masahiro Yamada CC: Alexey Dobriyan CC: Pekka Enberg CC: "Paul E. McKenney" CC: Lihao Liang CC: linux-kernel@vger.kernel.org --- include/linux/types.h | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/include/linux/types.h b/include/linux/types.h index 9834e90aa010..53609bbdcf0f 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -183,17 +183,29 @@ typedef struct { } atomic64_t; #endif +#ifdef CONFIG_PRMEM struct list_head { - struct list_head *next, *prev; -}; + struct list_head *next __aligned(sizeof(void *)); + struct list_head *prev __aligned(sizeof(void *)); +} __aligned(sizeof(void *)); -struct hlist_head { - struct hlist_node *first; +struct hlist_node { + struct hlist_node *next __aligned(sizeof(void *)); + struct hlist_node **pprev __aligned(sizeof(void *)); +} __aligned(sizeof(void *)); +#else +struct list_head { + struct list_head *next, *prev; }; struct hlist_node { struct hlist_node *next, **pprev; }; +#endif + +struct hlist_head { + struct hlist_node *first; +}; struct ustat { __kernel_daddr_t f_tfree; From patchwork Tue Oct 23 21:35:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653801 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6EE0F13A4 for ; Tue, 23 Oct 2018 21:38:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5BDC22A416 for ; Tue, 23 Oct 2018 21:38:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F2DD2A421; Tue, 23 Oct 2018 21:38:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 67FA52A416 for ; Tue, 23 Oct 2018 21:38:58 +0000 (UTC) Received: (qmail 15601 invoked by uid 550); 23 Oct 2018 21:36:29 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 15472 invoked from network); 23 Oct 2018 21:36:27 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=TAyei4tGiwzaEFtWD6qceoBqlcxrR+efj6ojDl9ZulY=; b=itzYzOZKVGSRr5rVXV9w3CTN+HNRtiP2lfwk2bdN5A02HCaZvCEbnax0J9Fu/juSNo EcY511Ql87pqbnb0nQ0U4qlvdE/WGUUTyhGDw3ocKVdXVrH6nq1j+VVbuuQZBZ3Y2I+u 29hbESNI1S7hLjecHmP8IOlcVCaZa30vVsEtLe64Cw9MI0v07p2NrG9jjY3s2KBpm8lg rRfTiwuxjwvwnfNey826fVQVWA/nhe5Q/2Tp+jYyyT3lcto+pe0S34JatMd3V+ebedOk MjWulDt7SOtNxlaCwZMuKPaoMCvmL2XsAbZoMRLzFuGwvYBHmVQ/0nqh9XiTQOyJz+ee 7Qwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=TAyei4tGiwzaEFtWD6qceoBqlcxrR+efj6ojDl9ZulY=; b=qGH+rOy4ksi6ai8nmYE/28SnZhI6zDuY6IRX8ia6hwaAxPOBzrJlMdiNXIZI/hid2R gVScjcDyxeRgeHI4EWuhrtH34KLh22vzz9Z5m+3EquqnsNxGIoCsSrsOwZEccpFVOHms fl4lbA/X/Z/v9iIHtkE3KT3jGGOQlj8F76rGTdt5S9EPOxcPKPOTU+se74DvvtDfRDja viR6q0axyhM7/O9zfXMhjH6ws/kZbxtG+SZMQasqHLi+vXVmLHfwMjV+FqxsOdkxuGbf VVtQXGgRga4ex4nutoPjxeFX4X87jdE0uHG7dqqEz2LYNl4JEOBgY5gjpLb0sU2GMHzM o0EQ== X-Gm-Message-State: AGRZ1gLUSfKLnzR+vvUHdFOpiD8bYmOn8NUrwwXAkxcvS97DZZwDSue0 UhaKGPd9zmppPDp1zbAP+N4= X-Google-Smtp-Source: ACcGV62jyvAifNg0UaxHUEFim2YZjRqmlefBY1q8HSVOsnALwybksvOWnPKQYCy+l29Mj8oEFgWb/A== X-Received: by 2002:a2e:3810:: with SMTP id f16-v6mr13247460lja.77.1540330576357; Tue, 23 Oct 2018 14:36:16 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Greg Kroah-Hartman , Andrew Morton , Masahiro Yamada , Alexey Dobriyan , Pekka Enberg , "Paul E. McKenney" , Lihao Liang , linux-kernel@vger.kernel.org Subject: [PATCH 13/17] prmem: linked list: disable layout randomization Date: Wed, 24 Oct 2018 00:35:00 +0300 Message-Id: <20181023213504.28905-14-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Some of the data structures used in list management are composed by two pointers. Since the kernel is now configured by default to randomize the layout of data structures soleley composed by pointers, this might prevent correct type punning between these structures and their write rare counterpart. It shouldn't be anyway a big loss, in terms of security: with only two fields, there is a 50% chance of guessing correctly the layout. The randomization is disabled only when write rare is enabled. Signed-off-by: Igor Stoppa CC: Kees Cook CC: Greg Kroah-Hartman CC: Andrew Morton CC: Masahiro Yamada CC: Alexey Dobriyan CC: Pekka Enberg CC: "Paul E. McKenney" CC: Lihao Liang CC: linux-kernel@vger.kernel.org --- include/linux/types.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/types.h b/include/linux/types.h index 53609bbdcf0f..a9f6f6515fdc 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -187,12 +187,12 @@ typedef struct { struct list_head { struct list_head *next __aligned(sizeof(void *)); struct list_head *prev __aligned(sizeof(void *)); -} __aligned(sizeof(void *)); +} __no_randomize_layout __aligned(sizeof(void *)); struct hlist_node { struct hlist_node *next __aligned(sizeof(void *)); struct hlist_node **pprev __aligned(sizeof(void *)); -} __aligned(sizeof(void *)); +} __no_randomize_layout __aligned(sizeof(void *)); #else struct list_head { struct list_head *next, *prev; From patchwork Tue Oct 23 21:35:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653803 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 59DF713A4 for ; Tue, 23 Oct 2018 21:39:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 464F12A421 for ; Tue, 23 Oct 2018 21:39:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39FA02A426; Tue, 23 Oct 2018 21:39:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 952612A427 for ; Tue, 23 Oct 2018 21:39:12 +0000 (UTC) Received: (qmail 15638 invoked by uid 550); 23 Oct 2018 21:36:29 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 15607 invoked from network); 23 Oct 2018 21:36:29 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=V7XavEgNJp0NdEehJNSe3L5RBOhoOdzq94ao4os78FM=; b=V+Ztyc+Da3zJSnZ7XoW9soiweQV8sOZWklUf/FFukP7HoBrffu2le+vVMI0gB51gGp NHcaEAU0xhkiJD8fpHZEsRoCAtNIXh3Exytu478Y/NOIgJb7BbCtQ+ECjibpVsjiAmLT wvgtd8I1JlHIWzSA7jV7gSQdw8hObYvjCTkF7RVZmTq3estHlOBX9yD3s/jZ2yu73xe0 5gZS2UMboiZ8gTO/LKJoXWQb/PJWGf+Z/p3hkg0CCa7lJPA6aKDuPUMETZi3imsgoY1p 74c6IAc6CxPzV54We4jwXqe1qQHAWqRQTjSGRzD/jDy6dWeE+oG1yPbZ/d75sC3Ahmn1 N2BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=V7XavEgNJp0NdEehJNSe3L5RBOhoOdzq94ao4os78FM=; b=MOetYwI6cTLItP9QGuGTD4kf2BJDIGd6bVi3mkmPpVIqCRdQx0NFpyd7vCelSi9rl3 jYia9mYA1KKtXq2OcPVy9aCO1XJIMmQywBBLiVVPXfvUzITnKjrmqpGZftXC6H/SSYgM gCEJoUXL3axom4o4Mk7AKx/+YRua15yiYeeZYMOFH07NMavrV0ZsyMxS+T3RM85fZzxo DmPJw5XtiVn24GnUjmAak+KUiJR/ls2puDxuNGwXFkEeBAAW3OxWJVGE/rc+EcnwRbc0 6KSTJP6UN1Q57jefEi5T0mC2UMJup31xTWnheZpOFh3bPpuJcx8HVbXipLRS9M12FoLd fLrw== X-Gm-Message-State: AGRZ1gK669ihtLsQ6S3dsFphG7qzL+ouR8rNm/Slc+OU4olhNPJQt2uV +RbOT5GU3ZdsqEG+RzXf0wQ= X-Google-Smtp-Source: AJdET5dRYBiV0gZeJUZ5q7QId0ofD4d9/OxP3H+4pyUTa8ejUzdUlMvtXe/sY7SieoSRgeugC6aVyA== X-Received: by 2002:a19:d5:: with SMTP id 204mr421236lfa.116.1540330577744; Tue, 23 Oct 2018 14:36:17 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Thomas Gleixner , Kate Stewart , "David S. Miller" , Greg Kroah-Hartman , Philippe Ombredanne , "Paul E. McKenney" , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , linux-kernel@vger.kernel.org Subject: [PATCH 14/17] prmem: llist, hlist, both plain and rcu Date: Wed, 24 Oct 2018 00:35:01 +0300 Message-Id: <20181023213504.28905-15-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP In some cases, all the data needing protection can be allocated from a pool in one go, as directly writable, then initialized and protected. The sequence is relatively short and it's acceptable to leave the entire data set unprotected. In other cases, this is not possible, because the data will trickle over a relatively long period of time, in a non predictable way, possibly for the entire duration of the operations. For these cases, the safe approach is to have the memory already write protected, when allocated. However, this will require replacing any direct assignment with calls to functions that can perform write rare. Since lists are one of the most commonly used data structures in kernel, they are a the first candidate for receiving write rare extensions. This patch implements basic functionality for altering said lists. Signed-off-by: Igor Stoppa CC: Thomas Gleixner CC: Kate Stewart CC: "David S. Miller" CC: Greg Kroah-Hartman CC: Philippe Ombredanne CC: "Paul E. McKenney" CC: Josh Triplett CC: Steven Rostedt CC: Mathieu Desnoyers CC: Lai Jiangshan CC: linux-kernel@vger.kernel.org --- MAINTAINERS | 1 + include/linux/prlist.h | 934 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 935 insertions(+) create mode 100644 include/linux/prlist.h diff --git a/MAINTAINERS b/MAINTAINERS index 246b1a1cc8bb..f5689c014e07 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9464,6 +9464,7 @@ F: mm/prmem.c F: mm/test_write_rare.c F: mm/test_pmalloc.c F: Documentation/core-api/prmem.rst +F: include/linux/prlist.h MEMORY MANAGEMENT L: linux-mm@kvack.org diff --git a/include/linux/prlist.h b/include/linux/prlist.h new file mode 100644 index 000000000000..0387c78f8be8 --- /dev/null +++ b/include/linux/prlist.h @@ -0,0 +1,934 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * prlist.h: Header for Protected Lists + * + * (C) Copyright 2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + * + * Code from and , adapted to perform + * writes on write-rare data. + * These functions and macros rely on data structures that allow the reuse + * of what is already provided for reading the content of their non-write + * rare variant. + */ + +#ifndef _LINUX_PRLIST_H +#define _LINUX_PRLIST_H + +#include +#include +#include + +/* --------------- Circular Protected Doubly Linked List --------------- */ +union prlist_head { + struct list_head list __aligned(sizeof(void *)); + struct { + union prlist_head *next __aligned(sizeof(void *)); + union prlist_head *prev __aligned(sizeof(void *)); + } __no_randomize_layout; +} __aligned(sizeof(void *)); + +static __always_inline +union prlist_head *to_prlist_head(struct list_head *list) +{ + return container_of(list, union prlist_head, list); +} + +#define PRLIST_HEAD_INIT(name) { \ + .list = LIST_HEAD_INIT(name), \ +} + +#define PRLIST_HEAD(name) \ + union prlist_head name __wr_after_init = PRLIST_HEAD_INIT(name.list) + +static __always_inline +struct pmalloc_pool *prlist_create_custom_pool(size_t refill, + unsigned short align_order) +{ + return pmalloc_create_custom_pool(refill, align_order, + PMALLOC_MODE_START_WR); +} + +static __always_inline struct pmalloc_pool *prlist_create_pool(void) +{ + return prlist_create_custom_pool(PMALLOC_REFILL_DEFAULT, + PMALLOC_ALIGN_ORDER_DEFAULT); +} + +static __always_inline +void prlist_set_prev(union prlist_head *head, + const union prlist_head *prev) +{ + wr_ptr(&head->prev, prev); +} + +static __always_inline +void prlist_set_next(union prlist_head *head, + const union prlist_head *next) +{ + wr_ptr(&head->next, next); +} + +static __always_inline void INIT_PRLIST_HEAD(union prlist_head *head) +{ + prlist_set_prev(head, head); + prlist_set_next(head, head); +} + +/* + * Insert a new entry between two known consecutive entries. + * + * This is only for internal list manipulation where we know + * the prev/next entries already! + */ +static __always_inline +void __prlist_add(union prlist_head *new, union prlist_head *prev, + union prlist_head *next) +{ + if (!__list_add_valid(&new->list, &prev->list, &next->list)) + return; + + prlist_set_prev(next, new); + prlist_set_next(new, next); + prlist_set_prev(new, prev); + prlist_set_next(prev, new); +} + +/** + * prlist_add - add a new entry + * @new: new entry to be added + * @head: prlist head to add it after + * + * Insert a new entry after the specified head. + * This is good for implementing stacks. + */ +static __always_inline +void prlist_add(union prlist_head *new, union prlist_head *head) +{ + __prlist_add(new, head, head->next); +} + +/** + * prlist_add_tail - add a new entry + * @new: new entry to be added + * @head: list head to add it before + * + * Insert a new entry before the specified head. + * This is useful for implementing queues. + */ +static __always_inline +void prlist_add_tail(union prlist_head *new, union prlist_head *head) +{ + __prlist_add(new, head->prev, head); +} + +/* + * Delete a prlist entry by making the prev/next entries + * point to each other. + * + * This is only for internal list manipulation where we know + * the prev/next entries already! + */ +static __always_inline +void __prlist_del(union prlist_head *prev, union prlist_head *next) +{ + prlist_set_prev(next, prev); + prlist_set_next(prev, next); +} + +/** + * prlist_del - deletes entry from list. + * @entry: the element to delete from the list. + * Note: list_empty() on entry does not return true after this, the entry is + * in an undefined state. + */ +static inline void __prlist_del_entry(union prlist_head *entry) +{ + if (!__list_del_entry_valid(&entry->list)) + return; + __prlist_del(entry->prev, entry->next); +} + +static __always_inline void prlist_del(union prlist_head *entry) +{ + __prlist_del_entry(entry); + prlist_set_next(entry, LIST_POISON1); + prlist_set_prev(entry, LIST_POISON2); +} + +/** + * prlist_replace - replace old entry by new one + * @old : the element to be replaced + * @new : the new element to insert + * + * If @old was empty, it will be overwritten. + */ +static __always_inline +void prlist_replace(union prlist_head *old, union prlist_head *new) +{ + prlist_set_next(new, old->next); + prlist_set_prev(new->next, new); + prlist_set_prev(new, old->prev); + prlist_set_next(new->prev, new); +} + +static __always_inline +void prlist_replace_init(union prlist_head *old, union prlist_head *new) +{ + prlist_replace(old, new); + INIT_PRLIST_HEAD(old); +} + +/** + * prlist_del_init - deletes entry from list and reinitialize it. + * @entry: the element to delete from the list. + */ +static __always_inline void prlist_del_init(union prlist_head *entry) +{ + __prlist_del_entry(entry); + INIT_PRLIST_HEAD(entry); +} + +/** + * prlist_move - delete from one list and add as another's head + * @list: the entry to move + * @head: the head that will precede our entry + */ +static __always_inline +void prlist_move(union prlist_head *list, union prlist_head *head) +{ + __prlist_del_entry(list); + prlist_add(list, head); +} + +/** + * prlist_move_tail - delete from one list and add as another's tail + * @list: the entry to move + * @head: the head that will follow our entry + */ +static __always_inline +void prlist_move_tail(union prlist_head *list, union prlist_head *head) +{ + __prlist_del_entry(list); + prlist_add_tail(list, head); +} + +/** + * prlist_rotate_left - rotate the list to the left + * @head: the head of the list + */ +static __always_inline void prlist_rotate_left(union prlist_head *head) +{ + union prlist_head *first; + + if (!list_empty(&head->list)) { + first = head->next; + prlist_move_tail(first, head); + } +} + +static __always_inline +void __prlist_cut_position(union prlist_head *list, union prlist_head *head, + union prlist_head *entry) +{ + union prlist_head *new_first = entry->next; + + prlist_set_next(list, head->next); + prlist_set_prev(list->next, list); + prlist_set_prev(list, entry); + prlist_set_next(entry, list); + prlist_set_next(head, new_first); + prlist_set_prev(new_first, head); +} + +/** + * prlist_cut_position - cut a list into two + * @list: a new list to add all removed entries + * @head: a list with entries + * @entry: an entry within head, could be the head itself + * and if so we won't cut the list + * + * This helper moves the initial part of @head, up to and + * including @entry, from @head to @list. You should + * pass on @entry an element you know is on @head. @list + * should be an empty list or a list you do not care about + * losing its data. + * + */ +static __always_inline +void prlist_cut_position(union prlist_head *list, union prlist_head *head, + union prlist_head *entry) +{ + if (list_empty(&head->list)) + return; + if (list_is_singular(&head->list) && + (head->next != entry && head != entry)) + return; + if (entry == head) + INIT_PRLIST_HEAD(list); + else + __prlist_cut_position(list, head, entry); +} + +/** + * prlist_cut_before - cut a list into two, before given entry + * @list: a new list to add all removed entries + * @head: a list with entries + * @entry: an entry within head, could be the head itself + * + * This helper moves the initial part of @head, up to but + * excluding @entry, from @head to @list. You should pass + * in @entry an element you know is on @head. @list should + * be an empty list or a list you do not care about losing + * its data. + * If @entry == @head, all entries on @head are moved to + * @list. + */ +static __always_inline +void prlist_cut_before(union prlist_head *list, union prlist_head *head, + union prlist_head *entry) +{ + if (head->next == entry) { + INIT_PRLIST_HEAD(list); + return; + } + prlist_set_next(list, head->next); + prlist_set_prev(list->next, list); + prlist_set_prev(list, entry->prev); + prlist_set_next(list->prev, list); + prlist_set_next(head, entry); + prlist_set_prev(entry, head); +} + +static __always_inline +void __prlist_splice(const union prlist_head *list, union prlist_head *prev, + union prlist_head *next) +{ + union prlist_head *first = list->next; + union prlist_head *last = list->prev; + + prlist_set_prev(first, prev); + prlist_set_next(prev, first); + prlist_set_next(last, next); + prlist_set_prev(next, last); +} + +/** + * prlist_splice - join two lists, this is designed for stacks + * @list: the new list to add. + * @head: the place to add it in the first list. + */ +static __always_inline +void prlist_splice(const union prlist_head *list, union prlist_head *head) +{ + if (!list_empty(&list->list)) + __prlist_splice(list, head, head->next); +} + +/** + * prlist_splice_tail - join two lists, each list being a queue + * @list: the new list to add. + * @head: the place to add it in the first list. + */ +static __always_inline +void prlist_splice_tail(union prlist_head *list, union prlist_head *head) +{ + if (!list_empty(&list->list)) + __prlist_splice(list, head->prev, head); +} + +/** + * prlist_splice_init - join two lists and reinitialise the emptied list. + * @list: the new list to add. + * @head: the place to add it in the first list. + * + * The list at @list is reinitialised + */ +static __always_inline +void prlist_splice_init(union prlist_head *list, union prlist_head *head) +{ + if (!list_empty(&list->list)) { + __prlist_splice(list, head, head->next); + INIT_PRLIST_HEAD(list); + } +} + +/** + * prlist_splice_tail_init - join 2 lists and reinitialise the emptied list + * @list: the new list to add. + * @head: the place to add it in the first list. + * + * Each of the lists is a queue. + * The list at @list is reinitialised + */ +static __always_inline +void prlist_splice_tail_init(union prlist_head *list, + union prlist_head *head) +{ + if (!list_empty(&list->list)) { + __prlist_splice(list, head->prev, head); + INIT_PRLIST_HEAD(list); + } +} + +/* ---- Protected Doubly Linked List with single pointer list head ---- */ +union prhlist_head { + struct hlist_head head __aligned(sizeof(void *)); + union prhlist_node *first __aligned(sizeof(void *)); +} __aligned(sizeof(void *)); + +union prhlist_node { + struct hlist_node node __aligned(sizeof(void *)) + ; + struct { + union prhlist_node *next __aligned(sizeof(void *)); + union prhlist_node **pprev __aligned(sizeof(void *)); + } __no_randomize_layout; +} __aligned(sizeof(void *)); + +#define PRHLIST_HEAD_INIT { \ + .head = HLIST_HEAD_INIT, \ +} + +#define PRHLIST_HEAD(name) \ + union prhlist_head name __wr_after_init = PRHLIST_HEAD_INIT + + +#define is_static(object) \ + unlikely(wr_check_boundaries(object, sizeof(*object))) + +static __always_inline +struct pmalloc_pool *prhlist_create_custom_pool(size_t refill, + unsigned short align_order) +{ + return pmalloc_create_custom_pool(refill, align_order, + PMALLOC_MODE_AUTO_WR); +} + +static __always_inline +struct pmalloc_pool *prhlist_create_pool(void) +{ + return prhlist_create_custom_pool(PMALLOC_REFILL_DEFAULT, + PMALLOC_ALIGN_ORDER_DEFAULT); +} + +static __always_inline +void prhlist_set_first(union prhlist_head *head, union prhlist_node *first) +{ + wr_ptr(&head->first, first); +} + +static __always_inline +void prhlist_set_next(union prhlist_node *node, union prhlist_node *next) +{ + wr_ptr(&node->next, next); +} + +static __always_inline +void prhlist_set_pprev(union prhlist_node *node, union prhlist_node **pprev) +{ + wr_ptr(&node->pprev, pprev); +} + +static __always_inline +void prhlist_set_prev(union prhlist_node *node, union prhlist_node *prev) +{ + wr_ptr(node->pprev, prev); +} + +static __always_inline void INIT_PRHLIST_HEAD(union prhlist_head *head) +{ + prhlist_set_first(head, NULL); +} + +static __always_inline void INIT_PRHLIST_NODE(union prhlist_node *node) +{ + prhlist_set_next(node, NULL); + prhlist_set_pprev(node, NULL); +} + +static __always_inline void __prhlist_del(union prhlist_node *n) +{ + union prhlist_node *next = n->next; + union prhlist_node **pprev = n->pprev; + + wr_ptr(pprev, next); + if (next) + prhlist_set_pprev(next, pprev); +} + +static __always_inline void prhlist_del(union prhlist_node *n) +{ + __prhlist_del(n); + prhlist_set_next(n, LIST_POISON1); + prhlist_set_pprev(n, LIST_POISON2); +} + +static __always_inline void prhlist_del_init(union prhlist_node *n) +{ + if (!hlist_unhashed(&n->node)) { + __prhlist_del(n); + INIT_PRHLIST_NODE(n); + } +} + +static __always_inline +void prhlist_add_head(union prhlist_node *n, union prhlist_head *h) +{ + union prhlist_node *first = h->first; + + prhlist_set_next(n, first); + if (first) + prhlist_set_pprev(first, &n->next); + prhlist_set_first(h, n); + prhlist_set_pprev(n, &h->first); +} + +/* next must be != NULL */ +static __always_inline +void prhlist_add_before(union prhlist_node *n, union prhlist_node *next) +{ + prhlist_set_pprev(n, next->pprev); + prhlist_set_next(n, next); + prhlist_set_pprev(next, &n->next); + prhlist_set_prev(n, n); +} + +static __always_inline +void prhlist_add_behind(union prhlist_node *n, union prhlist_node *prev) +{ + prhlist_set_next(n, prev->next); + prhlist_set_next(prev, n); + prhlist_set_pprev(n, &prev->next); + if (n->next) + prhlist_set_pprev(n->next, &n->next); +} + +/* after that we'll appear to be on some hlist and hlist_del will work */ +static __always_inline void prhlist_add_fake(union prhlist_node *n) +{ + prhlist_set_pprev(n, &n->next); +} + +/* + * Move a list from one list head to another. Fixup the pprev + * reference of the first entry if it exists. + */ +static __always_inline +void prhlist_move_list(union prhlist_head *old, union prhlist_head *new) +{ + prhlist_set_first(new, old->first); + if (new->first) + prhlist_set_pprev(new->first, &new->first); + prhlist_set_first(old, NULL); +} + +/* ------------------------ RCU list and hlist ------------------------ */ + +/* + * INIT_LIST_HEAD_RCU - Initialize a list_head visible to RCU readers + * @head: list to be initialized + * + * It is exactly equivalent to INIT_LIST_HEAD() + */ +static __always_inline void INIT_PRLIST_HEAD_RCU(union prlist_head *head) +{ + INIT_PRLIST_HEAD(head); +} + +/* + * Insert a new entry between two known consecutive entries. + * + * This is only for internal list manipulation where we know + * the prev/next entries already! + */ +static __always_inline +void __prlist_add_rcu(union prlist_head *new, union prlist_head *prev, + union prlist_head *next) +{ + if (!__list_add_valid(&new->list, &prev->list, &next->list)) + return; + prlist_set_next(new, next); + prlist_set_prev(new, prev); + wr_rcu_assign_pointer(list_next_rcu(&prev->list), new); + prlist_set_prev(next, new); +} + +/** + * prlist_add_rcu - add a new entry to rcu-protected prlist + * @new: new entry to be added + * @head: prlist head to add it after + * + * Insert a new entry after the specified head. + * This is good for implementing stacks. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another prlist-mutation primitive, such as prlist_add_rcu() + * or prlist_del_rcu(), running on this same list. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * list_for_each_entry_rcu(). + */ +static __always_inline +void prlist_add_rcu(union prlist_head *new, union prlist_head *head) +{ + __prlist_add_rcu(new, head, head->next); +} + +/** + * prlist_add_tail_rcu - add a new entry to rcu-protected prlist + * @new: new entry to be added + * @head: prlist head to add it before + * + * Insert a new entry before the specified head. + * This is useful for implementing queues. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another prlist-mutation primitive, such as prlist_add_tail_rcu() + * or prlist_del_rcu(), running on this same list. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * list_for_each_entry_rcu(). + */ +static __always_inline +void prlist_add_tail_rcu(union prlist_head *new, union prlist_head *head) +{ + __prlist_add_rcu(new, head->prev, head); +} + +/** + * prlist_del_rcu - deletes entry from prlist without re-initialization + * @entry: the element to delete from the prlist. + * + * Note: list_empty() on entry does not return true after this, + * the entry is in an undefined state. It is useful for RCU based + * lockfree traversal. + * + * In particular, it means that we can not poison the forward + * pointers that may still be used for walking the list. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another list-mutation primitive, such as prlist_del_rcu() + * or prlist_add_rcu(), running on this same prlist. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * list_for_each_entry_rcu(). + * + * Note that the caller is not permitted to immediately free + * the newly deleted entry. Instead, either synchronize_rcu() + * or call_rcu() must be used to defer freeing until an RCU + * grace period has elapsed. + */ +static __always_inline void prlist_del_rcu(union prlist_head *entry) +{ + __prlist_del_entry(entry); + prlist_set_prev(entry, LIST_POISON2); +} + +/** + * prhlist_del_init_rcu - deletes entry from hash list with re-initialization + * @n: the element to delete from the hash list. + * + * Note: list_unhashed() on the node return true after this. It is + * useful for RCU based read lockfree traversal if the writer side + * must know if the list entry is still hashed or already unhashed. + * + * In particular, it means that we can not poison the forward pointers + * that may still be used for walking the hash list and we can only + * zero the pprev pointer so list_unhashed() will return true after + * this. + * + * The caller must take whatever precautions are necessary (such as + * holding appropriate locks) to avoid racing with another + * list-mutation primitive, such as hlist_add_head_rcu() or + * hlist_del_rcu(), running on this same list. However, it is + * perfectly legal to run concurrently with the _rcu list-traversal + * primitives, such as hlist_for_each_entry_rcu(). + */ +static __always_inline void prhlist_del_init_rcu(union prhlist_node *n) +{ + if (!hlist_unhashed(&n->node)) { + __prhlist_del(n); + prhlist_set_pprev(n, NULL); + } +} + +/** + * prlist_replace_rcu - replace old entry by new one + * @old : the element to be replaced + * @new : the new element to insert + * + * The @old entry will be replaced with the @new entry atomically. + * Note: @old should not be empty. + */ +static __always_inline +void prlist_replace_rcu(union prlist_head *old, union prlist_head *new) +{ + prlist_set_next(new, old->next); + prlist_set_prev(new, old->prev); + wr_rcu_assign_pointer(list_next_rcu(&new->prev->list), new); + prlist_set_prev(new->next, new); + prlist_set_prev(old, LIST_POISON2); +} + +/** + * __prlist_splice_init_rcu - join an RCU-protected list into an existing list. + * @list: the RCU-protected list to splice + * @prev: points to the last element of the existing list + * @next: points to the first element of the existing list + * @sync: function to sync: synchronize_rcu(), synchronize_sched(), ... + * + * The list pointed to by @prev and @next can be RCU-read traversed + * concurrently with this function. + * + * Note that this function blocks. + * + * Important note: the caller must take whatever action is necessary to prevent + * any other updates to the existing list. In principle, it is possible to + * modify the list as soon as sync() begins execution. If this sort of thing + * becomes necessary, an alternative version based on call_rcu() could be + * created. But only if -really- needed -- there is no shortage of RCU API + * members. + */ +static __always_inline +void __prlist_splice_init_rcu(union prlist_head *list, + union prlist_head *prev, + union prlist_head *next, void (*sync)(void)) +{ + union prlist_head *first = list->next; + union prlist_head *last = list->prev; + + /* + * "first" and "last" tracking list, so initialize it. RCU readers + * have access to this list, so we must use INIT_LIST_HEAD_RCU() + * instead of INIT_LIST_HEAD(). + */ + + INIT_PRLIST_HEAD_RCU(list); + + /* + * At this point, the list body still points to the source list. + * Wait for any readers to finish using the list before splicing + * the list body into the new list. Any new readers will see + * an empty list. + */ + + sync(); + + /* + * Readers are finished with the source list, so perform splice. + * The order is important if the new list is global and accessible + * to concurrent RCU readers. Note that RCU readers are not + * permitted to traverse the prev pointers without excluding + * this function. + */ + + prlist_set_next(last, next); + wr_rcu_assign_pointer(list_next_rcu(&prev->list), first); + prlist_set_prev(first, prev); + prlist_set_prev(next, last); +} + +/** + * prlist_splice_init_rcu - splice an RCU-protected list into an existing + * list, designed for stacks. + * @list: the RCU-protected list to splice + * @head: the place in the existing list to splice the first list into + * @sync: function to sync: synchronize_rcu(), synchronize_sched(), ... + */ +static __always_inline +void prlist_splice_init_rcu(union prlist_head *list, + union prlist_head *head, + void (*sync)(void)) +{ + if (!list_empty(&list->list)) + __prlist_splice_init_rcu(list, head, head->next, sync); +} + +/** + * prlist_splice_tail_init_rcu - splice an RCU-protected list into an + * existing list, designed for queues. + * @list: the RCU-protected list to splice + * @head: the place in the existing list to splice the first list into + * @sync: function to sync: synchronize_rcu(), synchronize_sched(), ... + */ +static __always_inline +void prlist_splice_tail_init_rcu(union prlist_head *list, + union prlist_head *head, + void (*sync)(void)) +{ + if (!list_empty(&list->list)) + __prlist_splice_init_rcu(list, head->prev, head, sync); +} + +/** + * prhlist_del_rcu - deletes entry from hash list without re-initialization + * @n: the element to delete from the hash list. + * + * Note: list_unhashed() on entry does not return true after this, + * the entry is in an undefined state. It is useful for RCU based + * lockfree traversal. + * + * In particular, it means that we can not poison the forward + * pointers that may still be used for walking the hash list. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another list-mutation primitive, such as hlist_add_head_rcu() + * or hlist_del_rcu(), running on this same list. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * hlist_for_each_entry(). + */ +static __always_inline void prhlist_del_rcu(union prhlist_node *n) +{ + __prhlist_del(n); + prhlist_set_pprev(n, LIST_POISON2); +} + +/** + * prhlist_replace_rcu - replace old entry by new one + * @old : the element to be replaced + * @new : the new element to insert + * + * The @old entry will be replaced with the @new entry atomically. + */ +static __always_inline +void prhlist_replace_rcu(union prhlist_node *old, union prhlist_node *new) +{ + union prhlist_node *next = old->next; + + prhlist_set_next(new, next); + prhlist_set_pprev(new, old->pprev); + wr_rcu_assign_pointer(*(union prhlist_node __rcu **)new->pprev, new); + if (next) + prhlist_set_pprev(new->next, &new->next); + prhlist_set_pprev(old, LIST_POISON2); +} + +/** + * prhlist_add_head_rcu + * @n: the element to add to the hash list. + * @h: the list to add to. + * + * Description: + * Adds the specified element to the specified hlist, + * while permitting racing traversals. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another list-mutation primitive, such as hlist_add_head_rcu() + * or hlist_del_rcu(), running on this same list. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * hlist_for_each_entry_rcu(), used to prevent memory-consistency + * problems on Alpha CPUs. Regardless of the type of CPU, the + * list-traversal primitive must be guarded by rcu_read_lock(). + */ +static __always_inline +void prhlist_add_head_rcu(union prhlist_node *n, union prhlist_head *h) +{ + union prhlist_node *first = h->first; + + prhlist_set_next(n, first); + prhlist_set_pprev(n, &h->first); + wr_rcu_assign_pointer(hlist_first_rcu(&h->head), n); + if (first) + prhlist_set_pprev(first, &n->next); +} + +/** + * prhlist_add_tail_rcu + * @n: the element to add to the hash list. + * @h: the list to add to. + * + * Description: + * Adds the specified element to the specified hlist, + * while permitting racing traversals. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another list-mutation primitive, such as prhlist_add_head_rcu() + * or prhlist_del_rcu(), running on this same list. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * hlist_for_each_entry_rcu(), used to prevent memory-consistency + * problems on Alpha CPUs. Regardless of the type of CPU, the + * list-traversal primitive must be guarded by rcu_read_lock(). + */ +static __always_inline +void prhlist_add_tail_rcu(union prhlist_node *n, union prhlist_head *h) +{ + union prhlist_node *i, *last = NULL; + + /* Note: write side code, so rcu accessors are not needed. */ + for (i = h->first; i; i = i->next) + last = i; + + if (last) { + prhlist_set_next(n, last->next); + prhlist_set_pprev(n, &last->next); + wr_rcu_assign_pointer(hlist_next_rcu(&last->node), n); + } else { + prhlist_add_head_rcu(n, h); + } +} + +/** + * prhlist_add_before_rcu + * @n: the new element to add to the hash list. + * @next: the existing element to add the new element before. + * + * Description: + * Adds the specified element to the specified hlist + * before the specified node while permitting racing traversals. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another list-mutation primitive, such as prhlist_add_head_rcu() + * or prhlist_del_rcu(), running on this same list. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * hlist_for_each_entry_rcu(), used to prevent memory-consistency + * problems on Alpha CPUs. + */ +static __always_inline +void prhlist_add_before_rcu(union prhlist_node *n, union prhlist_node *next) +{ + prhlist_set_pprev(n, next->pprev); + prhlist_set_next(n, next); + wr_rcu_assign_pointer(hlist_pprev_rcu(&n->node), n); + prhlist_set_pprev(next, &n->next); +} + +/** + * prhlist_add_behind_rcu + * @n: the new element to add to the hash list. + * @prev: the existing element to add the new element after. + * + * Description: + * Adds the specified element to the specified hlist + * after the specified node while permitting racing traversals. + * + * The caller must take whatever precautions are necessary + * (such as holding appropriate locks) to avoid racing + * with another list-mutation primitive, such as prhlist_add_head_rcu() + * or prhlist_del_rcu(), running on this same list. + * However, it is perfectly legal to run concurrently with + * the _rcu list-traversal primitives, such as + * hlist_for_each_entry_rcu(), used to prevent memory-consistency + * problems on Alpha CPUs. + */ +static __always_inline +void prhlist_add_behind_rcu(union prhlist_node *n, union prhlist_node *prev) +{ + prhlist_set_next(n, prev->next); + prhlist_set_pprev(n, &prev->next); + wr_rcu_assign_pointer(hlist_next_rcu(&prev->node), n); + if (n->next) + prhlist_set_pprev(n->next, &n->next); +} +#endif From patchwork Tue Oct 23 21:35:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653805 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2502E14BB for ; Tue, 23 Oct 2018 21:39:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0FC532A426 for ; Tue, 23 Oct 2018 21:39:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 010A12A507; Tue, 23 Oct 2018 21:39:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 6A6F42A426 for ; Tue, 23 Oct 2018 21:39:31 +0000 (UTC) Received: (qmail 15860 invoked by uid 550); 23 Oct 2018 21:36:31 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 15748 invoked from network); 23 Oct 2018 21:36:30 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=5k2OYOiqWNcnfE7W+3KF/KBNiSrfTjcJhGVfescDD7o=; b=E9X2B2CpnKD8qa/LlwKbw4cqywWLq+kgQJxUHmDXA/7aJp+l2MwLvziy7JYXJR2pEv 9ibFPiGUYTS/Qlb0EaDxVRWYjcdvYGbvdL3im1xYKw1TGKh8dJ/tXwP/AUV0dYTUqYPO dbdOFG35QNYcyRS3e1oO/wuojHCxN7/EOLMwQSo1/ko2daDNHFMzlW5eLmfHKqdKi8+I fA4rUKveaRuwVhnXbvQvfkj9atodqyBsetdWmgi6drrvxPw36vSsWD/60sUqIbDKcDgf sGI6PCYGGEp8GrYUaYwWLruOvkU+/A2BBMDTX+SIeB6NVHmFiqj2jLq/1A9Atn3JB8T4 5plQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=5k2OYOiqWNcnfE7W+3KF/KBNiSrfTjcJhGVfescDD7o=; b=n7FQBTMuuQxXfubqQCXyZoREDQ8Eua8NxfPx2kU5g28NAOKiHNUNICpOMVUpmiPScD XNigyWJAbm4sDTLsPaJvbUYgcnEJl11Sz1c9rUvpRsCD7WZIkqWFHj58PfNMkgV4ZbSp Gfno/MK0lSIE7nQfl/a2lt03fDOw+eOeixM/anqX1uPFa0uhJQ4T1WQklf4xhyPSSsSR 1M2ePRP8UHGX0fVxcZcsJOvIj4Q6wk2s3tPRu1m6HrT6j8TG89GZgxzUJ8wti+jToi5m zJjTfdCVfMs82JN1nfIH+YdegNXE0SDGhpc39yYbU7cNZK6zK7mop6gxcuaPu7dn1Pz/ hWlg== X-Gm-Message-State: AGRZ1gIqukfayEJN6FabZ1OTVnPDcRXNUKg2gNf+8xsPpCzzQRi1agYN CddreW7xVNlc0k6CvJDVrpg= X-Google-Smtp-Source: ACcGV62ipIIRNsS/46sZsIuOqouun/oWdvVQjUzMek7vtdGeL9YSFlZUWQAvsBQSWnrWxYsmQTJLtA== X-Received: by 2002:a2e:8457:: with SMTP id u23-v6mr10422349ljh.154.1540330579056; Tue, 23 Oct 2018 14:36:19 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Kate Stewart , Philippe Ombredanne , Thomas Gleixner , Greg Kroah-Hartman , Edward Cree , linux-kernel@vger.kernel.org Subject: [PATCH 15/17] prmem: test cases for prlist and prhlist Date: Wed, 24 Oct 2018 00:35:02 +0300 Message-Id: <20181023213504.28905-16-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP These test cases focus on the basic operations required to operate both prlist and prhlist data, in particular creating, growing, shrinking, destroying. They can also be useful as reference for practical use of write-rare lists. Signed-off-by: Igor Stoppa CC: Kate Stewart CC: Philippe Ombredanne CC: Thomas Gleixner CC: Greg Kroah-Hartman CC: Edward Cree CC: linux-kernel@vger.kernel.org --- MAINTAINERS | 1 + lib/Kconfig.debug | 9 ++ lib/Makefile | 1 + lib/test_prlist.c | 252 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 263 insertions(+) create mode 100644 lib/test_prlist.c diff --git a/MAINTAINERS b/MAINTAINERS index f5689c014e07..e7f7cb1682a6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9465,6 +9465,7 @@ F: mm/test_write_rare.c F: mm/test_pmalloc.c F: Documentation/core-api/prmem.rst F: include/linux/prlist.h +F: lib/test_prlist.c MEMORY MANAGEMENT L: linux-mm@kvack.org diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 4966c4fbe7f7..40039992f05f 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2034,6 +2034,15 @@ config IO_STRICT_DEVMEM If in doubt, say Y. +config DEBUG_PRLIST_TEST + bool "Testcase for Protected Linked List" + depends on STRICT_KERNEL_RWX && PRMEM + help + This option enables the testing of an implementation of linked + list based on write rare memory. + The test cases can also be used as examples for how to use the + prlist data structure(s). + source "arch/$(SRCARCH)/Kconfig.debug" endmenu # Kernel hacking diff --git a/lib/Makefile b/lib/Makefile index 423876446810..fe7200e84c5f 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -270,3 +270,4 @@ obj-$(CONFIG_GENERIC_LIB_LSHRDI3) += lshrdi3.o obj-$(CONFIG_GENERIC_LIB_MULDI3) += muldi3.o obj-$(CONFIG_GENERIC_LIB_CMPDI2) += cmpdi2.o obj-$(CONFIG_GENERIC_LIB_UCMPDI2) += ucmpdi2.o +obj-$(CONFIG_DEBUG_PRLIST_TEST) += test_prlist.o diff --git a/lib/test_prlist.c b/lib/test_prlist.c new file mode 100644 index 000000000000..8ee46795d72a --- /dev/null +++ b/lib/test_prlist.c @@ -0,0 +1,252 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * test_prlist.c: Test cases for protected doubly linked list + * + * (C) Copyright 2018 Huawei Technologies Co. Ltd. + * Author: Igor Stoppa + */ + +#include +#include +#include +#include +#include + + +#ifdef pr_fmt +#undef pr_fmt +#endif + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +static struct pmalloc_pool *pool; + +static PRLIST_HEAD(test_prlist_head); + +/* ---------------------- prlist test functions ---------------------- */ +static bool test_init_prlist_head(void) +{ + if (WARN(test_prlist_head.prev != &test_prlist_head || + test_prlist_head.next != &test_prlist_head, + "static initialization of static prlist_head failed")) + return false; + wr_ptr(&test_prlist_head.next, NULL); + wr_ptr(&test_prlist_head.prev, NULL); + if (WARN(test_prlist_head.prev || test_prlist_head.next, + "resetting of static prlist_head failed")) + return false; + INIT_PRLIST_HEAD(&test_prlist_head); + if (WARN(test_prlist_head.prev != &test_prlist_head || + test_prlist_head.next != &test_prlist_head, + "initialization of static prlist_head failed")) + return false; + pr_info("initialization of static prlist_head passed"); + return true; +} + +struct prlist_data { + int d_int; + union prlist_head node; + unsigned long long d_ulonglong; +}; + + +#define LIST_INTERVAL 5 +#define LIST_INTERVALS 3 +#define LIST_NODES (LIST_INTERVALS * LIST_INTERVAL) +static bool test_build_prlist(void) +{ + short i; + struct prlist_data *data; + int delta; + + pool = prlist_create_pool(); + if (WARN(!pool, "could not create pool")) + return false; + + for (i = 0; i < LIST_NODES; i++) { + data = (struct prlist_data *)pmalloc(pool, sizeof(*data)); + if (WARN(!data, "Failed to allocate prlist node")) + goto out; + wr_int(&data->d_int, i); + wr_ulonglong(&data->d_ulonglong, i); + prlist_add_tail(&data->node, &test_prlist_head); + } + for (i = 1; i < LIST_NODES; i++) { + data = (struct prlist_data *)pmalloc(pool, sizeof(*data)); + if (WARN(!data, "Failed to allocate prlist node")) + goto out; + wr_int(&data->d_int, i); + wr_ulonglong(&data->d_ulonglong, i); + prlist_add(&data->node, &test_prlist_head); + } + i = LIST_NODES; + delta = -1; + list_for_each_entry(data, &test_prlist_head, node) { + i += delta; + if (!i) + delta = 1; + if (WARN(data->d_int != i || data->d_ulonglong != i, + "unexpected value in prlist, build test failed")) + goto out; + } + pr_info("build prlist test passed"); + return true; +out: + pmalloc_destroy_pool(pool); + return false; +} + +static bool test_teardown_prlist(void) +{ + short i; + + for (i = 0; !list_empty(&test_prlist_head.list); i++) + prlist_del(test_prlist_head.next); + if (WARN(i != LIST_NODES * 2 - 1, "teardown prlist test failed")) + return false; + pmalloc_destroy_pool(pool); + pr_info("teardown prlist test passed"); + return true; +} + +static bool test_prlist(void) +{ + if (WARN(!(test_init_prlist_head() && + test_build_prlist() && + test_teardown_prlist()), + "prlist test failed")) + return false; + pr_info("prlist test passed"); + return true; +} + +/* ---------------------- prhlist test functions ---------------------- */ +static PRHLIST_HEAD(test_prhlist_head); + +static bool test_init_prhlist_head(void) +{ + if (WARN(test_prhlist_head.first, + "static initialization of static prhlist_head failed")) + return false; + wr_ptr(&test_prhlist_head.first, (void *)-1); + if (WARN(!test_prhlist_head.first, + "resetting of static prhlist_head failed")) + return false; + INIT_PRHLIST_HEAD(&test_prhlist_head); + if (WARN(!test_prlist_head.prev, + "initialization of static prhlist_head failed")) + return false; + pr_info("initialization of static prlist_head passed"); + return true; +} + +struct prhlist_data { + int d_int; + union prhlist_node node; + unsigned long long d_ulonglong; +}; + +static bool test_build_prhlist(void) +{ + short i; + struct prhlist_data *data; + union prhlist_node *anchor; + + pool = prhlist_create_pool(); + if (WARN(!pool, "could not create pool")) + return false; + + for (i = 2 * LIST_INTERVAL - 1; i >= LIST_INTERVAL; i--) { + data = (struct prhlist_data *)pmalloc(pool, sizeof(*data)); + if (WARN(!data, "Failed to allocate prhlist node")) + goto out; + wr_int(&data->d_int, i); + wr_ulonglong(&data->d_ulonglong, i); + prhlist_add_head(&data->node, &test_prhlist_head); + } + anchor = test_prhlist_head.first; + for (i = 0; i < LIST_INTERVAL; i++) { + data = (struct prhlist_data *)pmalloc(pool, sizeof(*data)); + if (WARN(!data, "Failed to allocate prhlist node")) + goto out; + wr_int(&data->d_int, i); + wr_ulonglong(&data->d_ulonglong, i); + prhlist_add_before(&data->node, anchor); + } + hlist_for_each_entry(data, &test_prhlist_head, node) + if (!data->node.next) + anchor = &data->node; + for (i = 3 * LIST_INTERVAL - 1; i >= 2 * LIST_INTERVAL; i--) { + data = (struct prhlist_data *)pmalloc(pool, sizeof(*data)); + if (WARN(!data, "Failed to allocate prhlist node")) + goto out; + wr_int(&data->d_int, i); + wr_ulonglong(&data->d_ulonglong, i); + prhlist_add_behind(&data->node, anchor); + } + i = 0; + hlist_for_each_entry(data, &test_prhlist_head, node) { + if (WARN(data->d_int != i || data->d_ulonglong != i, + "unexpected value in prhlist, build test failed")) + goto out; + i++; + } + if (WARN(i != LIST_NODES, + "wrong number of nodes: %d, expectd %d", i, LIST_NODES)) + goto out; + pr_info("build prhlist test passed"); + return true; +out: + pmalloc_destroy_pool(pool); + return false; +} + +static bool test_teardown_prhlist(void) +{ + union prhlist_node **pnode; + bool retval = false; + + for (pnode = &test_prhlist_head.first->next; *pnode;) { + if (WARN(*(*pnode)->pprev != *pnode, + "inconsistent pprev value, delete test failed")) + goto err; + prhlist_del(*pnode); + } + prhlist_del(test_prhlist_head.first); + if (WARN(!hlist_empty(&test_prhlist_head.head), + "prhlist is not empty, delete test failed")) + goto err; + pr_info("deletion of prhlist passed"); + retval = true; +err: + pmalloc_destroy_pool(pool); + return retval; +} + +static bool test_prhlist(void) +{ + if (WARN(!(test_init_prhlist_head() && + test_build_prhlist() && + test_teardown_prhlist()), + "prhlist test failed")) + return false; + pr_info("prhlist test passed"); + return true; +} + +static int __init test_prlists_init_module(void) +{ + if (WARN(!(test_prlist() && + test_prhlist()), + "protected lists test failed")) + return -EFAULT; + pr_info("protected lists test passed"); + return 0; +} + +module_init(test_prlists_init_module); + +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Igor Stoppa "); +MODULE_DESCRIPTION("Test module for protected doubly linked list."); From patchwork Tue Oct 23 21:35:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653807 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEC7613A9 for ; Tue, 23 Oct 2018 21:39:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ABA642A426 for ; Tue, 23 Oct 2018 21:39:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9FBCC2A4FD; Tue, 23 Oct 2018 21:39:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id BC2F82A426 for ; Tue, 23 Oct 2018 21:39:46 +0000 (UTC) Received: (qmail 16009 invoked by uid 550); 23 Oct 2018 21:36:32 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 15863 invoked from network); 23 Oct 2018 21:36:32 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=8LbEKOKWe/FiMlUHhmCc7voGh1W3EYnicEfE4r2R4yA=; b=N8qUCIMyIvLATq4CE5wE0uPw31UaGBnKbzEM90Hb/Wj5TI1WQujrJdzbDP0qxRobgM ZCoMs/0ZzA4kG0cqvysWLv8m1q0Btspo1GbzYx1YZcfwsBvBLB347lafqVcLjNpuD7uh xNIhdaCGy21A02rCV1tCtVf8Qt9OVwhnyCylb8Rw+Qqs4nwLso7hSHPYpkPb2fncfVfs LOb4TQscLGzNtufTiVOunV6qW8ZGAQJeeXOSVZhjwVyf+ji2DRIiXWzgsXSlLdIPe+bb VhYGtK5SvxmDKoY1xlMKrg9FwOYZUbMyGD6luSZZPCqmzYSTidxTHU/+LpYhnSh1CdwT PDtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=8LbEKOKWe/FiMlUHhmCc7voGh1W3EYnicEfE4r2R4yA=; b=tovlIyL3mGv2OOER4YVrX6RLF9iLqVhCAwCajdCpERWWOEMlKZYDlugCZGOKfFqBp7 Jpoi3s/1T3U1whOqTeVxfynWhjWdzeRGSwF2pPLlCKdOhu/At0QKTbYBMt4RuUBPiIZD J2BdqOEN8nnd2XPklqQEblOY0DfLfNTT7mmjd6OYb3OAYKlq4BUC75eTVmQvPW/lOiOv tZ4KX4829nPmePWAl2uIBv51GkTc/rxcq3HaXeIjnJO3tW3k+aOkAaUS3fFW2TbEScX7 4vGPQoNdoIQIL93Fkkxx52DSGqXL3RoWPT/9vYRyPBehe1VOQJKDzZrvQOizMPLsDANM FXhg== X-Gm-Message-State: ABuFfojCAnsqbcehOJsqC+7sC7VmNTp1Y3P6UXp1lNmyhZnDXG8SAh00 oVxXi8WO4eo9HNBcek7OHVc= X-Google-Smtp-Source: ACcGV60VSV41bEasnudRwceqLHq3cQn21dkmbWSOYeDgxo5mc3i8TMNkXsnfHZ82wYQ9h6vmAtB3yg== X-Received: by 2002:a2e:80d2:: with SMTP id r18-v6mr37260197ljg.100.1540330580535; Tue, 23 Oct 2018 14:36:20 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Will Deacon , Peter Zijlstra , Boqun Feng , Arnd Bergmann , linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 16/17] prmem: pratomic-long Date: Wed, 24 Oct 2018 00:35:03 +0300 Message-Id: <20181023213504.28905-17-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP Minimalistic functionality for having the write rare version of atomic_long_t data. Signed-off-by: Igor Stoppa CC: Will Deacon CC: Peter Zijlstra CC: Boqun Feng CC: Arnd Bergmann CC: linux-arch@vger.kernel.org CC: linux-kernel@vger.kernel.org --- MAINTAINERS | 1 + include/linux/pratomic-long.h | 73 +++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 include/linux/pratomic-long.h diff --git a/MAINTAINERS b/MAINTAINERS index e7f7cb1682a6..9d72688d00a3 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9466,6 +9466,7 @@ F: mm/test_pmalloc.c F: Documentation/core-api/prmem.rst F: include/linux/prlist.h F: lib/test_prlist.c +F: include/linux/pratomic-long.h MEMORY MANAGEMENT L: linux-mm@kvack.org diff --git a/include/linux/pratomic-long.h b/include/linux/pratomic-long.h new file mode 100644 index 000000000000..8f1408593733 --- /dev/null +++ b/include/linux/pratomic-long.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Atomic operations for write rare memory */ +#ifndef _LINUX_PRATOMIC_LONG_H +#define _LINUX_PRATOMIC_LONG_H +#include +#include +#include + +struct pratomic_long_t { + atomic_long_t l __aligned(sizeof(atomic_long_t)); +} __aligned(sizeof(atomic_long_t)); + +#define PRATOMIC_LONG_INIT(i) { \ + .l = ATOMIC_LONG_INIT((i)), \ +} + +static __always_inline +bool __pratomic_long_op(bool inc, struct pratomic_long_t *l) +{ + struct page *page; + uintptr_t base; + uintptr_t offset; + unsigned long flags; + size_t size = sizeof(*l); + bool is_virt = __is_wr_after_init(l, size); + + if (WARN(!(is_virt || likely(__is_wr_pool(l, size))), + WR_ERR_RANGE_MSG)) + return false; + local_irq_save(flags); + if (is_virt) + page = virt_to_page(l); + else + vmalloc_to_page(l); + offset = (~PAGE_MASK) & (uintptr_t)l; + base = (uintptr_t)vmap(&page, 1, VM_MAP, PAGE_KERNEL); + if (WARN(!base, WR_ERR_PAGE_MSG)) { + local_irq_restore(flags); + return false; + } + if (inc) + atomic_long_inc((atomic_long_t *)(base + offset)); + else + atomic_long_dec((atomic_long_t *)(base + offset)); + vunmap((void *)base); + local_irq_restore(flags); + return true; + +} + +/** + * pratomic_long_inc - atomic increment of rare write long + * @l: address of the variable of type struct pratomic_long_t + * + * Return: true on success, false otherwise + */ +static __always_inline bool pratomic_long_inc(struct pratomic_long_t *l) +{ + return __pratomic_long_op(true, l); +} + +/** + * pratomic_long_inc - atomic decrement of rare write long + * @l: address of the variable of type struct pratomic_long_t + * + * Return: true on success, false otherwise + */ +static __always_inline bool pratomic_long_dec(struct pratomic_long_t *l) +{ + return __pratomic_long_op(false, l); +} + +#endif From patchwork Tue Oct 23 21:35:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Igor Stoppa X-Patchwork-Id: 10653809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E2E7314BB for ; Tue, 23 Oct 2018 21:40:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB60A2A4FF for ; Tue, 23 Oct 2018 21:40:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC08A2A507; Tue, 23 Oct 2018 21:40:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id F05AB2A4FF for ; Tue, 23 Oct 2018 21:39:59 +0000 (UTC) Received: (qmail 16136 invoked by uid 550); 23 Oct 2018 21:36:34 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 16064 invoked from network); 23 Oct 2018 21:36:33 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to; bh=9FjWk819tI8q4Of5/GPMym38mMpreAR6lAKVVURm4+E=; b=ZaGL7O9QhY2jV9QlUoGEx2RYYKVeDm+Qn8Dz7gfcUqyv0drW9KAQdyhAtBl5ScwmJ4 nDiWn/GV5yOAQihkjSTJnjT2aokUhrXlMJohwEiUETROgzQo7qqJsHjUCafOBfQvdMxH YHQR1GLLP/maXsuxnk63Cxq9HpqVTBMK2NI2bv900uXuZmBGqH51It6uGOsLJE2fWYfG 6pkXXBdjXQw5c81MPcYkUp/W2Fchea7WoyQOXNAdP3rX0RjytwLNxw0tBso2VK3Bnzzp +nDYJl8GRYYpT918e/BGp+3Qu4x67MFTUlCnXST536vZQUhX2hfdmUDtAqW8JAyluAPO vifw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to; bh=9FjWk819tI8q4Of5/GPMym38mMpreAR6lAKVVURm4+E=; b=S95PbY3gcawVbLOps7/8O7x6mZVhkwLEZcNLixM0bDy7Ye8Br5kenDQjYqYE+3QfNX tphq9EMsJ2ZrRdG8FdIKSW/GCfc6TO3vu4G62j1QnBcMwSXOl51CuvKcWQF9Wigs8ejQ 3OnqRFsWwQHNg9YPRQmCX7IrAjsQ7JYxOcxiFNdQgMhnHxQwdsyTbCod9zLPgo2J2aoD tGYbj5u5f+ydgUw7dBYzc7DJ7KBvaQQli9WFtFsu+6liOvEMVWelIEtsGWvhJ/Q09ptL qVyk3KxW5e7nTlGiAF01YCUP4xzAxTUttVGfnuwH35lPKT0kpjYNm/udVv01C4M1BOTf FlOg== X-Gm-Message-State: ABuFfoiGgdD1HEUx6xuZJPA+SLVchVK2TNM72ApPaDXU0Lxxj1MuBJ/L dM5NRxpcRTPfo2ICQukev0M= X-Google-Smtp-Source: AJdET5fStpo5iwjElaH4GwCWOgDJjeaJxYFI38XhqKxAW7VjqnSm921zQaz+uFEKHN4MgpOPwJPDsA== X-Received: by 2002:a2e:9f17:: with SMTP id u23-v6mr15051827ljk.53.1540330582091; Tue, 23 Oct 2018 14:36:22 -0700 (PDT) From: Igor Stoppa X-Google-Original-From: Igor Stoppa To: Mimi Zohar , Kees Cook , Matthew Wilcox , Dave Chinner , James Morris , Michal Hocko , kernel-hardening@lists.openwall.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org Cc: igor.stoppa@huawei.com, Dave Hansen , Jonathan Corbet , Laura Abbott , Dmitry Kasatkin , "Serge E. Hallyn" , linux-kernel@vger.kernel.org Subject: [PATCH 17/17] prmem: ima: turn the measurements list write rare Date: Wed, 24 Oct 2018 00:35:04 +0300 Message-Id: <20181023213504.28905-18-igor.stoppa@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181023213504.28905-1-igor.stoppa@huawei.com> References: <20181023213504.28905-1-igor.stoppa@huawei.com> X-Virus-Scanned: ClamAV using ClamSMTP The measurement list is moved to write rare memory, including related data structures. Various boilerplate linux data structures and related functions are replaced by their write-rare counterpart. Signed-off-by: Igor Stoppa CC: Mimi Zohar CC: Dmitry Kasatkin CC: James Morris CC: "Serge E. Hallyn" CC: linux-integrity@vger.kernel.org CC: linux-kernel@vger.kernel.org --- security/integrity/ima/ima.h | 18 ++++++++------ security/integrity/ima/ima_api.c | 29 +++++++++++++---------- security/integrity/ima/ima_fs.c | 12 +++++----- security/integrity/ima/ima_main.c | 6 +++++ security/integrity/ima/ima_queue.c | 28 +++++++++++++--------- security/integrity/ima/ima_template.c | 14 ++++++----- security/integrity/ima/ima_template_lib.c | 14 +++++++---- 7 files changed, 74 insertions(+), 47 deletions(-) diff --git a/security/integrity/ima/ima.h b/security/integrity/ima/ima.h index 67db9d9454ca..5f5959753bf5 100644 --- a/security/integrity/ima/ima.h +++ b/security/integrity/ima/ima.h @@ -24,6 +24,8 @@ #include #include #include +#include +#include #include #include "../integrity.h" @@ -84,7 +86,7 @@ struct ima_template_field { /* IMA template descriptor definition */ struct ima_template_desc { - struct list_head list; + union prlist_head list; char *name; char *fmt; int num_fields; @@ -100,11 +102,13 @@ struct ima_template_entry { }; struct ima_queue_entry { - struct hlist_node hnext; /* place in hash collision list */ - struct list_head later; /* place in ima_measurements list */ + union prhlist_node hnext; /* place in hash collision list */ + union prlist_head later; /* place in ima_measurements list */ struct ima_template_entry *entry; }; -extern struct list_head ima_measurements; /* list of all measurements */ + +/* list of all measurements */ +extern union prlist_head ima_measurements __wr_after_init; /* Some details preceding the binary serialized measurement list */ struct ima_kexec_hdr { @@ -160,9 +164,9 @@ void ima_init_template_list(void); extern spinlock_t ima_queue_lock; struct ima_h_table { - atomic_long_t len; /* number of stored measurements in the list */ - atomic_long_t violations; - struct hlist_head queue[IMA_MEASURE_HTABLE_SIZE]; + struct pratomic_long_t len; /* # of measurements in the list */ + struct pratomic_long_t violations; + union prhlist_head queue[IMA_MEASURE_HTABLE_SIZE]; }; extern struct ima_h_table ima_htable; diff --git a/security/integrity/ima/ima_api.c b/security/integrity/ima/ima_api.c index a02c5acfd403..4fc28c2478b0 100644 --- a/security/integrity/ima/ima_api.c +++ b/security/integrity/ima/ima_api.c @@ -19,9 +19,12 @@ #include #include #include +#include +#include #include "ima.h" +extern struct pmalloc_pool ima_pool; /* * ima_free_template_entry - free an existing template entry */ @@ -29,10 +32,10 @@ void ima_free_template_entry(struct ima_template_entry *entry) { int i; - for (i = 0; i < entry->template_desc->num_fields; i++) - kfree(entry->template_data[i].data); +// for (i = 0; i < entry->template_desc->num_fields; i++) +// kfree(entry->template_data[i].data); - kfree(entry); +// kfree(entry); } /* @@ -44,12 +47,13 @@ int ima_alloc_init_template(struct ima_event_data *event_data, struct ima_template_desc *template_desc = ima_template_desc_current(); int i, result = 0; - *entry = kzalloc(sizeof(**entry) + template_desc->num_fields * - sizeof(struct ima_field_data), GFP_NOFS); + *entry = pzalloc(&ima_pool, + sizeof(**entry) + template_desc->num_fields * + sizeof(struct ima_field_data)); if (!*entry) return -ENOMEM; - (*entry)->template_desc = template_desc; + wr_ptr(&((*entry)->template_desc), template_desc); for (i = 0; i < template_desc->num_fields; i++) { struct ima_template_field *field = template_desc->fields[i]; u32 len; @@ -59,9 +63,10 @@ int ima_alloc_init_template(struct ima_event_data *event_data, if (result != 0) goto out; - len = (*entry)->template_data[i].len; - (*entry)->template_data_len += sizeof(len); - (*entry)->template_data_len += len; + len = (*entry)->template_data_len + sizeof(len) + + (*entry)->template_data[i].len; + wr_memcpy(&(*entry)->template_data_len, &len, + sizeof(len)); } return 0; out: @@ -113,9 +118,9 @@ int ima_store_template(struct ima_template_entry *entry, audit_cause, result, 0); return result; } - memcpy(entry->digest, hash.hdr.digest, hash.hdr.length); + wr_memcpy(entry->digest, hash.hdr.digest, hash.hdr.length); } - entry->pcr = pcr; + wr_int(&entry->pcr, pcr); result = ima_add_template_entry(entry, violation, op, inode, filename); return result; } @@ -139,7 +144,7 @@ void ima_add_violation(struct file *file, const unsigned char *filename, int result; /* can overflow, only indicator */ - atomic_long_inc(&ima_htable.violations); + pratomic_long_inc(&ima_htable.violations); result = ima_alloc_init_template(&event_data, &entry); if (result < 0) { diff --git a/security/integrity/ima/ima_fs.c b/security/integrity/ima/ima_fs.c index ae9d5c766a3c..ab20da1161c7 100644 --- a/security/integrity/ima/ima_fs.c +++ b/security/integrity/ima/ima_fs.c @@ -57,7 +57,8 @@ static ssize_t ima_show_htable_violations(struct file *filp, char __user *buf, size_t count, loff_t *ppos) { - return ima_show_htable_value(buf, count, ppos, &ima_htable.violations); + return ima_show_htable_value(buf, count, ppos, + &ima_htable.violations.l); } static const struct file_operations ima_htable_violations_ops = { @@ -69,8 +70,7 @@ static ssize_t ima_show_measurements_count(struct file *filp, char __user *buf, size_t count, loff_t *ppos) { - return ima_show_htable_value(buf, count, ppos, &ima_htable.len); - + return ima_show_htable_value(buf, count, ppos, &ima_htable.len.l); } static const struct file_operations ima_measurements_count_ops = { @@ -86,7 +86,7 @@ static void *ima_measurements_start(struct seq_file *m, loff_t *pos) /* we need a lock since pos could point beyond last element */ rcu_read_lock(); - list_for_each_entry_rcu(qe, &ima_measurements, later) { + list_for_each_entry_rcu(qe, &ima_measurements.list, later.list) { if (!l--) { rcu_read_unlock(); return qe; @@ -303,7 +303,7 @@ static ssize_t ima_read_policy(char *path) size -= rc; } - vfree(data); +// vfree(data); if (rc < 0) return rc; else if (size) @@ -350,7 +350,7 @@ static ssize_t ima_write_policy(struct file *file, const char __user *buf, } mutex_unlock(&ima_write_mutex); out_free: - kfree(data); +// kfree(data); out: if (result < 0) valid_policy = 0; diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c index 2d31921fbda4..d52e59006781 100644 --- a/security/integrity/ima/ima_main.c +++ b/security/integrity/ima/ima_main.c @@ -29,6 +29,7 @@ #include #include #include +#include #include "ima.h" @@ -536,10 +537,15 @@ int ima_load_data(enum kernel_load_data_id id) return 0; } +struct pmalloc_pool ima_pool; + +#define IMA_POOL_ALLOC_CHUNK (16 * PAGE_SIZE) static int __init init_ima(void) { int error; + pmalloc_init_custom_pool(&ima_pool, IMA_POOL_ALLOC_CHUNK, 3, + PMALLOC_MODE_START_WR); ima_init_template_list(); hash_setup(CONFIG_IMA_DEFAULT_HASH); error = ima_init(); diff --git a/security/integrity/ima/ima_queue.c b/security/integrity/ima/ima_queue.c index b186819bd5aa..444c47b745d8 100644 --- a/security/integrity/ima/ima_queue.c +++ b/security/integrity/ima/ima_queue.c @@ -24,11 +24,14 @@ #include #include #include +#include +#include +#include #include "ima.h" #define AUDIT_CAUSE_LEN_MAX 32 -LIST_HEAD(ima_measurements); /* list of all measurements */ +PRLIST_HEAD(ima_measurements); /* list of all measurements */ #ifdef CONFIG_IMA_KEXEC static unsigned long binary_runtime_size; #else @@ -36,9 +39,9 @@ static unsigned long binary_runtime_size = ULONG_MAX; #endif /* key: inode (before secure-hashing a file) */ -struct ima_h_table ima_htable = { - .len = ATOMIC_LONG_INIT(0), - .violations = ATOMIC_LONG_INIT(0), +struct ima_h_table ima_htable __wr_after_init = { + .len = PRATOMIC_LONG_INIT(0), + .violations = PRATOMIC_LONG_INIT(0), .queue[0 ... IMA_MEASURE_HTABLE_SIZE - 1] = HLIST_HEAD_INIT }; @@ -58,7 +61,7 @@ static struct ima_queue_entry *ima_lookup_digest_entry(u8 *digest_value, key = ima_hash_key(digest_value); rcu_read_lock(); - hlist_for_each_entry_rcu(qe, &ima_htable.queue[key], hnext) { + hlist_for_each_entry_rcu(qe, &ima_htable.queue[key], hnext.node) { rc = memcmp(qe->entry->digest, digest_value, TPM_DIGEST_SIZE); if ((rc == 0) && (qe->entry->pcr == pcr)) { ret = qe; @@ -87,6 +90,8 @@ static int get_binary_runtime_size(struct ima_template_entry *entry) return size; } +extern struct pmalloc_pool ima_pool; + /* ima_add_template_entry helper function: * - Add template entry to the measurement list and hash table, for * all entries except those carried across kexec. @@ -99,20 +104,21 @@ static int ima_add_digest_entry(struct ima_template_entry *entry, struct ima_queue_entry *qe; unsigned int key; - qe = kmalloc(sizeof(*qe), GFP_KERNEL); + qe = pmalloc(&ima_pool, sizeof(*qe)); if (qe == NULL) { pr_err("OUT OF MEMORY ERROR creating queue entry\n"); return -ENOMEM; } - qe->entry = entry; + wr_ptr(&qe->entry, entry); + INIT_PRLIST_HEAD(&qe->later); + prlist_add_tail_rcu(&qe->later, &ima_measurements); + - INIT_LIST_HEAD(&qe->later); - list_add_tail_rcu(&qe->later, &ima_measurements); + pratomic_long_inc(&ima_htable.len); - atomic_long_inc(&ima_htable.len); if (update_htable) { key = ima_hash_key(entry->digest); - hlist_add_head_rcu(&qe->hnext, &ima_htable.queue[key]); + prhlist_add_head_rcu(&qe->hnext, &ima_htable.queue[key]); } if (binary_runtime_size != ULONG_MAX) { diff --git a/security/integrity/ima/ima_template.c b/security/integrity/ima/ima_template.c index 30db39b23804..40ae57a17d89 100644 --- a/security/integrity/ima/ima_template.c +++ b/security/integrity/ima/ima_template.c @@ -22,14 +22,15 @@ enum header_fields { HDR_PCR, HDR_DIGEST, HDR_TEMPLATE_NAME, HDR_TEMPLATE_DATA, HDR__LAST }; -static struct ima_template_desc builtin_templates[] = { +static struct ima_template_desc builtin_templates[] __wr_after_init = { {.name = IMA_TEMPLATE_IMA_NAME, .fmt = IMA_TEMPLATE_IMA_FMT}, {.name = "ima-ng", .fmt = "d-ng|n-ng"}, {.name = "ima-sig", .fmt = "d-ng|n-ng|sig"}, {.name = "", .fmt = ""}, /* placeholder for a custom format */ }; -static LIST_HEAD(defined_templates); +static PRLIST_HEAD(defined_templates); + static DEFINE_SPINLOCK(template_list); static struct ima_template_field supported_fields[] = { @@ -114,7 +115,8 @@ static struct ima_template_desc *lookup_template_desc(const char *name) int found = 0; rcu_read_lock(); - list_for_each_entry_rcu(template_desc, &defined_templates, list) { + list_for_each_entry_rcu(template_desc, &defined_templates.list, + list.list) { if ((strcmp(template_desc->name, name) == 0) || (strcmp(template_desc->fmt, name) == 0)) { found = 1; @@ -207,12 +209,12 @@ void ima_init_template_list(void) { int i; - if (!list_empty(&defined_templates)) + if (!list_empty(&defined_templates.list)) return; spin_lock(&template_list); for (i = 0; i < ARRAY_SIZE(builtin_templates); i++) { - list_add_tail_rcu(&builtin_templates[i].list, + prlist_add_tail_rcu(&builtin_templates[i].list, &defined_templates); } spin_unlock(&template_list); @@ -266,7 +268,7 @@ static struct ima_template_desc *restore_template_fmt(char *template_name) goto out; spin_lock(&template_list); - list_add_tail_rcu(&template_desc->list, &defined_templates); + prlist_add_tail_rcu(&template_desc->list, &defined_templates); spin_unlock(&template_list); out: return template_desc; diff --git a/security/integrity/ima/ima_template_lib.c b/security/integrity/ima/ima_template_lib.c index 43752002c222..a6d10eabf0e5 100644 --- a/security/integrity/ima/ima_template_lib.c +++ b/security/integrity/ima/ima_template_lib.c @@ -15,8 +15,12 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include +#include #include "ima_template_lib.h" +extern struct pmalloc_pool ima_pool; + static bool ima_template_hash_algo_allowed(u8 algo) { if (algo == HASH_ALGO_SHA1 || algo == HASH_ALGO_MD5) @@ -42,11 +46,11 @@ static int ima_write_template_field_data(const void *data, const u32 datalen, if (datafmt == DATA_FMT_STRING) buflen = datalen + 1; - buf = kzalloc(buflen, GFP_KERNEL); + buf = pzalloc(&ima_pool, buflen); if (!buf) return -ENOMEM; - memcpy(buf, data, datalen); + wr_memcpy(buf, data, datalen); /* * Replace all space characters with underscore for event names and @@ -58,11 +62,11 @@ static int ima_write_template_field_data(const void *data, const u32 datalen, if (datafmt == DATA_FMT_STRING) { for (buf_ptr = buf; buf_ptr - buf < datalen; buf_ptr++) if (*buf_ptr == ' ') - *buf_ptr = '_'; + wr_char(buf_ptr, '_'); } - field_data->data = buf; - field_data->len = buflen; + wr_ptr(&field_data->data, buf); + wr_memcpy(&field_data->len, &buflen, sizeof(buflen)); return 0; }