From patchwork Thu Jan 10 21:09:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10756951 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D302113B5 for ; Thu, 10 Jan 2019 21:13:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF9BC29B97 for ; Thu, 10 Jan 2019 21:13:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B1D1929BC9; Thu, 10 Jan 2019 21:13:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id AA5B529B97 for ; Thu, 10 Jan 2019 21:13:40 +0000 (UTC) Received: (qmail 30707 invoked by uid 550); 10 Jan 2019 21:13:21 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 30637 invoked from network); 10 Jan 2019 21:13:20 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=/IFEDzhdI0x1IZkLHUorzjqe46w6d+OUBpCZ00v5/kI=; b=JNUIA0ZiLYSbjyCwAVK6/QyBmq8FL72dUceoNqR3YqbxV3KpfLrJ4/o4shwgV/QQkpbT i+JrwBLXO2mzCBJzNUIf2RhgKPu2jg5XHRABDtU+LXQqrUZ6aGpK2NvIydlwKaJppCdP j+wdmrhR7TDmOqvKgoKXVAWk8vUVI1gtKEbYePfC6zugyaJRhLJtLxHH13kyG9h5AdLz qJSxuJEXRAMb7+qsXPG8fmHl/+VvjCI0MtbqDIbPMADcIhXDIUSs7Ix3Av5qePZBrkqB 7GL2jlWfvCsYjC3WOY/uiO9bWQO8zEc6KoJZ/k+sNM/QF+DyHlfh3pKl8iF6Z2h4GVQH Fw== From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, "Vasileios P . Kemerlis" , Juerg Haefliger , Tycho Andersen , Marco Benatto , David Woodhouse , Khalid Aziz Subject: [RFC PATCH v7 11/16] mm, x86: omit TLB flushing by default for XPFO page table modifications Date: Thu, 10 Jan 2019 14:09:43 -0700 Message-Id: <4e51a5d4409b54116968b8c0501f6d82c4eb9cb5.1547153058.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9132 signatures=668680 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901100164 X-Virus-Scanned: ClamAV using ClamSMTP From: Julian Stecklina XPFO carries a large performance overhead. In my tests, I saw >40% overhead for compiling a Linux kernel with XPFO enabled. The frequent TLB flushes that XPFO performs are the root cause of much of this overhead. TLB flushing is required for full paranoia mode where we don't want TLB entries of physmap pages to stick around potentially indefinitely. In reality, though, these TLB entries are going to be evicted pretty rapidly even without explicit flushing. That means omitting TLB flushes only marginally lowers the security benefits of XPFO. For kernel compile, omitting TLB flushes pushes the overhead below 3%. Change the default in XPFO to not flush TLBs unless the user explicitly requests to do so using a kernel parameter. Signed-off-by: Julian Stecklina Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis Cc: Juerg Haefliger Cc: Tycho Andersen Cc: Marco Benatto Cc: David Woodhouse Signed-off-by: Khalid Aziz --- mm/xpfo.c | 37 +++++++++++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/mm/xpfo.c b/mm/xpfo.c index 25fba05d01bd..e80374b0c78e 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -36,6 +36,7 @@ struct xpfo { }; DEFINE_STATIC_KEY_FALSE(xpfo_inited); +DEFINE_STATIC_KEY_FALSE(xpfo_do_tlb_flush); static bool xpfo_disabled __initdata; @@ -46,7 +47,15 @@ static int __init noxpfo_param(char *str) return 0; } +static int __init xpfotlbflush_param(char *str) +{ + static_branch_enable(&xpfo_do_tlb_flush); + + return 0; +} + early_param("noxpfo", noxpfo_param); +early_param("xpfotlbflush", xpfotlbflush_param); static bool __init need_xpfo(void) { @@ -76,6 +85,13 @@ bool __init xpfo_enabled(void) } EXPORT_SYMBOL(xpfo_enabled); + +static void xpfo_cond_flush_kernel_tlb(struct page *page, int order) +{ + if (static_branch_unlikely(&xpfo_do_tlb_flush)) + xpfo_flush_kernel_tlb(page, order); +} + static inline struct xpfo *lookup_xpfo(struct page *page) { struct page_ext *page_ext = lookup_page_ext(page); @@ -114,12 +130,17 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) "xpfo: already mapped page being allocated\n"); if ((gfp & GFP_HIGHUSER) == GFP_HIGHUSER) { - /* - * Tag the page as a user page and flush the TLB if it - * was previously allocated to the kernel. - */ - if (!test_and_set_bit(XPFO_PAGE_USER, &xpfo->flags)) - flush_tlb = 1; + if (static_branch_unlikely(&xpfo_do_tlb_flush)) { + /* + * Tag the page as a user page and flush the TLB if it + * was previously allocated to the kernel. + */ + if (!test_and_set_bit(XPFO_PAGE_USER, &xpfo->flags)) + flush_tlb = 1; + } else { + set_bit(XPFO_PAGE_USER, &xpfo->flags); + } + } else { /* Tag the page as a non-user (kernel) page */ clear_bit(XPFO_PAGE_USER, &xpfo->flags); @@ -127,7 +148,7 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) } if (flush_tlb) - xpfo_flush_kernel_tlb(page, order); + xpfo_cond_flush_kernel_tlb(page, order); } void xpfo_free_pages(struct page *page, int order) @@ -221,7 +242,7 @@ void xpfo_kunmap(void *kaddr, struct page *page) "xpfo: unmapping already unmapped page\n"); set_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); set_kpte(kaddr, page, __pgprot(0)); - xpfo_flush_kernel_tlb(page, 0); + xpfo_cond_flush_kernel_tlb(page, 0); } spin_unlock(&xpfo->maplock);