From patchwork Thu Jan 10 21:09:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10756917 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E6678159A for ; Thu, 10 Jan 2019 21:11:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D43D329BF7 for ; Thu, 10 Jan 2019 21:11:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D0BD629C04; Thu, 10 Jan 2019 21:11:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2C1AE29BF7 for ; Thu, 10 Jan 2019 21:11:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D9E88E0010; Thu, 10 Jan 2019 16:11:01 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 261C58E0008; Thu, 10 Jan 2019 16:11:01 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1067E8E0010; Thu, 10 Jan 2019 16:11:01 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id BEEC28E0008 for ; Thu, 10 Jan 2019 16:11:00 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id u20so8688970pfa.1 for ; Thu, 10 Jan 2019 13:11:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=/IFEDzhdI0x1IZkLHUorzjqe46w6d+OUBpCZ00v5/kI=; b=Ae5bfRAotBgQrI5wDAE4nSJqBxUT4Ev4WF80J6rYvlbjpwIvjxFXT469BXLoppEUgk cmz2zwtNaXcfh8u6kcLDFUfnVI+lS8qRCCdujVmRrPKtYP4tkRQjEp5O0KWGHAcX2UB9 73ZAzilRSrxKeGKkavFx12AN6cRi55TDmQPlDT1Faepce+SGq9oH1MZVb6cuF9pF4rpO vKrcmX/5p5vY+dYQaE5TRDmJxw7FAAkk8ifVobGaSwaR6dvpSt6YVfl5YPbWpfjU4wS0 U6PYshcLlpqEOUtYOjUm7a+bJTmn5h9DXTVoVoc/qGrz6EyoxdQbvtVw2UylESqmFtq+ 1hSQ== X-Gm-Message-State: AJcUuke3M1mZJbHaHRfLycZ4pgrBxUL2WuZrN2Jj8hitBc4BF+mEo1yl 8mhdKmI3/R0iESaVL2X12zaW+am7fl9ig7QLvhhXyhXfClE7RfZyiJ/6ol55Q1PCxnWzvSth+zr DJ56ngl5RKnRhmKCoj+zQs90QR/bQEPywKUAn1TBKk4vPT6UaV31BxvkH6HxCTGSYjQ== X-Received: by 2002:a62:7f93:: with SMTP id a141mr11864489pfd.96.1547154660403; Thu, 10 Jan 2019 13:11:00 -0800 (PST) X-Google-Smtp-Source: ALg8bN6K6MpdatgfvBfd2+tbX/H9Lc0JTMsAIwsjIrkEM4MiZ3v74p6hSiGRAnxFtOVZ2Ago45LK X-Received: by 2002:a62:7f93:: with SMTP id a141mr11864459pfd.96.1547154659724; Thu, 10 Jan 2019 13:10:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547154659; cv=none; d=google.com; s=arc-20160816; b=iftzBbhk+bdLG3oyhYTtMFgDgrWblvKrszk2fi8FNIpRX7lFrCmzV/jiCFDPAFKU0+ SG69w/mqVZ8FJhYGQy3bCR4vISQf/8/lO+5H0bhCywEnOdYNLsqnMsK1ommZKrrDZtL9 Qn+Zk/tPEy4qi8O1dRObQyzHJIeV+cSeGUk7l6ZfcRQNyCUbZON4apap+UmVISEOoZNK RNR3SvPhZw+IdNn9kLszBj/Imcdk9AO9pgd5/t62XTsVCXupCtc/8i5Sr9s8yqsYWcg9 JKixuSx0ImHFU/KGlnNm4ZOcIgio4P21Msf/poZmF0QNyF4x65wUPSuXATOcUoKVoS9s +XEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=/IFEDzhdI0x1IZkLHUorzjqe46w6d+OUBpCZ00v5/kI=; b=PdeDKhRSKQnm1WS9qCKK4y5SWZFlY/Bc3bbNgHD4OZtVj/TCcUnDO1U6MXdOJUjSI6 +07ZsslAVT1+OuJ3ExKPtkmHB+gR6L1J9uTM+zpVsD1La/RwLzk2aF2Su8cByV2uvYAE HpdD99ItRjW8B5X+jTJU0AjjuY4UV5ixKizY4VY2RK9m6MdDutjBSrnoXw60vIASlUjf 9lwaC2FPRBQhbLWmrg1vvpOtpdfk08WiK0I2PhqUSD0+om/X+7K7Qy8YmHTRpR7XH1i/ 7+Qalwf7nmIZmPlf/MKuXsv3+jDUMlyAYBZo1iULcVv9OYA4vK1O1s9VUlvh/nB5zUQC vcvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=JNUIA0Zi; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id s123si7078247pfb.274.2019.01.10.13.10.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 10 Jan 2019 13:10:59 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=JNUIA0Zi; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id x0AL9qN7187636; Thu, 10 Jan 2019 21:10:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=/IFEDzhdI0x1IZkLHUorzjqe46w6d+OUBpCZ00v5/kI=; b=JNUIA0ZiLYSbjyCwAVK6/QyBmq8FL72dUceoNqR3YqbxV3KpfLrJ4/o4shwgV/QQkpbT i+JrwBLXO2mzCBJzNUIf2RhgKPu2jg5XHRABDtU+LXQqrUZ6aGpK2NvIydlwKaJppCdP j+wdmrhR7TDmOqvKgoKXVAWk8vUVI1gtKEbYePfC6zugyaJRhLJtLxHH13kyG9h5AdLz qJSxuJEXRAMb7+qsXPG8fmHl/+VvjCI0MtbqDIbPMADcIhXDIUSs7Ix3Av5qePZBrkqB 7GL2jlWfvCsYjC3WOY/uiO9bWQO8zEc6KoJZ/k+sNM/QF+DyHlfh3pKl8iF6Z2h4GVQH Fw== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2130.oracle.com with ESMTP id 2ptj3e9thg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 10 Jan 2019 21:10:35 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x0ALAZgD012185 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 10 Jan 2019 21:10:35 GMT Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x0ALAYF0023487; Thu, 10 Jan 2019 21:10:34 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 10 Jan 2019 13:10:34 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, "Vasileios P . Kemerlis" , Juerg Haefliger , Tycho Andersen , Marco Benatto , David Woodhouse , Khalid Aziz Subject: [RFC PATCH v7 11/16] mm, x86: omit TLB flushing by default for XPFO page table modifications Date: Thu, 10 Jan 2019 14:09:43 -0700 Message-Id: <4e51a5d4409b54116968b8c0501f6d82c4eb9cb5.1547153058.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9132 signatures=668680 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901100164 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Julian Stecklina XPFO carries a large performance overhead. In my tests, I saw >40% overhead for compiling a Linux kernel with XPFO enabled. The frequent TLB flushes that XPFO performs are the root cause of much of this overhead. TLB flushing is required for full paranoia mode where we don't want TLB entries of physmap pages to stick around potentially indefinitely. In reality, though, these TLB entries are going to be evicted pretty rapidly even without explicit flushing. That means omitting TLB flushes only marginally lowers the security benefits of XPFO. For kernel compile, omitting TLB flushes pushes the overhead below 3%. Change the default in XPFO to not flush TLBs unless the user explicitly requests to do so using a kernel parameter. Signed-off-by: Julian Stecklina Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis Cc: Juerg Haefliger Cc: Tycho Andersen Cc: Marco Benatto Cc: David Woodhouse Signed-off-by: Khalid Aziz --- mm/xpfo.c | 37 +++++++++++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/mm/xpfo.c b/mm/xpfo.c index 25fba05d01bd..e80374b0c78e 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -36,6 +36,7 @@ struct xpfo { }; DEFINE_STATIC_KEY_FALSE(xpfo_inited); +DEFINE_STATIC_KEY_FALSE(xpfo_do_tlb_flush); static bool xpfo_disabled __initdata; @@ -46,7 +47,15 @@ static int __init noxpfo_param(char *str) return 0; } +static int __init xpfotlbflush_param(char *str) +{ + static_branch_enable(&xpfo_do_tlb_flush); + + return 0; +} + early_param("noxpfo", noxpfo_param); +early_param("xpfotlbflush", xpfotlbflush_param); static bool __init need_xpfo(void) { @@ -76,6 +85,13 @@ bool __init xpfo_enabled(void) } EXPORT_SYMBOL(xpfo_enabled); + +static void xpfo_cond_flush_kernel_tlb(struct page *page, int order) +{ + if (static_branch_unlikely(&xpfo_do_tlb_flush)) + xpfo_flush_kernel_tlb(page, order); +} + static inline struct xpfo *lookup_xpfo(struct page *page) { struct page_ext *page_ext = lookup_page_ext(page); @@ -114,12 +130,17 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) "xpfo: already mapped page being allocated\n"); if ((gfp & GFP_HIGHUSER) == GFP_HIGHUSER) { - /* - * Tag the page as a user page and flush the TLB if it - * was previously allocated to the kernel. - */ - if (!test_and_set_bit(XPFO_PAGE_USER, &xpfo->flags)) - flush_tlb = 1; + if (static_branch_unlikely(&xpfo_do_tlb_flush)) { + /* + * Tag the page as a user page and flush the TLB if it + * was previously allocated to the kernel. + */ + if (!test_and_set_bit(XPFO_PAGE_USER, &xpfo->flags)) + flush_tlb = 1; + } else { + set_bit(XPFO_PAGE_USER, &xpfo->flags); + } + } else { /* Tag the page as a non-user (kernel) page */ clear_bit(XPFO_PAGE_USER, &xpfo->flags); @@ -127,7 +148,7 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) } if (flush_tlb) - xpfo_flush_kernel_tlb(page, order); + xpfo_cond_flush_kernel_tlb(page, order); } void xpfo_free_pages(struct page *page, int order) @@ -221,7 +242,7 @@ void xpfo_kunmap(void *kaddr, struct page *page) "xpfo: unmapping already unmapped page\n"); set_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); set_kpte(kaddr, page, __pgprot(0)); - xpfo_flush_kernel_tlb(page, 0); + xpfo_cond_flush_kernel_tlb(page, 0); } spin_unlock(&xpfo->maplock);