From patchwork Thu Feb 14 00:01:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811353 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E46F3746 for ; Thu, 14 Feb 2019 00:02:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C74AD29C32 for ; Thu, 14 Feb 2019 00:02:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA4702D9EE; Thu, 14 Feb 2019 00:02:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 221B829C32 for ; Thu, 14 Feb 2019 00:02:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDA948E0003; Wed, 13 Feb 2019 19:02:35 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E889F8E0001; Wed, 13 Feb 2019 19:02:35 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D52058E0003; Wed, 13 Feb 2019 19:02:35 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 9D3F98E0001 for ; Wed, 13 Feb 2019 19:02:35 -0500 (EST) Received: by mail-pf1-f199.google.com with SMTP id x134so3201695pfd.18 for ; Wed, 13 Feb 2019 16:02:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=BnvBthvqyUR+BETbLnaqcFAixVQhrRUG4KWFDYKw33s=; b=RWLD4rjjJEkrTR69bU1UH1OaMS0YmArk7kwqEX0xDoTjrgrtu538mXNvvjlZGmhml7 ptKHKGnH+Q4t9qGO44gSL+/BTl3txyONkhcQQZgshu0dxxY5gzrMTfTUkcFbE8O2N2L7 xfwJjQlaOfnozUa7LjCbfAF24fOJWlKxYdT6PWEt2VZKsY95mmYz0FdnG+Lw2vvn8yKK u7MDmX715W9Mbg0b2zyaIIl6lsT3qSpI7H7FSBoC16iH0BI2VLB5ikDSj+T2cxI4vnR3 5OIkjhs313w9XKCpG9XBOu/OQTT+/t5xhl3dZ5jP5fy6WoxQkvjWgvwm3izRU99QQSyp pWpw== X-Gm-Message-State: AHQUAuZmjZzkmrxcnvnF/EZZQjVECDbzoAv3Atngf9sl5cYhgqBsQLMN OIiPQ9Z//w27AstBbuH3Vs6Jo9Kuf1MfOCYg/ceIFN+eQLHFYjxB672gZmTH10YJwchwCEyy+7R RI8x2ElQihdXhjbQghdMPLW4vlG5z1O6dgElMLUJi4kh2At7qtTRp2ED66gLXgHVznQ== X-Received: by 2002:a17:902:9a4a:: with SMTP id x10mr875244plv.93.1550102555297; Wed, 13 Feb 2019 16:02:35 -0800 (PST) X-Google-Smtp-Source: AHgI3IbI0p7iYDv1Xk070oeANnIK5ob4IHR89AGxU08wgydCYHnXeJG6Socn2IIwGCw2Nn+QFE7i X-Received: by 2002:a17:902:9a4a:: with SMTP id x10mr875173plv.93.1550102554498; Wed, 13 Feb 2019 16:02:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102554; cv=none; d=google.com; s=arc-20160816; b=KL6r+hRQrF7FALivu1sgpgYuYJ8vQ38W1i27nB9llC8dwUSNddIFb/F3g7l5stZ/jz zLpoQLXVRjq0e1SA60xM//5A4f19HGqrPIUvJEW2Wv9Ut+wdE5I5JpQgQWvnMDjhbDx0 xePLAoMYbtM1lUmLLsKrtJHZ0E9QejZmau/eNhkLnGLOG7s3Y3r0yPWXMcR3kHlOm7ie RaTcSfQ3T+wlCi3fS5TEJBrAl/iCza1PvLP2MKkT/AXsZ5vSjGcBtNkdo4c8qEumScR0 N98VU0GKIGheUQF37ad4Z1xECazvfz66QT1qzz2IJvMPeUG8roXhhRg1KNuWbx2rLozD PCdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=BnvBthvqyUR+BETbLnaqcFAixVQhrRUG4KWFDYKw33s=; b=gFGEQAVquROZ6kbRSru918i/14YxxEHaoFRkVKe8dFuF9p5T3Tlv9d1FIIs7egxfJD utoBfFhrmcil2AWy2U9vXAgmkbMF8QMzX9OzrgvnX8O8N7KlVeuQewKz674U87TZ5/En 48CcMtel7I2dvoAqlO33BT2GwUVEBsq3Iinoon7ZVT4xoPCjJ8d1JF6mRoby36seyPej W30CioFgDYySCTbhnt3ZEcB/yJHardV/Xw0HePJe9MiCFiXYRr5wVxQ1KWAD2KvdgVH6 Tw4RgjOiroyxL5NHhNXcWC4dtqmiOfSUDbicrvMxccJRzwPrD6L1ZfrGSiXuCt6bg2kD 4CyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=mcr+aLBj; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id m10si696301plt.295.2019.02.13.16.02.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:34 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=mcr+aLBj; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwsZ2100568; Thu, 14 Feb 2019 00:01:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=BnvBthvqyUR+BETbLnaqcFAixVQhrRUG4KWFDYKw33s=; b=mcr+aLBjXAXSptqDtlz0VxxfWE1bq8ULxWcJo1ATX2NiCgH2AWLADjbr/HqYkpk9odGK ChvFnz1ZxMVxKCJ0jfXdJW5TYWVBwj+Ty3/i6pxWwCEiACxkvV13/+T2YmPTX1CBTwJG UBQhh1BtAVY8KyQ7Wk/NDQM5Umuvxs3VEJF2d3Xv4chhe+jqnXx8PoRI61KUxuMIHH1j hzaCrG4RkCU1W8fxes/FLLMZ7uQSL15sc1ZI1d0I09k3vAa0Q8LpfMdnyskkbMB7yMLc eVkiHdh8fE/ZM3EnxXIAPz7Bn0aTkW7mrmcxMY3krFlvZUEFc5Jyqgv8meeyiWbyIGX9 Lw== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2130.oracle.com with ESMTP id 2qhre5n3u2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:01:59 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1E01wVi031042 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:01:58 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1E01tie014301; Thu, 14 Feb 2019 00:01:56 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:01:55 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Tycho Andersen , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v8 01/14] mm: add MAP_HUGETLB support to vm_mmap Date: Wed, 13 Feb 2019 17:01:24 -0700 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen vm_mmap is exported, which means kernel modules can use it. In particular, for testing XPFO support, we want to use it with the MAP_HUGETLB flag, so let's support it via vm_mmap. Signed-off-by: Tycho Andersen Tested-by: Marco Benatto Tested-by: Khalid Aziz --- include/linux/mm.h | 2 ++ mm/mmap.c | 19 +------------------ mm/util.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 18 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5411de93a363..30bddc7b3c75 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2361,6 +2361,8 @@ struct vm_unmapped_area_info { extern unsigned long unmapped_area(struct vm_unmapped_area_info *info); extern unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info); +struct file *map_hugetlb_setup(unsigned long *len, unsigned long flags); + /* * Search for an unmapped address range. * diff --git a/mm/mmap.c b/mm/mmap.c index 6c04292e16a7..c668d7d27c2b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1582,24 +1582,7 @@ unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len, if (unlikely(flags & MAP_HUGETLB && !is_file_hugepages(file))) goto out_fput; } else if (flags & MAP_HUGETLB) { - struct user_struct *user = NULL; - struct hstate *hs; - - hs = hstate_sizelog((flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); - if (!hs) - return -EINVAL; - - len = ALIGN(len, huge_page_size(hs)); - /* - * VM_NORESERVE is used because the reservations will be - * taken when vm_ops->mmap() is called - * A dummy user value is used because we are not locking - * memory so no accounting is necessary - */ - file = hugetlb_file_setup(HUGETLB_ANON_FILE, len, - VM_NORESERVE, - &user, HUGETLB_ANONHUGE_INODE, - (flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); + file = map_hugetlb_setup(&len, flags); if (IS_ERR(file)) return PTR_ERR(file); } diff --git a/mm/util.c b/mm/util.c index 8bf08b5b5760..536c14cf88ba 100644 --- a/mm/util.c +++ b/mm/util.c @@ -357,6 +357,29 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, return ret; } +struct file *map_hugetlb_setup(unsigned long *len, unsigned long flags) +{ + struct user_struct *user = NULL; + struct hstate *hs; + + hs = hstate_sizelog((flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); + if (!hs) + return ERR_PTR(-EINVAL); + + *len = ALIGN(*len, huge_page_size(hs)); + + /* + * VM_NORESERVE is used because the reservations will be + * taken when vm_ops->mmap() is called + * A dummy user value is used because we are not locking + * memory so no accounting is necessary + */ + return hugetlb_file_setup(HUGETLB_ANON_FILE, *len, + VM_NORESERVE, + &user, HUGETLB_ANONHUGE_INODE, + (flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK); +} + unsigned long vm_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flag, unsigned long offset) @@ -366,6 +389,15 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, if (unlikely(offset_in_page(offset))) return -EINVAL; + if (flag & MAP_HUGETLB) { + if (file) + return -EINVAL; + + file = map_hugetlb_setup(&len, flag); + if (IS_ERR(file)) + return PTR_ERR(file); + } + return vm_mmap_pgoff(file, addr, len, prot, flag, offset >> PAGE_SHIFT); } EXPORT_SYMBOL(vm_mmap); From patchwork Thu Feb 14 00:01:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811367 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3E2E746 for ; Thu, 14 Feb 2019 00:02:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF63329C32 for ; Thu, 14 Feb 2019 00:02:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A1B912D9EE; Thu, 14 Feb 2019 00:02:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 184DE29C32 for ; Thu, 14 Feb 2019 00:02:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B41E8E0009; Wed, 13 Feb 2019 19:02:46 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7392B8E0005; Wed, 13 Feb 2019 19:02:46 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 629288E0009; Wed, 13 Feb 2019 19:02:46 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 1DF948E0005 for ; Wed, 13 Feb 2019 19:02:46 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id 74so3217165pfk.12 for ; Wed, 13 Feb 2019 16:02:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=BVvtye26KZIs7muho5hLhXsAoOpu90yJN7XQD4ZHF6s=; b=unEHy70hhzKL0tnS8cl2jjcwPtuZ8nAdi+2+ZQxA53/mnKfq1lf361f8JpFliknMVw q+VKHOKYrIWshMTwqyHCzfPNxGW7jzKk9m7+GD94qOuEAlkkDIZVH6/+q9ztRoYL08aT ZjGGgivrWFEYI+onyKAHYm63o5BbaWrvlYMyRg263rGy4ieSjwfsqTJFMLualxZnjrD3 M3b/ER4tJKblvurWXwNFyj25By9VTWIKtgJ+vFl8aGsb23FFxPsZHIrBL8y5eHa+2/Wh NPo8cxOrKlrAIxL3mx8USNOgnfbpOCD3r5iQRvEbBJRtNiGZmv1Z2k5Esei1n7GNkhgc vVZw== X-Gm-Message-State: AHQUAuaXo9R887iYeGHHAfORMAc1Uytn6esLr9GlW3Gm4WRmBVCPqtNd DBr2IapWCFkCoWf1R0wBTa0Rj9hu2gLWDgMVJj4+u/QGUSHT+82wgnQFzIJrml5TiGFoq9+Xh1E PXXW68HpBUF6/KhtgSZX9cFl/uQX9LwBjP8G55x/eljjXeo7pwLqnyqqIcLV8WElj0Q== X-Received: by 2002:a62:2008:: with SMTP id g8mr849384pfg.121.1550102565739; Wed, 13 Feb 2019 16:02:45 -0800 (PST) X-Google-Smtp-Source: AHgI3IY0bGWEapqYPOqvqYfEjLTTSMiFMS6pORptQwrf6Vs35u3/jWcy70Rn4C4mHEH5GV2GZuGr X-Received: by 2002:a62:2008:: with SMTP id g8mr849303pfg.121.1550102564870; Wed, 13 Feb 2019 16:02:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102564; cv=none; d=google.com; s=arc-20160816; b=p7lKYOUnOjUBlob+qF7F5Pbl5qNZRkTdLzFHxXoJf3mODWUVquaKn/fP5mi3uRZPXb muamW4NPn/LdXnsSEftrEBjFvC/8nG34ujQgxP4776qftX9McWajN250q36zUwEDJ3SJ rQkrEdIJmXladTEl3bu5ACXWu02O390P580NSRChtHuV8hYsU88FZ3Vdv+9jyGG1wAHQ KgAsrDNs/zG+ZvfkH6F4ZMaVoAbIk99vgJdV55NqYgvXb9cUX6Wgi6IBwNy+bA9S4eeM 7oNWfo/UXg1+vPPq/3l2yQmFrLKeDHYpH5n47TD21uoh4zLoybTdbyqWexF/vHkMmNs7 8RAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=BVvtye26KZIs7muho5hLhXsAoOpu90yJN7XQD4ZHF6s=; b=JUyXU/ysUsre/OtLP6vOm/hQPpAzMO6QXgiUt5VPBXl5sQEMOF/uF4YeoE6CO2yC95 WfPlMDDcB5RcI4qm0HhAc8epv2TJTSFYamCBEbbAqdKtdEgMQwSThSg8Et66/zkltCTN xOxFSJeh+2oATEv8n0V67Jv1alp7qfF/l7NqbqWvx1G4uIcbUJgnO5S1OWBv4NV/K5Ga ADaSreMhhSysWHwQMJdHqTH1Ds1i73EP8gEu3V6SwojLJzSPSlv5JHsXH6Z5J6HKlLFO ZODAmLb0+7xufsOiBei+9IDjQnMNQoLW2551Gj4eBZ1+Ljev9/nU0zZYCPZ+dOeGWd1c YBNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=fZ7BPquZ; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id i5si712705pgg.279.2019.02.13.16.02.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:44 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=fZ7BPquZ; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwlSC100511; Thu, 14 Feb 2019 00:02:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=BVvtye26KZIs7muho5hLhXsAoOpu90yJN7XQD4ZHF6s=; b=fZ7BPquZinNkNfviJmu2aj5OWxvfaEW6ze9/DO2c6uAP65Cc4e4Cl3T1X2A+uS3/Cnnj FXw2tAJbgFiECJbQ4WvubRBn8pxAj4qhxLwEmJiaDQgis7w2FmPwW2j42M7DueNqyBGy EyJX2fW7sQZHaipQdaHrDl4Ch27fK6OjK5xHCrNaGpwnUD2AjFaKdF5KlkO1RjMlgz7E wO+hNUWeMh8kKPGfqr2rZVzyWdP8lbCxRvKmyeYtHEahIc8wLWsnl9jqQNQFXSxMWNlu 3RxQ4F/6tOJWAzRs4vywbeqWiVdLGC3PRvx47Ai+oOoMdEY1LwrLa/ajJI34d9cE7gWR uA== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2130.oracle.com with ESMTP id 2qhre5n3u7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:02 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E021Ic025030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:02 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E01w4G032279; Thu, 14 Feb 2019 00:02:00 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:01:57 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Tycho Andersen , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v8 02/14] x86: always set IF before oopsing from page fault Date: Wed, 13 Feb 2019 17:01:25 -0700 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=885 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen Oopsing might kill the task, via rewind_stack_do_exit() at the bottom, and that might sleep: Aug 23 19:30:27 xpfo kernel: [ 38.302714] BUG: sleeping function called from invalid context at ./include/linux/percpu-rwsem.h:33 Aug 23 19:30:27 xpfo kernel: [ 38.303837] in_atomic(): 0, irqs_disabled(): 1, pid: 1970, name: lkdtm_xpfo_test Aug 23 19:30:27 xpfo kernel: [ 38.304758] CPU: 3 PID: 1970 Comm: lkdtm_xpfo_test Tainted: G D 4.13.0-rc5+ #228 Aug 23 19:30:27 xpfo kernel: [ 38.305813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-1ubuntu1 04/01/2014 Aug 23 19:30:27 xpfo kernel: [ 38.306926] Call Trace: Aug 23 19:30:27 xpfo kernel: [ 38.307243] dump_stack+0x63/0x8b Aug 23 19:30:27 xpfo kernel: [ 38.307665] ___might_sleep+0xec/0x110 Aug 23 19:30:27 xpfo kernel: [ 38.308139] __might_sleep+0x45/0x80 Aug 23 19:30:27 xpfo kernel: [ 38.308593] exit_signals+0x21/0x1c0 Aug 23 19:30:27 xpfo kernel: [ 38.309046] ? blocking_notifier_call_chain+0x11/0x20 Aug 23 19:30:27 xpfo kernel: [ 38.309677] do_exit+0x98/0xbf0 Aug 23 19:30:27 xpfo kernel: [ 38.310078] ? smp_reader+0x27/0x40 [lkdtm] Aug 23 19:30:27 xpfo kernel: [ 38.310604] ? kthread+0x10f/0x150 Aug 23 19:30:27 xpfo kernel: [ 38.311045] ? read_user_with_flags+0x60/0x60 [lkdtm] Aug 23 19:30:27 xpfo kernel: [ 38.311680] rewind_stack_do_exit+0x17/0x20 To be safe, let's just always enable irqs. The particular case I'm hitting is: Aug 23 19:30:27 xpfo kernel: [ 38.278615] __bad_area_nosemaphore+0x1a9/0x1d0 Aug 23 19:30:27 xpfo kernel: [ 38.278617] bad_area_nosemaphore+0xf/0x20 Aug 23 19:30:27 xpfo kernel: [ 38.278618] __do_page_fault+0xd1/0x540 Aug 23 19:30:27 xpfo kernel: [ 38.278620] ? irq_work_queue+0x9b/0xb0 Aug 23 19:30:27 xpfo kernel: [ 38.278623] ? wake_up_klogd+0x36/0x40 Aug 23 19:30:27 xpfo kernel: [ 38.278624] trace_do_page_fault+0x3c/0xf0 Aug 23 19:30:27 xpfo kernel: [ 38.278625] do_async_page_fault+0x14/0x60 Aug 23 19:30:27 xpfo kernel: [ 38.278627] async_page_fault+0x28/0x30 When a fault is in kernel space which has been triggered by XPFO. Signed-off-by: Tycho Andersen CC: x86@kernel.org Tested-by: Khalid Aziz --- arch/x86/mm/fault.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 71d4b9d4d43f..ba51652fbd33 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -748,6 +748,12 @@ no_context(struct pt_regs *regs, unsigned long error_code, /* Executive summary in case the body of the oops scrolled away */ printk(KERN_DEFAULT "CR2: %016lx\n", address); + /* + * We're about to oops, which might kill the task. Make sure we're + * allowed to sleep. + */ + flags |= X86_EFLAGS_IF; + oops_end(flags, regs, sig); } From patchwork Thu Feb 14 00:01:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811359 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F4D4746 for ; Thu, 14 Feb 2019 00:02:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED52029C32 for ; Thu, 14 Feb 2019 00:02:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E001A2D9EE; Thu, 14 Feb 2019 00:02:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5EEEA29C32 for ; Thu, 14 Feb 2019 00:02:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62C5A8E0007; Wed, 13 Feb 2019 19:02:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 518088E0005; Wed, 13 Feb 2019 19:02:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 341EF8E0006; Wed, 13 Feb 2019 19:02:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id D6DA78E0004 for ; Wed, 13 Feb 2019 19:02:39 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id y12so1764668pll.15 for ; Wed, 13 Feb 2019 16:02:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=tGIuo+6XNLWxTeQwFkrLNJKDwNp+81usp+K2xaNMZbQ=; b=t4BTTMz0R7iFxHiVXjzOjq//DGTNytO61LmM2mdWEGknnFQAytzQ8pCG1AYmd/0/uq y3/x2Y+Z1Z9G5AcdMu5SINZZ7dq2r3XX0iQ2C60PuMGsE13/tvk8MoQrizqlIPe5YDY6 dtC+dMB//GT0v8asnOTEJJqta74u929m2ZRyKCLOo2i1sF1UWd0p08RpBHvSgcbU8XAv 9FSWrn7mrkKplFsIDOikjpbYViy3nZPuqiH2/lLAf8kLFoq6F8/3/ELqZzpynwg4wiZa aCy/HHVhs+O2AgwI0k8TSujyIPaAZDDi5MaVfeE3JOwBQLo/7HDmFlveb1JL0OeAS4zf DOfg== X-Gm-Message-State: AHQUAuZ0e3vPcOLos0yA3AmOjfYPPP/QIymow+rUsabBYLiSr46OTdlV S6y/HEVSnjAAKPWQDC9HltarNDuO2dkzGwbEqx06dKdfYnF0Hy227JtTh9mH0AyMJl5hkz6UWrQ vj/2uwxJ9N4IF6ahrnda3DDdGrydslgOqyVjdOklXZRcWzLuHKfXxfkMnm2tBsh2sIg== X-Received: by 2002:a63:4b60:: with SMTP id k32mr845414pgl.186.1550102559411; Wed, 13 Feb 2019 16:02:39 -0800 (PST) X-Google-Smtp-Source: AHgI3IbdUvmKsFyYVZ1P4KkXwBoyGe6x8XD6uQfKRc3wNnRfy488LkeKkjIS5cNwQZm9NsEcTlYd X-Received: by 2002:a63:4b60:: with SMTP id k32mr845257pgl.186.1550102557698; Wed, 13 Feb 2019 16:02:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102557; cv=none; d=google.com; s=arc-20160816; b=TNHKQUQ34RIQcl1vChvcLlo4Zy9ERQPDBqTMHdZWUBXe4wotmF0MXnAbTmuXME+9m1 JdmXoKJ4gxo0K6a9/DzfULTjE3hQUW+CGg4TiCt+SpozPF1QFSm2agFog3wzSg0Bkkn8 pxszS13UkmXCCiA0h9hh7ci4G5kTBoDfthlDqf/wayedYyb9/aLm/CAZGfRvBfn2PyEz bFND/1qEOYl5xX8dMX+JY2xyPV05Hs24VKniGvmASgjy17y005/vCZjbnbrPz2egyfSa CH93i/5DFb3mf3gUo13pZThjQ13xkt+CxmyMBqxN7Dq5vMI3g+hhQxYfUhXr47MERxeF +mfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=tGIuo+6XNLWxTeQwFkrLNJKDwNp+81usp+K2xaNMZbQ=; b=qhLrPPvGR8vPmI/2PVP+WsFlEPBbwBTfTdOO2yiIe0WnD1xPICMcnc34bhxAubkt8p XAtm7Z80g3sTAaC6HJNAMMZed2z9cWj76dZyv4ujkaxJt50Wy4hwuFSC+vfwT4CE8ANF f/3mr829qugStg81x+wNiSPIbGYwdvWbMC6ujrkVAhDeDrFPK5JREzi1l4xhiqc8UJr7 xTXNT8okyBH9okKFzofi9mo3sQYVjMn2f7kB/kZAiRqtaW5HR9NYOfYE9IXTVWhlujke TleeLKi0hhYnCt1V3j94mkvXLiL3zimvxZFoK1tZEwVPALd1glplxD9VyAkm19476cTp FaPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="EV0/Kezf"; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id a68si772715pla.267.2019.02.13.16.02.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:37 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="EV0/Kezf"; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwnGH100531; Thu, 14 Feb 2019 00:02:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=tGIuo+6XNLWxTeQwFkrLNJKDwNp+81usp+K2xaNMZbQ=; b=EV0/KezfENVHmr/AMNMxy2Nd61uEG5+pcnOu5DPfYMTva1/IdPx+bbb8siANlxD0fdgY sZukNc6EGxpttdwgr8JPyw6hQH4zlHCuBvnmuyXP71eeBk2S8t7slIaSOrF+P6M5wC5y 4Jv2X3fB+dQq9pFM9aC2bB4beiAYPs0WbvJHQz9UfOi+aD489bz7N77ZxE4TQOG/CrX2 05lcdk23um8iMP0q5FcGMFpG+98u25VFNMJtTzpAfxlaCt8/lCpSyfYuK/e5sd3YmYYB EWzKyo9eiVRG1uOsHGyawgrR7H9TeOXmHDB7Hd3VozbXXXOXFmJTQwVd1e+JN7Gs6ldm 1A== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2qhre5n3ud-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:07 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1E0224s007646 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:02 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E022w3018542; Thu, 14 Feb 2019 00:02:02 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:01 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Tycho Andersen , Marco Benatto Subject: [RFC PATCH v8 03/14] mm, x86: Add support for eXclusive Page Frame Ownership (XPFO) Date: Wed, 13 Feb 2019 17:01:26 -0700 Message-Id: <8275de2a7e6b72d19b1cd2ec5d71a42c2c7dd6c5.1550088114.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger This patch adds support for XPFO which protects against 'ret2dir' kernel attacks. The basic idea is to enforce exclusive ownership of page frames by either the kernel or userspace, unless explicitly requested by the kernel. Whenever a page destined for userspace is allocated, it is unmapped from physmap (the kernel's page table). When such a page is reclaimed from userspace, it is mapped back to physmap. Additional fields in the page_ext struct are used for XPFO housekeeping, specifically: - two flags to distinguish user vs. kernel pages and to tag unmapped pages. - a reference counter to balance kmap/kunmap operations. - a lock to serialize access to the XPFO fields. This patch is based on the work of Vasileios P. Kemerlis et al. who published their work in this paper: http://www.cs.columbia.edu/~vpk/papers/ret2dir.sec14.pdf v6: * use flush_tlb_kernel_range() instead of __flush_tlb_one, so we flush the tlb entry on all CPUs when unmapping it in kunmap * handle lookup_page_ext()/lookup_xpfo() returning NULL * drop lots of BUG()s in favor of WARN() * don't disable irqs in xpfo_kmap/xpfo_kunmap, export __split_large_page so we can do our own alloc_pages(GFP_ATOMIC) to pass it CC: x86@kernel.org Suggested-by: Vasileios P. Kemerlis Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Signed-off-by: Marco Benatto [jsteckli@amazon.de: rebased from v4.13 to v4.19] Signed-off-by: Julian Stecklina Reviewed-by: Khalid Aziz --- .../admin-guide/kernel-parameters.txt | 2 + arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 26 ++ arch/x86/mm/Makefile | 2 + arch/x86/mm/pageattr.c | 23 +- arch/x86/mm/xpfo.c | 119 ++++++++++ include/linux/highmem.h | 15 +- include/linux/xpfo.h | 48 ++++ mm/Makefile | 1 + mm/page_alloc.c | 2 + mm/page_ext.c | 4 + mm/xpfo.c | 223 ++++++++++++++++++ security/Kconfig | 19 ++ 13 files changed, 463 insertions(+), 22 deletions(-) create mode 100644 arch/x86/mm/xpfo.c create mode 100644 include/linux/xpfo.h create mode 100644 mm/xpfo.c diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index aefd358a5ca3..c4c62599f216 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2982,6 +2982,8 @@ nox2apic [X86-64,APIC] Do not enable x2APIC mode. + noxpfo [X86-64] Disable XPFO when CONFIG_XPFO is on. + cpu0_hotplug [X86] Turn on CPU0 hotplug feature when CONFIG_BOOTPARAM_HOTPLUG_CPU0 is off. Some features depend on CPU0. Known dependencies are: diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 8689e794a43c..d69d8cc6e57e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -207,6 +207,7 @@ config X86 select USER_STACKTRACE_SUPPORT select VIRT_TO_BUS select X86_FEATURE_NAMES if PROC_FS + select ARCH_SUPPORTS_XPFO if X86_64 config INSTRUCTION_DECODER def_bool y diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 40616e805292..f6eeb75c8a21 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1437,6 +1437,32 @@ static inline bool arch_has_pfn_modify_check(void) return boot_cpu_has_bug(X86_BUG_L1TF); } +/* + * The current flushing context - we pass it instead of 5 arguments: + */ +struct cpa_data { + unsigned long *vaddr; + pgd_t *pgd; + pgprot_t mask_set; + pgprot_t mask_clr; + unsigned long numpages; + int flags; + unsigned long pfn; + unsigned force_split : 1, + force_static_prot : 1; + int curpage; + struct page **pages; +}; + + +int +should_split_large_page(pte_t *kpte, unsigned long address, + struct cpa_data *cpa); +extern spinlock_t cpa_lock; +int +__split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, + struct page *base); + #include #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 4b101dd6e52f..93b0fdaf4a99 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -53,3 +53,5 @@ obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o + +obj-$(CONFIG_XPFO) += xpfo.o diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index a1bcde35db4c..84002442ab61 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -26,23 +26,6 @@ #include #include -/* - * The current flushing context - we pass it instead of 5 arguments: - */ -struct cpa_data { - unsigned long *vaddr; - pgd_t *pgd; - pgprot_t mask_set; - pgprot_t mask_clr; - unsigned long numpages; - int flags; - unsigned long pfn; - unsigned force_split : 1, - force_static_prot : 1; - int curpage; - struct page **pages; -}; - enum cpa_warn { CPA_CONFLICT, CPA_PROTECT, @@ -57,7 +40,7 @@ static const int cpa_warn_level = CPA_PROTECT; * entries change the page attribute in parallel to some other cpu * splitting a large page entry along with changing the attribute. */ -static DEFINE_SPINLOCK(cpa_lock); +DEFINE_SPINLOCK(cpa_lock); #define CPA_FLUSHTLB 1 #define CPA_ARRAY 2 @@ -869,7 +852,7 @@ static int __should_split_large_page(pte_t *kpte, unsigned long address, return 0; } -static int should_split_large_page(pte_t *kpte, unsigned long address, +int should_split_large_page(pte_t *kpte, unsigned long address, struct cpa_data *cpa) { int do_split; @@ -919,7 +902,7 @@ static void split_set_pte(struct cpa_data *cpa, pte_t *pte, unsigned long pfn, set_pte(pte, pfn_pte(pfn, ref_prot)); } -static int +int __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, struct page *base) { diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c new file mode 100644 index 000000000000..6c7502993351 --- /dev/null +++ b/arch/x86/mm/xpfo.c @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#include + +#include + +extern spinlock_t cpa_lock; + +/* Update a single kernel page table entry */ +inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) +{ + unsigned int level; + pgprot_t msk_clr; + pte_t *pte = lookup_address((unsigned long)kaddr, &level); + + if (unlikely(!pte)) { + WARN(1, "xpfo: invalid address %p\n", kaddr); + return; + } + + switch (level) { + case PG_LEVEL_4K: + set_pte_atomic(pte, pfn_pte(page_to_pfn(page), + canon_pgprot(prot))); + break; + case PG_LEVEL_2M: + case PG_LEVEL_1G: { + struct cpa_data cpa = { }; + int do_split; + + if (level == PG_LEVEL_2M) + msk_clr = pmd_pgprot(*(pmd_t *)pte); + else + msk_clr = pud_pgprot(*(pud_t *)pte); + + cpa.vaddr = kaddr; + cpa.pages = &page; + cpa.mask_set = prot; + cpa.mask_clr = msk_clr; + cpa.numpages = 1; + cpa.flags = 0; + cpa.curpage = 0; + cpa.force_split = 0; + + + do_split = should_split_large_page(pte, (unsigned long)kaddr, + &cpa); + if (do_split) { + struct page *base; + + base = alloc_pages(GFP_ATOMIC, 0); + if (!base) { + WARN(1, "xpfo: failed to split large page\n"); + break; + } + + if (!debug_pagealloc_enabled()) + spin_lock(&cpa_lock); + if (__split_large_page(&cpa, pte, + (unsigned long)kaddr, base) < 0) + WARN(1, "xpfo: failed to split large page\n"); + if (!debug_pagealloc_enabled()) + spin_unlock(&cpa_lock); + } + + break; + } + case PG_LEVEL_512G: + /* fallthrough, splitting infrastructure doesn't + * support 512G pages. + */ + default: + WARN(1, "xpfo: unsupported page level %x\n", level); + } + +} + +inline void xpfo_flush_kernel_tlb(struct page *page, int order) +{ + int level; + unsigned long size, kaddr; + + kaddr = (unsigned long)page_address(page); + + if (unlikely(!lookup_address(kaddr, &level))) { + WARN(1, "xpfo: invalid address to flush %lx %d\n", kaddr, + level); + return; + } + + switch (level) { + case PG_LEVEL_4K: + size = PAGE_SIZE; + break; + case PG_LEVEL_2M: + size = PMD_SIZE; + break; + case PG_LEVEL_1G: + size = PUD_SIZE; + break; + default: + WARN(1, "xpfo: unsupported page level %x\n", level); + return; + } + + flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); +} diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 0690679832d4..1fdae929e38b 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -8,6 +8,7 @@ #include #include #include +#include #include @@ -56,24 +57,34 @@ static inline struct page *kmap_to_page(void *addr) #ifndef ARCH_HAS_KMAP static inline void *kmap(struct page *page) { + void *kaddr; + might_sleep(); - return page_address(page); + kaddr = page_address(page); + xpfo_kmap(kaddr, page); + return kaddr; } static inline void kunmap(struct page *page) { + xpfo_kunmap(page_address(page), page); } static inline void *kmap_atomic(struct page *page) { + void *kaddr; + preempt_disable(); pagefault_disable(); - return page_address(page); + kaddr = page_address(page); + xpfo_kmap(kaddr, page); + return kaddr; } #define kmap_atomic_prot(page, prot) kmap_atomic(page) static inline void __kunmap_atomic(void *addr) { + xpfo_kunmap(addr, virt_to_page(addr)); pagefault_enable(); preempt_enable(); } diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h new file mode 100644 index 000000000000..b15234745fb4 --- /dev/null +++ b/include/linux/xpfo.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2017 Docker, Inc. + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * Tycho Andersen + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#ifndef _LINUX_XPFO_H +#define _LINUX_XPFO_H + +#include +#include + +struct page; + +#ifdef CONFIG_XPFO + +extern struct page_ext_operations page_xpfo_ops; + +void set_kpte(void *kaddr, struct page *page, pgprot_t prot); +void xpfo_dma_map_unmap_area(bool map, const void *addr, size_t size, + enum dma_data_direction dir); +void xpfo_flush_kernel_tlb(struct page *page, int order); + +void xpfo_kmap(void *kaddr, struct page *page); +void xpfo_kunmap(void *kaddr, struct page *page); +void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp); +void xpfo_free_pages(struct page *page, int order); + +#else /* !CONFIG_XPFO */ + +static inline void xpfo_kmap(void *kaddr, struct page *page) { } +static inline void xpfo_kunmap(void *kaddr, struct page *page) { } +static inline void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) { } +static inline void xpfo_free_pages(struct page *page, int order) { } + +#endif /* CONFIG_XPFO */ + +#endif /* _LINUX_XPFO_H */ diff --git a/mm/Makefile b/mm/Makefile index d210cc9d6f80..e99e1e6ae5ae 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -99,3 +99,4 @@ obj-$(CONFIG_HARDENED_USERCOPY) += usercopy.o obj-$(CONFIG_PERCPU_STATS) += percpu-stats.o obj-$(CONFIG_HMM) += hmm.o obj-$(CONFIG_MEMFD_CREATE) += memfd.o +obj-$(CONFIG_XPFO) += xpfo.o diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e95b5b7c9c3d..08e277790b5f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1038,6 +1038,7 @@ static __always_inline bool free_pages_prepare(struct page *page, kernel_poison_pages(page, 1 << order, 0); kernel_map_pages(page, 1 << order, 0); kasan_free_pages(page, order); + xpfo_free_pages(page, order); return true; } @@ -1915,6 +1916,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, kernel_map_pages(page, 1 << order, 1); kernel_poison_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); + xpfo_alloc_pages(page, order, gfp_flags); set_page_owner(page, order, gfp_flags); } diff --git a/mm/page_ext.c b/mm/page_ext.c index ae44f7adbe07..38e5013dcb9a 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -8,6 +8,7 @@ #include #include #include +#include /* * struct page extension @@ -68,6 +69,9 @@ static struct page_ext_operations *page_ext_ops[] = { #if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_XPFO + &page_xpfo_ops, +#endif }; static unsigned long total_usage; diff --git a/mm/xpfo.c b/mm/xpfo.c new file mode 100644 index 000000000000..24b33d3c20cb --- /dev/null +++ b/mm/xpfo.c @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2017 Docker, Inc. + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * Tycho Andersen + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#include +#include +#include +#include + +#include + +/* XPFO page state flags */ +enum xpfo_flags { + XPFO_PAGE_USER, /* Page is allocated to user-space */ + XPFO_PAGE_UNMAPPED, /* Page is unmapped from the linear map */ +}; + +/* Per-page XPFO house-keeping data */ +struct xpfo { + unsigned long flags; /* Page state */ + bool inited; /* Map counter and lock initialized */ + atomic_t mapcount; /* Counter for balancing map/unmap requests */ + spinlock_t maplock; /* Lock to serialize map/unmap requests */ +}; + +DEFINE_STATIC_KEY_FALSE(xpfo_inited); + +static bool xpfo_disabled __initdata; + +static int __init noxpfo_param(char *str) +{ + xpfo_disabled = true; + + return 0; +} + +early_param("noxpfo", noxpfo_param); + +static bool __init need_xpfo(void) +{ + if (xpfo_disabled) { + pr_info("XPFO disabled\n"); + return false; + } + + return true; +} + +static void init_xpfo(void) +{ + pr_info("XPFO enabled\n"); + static_branch_enable(&xpfo_inited); +} + +struct page_ext_operations page_xpfo_ops = { + .size = sizeof(struct xpfo), + .need = need_xpfo, + .init = init_xpfo, +}; + +static inline struct xpfo *lookup_xpfo(struct page *page) +{ + struct page_ext *page_ext = lookup_page_ext(page); + + if (unlikely(!page_ext)) { + WARN(1, "xpfo: failed to get page ext"); + return NULL; + } + + return (void *)page_ext + page_xpfo_ops.offset; +} + +void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) +{ + int i, flush_tlb = 0; + struct xpfo *xpfo; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + for (i = 0; i < (1 << order); i++) { + xpfo = lookup_xpfo(page + i); + if (!xpfo) + continue; + + WARN(test_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags), + "xpfo: unmapped page being allocated\n"); + + /* Initialize the map lock and map counter */ + if (unlikely(!xpfo->inited)) { + spin_lock_init(&xpfo->maplock); + atomic_set(&xpfo->mapcount, 0); + xpfo->inited = true; + } + WARN(atomic_read(&xpfo->mapcount), + "xpfo: already mapped page being allocated\n"); + + if ((gfp & GFP_HIGHUSER) == GFP_HIGHUSER) { + /* + * Tag the page as a user page and flush the TLB if it + * was previously allocated to the kernel. + */ + if (!test_and_set_bit(XPFO_PAGE_USER, &xpfo->flags)) + flush_tlb = 1; + } else { + /* Tag the page as a non-user (kernel) page */ + clear_bit(XPFO_PAGE_USER, &xpfo->flags); + } + } + + if (flush_tlb) + xpfo_flush_kernel_tlb(page, order); +} + +void xpfo_free_pages(struct page *page, int order) +{ + int i; + struct xpfo *xpfo; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + for (i = 0; i < (1 << order); i++) { + xpfo = lookup_xpfo(page + i); + if (!xpfo || unlikely(!xpfo->inited)) { + /* + * The page was allocated before page_ext was + * initialized, so it is a kernel page. + */ + continue; + } + + /* + * Map the page back into the kernel if it was previously + * allocated to user space. + */ + if (test_and_clear_bit(XPFO_PAGE_USER, &xpfo->flags)) { + clear_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); + set_kpte(page_address(page + i), page + i, + PAGE_KERNEL); + } + } +} + +void xpfo_kmap(void *kaddr, struct page *page) +{ + struct xpfo *xpfo; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + xpfo = lookup_xpfo(page); + + /* + * The page was allocated before page_ext was initialized (which means + * it's a kernel page) or it's allocated to the kernel, so nothing to + * do. + */ + if (!xpfo || unlikely(!xpfo->inited) || + !test_bit(XPFO_PAGE_USER, &xpfo->flags)) + return; + + spin_lock(&xpfo->maplock); + + /* + * The page was previously allocated to user space, so map it back + * into the kernel. No TLB flush required. + */ + if ((atomic_inc_return(&xpfo->mapcount) == 1) && + test_and_clear_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags)) + set_kpte(kaddr, page, PAGE_KERNEL); + + spin_unlock(&xpfo->maplock); +} +EXPORT_SYMBOL(xpfo_kmap); + +void xpfo_kunmap(void *kaddr, struct page *page) +{ + struct xpfo *xpfo; + + if (!static_branch_unlikely(&xpfo_inited)) + return; + + xpfo = lookup_xpfo(page); + + /* + * The page was allocated before page_ext was initialized (which means + * it's a kernel page) or it's allocated to the kernel, so nothing to + * do. + */ + if (!xpfo || unlikely(!xpfo->inited) || + !test_bit(XPFO_PAGE_USER, &xpfo->flags)) + return; + + spin_lock(&xpfo->maplock); + + /* + * The page is to be allocated back to user space, so unmap it from the + * kernel, flush the TLB and tag it as a user page. + */ + if (atomic_dec_return(&xpfo->mapcount) == 0) { + WARN(test_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags), + "xpfo: unmapping already unmapped page\n"); + set_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); + set_kpte(kaddr, page, __pgprot(0)); + xpfo_flush_kernel_tlb(page, 0); + } + + spin_unlock(&xpfo->maplock); +} +EXPORT_SYMBOL(xpfo_kunmap); diff --git a/security/Kconfig b/security/Kconfig index d9aa521b5206..8d0e4e303551 100644 --- a/security/Kconfig +++ b/security/Kconfig @@ -6,6 +6,25 @@ menu "Security options" source security/keys/Kconfig +config ARCH_SUPPORTS_XPFO + bool + +config XPFO + bool "Enable eXclusive Page Frame Ownership (XPFO)" + default n + depends on ARCH_SUPPORTS_XPFO + select PAGE_EXTENSION + help + This option offers protection against 'ret2dir' kernel attacks. + When enabled, every time a page frame is allocated to user space, it + is unmapped from the direct mapped RAM region in kernel space + (physmap). Similarly, when a page frame is freed/reclaimed, it is + mapped back to physmap. + + There is a slight performance impact when this option is enabled. + + If in doubt, say "N". + config SECURITY_DMESG_RESTRICT bool "Restrict unprivileged access to the kernel syslog" default n From patchwork Thu Feb 14 00:01:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811373 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5C2C8746 for ; Thu, 14 Feb 2019 00:03:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D4B629C32 for ; Thu, 14 Feb 2019 00:03:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A1E92D9EE; Thu, 14 Feb 2019 00:03:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97BF029C32 for ; Thu, 14 Feb 2019 00:03:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B5CF8E000A; Wed, 13 Feb 2019 19:02:55 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3917E8E0005; Wed, 13 Feb 2019 19:02:55 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 231548E000A; Wed, 13 Feb 2019 19:02:55 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id D41DC8E0005 for ; Wed, 13 Feb 2019 19:02:54 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id q62so2892852pgq.9 for ; Wed, 13 Feb 2019 16:02:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=MZqi6ournax3KQmG8JyBwhw5kF/1WOYT8qIrSyRL3+o=; b=hAxY21JeJ9n5xrmvqR5m2MVWxkb3GgM82X5C35G/s5WXVPmTnA2XK25F7y+UPlZLZm Lv8TRHJ05YWvCn6qBipVWWEPift5Ual0d+KgBuBLXseCw2w//lYINl7tDuOWfr3EY+vL 1ZCkbBpavNa1gZpiWfDiFfhoDArexQogXS1eYO83iXw8VG+VZl9ptMny642B/Waa2jZL Xp/4nsF1J0yAcdVeB/LqfcBZCfUlxLcSHCD3/XhCHSjneOLgvnXctEZsZw7/7KtB/hbW Prt5JdqLWjj7cqgwKjqAEZed0ugZkpC/XqBmtiZPZ36wf9mL5s0e8tyAoqND8Td+7DFJ 2V+Q== X-Gm-Message-State: AHQUAubisjRO2ip1ItF3B/cJW3upWou4cr7MqkUJVk8ghj3s0cVP/Yz5 W76T8t1Ucp987LUh7HwSuODv5R2g0++lOcbYgO9MnwaH1P/AZNnoRygm477q0kaDO6dO9mpOLf+ vkI/zAwxQGcolflOU0YkAJyGLU6BoMED5HkcQLwsn1KCQZuCjHnHvjSOJNIXte/Z3Rg== X-Received: by 2002:a63:4913:: with SMTP id w19mr830786pga.394.1550102574460; Wed, 13 Feb 2019 16:02:54 -0800 (PST) X-Google-Smtp-Source: AHgI3IZumfSNdSEuP6jo3DsbeF8AjN2J5keDEGTGt/0vvQU5ZzM0meawRRDmvL6BU6L54rHitR7O X-Received: by 2002:a63:4913:: with SMTP id w19mr830729pga.394.1550102573757; Wed, 13 Feb 2019 16:02:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102573; cv=none; d=google.com; s=arc-20160816; b=UZd/ufIeJrD8qVtbMhvEVgsdnkJBLzYnWhyryzCzRoKNMHgqGNnBc3jx/lqHpsoe5R AGgVEJ31baoWvTOZY9frdYueWhzsYD3x1oz8oJYyK+rkDLcMce++7Gx2Ogznr/dIlLii QcRgGza4stlbeo/57t534no/EYd2LpqfUBjh2q3nvJ3DWrAqN+DMaaHkPAPbBtgFt6+D Sc9nxHbPmU8N2B9T0FNgbV0UZPO4FmYytpL5m8TXXL16aVAtrjCipJaMlf+KaHCZBZC2 i/7uXE67wsk5NXcrpC7yU96TF1H/eyxsveUHXQeNXqUwwtPfQuR2lhC3K0Jf3xxIuZKd 8MBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=MZqi6ournax3KQmG8JyBwhw5kF/1WOYT8qIrSyRL3+o=; b=OJDiabx/7mpQIfqeOAKe4u1m+rYgzTWK5x6bHE0KTwmCIRwWx+RfV8JbDH1lV3p34y 1WE+LBtq3T/zKd753cqQCbasvuT91JTZHXy76lCCqqyf06KAFrLhI+8t0EvrbGsEo5Jv E7d8gQBij2vPKWzLoFxYt3s/upf4jIwnRvHnBnwm1ZCFMzSWMaBDAFzw6R3P2kpb7Z57 Yxv2EiJbIYlhdCsRgVZG8M18DKQoNVm8ZtB35Eoxoi5KcK4qZf0GDBZK4I3T68edG1qw RRd/A6E8KUVWu/0PzyxtUg3RG2UlJGnMhzbNjRlzIHMTvMBXHPpQBw7qLPDWr2z6yF2R QdiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=5c4qlL79; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id c6si720708plo.270.2019.02.13.16.02.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:53 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=5c4qlL79; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwfsg100210; Thu, 14 Feb 2019 00:02:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=MZqi6ournax3KQmG8JyBwhw5kF/1WOYT8qIrSyRL3+o=; b=5c4qlL7951pylpPk/LYjHCqBaQh+OE07SNEncySbv9VpXjItDk191jRy8R2qAS/Si7wk 5azy3SAjL1y5ZU+ayiqvpGeYyYPej1QFoowA8i7hY6LnuqWwGRdVeXP9R8NT7EnzXbG6 VmTdJcDS8tsN+qApiiulm6FgWZxe8R1C9yy4gLDgnBMg2ET8uYg5u30QJc9jnecDxXLZ Ws9u2vhvaPb5ONbVL/qJNVAlo/vUy0Ks8iSwt4n7n8NmKRBZtbPL12yeMBe53Z6nM1Be It7bqaU0WEZ+yaRGwGM5y3xEn/ALsicHwZUxBoSgDXDN78jBld75C6noZszpY4wmUtbI 7A== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2130.oracle.com with ESMTP id 2qhre5n3uq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:10 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E024xT025641 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:04 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E024Xu018660; Thu, 14 Feb 2019 00:02:04 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:04 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Tycho Andersen Subject: [RFC PATCH v8 04/14] swiotlb: Map the buffer if it was unmapped by XPFO Date: Wed, 13 Feb 2019 17:01:27 -0700 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=845 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger v6: * guard against lookup_xpfo() returning NULL CC: Konrad Rzeszutek Wilk Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Reviewed-by: Khalid Aziz Reviewed-by: Konrad Rzeszutek Wilk --- include/linux/xpfo.h | 4 ++++ kernel/dma/swiotlb.c | 3 ++- mm/xpfo.c | 15 +++++++++++++++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index b15234745fb4..cba37ffb09b1 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -36,6 +36,8 @@ void xpfo_kunmap(void *kaddr, struct page *page); void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp); void xpfo_free_pages(struct page *page, int order); +bool xpfo_page_is_unmapped(struct page *page); + #else /* !CONFIG_XPFO */ static inline void xpfo_kmap(void *kaddr, struct page *page) { } @@ -43,6 +45,8 @@ static inline void xpfo_kunmap(void *kaddr, struct page *page) { } static inline void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) { } static inline void xpfo_free_pages(struct page *page, int order) { } +static inline bool xpfo_page_is_unmapped(struct page *page) { return false; } + #endif /* CONFIG_XPFO */ #endif /* _LINUX_XPFO_H */ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 045930e32c0e..820a54b57491 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -396,8 +396,9 @@ static void swiotlb_bounce(phys_addr_t orig_addr, phys_addr_t tlb_addr, { unsigned long pfn = PFN_DOWN(orig_addr); unsigned char *vaddr = phys_to_virt(tlb_addr); + struct page *page = pfn_to_page(pfn); - if (PageHighMem(pfn_to_page(pfn))) { + if (PageHighMem(page) || xpfo_page_is_unmapped(page)) { /* The buffer does not have a mapping. Map it in and copy */ unsigned int offset = orig_addr & ~PAGE_MASK; char *buffer; diff --git a/mm/xpfo.c b/mm/xpfo.c index 24b33d3c20cb..67884736bebe 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -221,3 +221,18 @@ void xpfo_kunmap(void *kaddr, struct page *page) spin_unlock(&xpfo->maplock); } EXPORT_SYMBOL(xpfo_kunmap); + +bool xpfo_page_is_unmapped(struct page *page) +{ + struct xpfo *xpfo; + + if (!static_branch_unlikely(&xpfo_inited)) + return false; + + xpfo = lookup_xpfo(page); + if (unlikely(!xpfo) && !xpfo->inited) + return false; + + return test_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); +} +EXPORT_SYMBOL(xpfo_page_is_unmapped); From patchwork Thu Feb 14 00:01:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811375 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C0AB746 for ; Thu, 14 Feb 2019 00:03:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A09C29C32 for ; Thu, 14 Feb 2019 00:03:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F1B102D9EE; Thu, 14 Feb 2019 00:03:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5BC7C29C32 for ; Thu, 14 Feb 2019 00:03:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A9008E000C; Wed, 13 Feb 2019 19:02:59 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 05D488E0005; Wed, 13 Feb 2019 19:02:59 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8B2E8E000C; Wed, 13 Feb 2019 19:02:58 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 9C1458E0005 for ; Wed, 13 Feb 2019 19:02:58 -0500 (EST) Received: by mail-pg1-f198.google.com with SMTP id d3so2853978pgv.23 for ; Wed, 13 Feb 2019 16:02:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=iAUhMQrV5BcM9lvrtcT+KJw+jAn1o1A4vhSmYo6KfrI=; b=b6alK2/z+0CFtLq5myWxZGQPSl5JWSBB7fzSkU5aO+sjCRW33DjEP/fQCZROdCTNZr pqgv+mxEhf1EiXDuUTCDE/9xBLHbDLk1h0+gkTMClKMbjNdY49HjD2jDpQvjhboDGmoC PscJocaJS9MYz+AsoDkIHjg7U9akjVAzYq9HJ1DmGwI0rRaHheeRoO4bVD3qdmqoadKx crhfu6PluatTGl44DUerX4F3BJ/ppSPbUTmoWuCE3VZvrSQA5ST5Ysb7EAhP697Xxsv2 zC3dzFhzDPQQ8GaPUyw1UdYKLgkXvqi3aPvrtHmQSusQ2lk6yKkhtOwE0F+bimtbYvHM srRw== X-Gm-Message-State: AHQUAuYEgYbWLHK7CMo0Dfpu4xbXq4wCzgCtCbNC7zaQwq3S4OyvWsOd RoxLoaW6GB0P+qTh5negfvF/ipQ93WFB7bItXbUIrZ2MpTMfn0RBxzCMKi2ws8xZGrEXiBZVRT6 cJ9EByKKB23jeS7q035bNZFZSmguNDhJoYHcvPf2RMje+SqEfyJUGkQQcrc9Psp3Yrw== X-Received: by 2002:a17:902:28c1:: with SMTP id f59mr918824plb.37.1550102578275; Wed, 13 Feb 2019 16:02:58 -0800 (PST) X-Google-Smtp-Source: AHgI3IYVCohG/AKnTpehFrSpF5OZNDBmVXXiu82b41xr9olfcOzQgSumzSAfPFj80EwT/Pmz1Zbr X-Received: by 2002:a17:902:28c1:: with SMTP id f59mr918739plb.37.1550102577347; Wed, 13 Feb 2019 16:02:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102577; cv=none; d=google.com; s=arc-20160816; b=IrsZSwSqkY7uj5B+/wwhseskymOgz7FdrOy4FLXRibg4ijnP3KqDPFemG/lDbGInqv 6ROzyBc0dmzb6xVnkTDsbKLVScH2qvd83KYWc4QoaMuY59jK16Q1avqbstOnDYxSRHpL MmYO7K8B7IWqwGp/viaF19DmvuNW71tT38V/CNJSQNtmS4K3nwduwOCGvB0QCfNARpBK Zace/TBrQC0DsWzZ+0LvroBLB+ZUfgnyQZ1SYleJllxyXvUoxVDgdy44knymUYpQr0gc Pkef5Rl95dW+gauR3XFZMdia4573DupapsoRJqWKBTgPgdON7Vlp/vgQFpH+dx0y2qw2 OocQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=iAUhMQrV5BcM9lvrtcT+KJw+jAn1o1A4vhSmYo6KfrI=; b=a/dxRy5YLsOO5L5/dl/VrhvbAwxW+JQ+8Y9BwGxLmlUrU8FtGtDk2OUsxOVM3MEHNI Mfw7dNXqmCqtzckCuFTSIORYEn6LImbgd/diinI0WUgoAOt/qcxnM+9KDjF9Y4pqdwU4 ueLU3/UG4BBaVk3uhJ6dcUFVdy4nsHAQHlLmGdHXkhcZq0C5fVWGxd9Q/iQXdeCvjbQN Cj0s1hYQgqRJbHYMVkDcmoNoDJuTSoYbhOkwRCJcnF1Em9eEObmPfzZxzjSWxTEUzFoh qfFE+CdU12CS5nb56v1PcOkNWCfh5Y9j4yCbYEYs7yhg9YL/aLbsYu4wCtWqXZ+tengT m+Lg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="VjSTr/Aq"; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id y71si761140pfd.96.2019.02.13.16.02.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:57 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b="VjSTr/Aq"; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwvnk099503; Thu, 14 Feb 2019 00:02:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=iAUhMQrV5BcM9lvrtcT+KJw+jAn1o1A4vhSmYo6KfrI=; b=VjSTr/AqEcZZcnT253Enimw080YefgzZ6lczl9mWxonAMufYwkCcF3dwQch6ueHmg70i dJpR3VkpqqCo5MrMJg08cXPCQwRMg0ItEUkV50un7KCD9qy/0d+vUhhjmxELv5vDsXg3 Ec8x2cdR9pfsnnCSj5tyMVOUNuafN0UEGEnMrvyKFP6U+tH7zegPGoK2cHWcMcllXHpR 1PT/VRty4pVVnHKlT/GtTFe+lLr6OSdMuZAJLbsu2cPVgpX0UlpaWQ5YfTTR1EgTiKBX JspamAuYU1IEU+REu+zfOfbB5ugsNlyZSl+JykVJwE4cXEWuAqhvosNaLi5M380H3gN0 KA== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2qhree55mx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:10 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1E029GD008227 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:09 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1E027gu001545; Thu, 14 Feb 2019 00:02:07 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:06 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Tycho Andersen , Khalid Aziz Subject: [RFC PATCH v8 05/14] arm64/mm: Add support for XPFO Date: Wed, 13 Feb 2019 17:01:28 -0700 Message-Id: <4a7b373b438d5b93999349c648189bd5ba8f9198.1550088114.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger Enable support for eXclusive Page Frame Ownership (XPFO) for arm64 and provide a hook for updating a single kernel page table entry (which is required by the generic XPFO code). CC: linux-arm-kernel@lists.infradead.org Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Signed-off-by: Khalid Aziz --- v6: - use flush_tlb_kernel_range() instead of __flush_tlb_one() v8: - Add check for NULL pte in set_kpte() arch/arm64/Kconfig | 1 + arch/arm64/mm/Makefile | 2 ++ arch/arm64/mm/xpfo.c | 64 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) create mode 100644 arch/arm64/mm/xpfo.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index ea2ab0330e3a..f0a9c0007d23 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -171,6 +171,7 @@ config ARM64 select SWIOTLB select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK + select ARCH_SUPPORTS_XPFO help ARM 64-bit (AArch64) Linux support. diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 849c1df3d214..cca3808d9776 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -12,3 +12,5 @@ KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o KASAN_SANITIZE_kasan_init.o := n + +obj-$(CONFIG_XPFO) += xpfo.o diff --git a/arch/arm64/mm/xpfo.c b/arch/arm64/mm/xpfo.c new file mode 100644 index 000000000000..1f790f7746ad --- /dev/null +++ b/arch/arm64/mm/xpfo.c @@ -0,0 +1,64 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2017 Hewlett Packard Enterprise Development, L.P. + * Copyright (C) 2016 Brown University. All rights reserved. + * + * Authors: + * Juerg Haefliger + * Vasileios P. Kemerlis + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#include +#include + +#include + +/* + * Lookup the page table entry for a virtual address and return a pointer to + * the entry. Based on x86 tree. + */ +static pte_t *lookup_address(unsigned long addr) +{ + pgd_t *pgd; + pud_t *pud; + pmd_t *pmd; + + pgd = pgd_offset_k(addr); + if (pgd_none(*pgd)) + return NULL; + + pud = pud_offset(pgd, addr); + if (pud_none(*pud)) + return NULL; + + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) + return NULL; + + return pte_offset_kernel(pmd, addr); +} + +/* Update a single kernel page table entry */ +inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) +{ + pte_t *pte = lookup_address((unsigned long)kaddr); + + if (unlikely(!pte)) { + WARN(1, "xpfo: invalid address %p\n", kaddr); + return; + } + + set_pte(pte, pfn_pte(page_to_pfn(page), prot)); +} + +inline void xpfo_flush_kernel_tlb(struct page *page, int order) +{ + unsigned long kaddr = (unsigned long)page_address(page); + unsigned long size = PAGE_SIZE; + + flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); +} From patchwork Thu Feb 14 00:01:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811355 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF6116C2 for ; Thu, 14 Feb 2019 00:02:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD6D629C32 for ; Thu, 14 Feb 2019 00:02:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D0A6C2D9EE; Thu, 14 Feb 2019 00:02:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D3D929C32 for ; Thu, 14 Feb 2019 00:02:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21ECE8E0001; Wed, 13 Feb 2019 19:02:36 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 172648E0005; Wed, 13 Feb 2019 19:02:36 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F089E8E0002; Wed, 13 Feb 2019 19:02:35 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id A901C8E0002 for ; Wed, 13 Feb 2019 19:02:35 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id ay11so2883045plb.20 for ; Wed, 13 Feb 2019 16:02:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=pnPAMgXPoRW3Y+XGQjlBeFoy0dFQyjStoDUpBHTUk8o=; b=F/1sKKand5GMGVGhFIh2Yt9eviWHOtUEVzyowSYeQuTOHygjoxbDeebYA+as40QL7n 6BWYTNY2AwwHnu/Zmc1+WqiIZfUio8venC16ZxcyZ4qQUZ2d9n06WgDlGgNZQC6cK2FX Hj+XpxK96h4Lnvju30wOas2KMkDZ1fmB7CZreQmeV03FBknrgXzKNcFGpm4q0yz3M6qy dGWpL0jJZ/5gmocTTaE4VgVrWL6tpRaYV7rxGqJ7p8ZCwG3KGHp+gZmfk8buI5sAzql+ 1GZ5ku+8yBhiOG5p9O9MJrh07asykYv9MIYyM77NQD86fIQ0gvw2V6bhKTr9WyUCYU/l u+tg== X-Gm-Message-State: AHQUAuZMsopor5dFBZEiprLJkL5OCRRcGVM3rd2JswdWSyDPPFr6+lCj FPxLB9H7ubHVrDyhqpKcAtuDgKmdcwquzJIOQJr4QzkSx4+GOldJiPeU81FOogIGRxEIa6IwKvf 9fNTjw4FsZkBKitXSn5HH9excuu/Kkhxfi2uAOJyKdV/Y4xpSOuPR/fqjEiUnbKNv7g== X-Received: by 2002:a63:d347:: with SMTP id u7mr787744pgi.383.1550102555314; Wed, 13 Feb 2019 16:02:35 -0800 (PST) X-Google-Smtp-Source: AHgI3IaJxPJc4zz/jM4z1UYnLDTlDXBfimXXtelR0XPANxVQjJRR0AMTA/vp/VrCJrgU9qaq6Z8+ X-Received: by 2002:a63:d347:: with SMTP id u7mr787682pgi.383.1550102554496; Wed, 13 Feb 2019 16:02:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102554; cv=none; d=google.com; s=arc-20160816; b=hiocQ/au74FWa/9Ao2UwrlMSlYR8dpclTT/DqTnZpuI6QikP5LbZiX+jHIZpqtrECO kXFNOfbcfr0ht+y6RRs/u76oBVVnq+ScshhE4XnTzVAuJau+bropSaTywU4mMWOmeJ1Q D+Kx383g/Owzi3qf6/f9F7D8PdcncfP4x2SyL1F1FPYeUsrIbxtce5Tl+zQ51bd+YZF1 v9MPBSoSqmcRjuZRZEtmLJDJYnVodilRFUjOprZb5eMOkTJczKcKDmtvoriNRrdLYgIM Wnhontm08L4lC0oQA7439Sgmcj0DN2AlgndgNjnUwxrFvQhc2kxAvqzOfsykYAkF6ZYR Q1Gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=pnPAMgXPoRW3Y+XGQjlBeFoy0dFQyjStoDUpBHTUk8o=; b=W05gTG2Luu95qAD/9gYJZy+uFn0B5PExTfW+GWkbPwIF+OiKlh8Qa1U4ywU+7sU8KU wMYS3GLJzjybYOI9PqN0Co5Lbe/3j88WkCoN8t/vjjSClwsg62k9/y4FHl4SL8Mwu2MS hwWjpVCiZv2NUU/ArhDoUWI+NDw+LWa6P5OSokbJkpBxxUe1HW6KZ9fKKo3ZhhdsSrIh PF7gnibOKK1AIoZCQl89QfNL4YWHiAc1m6YwkmUw3beeswrMTjZfxqtl07nfwgl7kWc/ gAbsp9tlilaW15aUnMSKI7nVkXUHw4wveBCB7Ihj89bR7OWyucTD1dKMHAhRzFvt2WAx qxMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=P8MhcmBs; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id ct12si899208plb.422.2019.02.13.16.02.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:34 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=P8MhcmBs; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwuiN100575; Thu, 14 Feb 2019 00:02:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=pnPAMgXPoRW3Y+XGQjlBeFoy0dFQyjStoDUpBHTUk8o=; b=P8MhcmBsqowTapSCWseyQXycsL6swLz4umwP9SBw6+ulLYPgQxLYPAEZ/o9AhHRJONze oYae3uML40WaOv1a39/aVVlZlsxyHs9WIr9DlGXDRTDq832AMS1IhnxZR/SlXqxcRSe8 xVPdOvmpX2ulrj52wlQOEOi4pN4dGsXZqeE8pt03NqDXmySkTKp6Hnrv+rZewUQNaso7 ele2U7dCemABJxM/iTeQAVNSeCxzWjRiyAQZB4GRIUvXED192Fn5YsV8J3mHml/c/vsR 1A5jr/53DI9WF3/DXkm1vroHhlCMQMdkWRgRlraDEqbRGW8QDSSClGDzYcI1zO+9cv9T WQ== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2130.oracle.com with ESMTP id 2qhre5n3uv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:12 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02BWR032120 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:11 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E029p0032487; Thu, 14 Feb 2019 00:02:09 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:09 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Tycho Andersen , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v8 06/14] xpfo: add primitives for mapping underlying memory Date: Wed, 13 Feb 2019 17:01:29 -0700 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=956 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen In some cases (on arm64 DMA and data cache flushes) we may have unmapped the underlying pages needed for something via XPFO. Here are some primitives useful for ensuring the underlying memory is mapped/unmapped in the face of xpfo. Signed-off-by: Tycho Andersen Reviewed-by: Khalid Aziz --- include/linux/xpfo.h | 22 ++++++++++++++++++++++ mm/xpfo.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index cba37ffb09b1..1ae05756344d 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -38,6 +38,15 @@ void xpfo_free_pages(struct page *page, int order); bool xpfo_page_is_unmapped(struct page *page); +#define XPFO_NUM_PAGES(addr, size) \ + (PFN_UP((unsigned long) (addr) + (size)) - \ + PFN_DOWN((unsigned long) (addr))) + +void xpfo_temp_map(const void *addr, size_t size, void **mapping, + size_t mapping_len); +void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, + size_t mapping_len); + #else /* !CONFIG_XPFO */ static inline void xpfo_kmap(void *kaddr, struct page *page) { } @@ -47,6 +56,19 @@ static inline void xpfo_free_pages(struct page *page, int order) { } static inline bool xpfo_page_is_unmapped(struct page *page) { return false; } +#define XPFO_NUM_PAGES(addr, size) 0 + +static inline void xpfo_temp_map(const void *addr, size_t size, void **mapping, + size_t mapping_len) +{ +} + +static inline void xpfo_temp_unmap(const void *addr, size_t size, + void **mapping, size_t mapping_len) +{ +} + + #endif /* CONFIG_XPFO */ #endif /* _LINUX_XPFO_H */ diff --git a/mm/xpfo.c b/mm/xpfo.c index 67884736bebe..92ca6d1baf06 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -14,6 +14,7 @@ * the Free Software Foundation. */ +#include #include #include #include @@ -236,3 +237,32 @@ bool xpfo_page_is_unmapped(struct page *page) return test_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); } EXPORT_SYMBOL(xpfo_page_is_unmapped); + +void xpfo_temp_map(const void *addr, size_t size, void **mapping, + size_t mapping_len) +{ + struct page *page = virt_to_page(addr); + int i, num_pages = mapping_len / sizeof(mapping[0]); + + memset(mapping, 0, mapping_len); + + for (i = 0; i < num_pages; i++) { + if (page_to_virt(page + i) >= addr + size) + break; + + if (xpfo_page_is_unmapped(page + i)) + mapping[i] = kmap_atomic(page + i); + } +} +EXPORT_SYMBOL(xpfo_temp_map); + +void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, + size_t mapping_len) +{ + int i, num_pages = mapping_len / sizeof(mapping[0]); + + for (i = 0; i < num_pages; i++) + if (mapping[i]) + kunmap_atomic(mapping[i]); +} +EXPORT_SYMBOL(xpfo_temp_unmap); From patchwork Thu Feb 14 00:01:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811361 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 45AB3746 for ; Thu, 14 Feb 2019 00:02:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33BAA29C32 for ; Thu, 14 Feb 2019 00:02:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 23FB02D9EE; Thu, 14 Feb 2019 00:02:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 91C3629C32 for ; Thu, 14 Feb 2019 00:02:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F4298E0004; Wed, 13 Feb 2019 19:02:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 62A088E0006; Wed, 13 Feb 2019 19:02:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DEFC8E0004; Wed, 13 Feb 2019 19:02:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id E7C778E0005 for ; Wed, 13 Feb 2019 19:02:39 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id k14so2928416pls.2 for ; Wed, 13 Feb 2019 16:02:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=fEQ1MfpMn+jTbXDbToNZqpIPh0CEmp2s2jsy7mZAXXk=; b=G8vjkcFSKjmJjQOn96/hUgsiT2Xayw0kgpTEPJp6bneqIWlqhzt5tJnmSV3KnWj9+n EjPCjFf2nd1/qKiIzQhVeuqf0crF3bBk2+5hMQdqNwMQhSi18VzEDlWzNskqKsXG/Cvv fs43rFWow80mH13aGjsXKtEGROjcgzD2EGaOt4ygXUIIQs7ByjrlYOCNn6Irm0RCchCJ NMEf3OdJM334i7C8/hJSOInJxYgGkxXQmcFJwMflUcmUgSbeFwV+ICtmUguPWqkaYqUb 8Za3pZat+4ZEEFTwj0b0ENDwD40Cb9hMfVF82znXuLPlzeYfd2jKwEpU7r2vHRl2SzwL OuvQ== X-Gm-Message-State: AHQUAubOxE+6bJIKKWuzn2Hr4RcxYokVE9+h9vgiM0mCdwiMut7tOpRC kX+5d6PS397IS6PkYVMW0Hpl1udg7gU9e41Zp22bzOuCllaNx5xikViMcAe0mdR3FiPVR2izC9d qEKf5GaxSfSwsnxq2a6/mOZnqCT/EZy+o5xKIdlnVbs++jnHjn0FH8+5sGQHSE2KjCA== X-Received: by 2002:a17:902:b681:: with SMTP id c1mr918263pls.103.1550102559612; Wed, 13 Feb 2019 16:02:39 -0800 (PST) X-Google-Smtp-Source: AHgI3Iaf9gP2BsDXa7ks5ZEdGW5tGInj9eV41nTlhgFUlHdAkT5Opm6wv2MtP2uT2CI+CUWdiCrl X-Received: by 2002:a17:902:b681:: with SMTP id c1mr918169pls.103.1550102558692; Wed, 13 Feb 2019 16:02:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102558; cv=none; d=google.com; s=arc-20160816; b=XIvDZ2NKhD5vFLUeBTKQfMSWDu70eHEhDVxNv+/+IduDovcHgVkU+U83ltNPI/ZxEr kFsJ5uRdkRLIlpJE1bXt/fXXJBa0Q9bdX+G4GKxBKdMhX8S7EADz7x6moKX8k//dfWSB XbXORRbugiqdYMX64Q3Fvvc69ETDtUSdZMx/NCsnd9mEmKvhC0Y2u8sEzyk4hTADRkjI Ug0+RvtjUCxYDp8PNqwEzYK61xq61I2UCDHfzFI0VXYE5mjj0F/UqYWu2iR0J3Vli23i GDQYuiSaN+qiZ19aAozkTeoeJzpugzr0N30Qc/KmRCZAuSSsBQB9UAHW39JJ5xxwY7CR GYYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=fEQ1MfpMn+jTbXDbToNZqpIPh0CEmp2s2jsy7mZAXXk=; b=DcQb/bVcsLuwZvX0wkUetTTRYI1/9eg6w+npanppYUcdpy8icCgrHHw6OZ2H8TmksY wILUqvEPXshhluQ3h1regS+dkfB5WGuZ4d9NNR7NJdSOxk0vFC0jtK+n7R1dmtuDxhS8 V3OkudAjUaclGjKDI/SzPcWIV3OYcXTlLax+S6r8jFn/4CT+yi5PofIs5c9kQS1brb3Z YMroR51pvZydPK4VuJQjP1oFuICTOLFpmvdiSZVJE5WlsVxQC5RitEF4mKCg57FAMfa5 bnRdTllN8s6E3h2M1Ic4njGBXEx34vr33GrUJN0PpyEVcOCOz4cZddiXiqw7x+yPLVdY wnEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=clQp5FLl; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id l123si737718pfc.187.2019.02.13.16.02.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:38 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=clQp5FLl; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNxBx3100713; Thu, 14 Feb 2019 00:02:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=fEQ1MfpMn+jTbXDbToNZqpIPh0CEmp2s2jsy7mZAXXk=; b=clQp5FLlv6dBK7nLgBvO7z0XrNUeRlDkeSjrUlTYTSJbuNRLfOKl0R1BT5sDq21QFYX8 pSz8w5lOqXzP3bUCNpXzkHLjNtdbGgLEsSSt8Eua3YWbfcZEEcQkzBiGSUl4yar1bvHp q8fCHDNP/V5TBUNkjrEFGj4gq1sOoXWLJD0C9kdSdRCeEl+8pLBspnYCQCdJcqUzk5J3 M9AXZQEMVtI8t9Hvrz9GDsGdYXEpQKo1Pp1JwtMth2sBZOvx6EWbBElqk6cR1/ggkzPa cINSZoMWJK2sX83UednZi/f2IJhDrqSJBRYcN3DKtOMSwLUXe+sR+ATtY0MHyLaEGXWX Aw== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2130.oracle.com with ESMTP id 2qhre5n3v6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:17 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02CAr025949 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:12 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02C7V001627; Thu, 14 Feb 2019 00:02:12 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:11 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Tycho Andersen Subject: [RFC PATCH v8 07/14] arm64/mm, xpfo: temporarily map dcache regions Date: Wed, 13 Feb 2019 17:01:30 -0700 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=559 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger If the page is unmapped by XPFO, a data cache flush results in a fatal page fault, so let's temporarily map the region, flush the cache, and then unmap it. v6: actually flush in the face of xpfo, and temporarily map the underlying memory so it can be flushed correctly CC: linux-arm-kernel@lists.infradead.org Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen --- arch/arm64/mm/flush.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c index 30695a868107..fad09aafd9d5 100644 --- a/arch/arm64/mm/flush.c +++ b/arch/arm64/mm/flush.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -28,9 +29,15 @@ void sync_icache_aliases(void *kaddr, unsigned long len) { unsigned long addr = (unsigned long)kaddr; + unsigned long num_pages = XPFO_NUM_PAGES(addr, len); + void *mapping[num_pages]; if (icache_is_aliasing()) { + xpfo_temp_map(kaddr, len, mapping, + sizeof(mapping[0]) * num_pages); __clean_dcache_area_pou(kaddr, len); + xpfo_temp_unmap(kaddr, len, mapping, + sizeof(mapping[0]) * num_pages); __flush_icache_all(); } else { flush_icache_range(addr, addr + len); From patchwork Thu Feb 14 00:01:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811363 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 485CD6C2 for ; Thu, 14 Feb 2019 00:02:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3857029C32 for ; Thu, 14 Feb 2019 00:02:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2AFCE2DE30; Thu, 14 Feb 2019 00:02:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A009829C32 for ; Thu, 14 Feb 2019 00:02:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 712F28E0006; Wed, 13 Feb 2019 19:02:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6C3958E0005; Wed, 13 Feb 2019 19:02:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A1758E0006; Wed, 13 Feb 2019 19:02:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id F22B88E0005 for ; Wed, 13 Feb 2019 19:02:40 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id q21so3205466pfi.17 for ; Wed, 13 Feb 2019 16:02:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=B56xy1SykMvAYC6aqGnZB/bYPtFwWukcKOnvet3ifEU=; b=IqlchLeQxjj46gyvabWOVYKeSEnLm5QKzPDOzZjrRUufBDItDToinG5+/bD44kjSEr WEOvFFgDQkYwBeiFCJATzQYyNpwTYx0R0cKvV4zMpBBR+hCJETKdOjfPQDG/RNpMtfYB sUjIS1QEIVDmjuxTMoeMZPiHIwKAwWmbtLPHgXObCzs8wjkyXX7VYwdJPIxb/j59mxE7 etaBTETVtfc+qD740vnF2qL9bXcBxzRKw4C6kdHhD+9mI8SYfGi1lBnIq21MJVxZrZy5 QDhzz9mDBeArHdmr/qlYaQMd+cWhDRlwuoQp24gF/6lmYJxOT8LWquraZFmknIwULARL B7cQ== X-Gm-Message-State: AHQUAuYSUpeqAFJJI7YxRV9uk5nUvSqnJ/MOwhVDY0Fhl+O+IZWTBtMP AOOF+FdfXZHJpmKSCq7N26yhpzCyjCX/LNlSDZZs7Huv3I06tRNt5vo6HGPZad8P9KJT9el7Ez1 o/Ofx2sF03T9T2jT2ReBwtHnubpCgEOHg7oNvjMswsk/6+MHlRahEnnVinWbkHpOH4Q== X-Received: by 2002:a63:1143:: with SMTP id 3mr782675pgr.447.1550102560629; Wed, 13 Feb 2019 16:02:40 -0800 (PST) X-Google-Smtp-Source: AHgI3IaNigTvT93gwUe4QoSXqTSoyaZFKQwgL6tXJ8BWi+EHZSnmf0RC8bChkTpKmD21E7OHPGd1 X-Received: by 2002:a63:1143:: with SMTP id 3mr782599pgr.447.1550102559801; Wed, 13 Feb 2019 16:02:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102559; cv=none; d=google.com; s=arc-20160816; b=zaaAAds8vajyB/dwTMLB4EANOpNEvxxpr60Q27pavYiDQ3pbbAOv3VCwWAEOH8ffl/ kP1beHis6yedTLUFtZOlAkJCDqJZWe0I9dbdP09GxGS+64/7gszFdP9zuYFrllS0UylO YeTpfS5Ia/Zww9LkHYlbwkGjVvw3OO+zcqVpNN8NVWp1Bm9CAq0eBq4502c297h7Vn16 PSfswTwqnyFBa4V6QFFz7PUqjY3UuSbTMBYaeY0GDfmCSGAFvr5OsQvqs4okfovhBgaq 1BJ53Xm6G7Nh2SdzVaUwwKb1/u3pP0vUT0+2tyz6yQihWE2dCPrM29WauJOi0S3TIGoL 0LTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=B56xy1SykMvAYC6aqGnZB/bYPtFwWukcKOnvet3ifEU=; b=jG0JdoLrjKd8Hdop9RSyf2JVPUXY5Z2CfB84Aizionj1Ktqolc9Kx3HErIFBemhyxT QzFcNb6IGjYaTh2Y58UxGU7vWyq+ADo3hsxRuXZt5x0/Jz7hdJbzg9RjUCCeChyu6tOC 839CpaHmJOHAVunYRVj58xECGr4kSJz55x8yYR3W/OZ9ecUmpH45eQOZTVt2e4XwD+jA tRQU2prvJe18j2Yfo3oV/V7spNIAg9k34uJ/pKGgH1Pgl9t9F+8R1sKZbUS12gi1vPa+ F/+/zBhj6mfpguCW6RTxLFuWeqJGV9DTjrniDkloQY8+DSpLhR8raoK1pKg3w7WCutEO eD4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=uIx25zRg; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from aserp2130.oracle.com (aserp2130.oracle.com. [141.146.126.79]) by mx.google.com with ESMTPS id y12si716810pga.216.2019.02.13.16.02.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:39 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) client-ip=141.146.126.79; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=uIx25zRg; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwgkD100217; Thu, 14 Feb 2019 00:02:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=B56xy1SykMvAYC6aqGnZB/bYPtFwWukcKOnvet3ifEU=; b=uIx25zRg4V1mI7KE5rlBz/NOl4jvjuexvGmPkyZvqcfrmno71cCT5E+9JbWTSHC/QKat mwHwGgsn3lyUx7sgJV9UNpCRjTU6py7eGgk7lKgOpsDhpFjeerfPM2DbaBuKJ9UZcAKz DsSL/ZBoCBxdhR3YuaIpkJ3CIAyPWAuQv3sF7iGkSsqlEtLlb7Zeh/RNvN5hZ7YrOLsh sHsS6j1a4GAWGwWpXc9lQ+6XKaFyP9pfD/3U4ji5t9w3llUxgied6gcwwuUnMAESawHr XKG5wRxJo2PHsFrtNsUaPYHBCbF2txoevXeZNyWcby8O60DPcnZ1QRqSHrFDNLBVEldH TQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp2130.oracle.com with ESMTP id 2qhre5n3v1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:15 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02ERU026044 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:14 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02ELP001735; Thu, 14 Feb 2019 00:02:14 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:14 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Tycho Andersen , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v8 08/14] arm64/mm: disable section/contiguous mappings if XPFO is enabled Date: Wed, 13 Feb 2019 17:01:31 -0700 Message-Id: <0b9624b6c1fe5a31d73a6390e063d551bfebc321.1550088114.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen XPFO doesn't support section/contiguous mappings yet, so let's disable it if XPFO is turned on. Thanks to Laura Abbot for the simplification from v5, and Mark Rutland for pointing out we need NO_CONT_MAPPINGS too. CC: linux-arm-kernel@lists.infradead.org Signed-off-by: Tycho Andersen Reviewed-by: Khalid Aziz --- arch/arm64/mm/mmu.c | 2 +- include/linux/xpfo.h | 4 ++++ mm/xpfo.c | 6 ++++++ 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d1d6601b385d..f4dd27073006 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -451,7 +451,7 @@ static void __init map_mem(pgd_t *pgdp) struct memblock_region *reg; int flags = 0; - if (debug_pagealloc_enabled()) + if (debug_pagealloc_enabled() || xpfo_enabled()) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; /* diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 1ae05756344d..8b029918a958 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -47,6 +47,8 @@ void xpfo_temp_map(const void *addr, size_t size, void **mapping, void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, size_t mapping_len); +bool xpfo_enabled(void); + #else /* !CONFIG_XPFO */ static inline void xpfo_kmap(void *kaddr, struct page *page) { } @@ -69,6 +71,8 @@ static inline void xpfo_temp_unmap(const void *addr, size_t size, } +static inline bool xpfo_enabled(void) { return false; } + #endif /* CONFIG_XPFO */ #endif /* _LINUX_XPFO_H */ diff --git a/mm/xpfo.c b/mm/xpfo.c index 92ca6d1baf06..150784ae0f08 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -71,6 +71,12 @@ struct page_ext_operations page_xpfo_ops = { .init = init_xpfo, }; +bool __init xpfo_enabled(void) +{ + return !xpfo_disabled; +} +EXPORT_SYMBOL(xpfo_enabled); + static inline struct xpfo *lookup_xpfo(struct page *page) { struct page_ext *page_ext = lookup_page_ext(page); From patchwork Thu Feb 14 00:01:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ECD3F746 for ; Thu, 14 Feb 2019 00:03:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DBBFF2D923 for ; Thu, 14 Feb 2019 00:03:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CEA2C2DBB6; Thu, 14 Feb 2019 00:03:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BA382D923 for ; Thu, 14 Feb 2019 00:03:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B24F8E000D; Wed, 13 Feb 2019 19:03:07 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 561B88E0005; Wed, 13 Feb 2019 19:03:07 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 479E28E000D; Wed, 13 Feb 2019 19:03:07 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 08EDE8E0005 for ; Wed, 13 Feb 2019 19:03:07 -0500 (EST) Received: by mail-pg1-f198.google.com with SMTP id r13so2888166pgb.7 for ; Wed, 13 Feb 2019 16:03:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=TQlIsuP1ag8N8cWtIZarXg5qzwt4o6eHPtANA79RWaM=; b=FJDFL0tIt31vvIWsBhsll/glYzh2XtltNkoOr8UUzXNabRLZ+jj0Blt2+HEibQ2Ks1 TR2jT0dzh52K/N3ao6s72UswVdEKCOtlfEUg9KLfUK+TsTkO6babC+nfJJF7PXOOHNAu QiRwx5NfiS5lJcuKS3E8nyYBExWT/4Gt6AlxhhmvpxFvdZQYJAFMqKZraXN6szEVR4nV J7vRq+zm7D7zpG5oYj/hwHEwe6nOTTMyhyatgq4Ep4Ry4/PD3XfeMULHUByZNu7aN6gm DlwSD6GMTVKyL6CLVt959F95bWmYFRb0qU/amjTOQ9yQG6ZbvRLM+VaDCXf5Nvw4Yrna guBA== X-Gm-Message-State: AHQUAuZQ0iLEZRBzZafgIfgfo2uhdnE/ajgMTGn/eXWL36lmbAWgfZmN hhaJzBmE9CfDpQHpC4ku56CxC82iRXBoPgsuCIRT8Oe3helPNJ463/+GqGssbqjQoDs0dZatt0P kuc2LvGhpUAv/BB8iefLs6a6gpgkpLVunCUVUma5wRJcpPxqwttDb2vPEKVAv5HsVhA== X-Received: by 2002:a65:4383:: with SMTP id m3mr817829pgp.96.1550102586685; Wed, 13 Feb 2019 16:03:06 -0800 (PST) X-Google-Smtp-Source: AHgI3IYK/So7VUvbD5Kl5+s3Tt5+3ItVVG+1odnNT97rD9uWH6clHWBnrlA2rj+oE5+PymD9i+mB X-Received: by 2002:a65:4383:: with SMTP id m3mr817745pgp.96.1550102585748; Wed, 13 Feb 2019 16:03:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102585; cv=none; d=google.com; s=arc-20160816; b=A2HiSMhCcqMymU20980A0TPi6Mo3wmRUnGj3nmV0i5h78l53TmCqmrHjIZX9gF7EUk n70TNnmLHVx4E2569pVzy4Qq/ja8mXdknpAtZ2YwcnJnE5W3xfIfMs3ckbiaeU1Cmnbn 6lAh6Ntc1ftZMKQ3bUZllqHHCzUlDTxFvRfFGgYVgfykBYgbuyBq3NdxYvr/ptsb8OFu h+M5oUaZN4pd1rn+U7AyezBfUOrBycJU8xb+SIWII44P/xnEbEhC2b2ybQJoi/qrcOXm Jd2Dj5oe0aTiWbtL8E4sqKjsM60FBoHFoIAmGC+fQYw41/bI+gspAbq1cVbdn6gfbrbG sYZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=TQlIsuP1ag8N8cWtIZarXg5qzwt4o6eHPtANA79RWaM=; b=ola4CbZ1KEa3PyKWwWJC2mvaC8dnGg9ZQISt6DBZ6wbYu0e5Ri2rRGLopvbaad6ozU YixUeXMS/gWFzrNlRa3tb+EGqW1LbB+KJo0wuTprjiivd+EbIj3bMlOCdPZfU9QoNsoK alp1N+ST2biVRfhDMD+amHoeXyOqW8ug/yMZNO2qfVUrIFLmzmzZciiTI6vRP+xM8qkE 9gwvXfWM8InOHIntG6eEIztH+80388tT7mxfwmMH1SjL+C5s4ugzX7mtM0XjPp2wMRRg nvQ6+2f+qwDIsS0cMox17u5wfYXEwLGHhSe9Li5dIZgoZtwa//GF7QWvIuMci0vhlA1r FFog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=2PAnanOt; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id bb3si744153plb.160.2019.02.13.16.03.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:03:05 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=2PAnanOt; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNx9xx093658; Thu, 14 Feb 2019 00:02:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=TQlIsuP1ag8N8cWtIZarXg5qzwt4o6eHPtANA79RWaM=; b=2PAnanOtADcxei1cy8BlTb/qU5ZZ7i0wqibFXPa9bnInPG/bvReURBWdV7oqlHHKB04P eJe2e67nu+luxcdvC3UBRDtvbK4t0fpRO4v+csDCFP/QjO2HMEGx/ANYI0/HgjyJOXBQ A+HhQpvU1Zm+os48RbnhGhjXf13nrybGW9Q+s+JqmKbNdQ6NQ9nWd7faph98x/U5WFsQ GP3HR7Iu2pYrMuqLFfE3A+sddb/2CPIRd3PP80otI32ENy8xTiH9Z0CLXt9neXDuvRjw swjC9f3y4oeNAID085G4bmwvjmiQzHs2QY3PeVUXmX6gdeHyJcTl0BfB9Kn9iJyjhaP4 ZQ== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2130.oracle.com with ESMTP id 2qhrekn4kv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:20 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02I6f026212 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:19 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E02Hqw032540; Thu, 14 Feb 2019 00:02:17 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:16 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Tycho Andersen , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Khalid Aziz Subject: [RFC PATCH v8 09/14] mm: add a user_virt_to_phys symbol Date: Wed, 13 Feb 2019 17:01:32 -0700 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Tycho Andersen We need someting like this for testing XPFO. Since it's architecture specific, putting it in the test code is slightly awkward, so let's make it an arch-specific symbol and export it for use in LKDTM. CC: linux-arm-kernel@lists.infradead.org CC: x86@kernel.org Signed-off-by: Tycho Andersen Tested-by: Marco Benatto Signed-off-by: Khalid Aziz --- v6: * add a definition of user_virt_to_phys in the !CONFIG_XPFO case v7: * make user_virt_to_phys a GPL symbol arch/x86/mm/xpfo.c | 57 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/xpfo.h | 8 +++++++ 2 files changed, 65 insertions(+) diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c index 6c7502993351..e13b99019c47 100644 --- a/arch/x86/mm/xpfo.c +++ b/arch/x86/mm/xpfo.c @@ -117,3 +117,60 @@ inline void xpfo_flush_kernel_tlb(struct page *page, int order) flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); } + +/* Convert a user space virtual address to a physical address. + * Shamelessly copied from slow_virt_to_phys() and lookup_address() in + * arch/x86/mm/pageattr.c + */ +phys_addr_t user_virt_to_phys(unsigned long addr) +{ + phys_addr_t phys_addr; + unsigned long offset; + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + pgd = pgd_offset(current->mm, addr); + if (pgd_none(*pgd)) + return 0; + + p4d = p4d_offset(pgd, addr); + if (p4d_none(*p4d)) + return 0; + + if (p4d_large(*p4d) || !p4d_present(*p4d)) { + phys_addr = (unsigned long)p4d_pfn(*p4d) << PAGE_SHIFT; + offset = addr & ~P4D_MASK; + goto out; + } + + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return 0; + + if (pud_large(*pud) || !pud_present(*pud)) { + phys_addr = (unsigned long)pud_pfn(*pud) << PAGE_SHIFT; + offset = addr & ~PUD_MASK; + goto out; + } + + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) + return 0; + + if (pmd_large(*pmd) || !pmd_present(*pmd)) { + phys_addr = (unsigned long)pmd_pfn(*pmd) << PAGE_SHIFT; + offset = addr & ~PMD_MASK; + goto out; + } + + pte = pte_offset_kernel(pmd, addr); + phys_addr = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT; + offset = addr & ~PAGE_MASK; + +out: + return (phys_addr_t)(phys_addr | offset); +} +EXPORT_SYMBOL_GPL(user_virt_to_phys); diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 8b029918a958..117869991d5b 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -24,6 +24,10 @@ struct page; #ifdef CONFIG_XPFO +#include + +#include + extern struct page_ext_operations page_xpfo_ops; void set_kpte(void *kaddr, struct page *page, pgprot_t prot); @@ -49,6 +53,8 @@ void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, bool xpfo_enabled(void); +phys_addr_t user_virt_to_phys(unsigned long addr); + #else /* !CONFIG_XPFO */ static inline void xpfo_kmap(void *kaddr, struct page *page) { } @@ -73,6 +79,8 @@ static inline void xpfo_temp_unmap(const void *addr, size_t size, static inline bool xpfo_enabled(void) { return false; } +static inline phys_addr_t user_virt_to_phys(unsigned long addr) { return 0; } + #endif /* CONFIG_XPFO */ #endif /* _LINUX_XPFO_H */ From patchwork Thu Feb 14 00:01:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811365 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F606746 for ; Thu, 14 Feb 2019 00:02:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D25F2D923 for ; Thu, 14 Feb 2019 00:02:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 800272DBB6; Thu, 14 Feb 2019 00:02:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BEE432D923 for ; Thu, 14 Feb 2019 00:02:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AA468E0008; Wed, 13 Feb 2019 19:02:42 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 989F18E0005; Wed, 13 Feb 2019 19:02:42 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 849CE8E0008; Wed, 13 Feb 2019 19:02:42 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 468308E0005 for ; Wed, 13 Feb 2019 19:02:42 -0500 (EST) Received: by mail-pf1-f199.google.com with SMTP id 74so3216931pfk.12 for ; Wed, 13 Feb 2019 16:02:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=1tfiao/L4nkY1/BQ61imQ3duc4RawqVwLZqH4dWuQjI=; b=TNoT1ZSd7krsnLZWu9XdkmMHsgSwZ82joM5m5dcSw8Fo388bpMmQgHwoW8vCkFEPwz ghmhBpZTYzG9wpQczRJPlqGiKXxUfDi5VRA0rwDhghVDS+AIMdW0dwM7bOz8Hj81Frsx GfN3vRyXCrF8TaCl+0Cvxng93snmCh0JwKwcuiq+iQUR4wueCCAy2dmcJAny70F1G2DA VV3LRo+3mlhsgPFUrY83/hwLD0B5Mo5brzSJNn0ImLWbGvthPsp0HajuJWzz4h6/2qHa TewBEDjwjb0dEiUQ7cTl1dHvHmdxx14z4FbjRqk8Npu/Sx5//vQxtgiVWEs+HyfNXUK8 W27A== X-Gm-Message-State: AHQUAubO6Dtrr0ftbyYNjDiPiYGeG2q8B6umyPWIF5Ph6Dso0JxKj2X8 NaBluu46fquB+BcD9twp3/eW2DLs+H3/CtiWNIaVa7qe+pI4WDeNJbw2zZePzHDUOQDjqLeirIa IRUYLCnMe2B5tTOTbsW0BCT2KOIlCuB2DlI/9RyiJkXGH8ONaEiLIpFtSEOAOWPY+6w== X-Received: by 2002:a62:4e83:: with SMTP id c125mr874401pfb.101.1550102561914; Wed, 13 Feb 2019 16:02:41 -0800 (PST) X-Google-Smtp-Source: AHgI3Ia+H3u2zlPQ/oD4AKqiFu+Nb5DZxi0S7XciiJE3gkOFUnunKa/PcmImn0Ye0QVyYH9iGsBI X-Received: by 2002:a62:4e83:: with SMTP id c125mr874302pfb.101.1550102560930; Wed, 13 Feb 2019 16:02:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102560; cv=none; d=google.com; s=arc-20160816; b=jHkiAtC70BgtITwly7XrLEGzJ8KYr861KJ/fXgvwAtFCw5hzL84JvtxarlR/28TWNd SBBQAJQm/w+Gi8tYtooWRlY6PmFca/TknIl5vsOVv8KGeH9NauPrA2v1gXPd75N2bRs7 Puv4NuUg8SUQ5FftPazpGJ+bAIfBk8TqOfdVyHsEgyqBFe2Q6Fkf9BK1zkaY8tODm+kO wo6W5dBOk5w1Mj0e4XU+92z8d0MIw3wrErvUbx8rHs6/luuUEVyX5MeX5tbYnf8vm7az fTJKRl/k69B/s/te7pQraFkByDMpYPz5+RdcrTsfETl69UoK5tkEWQxqHDnK9chkCHMz j2Cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=1tfiao/L4nkY1/BQ61imQ3duc4RawqVwLZqH4dWuQjI=; b=O0FuCsRoKc0FUTgTTMA8AQ+7/coe0WpW1xEbpzPPS0AHxk6AvpJmNSihLP4RkIwyiz CABwSc+TQX7W7dlZ6H35sxiGVmkmLSlmz52su8f+CMgfdOYobAstfMtVgfg8r0shAxX1 Ygu0Fq62Q8RtrImlRuv5kYxpScCfa7qp7YsWcp8D5I4Jbew/Ljb9DXhFrJxbGZnWjPcW jcqAfCQnFWaOqDU8vHaStnkp4EQcwQiHuMeYHRmn1+uDAZE+b3qDvyx0Nb7UOplsNOwT G9b4d2EXl/cR6rRoJ0Wk13rPoxn7rKjWgYJNAOSiJ8yWzDNiwKJDGyB4i5VRUh9bkejN VQyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=D+VWajgN; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id g8si788556pgo.166.2019.02.13.16.02.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:40 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=D+VWajgN; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwhvM099390; Thu, 14 Feb 2019 00:02:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=1tfiao/L4nkY1/BQ61imQ3duc4RawqVwLZqH4dWuQjI=; b=D+VWajgNZnnbCBmJd6hRmQKSxbcltQMbhcmtC6zmq5GAiZd2odC4bM6E90rcz6FYx3g3 p7y7UNpH69a0nk3zZf710qRnzwfsm4El++cnOGfwCXpBM+jvTdVH6Sf1YTW1mX20YwrC xhahrxss9iSQvxnblHxengOYGhfFgrNit8QWBepNbxggligImNFYyOIvecSsajODeFCA QYzSeuNjvM9sbPmvcKLqraQPt2jiXMulJIq2+crPTnfNMyeJj3D2nJzTpOUsVOy0xI7m BqAYotYlxCLKc5WX7QxIgpOoO3+kFRcKTeD7C15b1QdehKVErxfhN/MX2ZEDG3Kpwb4g ZA== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2qhree55n9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:20 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02JKR001195 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:20 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E02J3c018809; Thu, 14 Feb 2019 00:02:19 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:19 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Juerg Haefliger , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Tycho Andersen Subject: [RFC PATCH v8 10/14] lkdtm: Add test for XPFO Date: Wed, 13 Feb 2019 17:01:33 -0700 Message-Id: <1af3c61568d36cad4a2b2fece978336620701b86.1550088114.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Juerg Haefliger This test simply reads from userspace memory via the kernel's linear map. v6: * drop an #ifdef, just let the test fail if XPFO is not supported * add XPFO_SMP test to try and test the case when one CPU does an xpfo unmap of an address, that it can't be used accidentally by other CPUs. Signed-off-by: Juerg Haefliger Signed-off-by: Tycho Andersen Tested-by: Marco Benatto [jsteckli@amazon.de: rebased from v4.13 to v4.19] Signed-off-by: Julian Stecklina Tested-by: Khalid Aziz --- drivers/misc/lkdtm/Makefile | 1 + drivers/misc/lkdtm/core.c | 3 + drivers/misc/lkdtm/lkdtm.h | 5 + drivers/misc/lkdtm/xpfo.c | 194 ++++++++++++++++++++++++++++++++++++ 4 files changed, 203 insertions(+) create mode 100644 drivers/misc/lkdtm/xpfo.c diff --git a/drivers/misc/lkdtm/Makefile b/drivers/misc/lkdtm/Makefile index 951c984de61a..97c6b7818cce 100644 --- a/drivers/misc/lkdtm/Makefile +++ b/drivers/misc/lkdtm/Makefile @@ -9,6 +9,7 @@ lkdtm-$(CONFIG_LKDTM) += refcount.o lkdtm-$(CONFIG_LKDTM) += rodata_objcopy.o lkdtm-$(CONFIG_LKDTM) += usercopy.o lkdtm-$(CONFIG_LKDTM) += stackleak.o +lkdtm-$(CONFIG_LKDTM) += xpfo.o KASAN_SANITIZE_stackleak.o := n KCOV_INSTRUMENT_rodata.o := n diff --git a/drivers/misc/lkdtm/core.c b/drivers/misc/lkdtm/core.c index 2837dc77478e..25f4ab4ebf50 100644 --- a/drivers/misc/lkdtm/core.c +++ b/drivers/misc/lkdtm/core.c @@ -185,6 +185,9 @@ static const struct crashtype crashtypes[] = { CRASHTYPE(USERCOPY_KERNEL), CRASHTYPE(USERCOPY_KERNEL_DS), CRASHTYPE(STACKLEAK_ERASING), + CRASHTYPE(XPFO_READ_USER), + CRASHTYPE(XPFO_READ_USER_HUGE), + CRASHTYPE(XPFO_SMP), }; diff --git a/drivers/misc/lkdtm/lkdtm.h b/drivers/misc/lkdtm/lkdtm.h index 3c6fd327e166..6b31ff0c7f8f 100644 --- a/drivers/misc/lkdtm/lkdtm.h +++ b/drivers/misc/lkdtm/lkdtm.h @@ -87,4 +87,9 @@ void lkdtm_USERCOPY_KERNEL_DS(void); /* lkdtm_stackleak.c */ void lkdtm_STACKLEAK_ERASING(void); +/* lkdtm_xpfo.c */ +void lkdtm_XPFO_READ_USER(void); +void lkdtm_XPFO_READ_USER_HUGE(void); +void lkdtm_XPFO_SMP(void); + #endif diff --git a/drivers/misc/lkdtm/xpfo.c b/drivers/misc/lkdtm/xpfo.c new file mode 100644 index 000000000000..d903063bdd0b --- /dev/null +++ b/drivers/misc/lkdtm/xpfo.c @@ -0,0 +1,194 @@ +/* + * This is for all the tests related to XPFO (eXclusive Page Frame Ownership). + */ + +#include "lkdtm.h" + +#include +#include +#include +#include +#include + +#include +#include + +#define XPFO_DATA 0xdeadbeef + +static unsigned long do_map(unsigned long flags) +{ + unsigned long user_addr, user_data = XPFO_DATA; + + user_addr = vm_mmap(NULL, 0, PAGE_SIZE, + PROT_READ | PROT_WRITE | PROT_EXEC, + flags, 0); + if (user_addr >= TASK_SIZE) { + pr_warn("Failed to allocate user memory\n"); + return 0; + } + + if (copy_to_user((void __user *)user_addr, &user_data, + sizeof(user_data))) { + pr_warn("copy_to_user failed\n"); + goto free_user; + } + + return user_addr; + +free_user: + vm_munmap(user_addr, PAGE_SIZE); + return 0; +} + +static unsigned long *user_to_kernel(unsigned long user_addr) +{ + phys_addr_t phys_addr; + void *virt_addr; + + phys_addr = user_virt_to_phys(user_addr); + if (!phys_addr) { + pr_warn("Failed to get physical address of user memory\n"); + return NULL; + } + + virt_addr = phys_to_virt(phys_addr); + if (phys_addr != virt_to_phys(virt_addr)) { + pr_warn("Physical address of user memory seems incorrect\n"); + return NULL; + } + + return virt_addr; +} + +static void read_map(unsigned long *virt_addr) +{ + pr_info("Attempting bad read from kernel address %p\n", virt_addr); + if (*(unsigned long *)virt_addr == XPFO_DATA) + pr_err("FAIL: Bad read succeeded?!\n"); + else + pr_err("FAIL: Bad read didn't fail but data is incorrect?!\n"); +} + +static void read_user_with_flags(unsigned long flags) +{ + unsigned long user_addr, *kernel; + + user_addr = do_map(flags); + if (!user_addr) { + pr_err("FAIL: map failed\n"); + return; + } + + kernel = user_to_kernel(user_addr); + if (!kernel) { + pr_err("FAIL: user to kernel conversion failed\n"); + goto free_user; + } + + read_map(kernel); + +free_user: + vm_munmap(user_addr, PAGE_SIZE); +} + +/* Read from userspace via the kernel's linear map. */ +void lkdtm_XPFO_READ_USER(void) +{ + read_user_with_flags(MAP_PRIVATE | MAP_ANONYMOUS); +} + +void lkdtm_XPFO_READ_USER_HUGE(void) +{ + read_user_with_flags(MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB); +} + +struct smp_arg { + unsigned long *virt_addr; + unsigned int cpu; +}; + +static int smp_reader(void *parg) +{ + struct smp_arg *arg = parg; + unsigned long *virt_addr; + + if (arg->cpu != smp_processor_id()) { + pr_err("FAIL: scheduled on wrong CPU?\n"); + return 0; + } + + virt_addr = smp_cond_load_acquire(&arg->virt_addr, VAL != NULL); + read_map(virt_addr); + + return 0; +} + +#ifdef CONFIG_X86 +#define XPFO_SMP_KILLED SIGKILL +#elif CONFIG_ARM64 +#define XPFO_SMP_KILLED SIGSEGV +#else +#error unsupported arch +#endif + +/* The idea here is to read from the kernel's map on a different thread than + * did the mapping (and thus the TLB flushing), to make sure that the page + * faults on other cores too. + */ +void lkdtm_XPFO_SMP(void) +{ + unsigned long user_addr, *virt_addr; + struct task_struct *thread; + int ret; + struct smp_arg arg; + + if (num_online_cpus() < 2) { + pr_err("not enough to do a multi cpu test\n"); + return; + } + + arg.virt_addr = NULL; + arg.cpu = (smp_processor_id() + 1) % num_online_cpus(); + thread = kthread_create(smp_reader, &arg, "lkdtm_xpfo_test"); + if (IS_ERR(thread)) { + pr_err("couldn't create kthread? %ld\n", PTR_ERR(thread)); + return; + } + + kthread_bind(thread, arg.cpu); + get_task_struct(thread); + wake_up_process(thread); + + user_addr = do_map(MAP_PRIVATE | MAP_ANONYMOUS); + if (!user_addr) + goto kill_thread; + + virt_addr = user_to_kernel(user_addr); + if (!virt_addr) { + /* + * let's store something that will fail, so we can unblock the + * thread + */ + smp_store_release(&arg.virt_addr, &arg); + goto free_user; + } + + smp_store_release(&arg.virt_addr, virt_addr); + + /* there must be a better way to do this. */ + while (1) { + if (thread->exit_state) + break; + msleep_interruptible(100); + } + +free_user: + if (user_addr) + vm_munmap(user_addr, PAGE_SIZE); + +kill_thread: + ret = kthread_stop(thread); + if (ret != XPFO_SMP_KILLED) + pr_err("FAIL: thread wasn't killed: %d\n", ret); + put_task_struct(thread); +} From patchwork Thu Feb 14 00:01:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811421 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 56AEC13B4 for ; Thu, 14 Feb 2019 00:04:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 436732DCD3 for ; Thu, 14 Feb 2019 00:04:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 34FA12DD6F; Thu, 14 Feb 2019 00:04:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07FE22DCD3 for ; Thu, 14 Feb 2019 00:04:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8F068E0003; Wed, 13 Feb 2019 19:04:50 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C3F1C8E0001; Wed, 13 Feb 2019 19:04:50 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB8E48E0003; Wed, 13 Feb 2019 19:04:50 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 5669C8E0001 for ; Wed, 13 Feb 2019 19:04:50 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id l9so2912853plt.7 for ; Wed, 13 Feb 2019 16:04:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=TYTmjWk1Ud9xFO4O2CDjqPHQ3NzvMn4w1OlUR3qol7g=; b=cxzt8Z1MPCjQkgz/p7NNoNJroLuK804G8lKxHe7xnVDKbQ41s3OKbI3GoR4m9xMnTX FRV0sssBabu63dq9jLvnAHecqys+BI6SHB5A7adjf13PpNcNJ1GktBk21bdVVm6oNMht ruYP6dOdku46ZO1Ap/nqXw02Am/LocoZHAzpUUZSpGj0EVZuzK2Iz9tgj5Gt2Dx0VgXL ts8nlMWvketSQFfmF3lnWRL34J0VDU0PJzBjsF53lHcALumR9VfO0G7yTRnJWcft9Gbr GyO5/tcAiKdXMHuvIlxTF1O54BhPv2S2xJlNGGz0RFyfBwHOysl9ZiHduaU14VggsfcH xKag== X-Gm-Message-State: AHQUAua4WP2CH5xfSnQ2rqo7tMhkXMVw/M//MzZsaWp227lLUCg8HOJX oqgYUwIM+3DaP0ZiWgFQEMoOsofbltFV1VizedLw0CYHfLvVGqN2rA3G1dkrD7Ybns01dTBmMNJ bM4q60HOryexhTFRR/g5FflBNJ7b5fpQUDIhK9dVbL+FxufjlNnGnYx3JYVPh/u3kBA== X-Received: by 2002:a62:1f5c:: with SMTP id f89mr876698pff.137.1550102689953; Wed, 13 Feb 2019 16:04:49 -0800 (PST) X-Google-Smtp-Source: AHgI3IatOIMwZKvTV/0gIgw1/4fTn+yaqRyXOc/p6SUSo4/gcg2JdB5HmL9SpB8clQ8IvXgdH1x2 X-Received: by 2002:a62:1f5c:: with SMTP id f89mr876604pff.137.1550102689015; Wed, 13 Feb 2019 16:04:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102689; cv=none; d=google.com; s=arc-20160816; b=iLvAwNXpaUApFp/pQcV98O86sTD2mTce5mAeT9ccbH6MEV3S6TOC61pU0eiNTa8Q0Z Rx/CxwYe7Clx6ZO/ueFtkWrN9WcBDPslnW9M6TmtM+W4ekZL/lvVJl1pmR3O1+oDHfSh Qal1cBbvVW/exl8nSjepUPBrgHnWK/zxvHOz0WIg+rmTQsUo7WUfU8Ml8F1FLIqCF9X8 G+W9ThLHQk1wT7Bl2Ut0iRdGmX1CCzALc7BYXRkS8BNnttm+S+Qsms3R23l1MKLR1tKx CLcTzQiIb3HMyHTj7L2mkZFQOrhCGZXsieu73gEMi3suTD1wJZxI27AhGz1RjnOWRACD 9/yA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=TYTmjWk1Ud9xFO4O2CDjqPHQ3NzvMn4w1OlUR3qol7g=; b=ygkNdolWzwSvcGU52AF5BHHeMxDjTzKGEb/BYLPMa/okj4R1liyYFy/oFpkm2FwhQr iGox9thM9fsuLXoXQj78kravvIBmFAfevduD0N17MqZQDDuwKwIpNO5AOSoloE18yzTS 8IRF7wFgYQUO6cYfL/qW2GV1hgDjtbA9llfp5KD6kDmRdQrbClAcS4Mev+/e0V53hJ6n +eLrZtSWdBhNOinzDayWaJyGObGQpWHQ2kGegFkjD+DiZyoDLotNKmKN2mLTp4WbNhca T9ERsop2zqQIADedcVXzsSZBJGOnifgiDbxF7WzaA7iU7RQ238e9ZiClUB07vBo92hZ9 ZslA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=xwYy2aXo; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id o11si708738pls.374.2019.02.13.16.04.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:04:49 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=xwYy2aXo; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwjFp099399; Thu, 14 Feb 2019 00:02:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=TYTmjWk1Ud9xFO4O2CDjqPHQ3NzvMn4w1OlUR3qol7g=; b=xwYy2aXo7t5xJC9+w32YfLL7cjgc3iIpBAE0V7wYjGFAQ06Se7GHUmGKKd3a8t4ViTPM hgsMfdwTc6d6rvLVimRpcx4hxWiqZKD+nUkx1x33WRzAvkWQYxjujKv9YZ4u+vG2ZrRK TTxa/WMoUVhUwZz7hR+pi68WsOnVaDmCXKCF/3R5yqBQTLOj2LJn4XmBTIh74W10CKhs UxA1QSEodqXxzu7zAkdK1fGxw33J2M/yvMyFF+CSQBOMgasBZ7a33Dqq6g4Aspx/2pF5 Rsiuq3kqdpq0p0SsA48rdZvbLBUg/DgjD6G/t8Bot0UQ0KJBENA5cO5yi+dmHCm/oK5p 4g== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2qhree55nk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:29 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02Nfp026321 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:23 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E02Mau018822; Thu, 14 Feb 2019 00:02:22 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:21 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, "Vasileios P . Kemerlis" , Juerg Haefliger , Tycho Andersen , Marco Benatto , David Woodhouse Subject: [RFC PATCH v8 11/14] xpfo, mm: remove dependency on CONFIG_PAGE_EXTENSION Date: Wed, 13 Feb 2019 17:01:34 -0700 Message-Id: <465161d96733a9bda8dfbae9d2028912ee383faa.1550088114.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Julian Stecklina Instead of using the page extension debug feature, encode all information, we need for XPFO in struct page. This allows to get rid of some checks in the hot paths and there are also no pages anymore that are allocated before XPFO is enabled. Also make debugging aids configurable for maximum performance. Signed-off-by: Julian Stecklina Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis Cc: Juerg Haefliger Cc: Tycho Andersen Cc: Marco Benatto Cc: David Woodhouse Reviewed-by: Khalid Aziz --- include/linux/mm_types.h | 8 ++ include/linux/page-flags.h | 13 +++ include/linux/xpfo.h | 3 +- include/trace/events/mmflags.h | 10 ++- mm/page_alloc.c | 3 +- mm/page_ext.c | 4 - mm/xpfo.c | 159 ++++++++------------------------- security/Kconfig | 12 ++- 8 files changed, 80 insertions(+), 132 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2c471a2c43fa..d17d33f36a01 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -204,6 +204,14 @@ struct page { #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS int _last_cpupid; #endif + +#ifdef CONFIG_XPFO + /* Counts the number of times this page has been kmapped. */ + atomic_t xpfo_mapcount; + + /* Serialize kmap/kunmap of this page */ + spinlock_t xpfo_lock; +#endif } _struct_page_alignment; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 50ce1bddaf56..a532063f27b5 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -101,6 +101,10 @@ enum pageflags { #if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT) PG_young, PG_idle, +#endif +#ifdef CONFIG_XPFO + PG_xpfo_user, /* Page is allocated to user-space */ + PG_xpfo_unmapped, /* Page is unmapped from the linear map */ #endif __NR_PAGEFLAGS, @@ -398,6 +402,15 @@ TESTCLEARFLAG(Young, young, PF_ANY) PAGEFLAG(Idle, idle, PF_ANY) #endif +#ifdef CONFIG_XPFO +PAGEFLAG(XpfoUser, xpfo_user, PF_ANY) +TESTCLEARFLAG(XpfoUser, xpfo_user, PF_ANY) +TESTSETFLAG(XpfoUser, xpfo_user, PF_ANY) +PAGEFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) +TESTCLEARFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) +TESTSETFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) +#endif + /* * On an anonymous page mapped into a user virtual memory area, * page->mapping points to its anon_vma, not to a struct address_space; diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 117869991d5b..1dd590ff1a1f 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -28,7 +28,7 @@ struct page; #include -extern struct page_ext_operations page_xpfo_ops; +void xpfo_init_single_page(struct page *page); void set_kpte(void *kaddr, struct page *page, pgprot_t prot); void xpfo_dma_map_unmap_area(bool map, const void *addr, size_t size, @@ -57,6 +57,7 @@ phys_addr_t user_virt_to_phys(unsigned long addr); #else /* !CONFIG_XPFO */ +static inline void xpfo_init_single_page(struct page *page) { } static inline void xpfo_kmap(void *kaddr, struct page *page) { } static inline void xpfo_kunmap(void *kaddr, struct page *page) { } static inline void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) { } diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index a1675d43777e..6bb000bb366f 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -79,6 +79,12 @@ #define IF_HAVE_PG_IDLE(flag,string) #endif +#ifdef CONFIG_XPFO +#define IF_HAVE_PG_XPFO(flag,string) ,{1UL << flag, string} +#else +#define IF_HAVE_PG_XPFO(flag,string) +#endif + #define __def_pageflag_names \ {1UL << PG_locked, "locked" }, \ {1UL << PG_waiters, "waiters" }, \ @@ -105,7 +111,9 @@ IF_HAVE_PG_MLOCK(PG_mlocked, "mlocked" ) \ IF_HAVE_PG_UNCACHED(PG_uncached, "uncached" ) \ IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \ IF_HAVE_PG_IDLE(PG_young, "young" ) \ -IF_HAVE_PG_IDLE(PG_idle, "idle" ) +IF_HAVE_PG_IDLE(PG_idle, "idle" ) \ +IF_HAVE_PG_XPFO(PG_xpfo_user, "xpfo_user" ) \ +IF_HAVE_PG_XPFO(PG_xpfo_unmapped, "xpfo_unmapped" ) \ #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 08e277790b5f..d00382b20001 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1024,6 +1024,7 @@ static __always_inline bool free_pages_prepare(struct page *page, if (bad) return false; + xpfo_free_pages(page, order); page_cpupid_reset_last(page); page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); @@ -1038,7 +1039,6 @@ static __always_inline bool free_pages_prepare(struct page *page, kernel_poison_pages(page, 1 << order, 0); kernel_map_pages(page, 1 << order, 0); kasan_free_pages(page, order); - xpfo_free_pages(page, order); return true; } @@ -1191,6 +1191,7 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, if (!is_highmem_idx(zone)) set_page_address(page, __va(pfn << PAGE_SHIFT)); #endif + xpfo_init_single_page(page); } #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT diff --git a/mm/page_ext.c b/mm/page_ext.c index 38e5013dcb9a..ae44f7adbe07 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -8,7 +8,6 @@ #include #include #include -#include /* * struct page extension @@ -69,9 +68,6 @@ static struct page_ext_operations *page_ext_ops[] = { #if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT) &page_idle_ops, #endif -#ifdef CONFIG_XPFO - &page_xpfo_ops, -#endif }; static unsigned long total_usage; diff --git a/mm/xpfo.c b/mm/xpfo.c index 150784ae0f08..dc03c423c52f 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -17,113 +17,58 @@ #include #include #include -#include #include #include -/* XPFO page state flags */ -enum xpfo_flags { - XPFO_PAGE_USER, /* Page is allocated to user-space */ - XPFO_PAGE_UNMAPPED, /* Page is unmapped from the linear map */ -}; - -/* Per-page XPFO house-keeping data */ -struct xpfo { - unsigned long flags; /* Page state */ - bool inited; /* Map counter and lock initialized */ - atomic_t mapcount; /* Counter for balancing map/unmap requests */ - spinlock_t maplock; /* Lock to serialize map/unmap requests */ -}; - -DEFINE_STATIC_KEY_FALSE(xpfo_inited); - -static bool xpfo_disabled __initdata; +DEFINE_STATIC_KEY_TRUE(xpfo_inited); static int __init noxpfo_param(char *str) { - xpfo_disabled = true; + static_branch_disable(&xpfo_inited); return 0; } early_param("noxpfo", noxpfo_param); -static bool __init need_xpfo(void) -{ - if (xpfo_disabled) { - pr_info("XPFO disabled\n"); - return false; - } - - return true; -} - -static void init_xpfo(void) -{ - pr_info("XPFO enabled\n"); - static_branch_enable(&xpfo_inited); -} - -struct page_ext_operations page_xpfo_ops = { - .size = sizeof(struct xpfo), - .need = need_xpfo, - .init = init_xpfo, -}; - bool __init xpfo_enabled(void) { - return !xpfo_disabled; + if (!static_branch_unlikely(&xpfo_inited)) + return false; + else + return true; } -EXPORT_SYMBOL(xpfo_enabled); -static inline struct xpfo *lookup_xpfo(struct page *page) +void __meminit xpfo_init_single_page(struct page *page) { - struct page_ext *page_ext = lookup_page_ext(page); - - if (unlikely(!page_ext)) { - WARN(1, "xpfo: failed to get page ext"); - return NULL; - } - - return (void *)page_ext + page_xpfo_ops.offset; + spin_lock_init(&page->xpfo_lock); } void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) { int i, flush_tlb = 0; - struct xpfo *xpfo; if (!static_branch_unlikely(&xpfo_inited)) return; for (i = 0; i < (1 << order); i++) { - xpfo = lookup_xpfo(page + i); - if (!xpfo) - continue; - - WARN(test_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags), - "xpfo: unmapped page being allocated\n"); - - /* Initialize the map lock and map counter */ - if (unlikely(!xpfo->inited)) { - spin_lock_init(&xpfo->maplock); - atomic_set(&xpfo->mapcount, 0); - xpfo->inited = true; - } - WARN(atomic_read(&xpfo->mapcount), - "xpfo: already mapped page being allocated\n"); - +#ifdef CONFIG_XPFO_DEBUG + BUG_ON(PageXpfoUser(page + i)); + BUG_ON(PageXpfoUnmapped(page + i)); + BUG_ON(spin_is_locked(&(page + i)->xpfo_lock)); + BUG_ON(atomic_read(&(page + i)->xpfo_mapcount)); +#endif if ((gfp & GFP_HIGHUSER) == GFP_HIGHUSER) { /* * Tag the page as a user page and flush the TLB if it * was previously allocated to the kernel. */ - if (!test_and_set_bit(XPFO_PAGE_USER, &xpfo->flags)) + if (!TestSetPageXpfoUser(page + i)) flush_tlb = 1; } else { /* Tag the page as a non-user (kernel) page */ - clear_bit(XPFO_PAGE_USER, &xpfo->flags); + ClearPageXpfoUser(page + i); } } @@ -134,27 +79,21 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) void xpfo_free_pages(struct page *page, int order) { int i; - struct xpfo *xpfo; if (!static_branch_unlikely(&xpfo_inited)) return; for (i = 0; i < (1 << order); i++) { - xpfo = lookup_xpfo(page + i); - if (!xpfo || unlikely(!xpfo->inited)) { - /* - * The page was allocated before page_ext was - * initialized, so it is a kernel page. - */ - continue; - } +#ifdef CONFIG_XPFO_DEBUG + BUG_ON(atomic_read(&(page + i)->xpfo_mapcount)); +#endif /* * Map the page back into the kernel if it was previously * allocated to user space. */ - if (test_and_clear_bit(XPFO_PAGE_USER, &xpfo->flags)) { - clear_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); + if (TestClearPageXpfoUser(page + i)) { + ClearPageXpfoUnmapped(page + i); set_kpte(page_address(page + i), page + i, PAGE_KERNEL); } @@ -163,84 +102,56 @@ void xpfo_free_pages(struct page *page, int order) void xpfo_kmap(void *kaddr, struct page *page) { - struct xpfo *xpfo; - if (!static_branch_unlikely(&xpfo_inited)) return; - xpfo = lookup_xpfo(page); - - /* - * The page was allocated before page_ext was initialized (which means - * it's a kernel page) or it's allocated to the kernel, so nothing to - * do. - */ - if (!xpfo || unlikely(!xpfo->inited) || - !test_bit(XPFO_PAGE_USER, &xpfo->flags)) + if (!PageXpfoUser(page)) return; - spin_lock(&xpfo->maplock); + spin_lock(&page->xpfo_lock); /* * The page was previously allocated to user space, so map it back * into the kernel. No TLB flush required. */ - if ((atomic_inc_return(&xpfo->mapcount) == 1) && - test_and_clear_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags)) + if ((atomic_inc_return(&page->xpfo_mapcount) == 1) && + TestClearPageXpfoUnmapped(page)) set_kpte(kaddr, page, PAGE_KERNEL); - spin_unlock(&xpfo->maplock); + spin_unlock(&page->xpfo_lock); } EXPORT_SYMBOL(xpfo_kmap); void xpfo_kunmap(void *kaddr, struct page *page) { - struct xpfo *xpfo; - if (!static_branch_unlikely(&xpfo_inited)) return; - xpfo = lookup_xpfo(page); - - /* - * The page was allocated before page_ext was initialized (which means - * it's a kernel page) or it's allocated to the kernel, so nothing to - * do. - */ - if (!xpfo || unlikely(!xpfo->inited) || - !test_bit(XPFO_PAGE_USER, &xpfo->flags)) + if (!PageXpfoUser(page)) return; - spin_lock(&xpfo->maplock); + spin_lock(&page->xpfo_lock); /* * The page is to be allocated back to user space, so unmap it from the * kernel, flush the TLB and tag it as a user page. */ - if (atomic_dec_return(&xpfo->mapcount) == 0) { - WARN(test_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags), - "xpfo: unmapping already unmapped page\n"); - set_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); + if (atomic_dec_return(&page->xpfo_mapcount) == 0) { +#ifdef CONFIG_XPFO_DEBUG + BUG_ON(PageXpfoUnmapped(page)); +#endif + SetPageXpfoUnmapped(page); set_kpte(kaddr, page, __pgprot(0)); xpfo_flush_kernel_tlb(page, 0); } - spin_unlock(&xpfo->maplock); + spin_unlock(&page->xpfo_lock); } EXPORT_SYMBOL(xpfo_kunmap); bool xpfo_page_is_unmapped(struct page *page) { - struct xpfo *xpfo; - - if (!static_branch_unlikely(&xpfo_inited)) - return false; - - xpfo = lookup_xpfo(page); - if (unlikely(!xpfo) && !xpfo->inited) - return false; - - return test_bit(XPFO_PAGE_UNMAPPED, &xpfo->flags); + return PageXpfoUnmapped(page); } EXPORT_SYMBOL(xpfo_page_is_unmapped); diff --git a/security/Kconfig b/security/Kconfig index 8d0e4e303551..c7c581bac963 100644 --- a/security/Kconfig +++ b/security/Kconfig @@ -13,7 +13,6 @@ config XPFO bool "Enable eXclusive Page Frame Ownership (XPFO)" default n depends on ARCH_SUPPORTS_XPFO - select PAGE_EXTENSION help This option offers protection against 'ret2dir' kernel attacks. When enabled, every time a page frame is allocated to user space, it @@ -25,6 +24,17 @@ config XPFO If in doubt, say "N". +config XPFO_DEBUG + bool "Enable debugging of XPFO" + default n + depends on XPFO + help + Enables additional checking of XPFO data structures that help find + bugs in the XPFO implementation. This option comes with a slight + performance cost. + + If in doubt, say "N". + config SECURITY_DMESG_RESTRICT bool "Restrict unprivileged access to the kernel syslog" default n From patchwork Thu Feb 14 00:01:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811417 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 62E7F13B4 for ; Thu, 14 Feb 2019 00:04:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 509F52DD5B for ; Thu, 14 Feb 2019 00:04:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 438DC2DD6F; Thu, 14 Feb 2019 00:04:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB3372DD5B for ; Thu, 14 Feb 2019 00:04:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D51C68E0002; Wed, 13 Feb 2019 19:04:44 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D01668E0001; Wed, 13 Feb 2019 19:04:44 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7BA68E0002; Wed, 13 Feb 2019 19:04:44 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 70FEF8E0001 for ; Wed, 13 Feb 2019 19:04:44 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id a72so1285283pfj.19 for ; Wed, 13 Feb 2019 16:04:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=sphrv7BG0r4M1etaGHVdl0ES/afYZFzSiF3kwKx/qUc=; b=o3OANTUU5h9sEJ8/qtNGY54K3qFrR1dW33WBP+w0mT6BMQco7Eu0ZmG/6EaTEWFrUC 2E3OZR24YSn6d3woFFTpNoeVq+Oa2sjwOTqBOQ3xWWSEUdjRztGPXHf89GEkE/+EpMMA 5FcmbBYyHCL9TP0oIizwQCkHZI6ZCo1HAp7EcO0nIdYuM81f9kb/LtMahDAXfx7jYH6t /jx8Ffd7/fJAHs9njYaIo+H3/1Sm/JcExxPSIsvSabX5sQO0Nj++RGuefTuVlo+v3CsI cErY95mazV1sJkes2Bs2vE8PNnhwgUweGtpxwNP7BWcrPNFmr+RxxvAxMlMuAvUEkSz1 ewoA== X-Gm-Message-State: AHQUAuZFc/cKbvePyqs324H9uKmtjnkMMXTTY3q2PUNjfb+2YKFdTfs3 kmrNd6rIfIMVLh/I3xrzSUiJmIJehjlAGthn+EkAztjxMgHXLtI2t7N/fe4nTiRHseeq3pbmkFx PiAZBXTFFk2MccIKzFBNlPGdAz+3+btZEiL4vZe2ihclJW1WrMchULYC5DThIj773YQ== X-Received: by 2002:a62:560f:: with SMTP id k15mr847520pfb.231.1550102684064; Wed, 13 Feb 2019 16:04:44 -0800 (PST) X-Google-Smtp-Source: AHgI3IbQ2m6yRlMun+ynZ8lGjJ0jr00kKDBZtXMGJGzlOEK/zd5IxWRxjyxU24rnsQAPOVnVfMQz X-Received: by 2002:a62:560f:: with SMTP id k15mr847461pfb.231.1550102683359; Wed, 13 Feb 2019 16:04:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102683; cv=none; d=google.com; s=arc-20160816; b=ZVNMIdg9RHAxkM7t8wdoaDzXT0Xo1MUk9GqOyLUEMQFUIN07RsYmbyI47ItnOWkLxF BZvVdxLIiWAM2PAfj/ls1YASv/CfgiBDdbkQSfiI5oR7Gri1WoUGkH92/epw8iIDxXOO fy0ocoLwECyCLpN5CYK9mwEk21d17gxFTUwKeM7TNBvS4W0fO4aJe8BexRgSptlIzf6E Dru+mw2klD17SCHwy9K7e8ALdcD97rtJnYq3LL1yjy8z3jJNnhpW426SY3+oPsWrumT2 N3KUfrXvJZtoNKgSIFWQmDlpPQ9JoyzRv4Nf1pO9751cbr5Fr+4PkuBOBR4V76Tnoym9 tWYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=sphrv7BG0r4M1etaGHVdl0ES/afYZFzSiF3kwKx/qUc=; b=QWVatsAzwo3Ud1AVzAVEs2vAiP33sPdrcXJsiuNzTFxW3eCx8Elo0AN1P+WG1QiAeJ yYLcWdZa/FZXA/ZezM0j/t8ANU2qZIo0pgT2wvy5PjQkP4N6n8Q+Tmm+2eEVoWBfcuGe qBCNfJ5y0U0DzLZJ++WpO0waxkZ1rLAhl56++33+gmUXPslb0zZ+uk1D1xny6kozEhIY MPjrCIQnbVewYNh5OA5+BZhQ7VHmjrGg7mpCEe6XZrd71IXJcg5hSSNS/ybElc+Qosjr ujrKLbc2vFR45kZu7tOj6wJaqe+oxvWo6vP4NMr0Do5jWzuYOHiU+3CEn7Jq777VPgzH LiEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=rq32CuBk; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id 3si754247plv.258.2019.02.13.16.04.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:04:43 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=rq32CuBk; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNxBhK099616; Thu, 14 Feb 2019 00:02:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=sphrv7BG0r4M1etaGHVdl0ES/afYZFzSiF3kwKx/qUc=; b=rq32CuBk5X2+Xev+zFjtWbTbukjVYgp052VQoUtKNOTE4Cda9Cy4Byz7X7TyDaf3TQ2I 6Xq6h/PXDcwuH55t84HlJx++IE29KeYzxGnud9iAcqceAw++4jace2g3nP5q9+tVQuDP JKmeME1MBZxG+r3eijrKz+I+/dFivoWQvUM6OVeVIggnFDVOUMogq4PMOpJhjY4t+4Ty T/Rv6uXJl+x2rhUlCA5dHNltlK7OXjBzMdts964clZe9v6/2TNndXTWUZGdIWr0yX2HO Q8vLkyWbeiF65glX8gZob//mF1Popwb6C6Z2TLKwVkfbusnbRd8Lf2e8RxtLEG/+nZ1T cg== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2qhree55ng-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:27 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02QQh001435 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:26 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02OhA014525; Thu, 14 Feb 2019 00:02:25 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:24 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Khalid Aziz , "Vasileios P . Kemerlis" , Juerg Haefliger , Tycho Andersen , Marco Benatto , David Woodhouse Subject: [RFC PATCH v8 12/14] xpfo, mm: optimize spinlock usage in xpfo_kunmap Date: Wed, 13 Feb 2019 17:01:35 -0700 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Julian Stecklina Only the xpfo_kunmap call that needs to actually unmap the page needs to be serialized. We need to be careful to handle the case, where after the atomic decrement of the mapcount, a xpfo_kmap increased the mapcount again. In this case, we can safely skip modifying the page table. Model-checked with up to 4 concurrent callers with Spin. Signed-off-by: Julian Stecklina Signed-off-by: Khalid Aziz Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis Cc: Juerg Haefliger Cc: Tycho Andersen Cc: Marco Benatto Cc: David Woodhouse --- mm/xpfo.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/mm/xpfo.c b/mm/xpfo.c index dc03c423c52f..5157cbebce4b 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -124,28 +124,35 @@ EXPORT_SYMBOL(xpfo_kmap); void xpfo_kunmap(void *kaddr, struct page *page) { + bool flush_tlb = false; + if (!static_branch_unlikely(&xpfo_inited)) return; if (!PageXpfoUser(page)) return; - spin_lock(&page->xpfo_lock); - /* * The page is to be allocated back to user space, so unmap it from the * kernel, flush the TLB and tag it as a user page. */ if (atomic_dec_return(&page->xpfo_mapcount) == 0) { -#ifdef CONFIG_XPFO_DEBUG - BUG_ON(PageXpfoUnmapped(page)); -#endif - SetPageXpfoUnmapped(page); - set_kpte(kaddr, page, __pgprot(0)); - xpfo_flush_kernel_tlb(page, 0); + spin_lock(&page->xpfo_lock); + + /* + * In the case, where we raced with kmap after the + * atomic_dec_return, we must not nuke the mapping. + */ + if (atomic_read(&page->xpfo_mapcount) == 0) { + SetPageXpfoUnmapped(page); + set_kpte(kaddr, page, __pgprot(0)); + flush_tlb = true; + } + spin_unlock(&page->xpfo_lock); } - spin_unlock(&page->xpfo_lock); + if (flush_tlb) + xpfo_flush_kernel_tlb(page, 0); } EXPORT_SYMBOL(xpfo_kunmap); From patchwork Thu Feb 14 00:01:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811377 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 090426C2 for ; Thu, 14 Feb 2019 00:03:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E8B1F2D923 for ; Thu, 14 Feb 2019 00:03:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DBF502DBB6; Thu, 14 Feb 2019 00:03:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C0E92D923 for ; Thu, 14 Feb 2019 00:03:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 390428E000B; Wed, 13 Feb 2019 19:02:59 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 368A28E0005; Wed, 13 Feb 2019 19:02:59 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11C978E000D; Wed, 13 Feb 2019 19:02:59 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id C52A28E000B for ; Wed, 13 Feb 2019 19:02:58 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id f5so2878506pgh.14 for ; Wed, 13 Feb 2019 16:02:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=lijI2dkmHOz8IBbMljD0kgh/GfNNHRB/Ww5ZCLiDpC8=; b=PQpXkCFKv11Pc2WsTE42kzC4WJkmmr+KY2Lk85LInIUxtKrrPKX/z2X2rp6VDM2rYf VvHQWr8UHFdTONiUzIsE+XLLL1BCwvv7yCdcpH406VeKZcq6M8hG42Ox6xlHlgKBz9eM a8iGPd/Q/jnSZ0u68LAA4Jj556yL2mTPKGZcjc6WgWS1sX/HLc3WYyMh5HkDBVMVJKHN G+Pr6Lsh7HJ084rap09+w+CqJMqUMatIL5lPRd6/SkyjA/bjkDWc2nRogBpW/seglBCK /WZqObiL08z167YIIWnwFKzBdynwpkw96fcBGf/vOuAn4pxLhLfpgFrZannPjLyrKVAU dMpw== X-Gm-Message-State: AHQUAuZhWYGASrUglVaqFRWHV7+Pa32YqBzVePpBgu8yJbX/UX5epJF8 17qK/Pw+P+FHzFX7bG92Bn2u6+l/BS6cO1/fZcBwZKWTmhuntRvNliyzTNxpMlipM0w7xMXQXPQ Bj5sSQKeuXeF7aMM0ihkkMIzU0gQxRuTnRsXxwPglQre1/yUub+r0U1ek7vflM/Ordg== X-Received: by 2002:a63:134f:: with SMTP id 15mr824723pgt.19.1550102578416; Wed, 13 Feb 2019 16:02:58 -0800 (PST) X-Google-Smtp-Source: AHgI3IbRT1ESkPwtpXFfnAPI4TixmN1AbfGxXFmHG3DaSLL/PrUznMDITRhuS9y79sWI+ecDhZtF X-Received: by 2002:a63:134f:: with SMTP id 15mr824641pgt.19.1550102577486; Wed, 13 Feb 2019 16:02:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102577; cv=none; d=google.com; s=arc-20160816; b=NhhcThI1p//ScKGoOG5tO/V5U6KFPvYUzH8r+nsVyu2T5ikCMajllgSjA3PcEwwJx3 MST1fLoW5NgdUPU9PBeOnmhaTZMooBKw7m6TuNOHrjnR3Pmdhb7hyUzl+vOOl7ABU+J2 KztSFWo+Wvd6mQmVI+t4nShdYLBYtuyMEhHFLc3PXHfQw1DyjXu2VSDyz0KrlVOZWb+l z0oA7J3VKJIgcQjkFKhVU+9odUVKE3+aJbnES0essO3zc7yLoQFlyd/4cVbJJH6vZeEJ g1rLtcZ2f3DyoGjJ1Wq1GKCkKGiUGetrJdmKcheeeNqvDAh/WISOnXk1vlgojseQc9Wx Fvvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=lijI2dkmHOz8IBbMljD0kgh/GfNNHRB/Ww5ZCLiDpC8=; b=dQJISXBH8g9kyj3gJbtl/6D/ZWRZUsDJNhyoRurmzFX4rpN75fmbQL/KsugGM4VMiV RvFOBg8Yo6Z4Fa5I+c2rbnO5UvvP2PEseFDa3/rkqIygfgNWHP4+dAWbEC5v7ECXX7dx sN/7uDMefgPHUsQ8ol26RLpwGTggB796bJhvyVSzDeTCE9ytX2fy/tkMd2OSEczfzON/ b2LjuXnEUUcrUZcqD6TrUYYonWjmatbwFneg7F3Ia5DFlc1hJxdJknvfEc5J66SHCeo2 BHFe5269tRfNhAM5QxPSEyHjzXOUXGRrK7gVpwTybS7ACrnWBISKL/xSXI9MB2xLsM1v WgSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Eq0NRf9C; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id u10si678993pgi.515.2019.02.13.16.02.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:02:57 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=Eq0NRf9C; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNworQ099417; Thu, 14 Feb 2019 00:02:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=lijI2dkmHOz8IBbMljD0kgh/GfNNHRB/Ww5ZCLiDpC8=; b=Eq0NRf9CvVjMRyqL/pJwd5gMUZxni+bHvlB5+zwob5e+xiwZT4ra0AyBxJ92DivdFLna 97cYTFa2qe9QQ6wYiXHornwjF1eTBScwRD03REBGbSCxrGe1sHulQ2gDVu4xX3mxM/36 3SvBa2fAgAWdU0U1FEda7/JATU3IVFLn4IveEsG+K9h5CqxCBFKvEGRJGGdkfDkYamdp PsnFndFcwiVrKpuBYeXNhI81MuZO4NIfCeGp/zunHR7sbKkLh0NDGuO1bP7hoZRAZp8q tczxkn156C41JsFvXIo8bWbOeUspWxI2rXu9qB8VJta8M6C6Ic/459RKF/2UsrhhQ7Ps Gw== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2qhree55nh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:28 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02R97032757 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:27 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E02RfT018893; Thu, 14 Feb 2019 00:02:27 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:26 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Khalid Aziz , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v8 13/14] xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only) Date: Wed, 13 Feb 2019 17:01:36 -0700 Message-Id: <98134cb73e911b2f0b59ffb76243a7777963d218.1550088114.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP XPFO flushes kernel space TLB entries for pages that are now mapped in userspace on not only the current CPU but also all other CPUs synchronously. Processes on each core allocating pages causes a flood of IPI messages to all other cores to flush TLB entries. Many of these messages are to flush the entire TLB on the core if the number of entries being flushed from local core exceeds tlb_single_page_flush_ceiling. The cost of TLB flush caused by unmapping pages from physmap goes up dramatically on machines with high core count. This patch flushes relevant TLB entries for current process or entire TLB depending upon number of entries for the current CPU and posts a pending TLB flush on all other CPUs when a page is unmapped from kernel space and mapped in userspace. Each core checks the pending TLB flush flag for itself on every context switch, flushes its TLB if the flag is set and clears it. This patch potentially aggregates multiple TLB flushes into one. This has very significant impact especially on machines with large core counts. To illustrate this, kernel was compiled with -j on two classes of machines - a server with high core count and large amount of memory, and a desktop class machine with more modest specs. System time from "make -j" from vanilla 4.20 kernel, 4.20 with XPFO patches before applying this patch and after applying this patch are below: Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM make -j60 all 4.20 950.966s 4.20+XPFO 25073.169s 26.366x 4.20+XPFO+Deferred flush 1372.874s 1.44x Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM make -j4 all 4.20 607.671s 4.20+XPFO 1588.646s 2.614x 4.20+XPFO+Deferred flush 803.989s 1.32x This patch could use more optimization. Batching more TLB entry flushes, as was suggested for earlier version of these patches, can help reduce these cases. This same code should be implemented for other architectures as well once finalized. Signed-off-by: Khalid Aziz --- arch/x86/include/asm/tlbflush.h | 1 + arch/x86/mm/tlb.c | 38 +++++++++++++++++++++++++++++++++ arch/x86/mm/xpfo.c | 2 +- 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index f4204bf377fc..92d23629d01d 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -561,6 +561,7 @@ extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); +extern void xpfo_flush_tlb_kernel_range(unsigned long start, unsigned long end); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 03b6b4c2238d..c907b643eecb 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -35,6 +35,15 @@ */ #define LAST_USER_MM_IBPB 0x1UL +/* + * When a full TLB flush is needed to flush stale TLB entries + * for pages that have been mapped into userspace and unmapped + * from kernel space, this TLB flush will be delayed until the + * task is scheduled on that CPU. Keep track of CPUs with + * pending full TLB flush forced by xpfo. + */ +static cpumask_t pending_xpfo_flush; + /* * We get here when we do something requiring a TLB invalidation * but could not go invalidate all of the contexts. We do the @@ -319,6 +328,15 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, __flush_tlb_all(); } #endif + + /* If there is a pending TLB flush for this CPU due to XPFO + * flush, do it now. + */ + if (cpumask_test_and_clear_cpu(cpu, &pending_xpfo_flush)) { + count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED); + __flush_tlb_all(); + } + this_cpu_write(cpu_tlbstate.is_lazy, false); /* @@ -801,6 +819,26 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) } } +void xpfo_flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + struct cpumask tmp_mask; + + /* Balance as user space task's flush, a bit conservative */ + if (end == TLB_FLUSH_ALL || + (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) { + do_flush_tlb_all(NULL); + } else { + struct flush_tlb_info info; + + info.start = start; + info.end = end; + do_kernel_range_flush(&info); + } + cpumask_setall(&tmp_mask); + cpumask_clear_cpu(smp_processor_id(), &tmp_mask); + cpumask_or(&pending_xpfo_flush, &pending_xpfo_flush, &tmp_mask); +} + void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { struct flush_tlb_info info = { diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c index e13b99019c47..d3833532bfdc 100644 --- a/arch/x86/mm/xpfo.c +++ b/arch/x86/mm/xpfo.c @@ -115,7 +115,7 @@ inline void xpfo_flush_kernel_tlb(struct page *page, int order) return; } - flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); + xpfo_flush_tlb_kernel_range(kaddr, kaddr + (1 << order) * size); } /* Convert a user space virtual address to a physical address. From patchwork Thu Feb 14 00:01:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10811385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BCB4A746 for ; Thu, 14 Feb 2019 00:03:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAEE22D9EE for ; Thu, 14 Feb 2019 00:03:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E5832DE30; Thu, 14 Feb 2019 00:03:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5A542D9EE for ; Thu, 14 Feb 2019 00:03:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90FFD8E000E; Wed, 13 Feb 2019 19:03:11 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8BDF38E0005; Wed, 13 Feb 2019 19:03:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7862D8E000E; Wed, 13 Feb 2019 19:03:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 2B63B8E0005 for ; Wed, 13 Feb 2019 19:03:11 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id o23so2934995pll.0 for ; Wed, 13 Feb 2019 16:03:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:in-reply-to:references; bh=Ov6dF12doUxji6ePINUtaYIJxqhm2/VTB1zpiOjT1tA=; b=SjrTDlLh/cFlnIEaXChbwHfsziqg/HikigVY96AjT6IKcEb5p30jMKREYDOfTlXE+X UHp4QpYjKmUwIg73Koqs9xrTuZlS3g3JoS0USMUgrUJcUyqocVQYxZxmXHFpErp08Tyo mUKBkE9ca5DTJwHECWvFcEe8Vm3CG9rvcZ1qLEPHCToMwAHArCDMVmEF8wABbDJVu7lF wfrwpjjmmL2+zipFzDoVdvwvbD5Q7nokQJI4qjQqPu8grYYp9dlZmlmjqT8ISe3P1UJN KAyvU3O2OWmdGysSbmbHayJftTHzsnEbHnrfVVrrfdXkNps8SWFeQJyxnRYofuqdWzr6 mH0A== X-Gm-Message-State: AHQUAuZHxAHSV0AykT6e+3Sx7BHcOKIvjOmgHwEzHIzflbM4d7SUSlgz iVUc6M74iuEzyb81+abvtUWV3kAkwPCsuCBDRyXB6+yMsuFBa07PZ7ynn6dqktkqiI0cJL/JGPc 04ZsvTWY1LdNZOoSECg6+iR/WM6nxLqeqlg5UV+lkCeHCW/LF0KPXRYf/lol7zkNWsw== X-Received: by 2002:a17:902:6bc3:: with SMTP id m3mr953330plt.24.1550102590794; Wed, 13 Feb 2019 16:03:10 -0800 (PST) X-Google-Smtp-Source: AHgI3IYnhDXDmQeR70fzHO6nCgYYEZLegMK0E1UsDUIK2BFSuGa9zGyO+9sE+pi6Lm/e4OidV6ZD X-Received: by 2002:a17:902:6bc3:: with SMTP id m3mr953239plt.24.1550102589925; Wed, 13 Feb 2019 16:03:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550102589; cv=none; d=google.com; s=arc-20160816; b=gyzNTKKmWwXcB58rMoNns+R8WuRAbq7XE5MtrnZ1R+YvdHeQCg34fNiM0igO17OWmD soYwTEiuXMABXjb5MvTrVdaPW7iV6qF12Xo+oPpOU0324iV7QaLd1RnyndDsmnDkMajc qKTnv3CFtzsAhvC59OFL8VleEO4SeMuVH4GkFd01CbIc+LnGG3Rjz4ej9BG5Ee20LR2d wNycXBxASf2MlbXxpsnXsCyfI7wCyEYq9ruxB8QuzFG0mMIr47sEnMKrhDW5Of7CQGjE 1k4PzXQR2C8APvOjSAHZyJQ1zQ45t6TZP5N6yTeMRfdlnjqP1VlGsfZt0ZscUml33xtf ypGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=Ov6dF12doUxji6ePINUtaYIJxqhm2/VTB1zpiOjT1tA=; b=mTbqnN6h664G8FiN+/HSWp/tGcxGUWF2cXIwHjqRVQg8dAlevwAcq2c2J4ojaVX+zD aeP0i7ZDV4Kvcqp8i3pRb8MVu61AZCA0VQbRqcxj0btFR6MXrYK0r98fVBnFVV/dHuDT 7Rwh8mJMvDFvAaIouv/g0AY1CejuJclTD08dBWJhTl5nXcZcRWQQxc1Di9n3etTUTeUy UZWI6PvhzyYMgfb0vN0ebT0Ap5jZlVDd+pqifc1CqWS2aY+9++DrJYK3CoLPVBUA2yJ3 ObDEvAOM6xC4gE0rSYTyeRKkrgXwGOG1TeDipkBpoCaeIpnGvM89s2Xix12MpVO/yDt6 hdZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=HWgMzsoV; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2120.oracle.com (userp2120.oracle.com. [156.151.31.85]) by mx.google.com with ESMTPS id y12si718191pga.216.2019.02.13.16.03.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Feb 2019 16:03:09 -0800 (PST) Received-SPF: pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) client-ip=156.151.31.85; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=HWgMzsoV; spf=pass (google.com: domain of khalid.aziz@oracle.com designates 156.151.31.85 as permitted sender) smtp.mailfrom=khalid.aziz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1DNwk6t099402; Thu, 14 Feb 2019 00:02:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=Ov6dF12doUxji6ePINUtaYIJxqhm2/VTB1zpiOjT1tA=; b=HWgMzsoVLWuxYs5bJDKNiffhWGRMwUJsNHgz0cMMCPscT4fdXMdRnfNdp9sKENEJvwft TXF3uWQrqHXeKNtriV1PxiSLggLWtrKHDZbqDPKfC2vSDmyGgCvykt4L33owldrq6md1 TQQBpA6dSqxhEIVIk5w1iCvxCAnQfbKcOuFvv8SErUpmXO8Gu4GIlFIIKLHDLf3O5o4+ ufPr2YjYEzK5U4hHeZSfwBGrufuBkFk0aiuvWEs187yLF1EZ2nbUxudhQYjrx/FwpdC3 jHTyqjS/1miNz9E4XceIQ4rlIkCOYGq8Vpq/8qTkxjsEyQH8SgC1/wxD+ys2W18dm4Sm 2A== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2qhree55nq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:30 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1E02TBh001559 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Feb 2019 00:02:30 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1E02TYJ018941; Thu, 14 Feb 2019 00:02:29 GMT Received: from concerto.internal (/24.9.64.241) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 16:02:29 -0800 From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Khalid Aziz , deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v8 14/14] xpfo, mm: Optimize XPFO TLB flushes by batching them together Date: Wed, 13 Feb 2019 17:01:37 -0700 Message-Id: <6a92971cd9b360ec1b0ae75887f33f67774d681a.1550088114.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9166 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When XPFO forces a TLB flush on all cores, the performance impact is very significant. Batching as many of these TLB updates as possible can help lower this impact. When a userspace allocates a page, kernel tries to get that page from the per-cpu free list. This free list is replenished in bulk when it runs low. Free list is being replenished for future allocation to userspace is a good opportunity to update TLB entries in batch and reduce the impact of multiple TLB flushes later. This patch adds new tags for the page so a page can be marked as available for userspace allocation and unmapped from kernel address space. All such pages are removed from kernel address space in bulk at the time they are added to per-cpu free list. This patch when combined with deferred TLB flushes improves performance further. Using the same benchmark as before of building kernel in parallel, here are the system times on two differently sized systems: Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM make -j60 all 4.20 950.966s 4.20+XPFO 25073.169s 26.366x 4.20+XPFO+Deferred flush 1372.874s 1.44x 4.20+XPFO+Deferred flush+Batch update 1255.021s 1.32x Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM make -j4 all 4.20 607.671s 4.20+XPFO 1588.646s 2.614x 4.20+XPFO+Deferred flush 803.989s 1.32x 4.20+XPFO+Deferred flush+Batch update 795.728s 1.31x Signed-off-by: Khalid Aziz Signed-off-by: Tycho Andersen --- arch/x86/mm/xpfo.c | 5 +++++ include/linux/page-flags.h | 5 ++++- include/linux/xpfo.h | 8 ++++++++ mm/page_alloc.c | 4 ++++ mm/xpfo.c | 35 +++++++++++++++++++++++++++++++++-- 5 files changed, 54 insertions(+), 3 deletions(-) diff --git a/arch/x86/mm/xpfo.c b/arch/x86/mm/xpfo.c index d3833532bfdc..fb06bb3cb718 100644 --- a/arch/x86/mm/xpfo.c +++ b/arch/x86/mm/xpfo.c @@ -87,6 +87,11 @@ inline void set_kpte(void *kaddr, struct page *page, pgprot_t prot) } +void xpfo_flush_tlb_all(void) +{ + xpfo_flush_tlb_kernel_range(0, TLB_FLUSH_ALL); +} + inline void xpfo_flush_kernel_tlb(struct page *page, int order) { int level; diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a532063f27b5..fdf7e14cbc96 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -406,9 +406,11 @@ PAGEFLAG(Idle, idle, PF_ANY) PAGEFLAG(XpfoUser, xpfo_user, PF_ANY) TESTCLEARFLAG(XpfoUser, xpfo_user, PF_ANY) TESTSETFLAG(XpfoUser, xpfo_user, PF_ANY) +#define __PG_XPFO_USER (1UL << PG_xpfo_user) PAGEFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) TESTCLEARFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) TESTSETFLAG(XpfoUnmapped, xpfo_unmapped, PF_ANY) +#define __PG_XPFO_UNMAPPED (1UL << PG_xpfo_unmapped) #endif /* @@ -787,7 +789,8 @@ static inline void ClearPageSlabPfmemalloc(struct page *page) * alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (((1UL << NR_PAGEFLAGS) - 1) & ~__PG_HWPOISON) + (((1UL << NR_PAGEFLAGS) - 1) & ~__PG_HWPOISON & ~__PG_XPFO_USER & \ + ~__PG_XPFO_UNMAPPED) #define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2) diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index 1dd590ff1a1f..c4f6c99e7380 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -34,6 +34,7 @@ void set_kpte(void *kaddr, struct page *page, pgprot_t prot); void xpfo_dma_map_unmap_area(bool map, const void *addr, size_t size, enum dma_data_direction dir); void xpfo_flush_kernel_tlb(struct page *page, int order); +void xpfo_flush_tlb_all(void); void xpfo_kmap(void *kaddr, struct page *page); void xpfo_kunmap(void *kaddr, struct page *page); @@ -55,6 +56,8 @@ bool xpfo_enabled(void); phys_addr_t user_virt_to_phys(unsigned long addr); +bool xpfo_pcp_refill(struct page *page, enum migratetype migratetype, + int order); #else /* !CONFIG_XPFO */ static inline void xpfo_init_single_page(struct page *page) { } @@ -82,6 +85,11 @@ static inline bool xpfo_enabled(void) { return false; } static inline phys_addr_t user_virt_to_phys(unsigned long addr) { return 0; } +static inline bool xpfo_pcp_refill(struct page *page, + enum migratetype migratetype, int order) +{ +} + #endif /* CONFIG_XPFO */ #endif /* _LINUX_XPFO_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d00382b20001..5702b6fa435c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2478,6 +2478,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, int migratetype) { int i, alloced = 0; + bool flush_tlb = false; spin_lock(&zone->lock); for (i = 0; i < count; ++i) { @@ -2503,6 +2504,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, if (is_migrate_cma(get_pcppage_migratetype(page))) __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -(1 << order)); + flush_tlb |= xpfo_pcp_refill(page, migratetype, order); } /* @@ -2513,6 +2515,8 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, */ __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); spin_unlock(&zone->lock); + if (flush_tlb) + xpfo_flush_tlb_all(); return alloced; } diff --git a/mm/xpfo.c b/mm/xpfo.c index 5157cbebce4b..7f78d00df002 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -47,7 +47,8 @@ void __meminit xpfo_init_single_page(struct page *page) void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) { - int i, flush_tlb = 0; + int i; + bool flush_tlb = false; if (!static_branch_unlikely(&xpfo_inited)) return; @@ -65,7 +66,7 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) * was previously allocated to the kernel. */ if (!TestSetPageXpfoUser(page + i)) - flush_tlb = 1; + flush_tlb = true; } else { /* Tag the page as a non-user (kernel) page */ ClearPageXpfoUser(page + i); @@ -74,6 +75,8 @@ void xpfo_alloc_pages(struct page *page, int order, gfp_t gfp) if (flush_tlb) xpfo_flush_kernel_tlb(page, order); + + return; } void xpfo_free_pages(struct page *page, int order) @@ -190,3 +193,31 @@ void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, kunmap_atomic(mapping[i]); } EXPORT_SYMBOL(xpfo_temp_unmap); + +bool xpfo_pcp_refill(struct page *page, enum migratetype migratetype, + int order) +{ + int i; + bool flush_tlb = false; + + if (!static_branch_unlikely(&xpfo_inited)) + return false; + + for (i = 0; i < 1 << order; i++) { + if (migratetype == MIGRATE_MOVABLE) { + /* GPF_HIGHUSER ** + * Tag the page as a user page, mark it as unmapped + * in kernel space and flush the TLB if it was + * previously allocated to the kernel. + */ + if (!TestSetPageXpfoUnmapped(page + i)) + flush_tlb = true; + SetPageXpfoUser(page + i); + } else { + /* Tag the page as a non-user (kernel) page */ + ClearPageXpfoUser(page + i); + } + } + + return(flush_tlb); +}