From patchwork Thu Jan 10 21:09:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khalid Aziz X-Patchwork-Id: 10756943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E94BC159A for ; Thu, 10 Jan 2019 21:12:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D7FF429BCC for ; Thu, 10 Jan 2019 21:12:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CBB3629BE7; Thu, 10 Jan 2019 21:12:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from mother.openwall.net (mother.openwall.net [195.42.179.200]) by mail.wl.linuxfoundation.org (Postfix) with SMTP id 93F0329BCC for ; Thu, 10 Jan 2019 21:12:51 +0000 (UTC) Received: (qmail 14285 invoked by uid 550); 10 Jan 2019 21:11:23 -0000 Mailing-List: contact kernel-hardening-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Delivered-To: mailing list kernel-hardening@lists.openwall.com Received: (qmail 14167 invoked from network); 10 Jan 2019 21:11:22 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references; s=corp-2018-07-02; bh=Hk38+nWLDl3nLi7YTCJ6Yd0tzUi3GaMmOjk//H1EH/4=; b=uhYxXSO/+zuWC4vaHaHuMFTavxikTbqRP/9BSmynD6o6Qfs9s/b9ru9AGepw0APjML8L t/zcQ5raQgd2hxnEuOIeQ3tDxSbyqTKVT6FNZFFa7ljeaVBa3CQAqtwsFUa9oJgUHMEN J1/5tStu0S/5soyx7I4gstQJ0JFuHXHoL4NIlQduCj46WdjUo+HRUfM3GlbCBZPjpZCP meRR+n3PdKItdFWjds7Pzwo05wntxRAKFMVuyxYxPHw2u2pUYySmPVsmEqcZwrgyZXTm Br+2+S7rp8VHwZgDUt9kxwdqx34UbClUTG1UlvtiwhHrxvZFpc1GovmwTm8BchLw/gP8 qQ== From: Khalid Aziz To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, joao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, "Vasileios P . Kemerlis" , Juerg Haefliger , Tycho Andersen , Marco Benatto , David Woodhouse , Khalid Aziz Subject: [RFC PATCH v7 14/16] EXPERIMENTAL: xpfo, mm: optimize spin lock usage in xpfo_kmap Date: Thu, 10 Jan 2019 14:09:46 -0700 Message-Id: <7e8e17f519ae87a91fc6cbb57b8b27094c96305c.1547153058.git.khalid.aziz@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: In-Reply-To: References: X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9132 signatures=668680 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901100164 X-Virus-Scanned: ClamAV using ClamSMTP From: Julian Stecklina We can reduce spin lock usage in xpfo_kmap to the 0->1 transition of the mapcount. This means that xpfo_kmap() can now race and that we get spurious page faults. The page fault handler helps the system make forward progress by fixing the page table instead of allowing repeated page faults until the right xpfo_kmap went through. Model-checked with up to 4 concurrent callers with Spin. Signed-off-by: Julian Stecklina Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis Cc: Juerg Haefliger Cc: Tycho Andersen Cc: Marco Benatto Cc: David Woodhouse Signed-off-by: Khalid Aziz --- arch/x86/mm/fault.c | 4 ++++ include/linux/xpfo.h | 4 ++++ mm/xpfo.c | 50 +++++++++++++++++++++++++++++++++++++------- 3 files changed, 51 insertions(+), 7 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index ba51652fbd33..207081dcd572 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -18,6 +18,7 @@ #include /* faulthandler_disabled() */ #include /* efi_recover_from_page_fault()*/ #include +#include #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1218,6 +1219,9 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, if (kprobes_fault(regs)) return; + if (xpfo_spurious_fault(address)) + return; + /* * Note, despite being a "bad area", there are quite a few * acceptable reasons to get here, such as erratum fixups diff --git a/include/linux/xpfo.h b/include/linux/xpfo.h index ea5188882f49..58dd243637d2 100644 --- a/include/linux/xpfo.h +++ b/include/linux/xpfo.h @@ -54,6 +54,8 @@ bool xpfo_enabled(void); phys_addr_t user_virt_to_phys(unsigned long addr); +bool xpfo_spurious_fault(unsigned long addr); + #else /* !CONFIG_XPFO */ static inline void xpfo_init_single_page(struct page *page) { } @@ -81,6 +83,8 @@ static inline bool xpfo_enabled(void) { return false; } static inline phys_addr_t user_virt_to_phys(unsigned long addr) { return 0; } +static inline bool xpfo_spurious_fault(unsigned long addr) { return false; } + #endif /* CONFIG_XPFO */ #endif /* _LINUX_XPFO_H */ diff --git a/mm/xpfo.c b/mm/xpfo.c index dbf20efb0499..85079377c91d 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -119,6 +119,16 @@ void xpfo_free_pages(struct page *page, int order) } } +static void xpfo_do_map(void *kaddr, struct page *page) +{ + spin_lock(&page->xpfo_lock); + if (PageXpfoUnmapped(page)) { + set_kpte(kaddr, page, PAGE_KERNEL); + ClearPageXpfoUnmapped(page); + } + spin_unlock(&page->xpfo_lock); +} + void xpfo_kmap(void *kaddr, struct page *page) { if (!static_branch_unlikely(&xpfo_inited)) @@ -127,17 +137,12 @@ void xpfo_kmap(void *kaddr, struct page *page) if (!PageXpfoUser(page)) return; - spin_lock(&page->xpfo_lock); - /* * The page was previously allocated to user space, so map it back * into the kernel. No TLB flush required. */ - if ((atomic_inc_return(&page->xpfo_mapcount) == 1) && - TestClearPageXpfoUnmapped(page)) - set_kpte(kaddr, page, PAGE_KERNEL); - - spin_unlock(&page->xpfo_lock); + if (atomic_inc_return(&page->xpfo_mapcount) == 1) + xpfo_do_map(kaddr, page); } EXPORT_SYMBOL(xpfo_kmap); @@ -204,3 +209,34 @@ void xpfo_temp_unmap(const void *addr, size_t size, void **mapping, kunmap_atomic(mapping[i]); } EXPORT_SYMBOL(xpfo_temp_unmap); + +bool xpfo_spurious_fault(unsigned long addr) +{ + struct page *page; + bool spurious; + int mapcount; + + if (!static_branch_unlikely(&xpfo_inited)) + return false; + + /* XXX Is this sufficient to guard against calling virt_to_page() on a + * virtual address that has no corresponding struct page? */ + if (!virt_addr_valid(addr)) + return false; + + page = virt_to_page(addr); + mapcount = atomic_read(&page->xpfo_mapcount); + spurious = PageXpfoUser(page) && mapcount; + + /* Guarantee forward progress in case xpfo_kmap() raced. */ + if (spurious && PageXpfoUnmapped(page)) { + xpfo_do_map((void *)(addr & PAGE_MASK), page); + } + + if (unlikely(!spurious)) + printk("XPFO non-spurious fault %lx user=%d unmapped=%d mapcount=%d\n", + addr, PageXpfoUser(page), PageXpfoUnmapped(page), + mapcount); + + return spurious; +}