From patchwork Wed Jul 11 11:29:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joerg Roedel X-Patchwork-Id: 10519459 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 052976032A for ; Wed, 11 Jul 2018 11:31:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A36A26861 for ; Wed, 11 Jul 2018 11:31:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F192928844; Wed, 11 Jul 2018 11:31:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9466328837 for ; Wed, 11 Jul 2018 11:31:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 934A26B027C; Wed, 11 Jul 2018 07:30:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 911096B027E; Wed, 11 Jul 2018 07:30:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 785DB6B027D; Wed, 11 Jul 2018 07:30:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 151206B027E for ; Wed, 11 Jul 2018 07:30:12 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id b12-v6so9468576edi.12 for ; Wed, 11 Jul 2018 04:30:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=POHKYbt8MShTfvkl0eO3rOoXgBwoHCSXfNgMI6pa5yE=; b=tS9jejqhDy0cLw0HHXyfVRkYFjMEQl2Et4C5hSNt2GjRh/pYmruFWFeNWk/m9Ckgdf vtRl5J6uO22SOtvTi/s+7uyyYv3IZCuiqV5J7+A+G1ntRT5zGwCh9ECDLhC+n8Y8dfaq du4v65iVDJlfVtFp8p3pxRJoxFfsBhtQ0J564aErygnIgJe24jMiWfFnIXzkZTiNuGTU EEVMvBn3XnWDLVkG82p5clXcbt2fflr3mZaF7chjYwI3Lwbs/xH1JxU87NHhsK0eJxfg 5/sIcjp3vAnxpvfxKUmZYYsfZVZROJReSRxEv6Zm4V+j9xouI3T085/u1PKxuw5Nu+Xb yRhA== X-Gm-Message-State: APt69E0GK+0aCNKFJLW+bS+RAm+MfkzTWxZmvsLl9TKSVlTqQjTUy/Wr UwpwfTTqdLV9tu52ZWHNIZN2hordksmtyl4qSR+rEScoA11GwHL76d1Kh4nlTTaYdbzR5zImKQT 3gA6uP8ZIin/QD+EW/T92wwagfqv62ti4SoyxKFsnqsNf+UEV6EgpgWyNlY3SdB7FKQ== X-Received: by 2002:aa7:d385:: with SMTP id x5-v6mr31280112edq.104.1531308611651; Wed, 11 Jul 2018 04:30:11 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfC+yYSR20lNJuZfhYxhM7WJdk4KmDqP7TBM+PMq6L0UGAY4tLIJgqLYtg55IFbM3s6reYD X-Received: by 2002:aa7:d385:: with SMTP id x5-v6mr31280073edq.104.1531308611042; Wed, 11 Jul 2018 04:30:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531308611; cv=none; d=google.com; s=arc-20160816; b=TClckgJmAeKJwh2PsfJyvP2oT5dNdHusXcYwL4Pp6TXdatf/6L+Az0Pk/T0yHF9VLG koEqJ6YWFuArtBNxQ8ND48brhbxkd9ViUuezll3danNEDK9DqTrwQRTFAjFd/res4MEs EtBGKDoFXss01JXUsQ64IbzpeSKVYaaMcpQXcGL3Ufm6kA66owHpxMo17Ua2mkKLApu4 D7C1Ao7bAVioOCQQUvWVFWMSLSMhwZzkwCxvvBjAopnb6ezCUzzWJQRTRmEwqdILvxA2 nOmue5T5m9P9mjTmi+PVKbX8wfOWihjZt5PpqWYUk/GSzCkIPgPTeZlTgywEAyG7ZQx/ 3OsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=POHKYbt8MShTfvkl0eO3rOoXgBwoHCSXfNgMI6pa5yE=; b=aIqfGZqKmx8DFLpFsKDenIJNopDpK6zKNMZ/bDGV5xAmfEAATPto1HTbN9Se/eeOaT MiOPsu33Slt71VDVwnXvVUbiV2kj3tPa2sgpr7P3+UODniXhKwzo0YUrlPqos8e5kvak v9lIVEhJAX9xgFxMeRekpDMdV0RpYwuJWZ7uGWAZcJ72agEGNJ+u/PrFDiDlzhSjU8nK uGp4FHB9LgBqoi4CH4R6nujzOu80y7xaMwFxbfF4mpIYTr0IbNsinJBNE+FB+CYqFdIN Q6q8jalh8jVheLG6tEvDYCGZ3CMM4u3ICxLboJjVhVsxKy3iig4aNRHFb4F884Iordti hA7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass (test mode) header.i=@8bytes.org header.s=mail-1 header.b=LgiJZBxZ; spf=pass (google.com: domain of joro@8bytes.org designates 2a01:238:4383:600:38bc:a715:4b6d:a889 as permitted sender) smtp.mailfrom=joro@8bytes.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=8bytes.org Received: from theia.8bytes.org (8bytes.org. [2a01:238:4383:600:38bc:a715:4b6d:a889]) by mx.google.com with ESMTPS id s6-v6si3880220edj.407.2018.07.11.04.30.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Jul 2018 04:30:11 -0700 (PDT) Received-SPF: pass (google.com: domain of joro@8bytes.org designates 2a01:238:4383:600:38bc:a715:4b6d:a889 as permitted sender) client-ip=2a01:238:4383:600:38bc:a715:4b6d:a889; Authentication-Results: mx.google.com; dkim=pass (test mode) header.i=@8bytes.org header.s=mail-1 header.b=LgiJZBxZ; spf=pass (google.com: domain of joro@8bytes.org designates 2a01:238:4383:600:38bc:a715:4b6d:a889 as permitted sender) smtp.mailfrom=joro@8bytes.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=8bytes.org Received: by theia.8bytes.org (Postfix, from userid 1000) id D57C28EB; Wed, 11 Jul 2018 13:30:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=8bytes.org; s=mail-1; t=1531308602; bh=JKbLaug3RNyDfvH2yfAePqSaHIh069+G0E3eJSS/Rlc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LgiJZBxZMlHKvRElzT/5InVNkummgbuCYIvdGwWxmqxI2NRc7WmbFPkyx9u4HvfLt gwh5woopcKx4TyQhwdINvfeLO7+Z/Vs9Y4r+XNvhmtgGINo5iSqgmrv3LPC+QvcLW5 NPsh5CxiKv3MWlDkd+wFGguOV9EvIYH1R4wEQfho1WfjpglKy3IO6gnmXI/DVyJM1V f1ocq3qs/sai+snh3NP/G2zD0riq24KbMFgbyasP92CVIbkHdBFbv+y5VJIiYJAALx fdxB0QisG9VADs581T3cxlPxfYvkyU75Glhf18DynPTreBl4IvqGV7zoBj4Nip0FPh DyRtsL4xQ570w== From: Joerg Roedel To: Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds , Andy Lutomirski , Dave Hansen , Josh Poimboeuf , Juergen Gross , Peter Zijlstra , Borislav Petkov , Jiri Kosina , Boris Ostrovsky , Brian Gerst , David Laight , Denys Vlasenko , Eduardo Valentin , Greg KH , Will Deacon , aliguori@amazon.com, daniel.gruss@iaik.tugraz.at, hughd@google.com, keescook@google.com, Andrea Arcangeli , Waiman Long , Pavel Machek , "David H . Gutteridge" , jroedel@suse.de, joro@8bytes.org Subject: [PATCH 14/39] x86/entry/32: Add PTI cr3 switches to NMI handler code Date: Wed, 11 Jul 2018 13:29:21 +0200 Message-Id: <1531308586-29340-15-git-send-email-joro@8bytes.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531308586-29340-1-git-send-email-joro@8bytes.org> References: <1531308586-29340-1-git-send-email-joro@8bytes.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Joerg Roedel The NMI handler is special, as it needs to leave with the same cr3 as it was entered with. We need to do this because we could enter the NMI handler from kernel code with user-cr3 already loaded. Signed-off-by: Joerg Roedel --- arch/x86/entry/entry_32.S | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 311aefa..a368583 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -77,6 +77,8 @@ #endif .endm +#define PTI_SWITCH_MASK (1 << PAGE_SHIFT) + /* * User gs save/restore * @@ -213,8 +215,19 @@ .endm -.macro SAVE_ALL_NMI +.macro SAVE_ALL_NMI cr3_reg:req SAVE_ALL + + /* + * Now switch the CR3 when PTI is enabled. + * + * We can enter with either user or kernel cr3, the code will + * store the old cr3 in \cr3_reg and switches to the kernel cr3 + * if necessary. + */ + SWITCH_TO_KERNEL_CR3 scratch_reg=\cr3_reg + +.Lend_\@: .endm /* * This is a sneaky trick to help the unwinder find pt_regs on the stack. The @@ -262,7 +275,23 @@ POP_GS_EX .endm -.macro RESTORE_ALL_NMI pop=0 +.macro RESTORE_ALL_NMI cr3_reg:req pop=0 + /* + * Now switch the CR3 when PTI is enabled. + * + * We enter with kernel cr3 and switch the cr3 to the value + * stored on \cr3_reg, which is either a user or a kernel cr3. + */ + ALTERNATIVE "jmp .Lswitched_\@", "", X86_FEATURE_PTI + + testl $PTI_SWITCH_MASK, \cr3_reg + jz .Lswitched_\@ + + /* User cr3 in \cr3_reg - write it to hardware cr3 */ + movl \cr3_reg, %cr3 + +.Lswitched_\@: + RESTORE_REGS pop=\pop .endm @@ -1333,7 +1362,7 @@ ENTRY(nmi) #endif pushl %eax # pt_regs->orig_ax - SAVE_ALL_NMI + SAVE_ALL_NMI cr3_reg=%edi ENCODE_FRAME_POINTER xorl %edx, %edx # zero error code movl %esp, %eax # pt_regs pointer @@ -1361,7 +1390,7 @@ ENTRY(nmi) .Lnmi_return: CHECK_AND_APPLY_ESPFIX - RESTORE_ALL_NMI pop=4 + RESTORE_ALL_NMI cr3_reg=%edi pop=4 jmp .Lirq_return #ifdef CONFIG_X86_ESPFIX32 @@ -1377,12 +1406,12 @@ ENTRY(nmi) pushl 16(%esp) .endr pushl %eax - SAVE_ALL_NMI + SAVE_ALL_NMI cr3_reg=%edi ENCODE_FRAME_POINTER FIXUP_ESPFIX_STACK # %eax == %esp xorl %edx, %edx # zero error code call do_nmi - RESTORE_ALL_NMI + RESTORE_ALL_NMI cr3_reg=%edi lss 12+4(%esp), %esp # back to espfix stack jmp .Lirq_return #endif