From patchwork Wed Jul 18 09:40:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joerg Roedel X-Patchwork-Id: 10531837 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0996B600F4 for ; Wed, 18 Jul 2018 09:42:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC8841FF0B for ; Wed, 18 Jul 2018 09:42:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E06FB207A7; Wed, 18 Jul 2018 09:42:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 652A31FF0B for ; Wed, 18 Jul 2018 09:42:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1C946B0279; Wed, 18 Jul 2018 05:41:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EA68D6B027B; Wed, 18 Jul 2018 05:41:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B253F6B0279; Wed, 18 Jul 2018 05:41:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 431876B0278 for ; Wed, 18 Jul 2018 05:41:31 -0400 (EDT) Received: by mail-ed1-f70.google.com with SMTP id r9-v6so1701557edh.14 for ; Wed, 18 Jul 2018 02:41:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=IphXzokAOSDFC9Da5H3LqdIoHOyY4+I/IXyH/kFcjNc=; b=noUESAy2xsSWhelWpyMVO+7GaAzH2on+50w3iF+yk1Zzzw8du5Z5hdqAQkv7siMmDI HZGWtNZdknLDsviCnX9LKnow0psDXBjK9Sn2TeDl1erUqj2PUtsnJ2mE7fjszvL4Re17 ra2fhebXwU6c7cg3tspomBRsdEoFSyFN1OYXQSoY82/V0wHZED2xvw04NF27o9HK5XmK GqJWXoMcjjnzplYPXZHyNCJ/5DthseUT6WC86/ReWaD5/7xkjt+NuPbS6+BoLdiQvicU nWkp1mEfcxhPuiBscnrKq7J+CPZiGSau0Nom6Cix4BcSVCwmeerD6Qfe1wsW3a47y09C pRKg== X-Gm-Message-State: AOUpUlH+B/vt/6zEUkiC3geQonJ/4hi1UrS3C8aGPX7/q0NXF6lT15lK 8R/Hu0yfqT5we2K6hl53U13+pF0jf5+0NQqJejciQ/CJP3Rb2l+krG6W/18RixUN6HdtI8chSM6 EigRlRfeol1e0hl3+u/77T9ebu+dB/qk4nRzUeodg4Lyylkq2ZZr2S2WLcpakIZ4pQw== X-Received: by 2002:aa7:da9a:: with SMTP id q26-v6mr6215327eds.115.1531906890832; Wed, 18 Jul 2018 02:41:30 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeEN8R/wcWsEehxAa4/CCFdMolOVoYnh9W2w0jU9hWYkVRml6qY4ebBCBAHX9S6bBOBZUif X-Received: by 2002:aa7:da9a:: with SMTP id q26-v6mr6215298eds.115.1531906890205; Wed, 18 Jul 2018 02:41:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1531906890; cv=none; d=google.com; s=arc-20160816; b=iiiQCGS+bWxnYQyidx5VWMsCQz9+OIYqInskvZ8luAyR8s5mQk+SlX9xWh+hRDglZk 8eUVDV7tx0n6bLOOn8xyhjrdxae7e6K1m4oJ4yOV/O9O4gehpY1HiFJdnYuDgocYOmQm 5Tn0hM/UDC0djESDx+wR/z4QPLXGuKBptFvCExKdkynzA+8qwvnajIl58e5HQbWcRjH/ c3/E91DwIqzpYfcAcdCqOuOJTPihdF3DnHkFm2OGOShZ058hdYuaGHeFbYaiJxjc1g8Q JuSBH6Z+pE/02/H+8IK7ygi0nFyrkL3Dj9nynYNjf7ToHcB7sgq2XeiinSO9B97ipHNm kQcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=IphXzokAOSDFC9Da5H3LqdIoHOyY4+I/IXyH/kFcjNc=; b=i3LF2w9greKSWV2QG5TOqpiqywLeZz2rUJ4rozTvB+1v3J/iB5VRuIKJ+jd3NxxaMl 8nqzfA25s20TVL2LOYW7fLG/ntNeItLLUKkyoYj/AT0o1SaUPpaWgQBDN1X6E6T9TUj7 wjwISkQrm10VKguufrAdYXAuvEPJVs+BZMuv0tbSMNZRstGbtg5liElr/3NJVhCeiowK +XrW6ArUdiE0ULW53hteNFkkoCQRsRTdTPIlpVS8LrIxaM4Ya50gyhj5hcWqF+vRhY7B BT0KdA+Lp0bm91UBFgp+ZsDU3X9BGWOD7gwp7wEl1iGJEFC2EakG35BOH6eUgs7JfvHg aUCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass (test mode) header.i=@8bytes.org header.s=mail-1 header.b=YzGEBYWg; spf=pass (google.com: domain of joro@8bytes.org designates 2a01:238:4383:600:38bc:a715:4b6d:a889 as permitted sender) smtp.mailfrom=joro@8bytes.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=8bytes.org Received: from theia.8bytes.org (8bytes.org. [2a01:238:4383:600:38bc:a715:4b6d:a889]) by mx.google.com with ESMTPS id q4-v6si28349edc.335.2018.07.18.02.41.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 18 Jul 2018 02:41:30 -0700 (PDT) Received-SPF: pass (google.com: domain of joro@8bytes.org designates 2a01:238:4383:600:38bc:a715:4b6d:a889 as permitted sender) client-ip=2a01:238:4383:600:38bc:a715:4b6d:a889; Authentication-Results: mx.google.com; dkim=pass (test mode) header.i=@8bytes.org header.s=mail-1 header.b=YzGEBYWg; spf=pass (google.com: domain of joro@8bytes.org designates 2a01:238:4383:600:38bc:a715:4b6d:a889 as permitted sender) smtp.mailfrom=joro@8bytes.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=8bytes.org Received: by theia.8bytes.org (Postfix, from userid 1000) id 00917590; Wed, 18 Jul 2018 11:41:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=8bytes.org; s=mail-1; t=1531906881; bh=p/4K8svw0aLeeaY4Rzc18GdQS6q0YwgiqWeWOcg2qUI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YzGEBYWgw2/IkH2tFshC4gCXsQY/S9MMEKbYL4Tf40q4x8lWwkxTuyyzdeq+tS8dv ysLF6pOe8IPQCJulS/4nY7KKSA4MNlPFNLaaZ/o+5KRzaOpsFyeYJSDgXsA13FS9P1 xblXaR6t2nj6QUggywSWoctEQSm7mps3CRyFtcHPCQ9CHKmV4DHtNgA9wFC/s5HL7J a3UF5lWw3RFcOVhkSI3WgIiouoAuOwH6OObfYZOR0Quf8ShjyN76m8D+vfAmWrGkow 67ArhRlVD6LGMV1cVPq9l7eVJeA35OoRQkCoNkR5VPND2XmEmgWxYF793Him89gZND 39iReZsNq1YNA== From: Joerg Roedel To: Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Linus Torvalds , Andy Lutomirski , Dave Hansen , Josh Poimboeuf , Juergen Gross , Peter Zijlstra , Borislav Petkov , Jiri Kosina , Boris Ostrovsky , Brian Gerst , David Laight , Denys Vlasenko , Eduardo Valentin , Greg KH , Will Deacon , aliguori@amazon.com, daniel.gruss@iaik.tugraz.at, hughd@google.com, keescook@google.com, Andrea Arcangeli , Waiman Long , Pavel Machek , "David H . Gutteridge" , jroedel@suse.de, joro@8bytes.org Subject: [PATCH 13/39] x86/entry/32: Add PTI cr3 switches to NMI handler code Date: Wed, 18 Jul 2018 11:40:50 +0200 Message-Id: <1531906876-13451-14-git-send-email-joro@8bytes.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531906876-13451-1-git-send-email-joro@8bytes.org> References: <1531906876-13451-1-git-send-email-joro@8bytes.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Joerg Roedel The NMI handler is special, as it needs to leave with the same cr3 as it was entered with. We need to do this because we could enter the NMI handler from kernel code with user-cr3 already loaded. Signed-off-by: Joerg Roedel --- arch/x86/entry/entry_32.S | 39 +++++++++++++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 6 deletions(-) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 60b28df..b1541c7 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -210,8 +210,19 @@ .endm -.macro SAVE_ALL_NMI +.macro SAVE_ALL_NMI cr3_reg:req SAVE_ALL + + /* + * Now switch the CR3 when PTI is enabled. + * + * We can enter with either user or kernel cr3, the code will + * store the old cr3 in \cr3_reg and switches to the kernel cr3 + * if necessary. + */ + SWITCH_TO_KERNEL_CR3 scratch_reg=\cr3_reg + +.Lend_\@: .endm /* * This is a sneaky trick to help the unwinder find pt_regs on the stack. The @@ -259,7 +270,23 @@ POP_GS_EX .endm -.macro RESTORE_ALL_NMI pop=0 +.macro RESTORE_ALL_NMI cr3_reg:req pop=0 + /* + * Now switch the CR3 when PTI is enabled. + * + * We enter with kernel cr3 and switch the cr3 to the value + * stored on \cr3_reg, which is either a user or a kernel cr3. + */ + ALTERNATIVE "jmp .Lswitched_\@", "", X86_FEATURE_PTI + + testl $PTI_SWITCH_MASK, \cr3_reg + jz .Lswitched_\@ + + /* User cr3 in \cr3_reg - write it to hardware cr3 */ + movl \cr3_reg, %cr3 + +.Lswitched_\@: + RESTORE_REGS pop=\pop .endm @@ -1331,7 +1358,7 @@ ENTRY(nmi) #endif pushl %eax # pt_regs->orig_ax - SAVE_ALL_NMI + SAVE_ALL_NMI cr3_reg=%edi ENCODE_FRAME_POINTER xorl %edx, %edx # zero error code movl %esp, %eax # pt_regs pointer @@ -1359,7 +1386,7 @@ ENTRY(nmi) .Lnmi_return: CHECK_AND_APPLY_ESPFIX - RESTORE_ALL_NMI pop=4 + RESTORE_ALL_NMI cr3_reg=%edi pop=4 jmp .Lirq_return #ifdef CONFIG_X86_ESPFIX32 @@ -1375,12 +1402,12 @@ ENTRY(nmi) pushl 16(%esp) .endr pushl %eax - SAVE_ALL_NMI + SAVE_ALL_NMI cr3_reg=%edi ENCODE_FRAME_POINTER FIXUP_ESPFIX_STACK # %eax == %esp xorl %edx, %edx # zero error code call do_nmi - RESTORE_ALL_NMI + RESTORE_ALL_NMI cr3_reg=%edi lss 12+4(%esp), %esp # back to espfix stack jmp .Lirq_return #endif