From patchwork Fri Jan 15 23:10:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 12024373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B60EC433E0 for ; Fri, 15 Jan 2021 23:11:35 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EBA7323382 for ; Fri, 15 Jan 2021 23:11:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EBA7323382 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68831.123351 (Exim 4.92) (envelope-from ) id 1l0YFQ-0005yB-Hq; Fri, 15 Jan 2021 23:11:16 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68831.123351; Fri, 15 Jan 2021 23:11:16 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0YFQ-0005y1-Ar; Fri, 15 Jan 2021 23:11:16 +0000 Received: by outflank-mailman (input) for mailman id 68831; Fri, 15 Jan 2021 23:11:15 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0YFP-0005vh-D3 for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 23:11:15 +0000 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 1f55a9db-672a-42a0-9d96-32b0276fc11a; Fri, 15 Jan 2021 23:11:07 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 1f55a9db-672a-42a0-9d96-32b0276fc11a DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610752267; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fygCPvuaHloKWc6XP3jLPfZcdHpf7Ee8rM9SEUn8eHs=; b=Aek3vRHuYtvm0kW6tbjksWrpqvOsRLefXsKVBQJaBs79tTbn3/uSKmaL NpVA+BDJm0IyvrbXw8hkKsjfFCaqQrQK8zQu7IOS1i6/kNl/KlImg5a6B iBn95Umy45sGVx7csTX/EsmcrcrthGpq30CbT2f36N5C4uCmCaNImrZd3 Y=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: U7jt1l8EztSXAKskEQujdVsvtWXihy4Dpggy/GRYdu6r7tGXptQfjzIRIw5AjEclWW34q+m+VX be6mfP0/alNAyMiEUCnIN30Gz4cp3JNNE0ZFqm0w9IuDfion2X6ki1AYq/+E63w7Ny86FxbnXO lrTrukgmz+/VyUTzZNQz6OFS5t26MEsgb/HJgHZsP/+0SP9kgGj+WpaMlNNL58yMjhxcWLbofo uc16V+vXJ1ZSTlYsNdhYUv4NdYMPrj+K/5ct084kTDh1Fq1r7vyPhyRGOPb4hx+bNa6j3eZSbZ Fc0= X-SBRS: 5.1 X-MesageID: 36513456 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="36513456" From: Andrew Cooper To: Xen-devel CC: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu , Marek Kasiewicz , =?utf-8?q?Norbert_Kami=C5=84ski?= , Michal Zygowski , Piotr Krol , Krystian Hebel , "Daniel P . Smith" , Rich Persaud , Christopher Clark Subject: [PATCH 1/3] x86/smpboot: Re-position the call to tboot_wake_ap() Date: Fri, 15 Jan 2021 23:10:44 +0000 Message-ID: <20210115231046.31785-2-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210115231046.31785-1-andrew.cooper3@citrix.com> References: <20210115231046.31785-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 So all the moving parts are in one function. No functional change. Signed-off-by: Andrew Cooper Acked-by: Jan Beulich --- CC: Jan Beulich CC: Roger Pau Monné CC: Wei Liu CC: Marek Kasiewicz CC: Norbert Kamiński CC: Michal Zygowski CC: Piotr Krol CC: Krystian Hebel CC: Daniel P. Smith CC: Rich Persaud CC: Christopher Clark --- xen/arch/x86/smpboot.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index 67e727cebd..9eca452ce1 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -426,6 +426,13 @@ static int wakeup_secondary_cpu(int phys_apicid, unsigned long start_eip) int maxlvt, timeout, i; /* + * Some versions of tboot might be able to handle the entire wake sequence + * on our behalf. + */ + if ( tboot_in_measured_env() && tboot_wake_ap(phys_apicid, start_eip) ) + return 0; + + /* * Be paranoid about clearing APIC errors. */ apic_write(APIC_ESR, 0); @@ -570,8 +577,7 @@ static int do_boot_cpu(int apicid, int cpu) set_cpu_state(CPU_STATE_INIT); /* Starting actual IPI sequence... */ - if ( !tboot_in_measured_env() || tboot_wake_ap(apicid, start_eip) ) - boot_error = wakeup_secondary_cpu(apicid, start_eip); + boot_error = wakeup_secondary_cpu(apicid, start_eip); if ( !boot_error ) { From patchwork Fri Jan 15 23:10:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 12024377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92B62C433E9 for ; Fri, 15 Jan 2021 23:11:36 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 527FA23382 for ; Fri, 15 Jan 2021 23:11:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 527FA23382 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68829.123332 (Exim 4.92) (envelope-from ) id 1l0YFL-0005vt-PR; Fri, 15 Jan 2021 23:11:11 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68829.123332; Fri, 15 Jan 2021 23:11:11 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0YFL-0005vm-Kl; Fri, 15 Jan 2021 23:11:11 +0000 Received: by outflank-mailman (input) for mailman id 68829; Fri, 15 Jan 2021 23:11:10 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0YFK-0005vh-Gk for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 23:11:10 +0000 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 45cce534-f808-470b-90a9-362a30d291b2; Fri, 15 Jan 2021 23:11:09 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 45cce534-f808-470b-90a9-362a30d291b2 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610752269; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dXS/3Rq9xVyxzLPRXvqV3AmMCV9sf33JIKkKChDLoGg=; b=blccXMiCyvHaDnAQHGK2DTidJzZEjaYtKx+YaMOrouHXMf64tntRSgN1 Pj1J8pX4MVFGWuix2LQlm//8b7ZzBQiLcrIXJWxFAvjnOzIFONLfu5PNU EDnZwrsugVO0S1lsyWDuyMCHXGyvP3BVuWilcqAWvHoD8FY4Kivclg+Cu c=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: uqeMXLqjdoNfVtZrUpv2fgraaZMjdZdyYMgYwj/wMu2Iw9OOwcdLIDBgPiwg9nesudjW+R8UR2 iRm7pIBHkGQcB6CY7h+td6xlXiKVrPOZS2/c2c92rBNIMzrAMR0xPSvksUkLUKhx229BrYl3QU w3l1XJZybR7fJX/1ShELp6QMq/ohMW7VyVdPZaZVLXulGszGWz2lzUYZHxrgmnL9CR7F4W+IWI 2uCKE4yy+yEWLVXpJ3E4opkYTvBmVwiZlRFQBJZOr/hBKFLVJsrZ4BHnFcWMfTZPAEtYidCWQh ZyE= X-SBRS: 5.1 X-MesageID: 35207499 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="35207499" From: Andrew Cooper To: Xen-devel CC: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu , Marek Kasiewicz , =?utf-8?q?Norbert_Kami=C5=84ski?= , Michal Zygowski , Piotr Krol , Krystian Hebel , "Daniel P . Smith" , Rich Persaud , Christopher Clark Subject: [PATCH 2/3] x86/smpboot: Allow making an INIT IPI conditional Date: Fri, 15 Jan 2021 23:10:45 +0000 Message-ID: <20210115231046.31785-3-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210115231046.31785-1-andrew.cooper3@citrix.com> References: <20210115231046.31785-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 A subsequent change is going to introduce SKINIT support, wherein the APs will be already be in the wait-for-SIPI state, and an INIT must not be sent. Introduce a send_INIT boolean, so we can control sending an INIT IPI separately from sending SIPIs. No functional change. Signed-off-by: Andrew Cooper Acked-by: Roger Pau Monné --- CC: Jan Beulich CC: Roger Pau Monné CC: Wei Liu CC: Marek Kasiewicz CC: Norbert Kamiński CC: Michal Zygowski CC: Piotr Krol CC: Krystian Hebel CC: Daniel P. Smith CC: Rich Persaud CC: Christopher Clark --- xen/arch/x86/smpboot.c | 78 ++++++++++++++++++++++++++------------------------ 1 file changed, 41 insertions(+), 37 deletions(-) diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index 9eca452ce1..195e3681b4 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -424,6 +424,7 @@ static int wakeup_secondary_cpu(int phys_apicid, unsigned long start_eip) { unsigned long send_status = 0, accept_status = 0; int maxlvt, timeout, i; + bool send_INIT = true; /* * Some versions of tboot might be able to handle the entire wake sequence @@ -438,49 +439,52 @@ static int wakeup_secondary_cpu(int phys_apicid, unsigned long start_eip) apic_write(APIC_ESR, 0); apic_read(APIC_ESR); - Dprintk("Asserting INIT.\n"); + if ( send_INIT ) + { + Dprintk("Asserting INIT.\n"); - /* - * Turn INIT on target chip via IPI - */ - apic_icr_write(APIC_INT_LEVELTRIG | APIC_INT_ASSERT | APIC_DM_INIT, - phys_apicid); + /* + * Turn INIT on target chip via IPI + */ + apic_icr_write(APIC_INT_LEVELTRIG | APIC_INT_ASSERT | APIC_DM_INIT, + phys_apicid); - if ( !x2apic_enabled ) - { - Dprintk("Waiting for send to finish...\n"); - timeout = 0; - do { - Dprintk("+"); - udelay(100); - send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY; - } while ( send_status && (timeout++ < 1000) ); + if ( !x2apic_enabled ) + { + Dprintk("Waiting for send to finish...\n"); + timeout = 0; + do { + Dprintk("+"); + udelay(100); + send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY; + } while ( send_status && (timeout++ < 1000) ); - mdelay(10); + mdelay(10); - Dprintk("Deasserting INIT.\n"); + Dprintk("Deasserting INIT.\n"); - apic_icr_write(APIC_INT_LEVELTRIG | APIC_DM_INIT, phys_apicid); + apic_icr_write(APIC_INT_LEVELTRIG | APIC_DM_INIT, phys_apicid); - Dprintk("Waiting for send to finish...\n"); - timeout = 0; - do { - Dprintk("+"); - udelay(100); - send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY; - } while ( send_status && (timeout++ < 1000) ); - } - else if ( tboot_in_measured_env() ) - { - /* - * With tboot AP is actually spinning in a mini-guest before - * receiving INIT. Upon receiving INIT ipi, AP need time to VMExit, - * update VMCS to tracking SIPIs and VMResume. - * - * While AP is in root mode handling the INIT the CPU will drop - * any SIPIs - */ - udelay(10); + Dprintk("Waiting for send to finish...\n"); + timeout = 0; + do { + Dprintk("+"); + udelay(100); + send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY; + } while ( send_status && (timeout++ < 1000) ); + } + else if ( tboot_in_measured_env() ) + { + /* + * With tboot AP is actually spinning in a mini-guest before + * receiving INIT. Upon receiving INIT ipi, AP need time to VMExit, + * update VMCS to tracking SIPIs and VMResume. + * + * While AP is in root mode handling the INIT the CPU will drop + * any SIPIs + */ + udelay(10); + } } maxlvt = get_maxlvt(); From patchwork Fri Jan 15 23:10:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 12024379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA806C433E6 for ; Fri, 15 Jan 2021 23:11:35 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7E59723382 for ; Fri, 15 Jan 2021 23:11:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E59723382 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.68832.123368 (Exim 4.92) (envelope-from ) id 1l0YFV-00063b-Se; Fri, 15 Jan 2021 23:11:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 68832.123368; Fri, 15 Jan 2021 23:11:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0YFV-00063T-Mt; Fri, 15 Jan 2021 23:11:21 +0000 Received: by outflank-mailman (input) for mailman id 68832; Fri, 15 Jan 2021 23:11:20 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l0YFU-0005vh-DA for xen-devel@lists.xenproject.org; Fri, 15 Jan 2021 23:11:20 +0000 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ac0a538e-3bc2-4d97-8358-958fcbeebe0d; Fri, 15 Jan 2021 23:11:11 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ac0a538e-3bc2-4d97-8358-958fcbeebe0d DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1610752270; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VKbjRpGNczWYULk90MvOqyNNrgUHfZhx9ZjhjLAEyLs=; b=Rmv18cAKcK0Ywlkb7eQAu1VAAzjndgVEr+H02zytrFUKxmvAIXFgQkB8 kie97lsE76tJ5GrBr2/fkFgKggsJopqrPWddVeqo5XDgJ/FIzPxt/ScGd EnhMdnchxmijg2cp5stvSK9KSCfKONkC7Us5X3Gkib7IT6xFPiw8SL798 8=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: uCzw+f9s4rOBMDS813uY3+Ky8SjSrh0lPMgIJzf33qvYtIn0KVRHcXGyDpWt2hv+0CwzJxxuR0 E61YrCVrElz9OfgX3xOeSUWVv0PHjuhrtP+x0C4gkRM+3GqUHR54pfTV0MCo0W0pmIJTY9TqVc 3zhLVPRvLS+diaXaqTXexLJ+ZQrSP5eOpvwkXCo9Ubguotzn34T2xO1SAx3ZryJGVy3cTK1k/0 rJcTXioeqLawiQmzlnj4Q8/W4xdzBJ6ErC+uvZPYhc1RYk0cZsJDpO0kWGvMKH1KwOR4GieEOn UGk= X-SBRS: 5.1 X-MesageID: 36513458 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,350,1602561600"; d="scan'208";a="36513458" From: Andrew Cooper To: Xen-devel CC: =?utf-8?q?Norbert_Kami=C5=84ski?= , "Marek Kasiewicz" , Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu , Michal Zygowski , Piotr Krol , Krystian Hebel , "Daniel P . Smith" , Rich Persaud , Christopher Clark Subject: [PATCH 3/3] x86: Support booting under Secure Startup via SKINIT Date: Fri, 15 Jan 2021 23:10:46 +0000 Message-ID: <20210115231046.31785-4-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210115231046.31785-1-andrew.cooper3@citrix.com> References: <20210115231046.31785-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 From: Norbert Kamiński For now, this is simply enough logic to let Xen come up after the bootloader has executed an SKINIT instruction to begin a Secure Startup. During a Secure Startup, the BSP operates with the GIF clear (blocks all external interrupts, even SMI/NMI), and INIT_REDIRECTION active (converts INIT IPIs to #SX exceptions, if e.g. the platform needs to scrub secrets before resetting). To afford APs the same Secure Startup protections as the BSP, the INIT IPI must be skipped, and SIPI must be the first interrupt seen. Full details are available in AMD APM Vol2 15.27 "Secure Startup with SKINIT" Introduce skinit_enable_intr() and call it from cpu_init(), next to the enable_nmis() which performs a related function for tboot startups. Also introduce ap_boot_method to control the sequence of actions for AP boot. Signed-off-by: Marek Kasiewicz Signed-off-by: Norbert Kamiński Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monné CC: Wei Liu CC: Marek Kasiewicz CC: Norbert Kamiński CC: Michal Zygowski CC: Piotr Krol CC: Krystian Hebel CC: Daniel P. Smith CC: Rich Persaud CC: Christopher Clark --- xen/arch/x86/cpu/common.c | 32 ++++++++++++++++++++++++++++++++ xen/arch/x86/smpboot.c | 12 +++++++++++- xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/msr-index.h | 1 + xen/include/asm-x86/processor.h | 6 ++++++ 5 files changed, 51 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c index a684519a20..d9a103e721 100644 --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -834,6 +834,29 @@ void load_system_tables(void) BUG_ON(system_state != SYS_STATE_early_boot && (stack_bottom & 0xf)); } +static void skinit_enable_intr(void) +{ + uint64_t val; + + /* + * If the platform is performing a Secure Launch via SKINIT + * INIT_REDIRECTION flag will be active. + */ + if ( !cpu_has_skinit || rdmsr_safe(MSR_K8_VM_CR, val) || + !(val & VM_CR_INIT_REDIRECTION) ) + return; + + ap_boot_method = AP_BOOT_SKINIT; + + /* + * We don't yet handle #SX. Disable INIT_REDIRECTION first, before + * enabling GIF, so a pending INIT resets us, rather than causing a + * panic due to an unknown exception. + */ + wrmsr_safe(MSR_K8_VM_CR, val & ~VM_CR_INIT_REDIRECTION); + asm volatile ( ".byte 0x0f,0x01,0xdc" /* STGI */ ::: "memory" ); +} + /* * cpu_init() initializes state that is per-CPU. Some data is already * initialized (naturally) in the bootstrap process, such as the GDT @@ -865,6 +888,15 @@ void cpu_init(void) write_debugreg(6, X86_DR6_DEFAULT); write_debugreg(7, X86_DR7_DEFAULT); + /* + * If the platform is performing a Secure Launch via SKINIT, GIF is + * clear to prevent external interrupts interfering with Secure + * Startup. Re-enable all interrupts now that we are suitably set up. + * + * Refer to AMD APM Vol2 15.27 "Secure Startup with SKINIT". + */ + skinit_enable_intr(); + /* Enable NMIs. Our loader (e.g. Tboot) may have left them disabled. */ enable_nmis(); } diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index 195e3681b4..0f11fea7be 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -49,6 +49,7 @@ #include unsigned long __read_mostly trampoline_phys; +enum ap_boot_method __read_mostly ap_boot_method = AP_BOOT_NORMAL; /* representing HT siblings of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask); @@ -424,7 +425,16 @@ static int wakeup_secondary_cpu(int phys_apicid, unsigned long start_eip) { unsigned long send_status = 0, accept_status = 0; int maxlvt, timeout, i; - bool send_INIT = true; + + /* + * Normal AP startup uses an INIT-SIPI-SIPI sequence. + * + * When using SKINIT for Secure Startup, the INIT IPI must be skipped, so + * that SIPI is the first interrupt the AP sees. + * + * Refer to AMD APM Vol2 15.27 "Secure Startup with SKINIT". + */ + bool send_INIT = ap_boot_method != AP_BOOT_SKINIT; /* * Some versions of tboot might be able to handle the entire wake sequence diff --git a/xen/include/asm-x86/cpufeature.h b/xen/include/asm-x86/cpufeature.h index ad3d84bdde..f62e526a96 100644 --- a/xen/include/asm-x86/cpufeature.h +++ b/xen/include/asm-x86/cpufeature.h @@ -76,6 +76,7 @@ #define cpu_has_svm boot_cpu_has(X86_FEATURE_SVM) #define cpu_has_sse4a boot_cpu_has(X86_FEATURE_SSE4A) #define cpu_has_xop boot_cpu_has(X86_FEATURE_XOP) +#define cpu_has_skinit boot_cpu_has(X86_FEATURE_SKINIT) #define cpu_has_fma4 boot_cpu_has(X86_FEATURE_FMA4) #define cpu_has_tbm boot_cpu_has(X86_FEATURE_TBM) diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h index ff583cf0ed..1f5a5d0e38 100644 --- a/xen/include/asm-x86/msr-index.h +++ b/xen/include/asm-x86/msr-index.h @@ -117,6 +117,7 @@ #define PASID_VALID (_AC(1, ULL) << 31) #define MSR_K8_VM_CR 0xc0010114 +#define VM_CR_INIT_REDIRECTION (_AC(1, ULL) << 1) #define VM_CR_SVM_DISABLE (_AC(1, ULL) << 4) /* diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h index 9acb80fdcd..d5f467d245 100644 --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -631,6 +631,12 @@ static inline uint8_t get_cpu_family(uint32_t raw, uint8_t *model, extern int8_t opt_tsx, cpu_has_tsx_ctrl; void tsx_init(void); +enum ap_boot_method { + AP_BOOT_NORMAL, + AP_BOOT_SKINIT, +}; +extern enum ap_boot_method ap_boot_method; + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_X86_PROCESSOR_H */ From patchwork Fri Jan 29 10:45:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 12055695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 089B2C433DB for ; Fri, 29 Jan 2021 10:46:10 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AC3B664ECB for ; Fri, 29 Jan 2021 10:46:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AC3B664ECB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.77956.141574 (Exim 4.92) (envelope-from ) id 1l5RHt-0000IX-8J; Fri, 29 Jan 2021 10:46:01 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 77956.141574; Fri, 29 Jan 2021 10:46:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l5RHt-0000IQ-5I; Fri, 29 Jan 2021 10:46:01 +0000 Received: by outflank-mailman (input) for mailman id 77956; Fri, 29 Jan 2021 10:46:00 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1l5RHs-0000II-61 for xen-devel@lists.xenproject.org; Fri, 29 Jan 2021 10:46:00 +0000 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 3b96bfaa-ea30-43aa-90ae-e97994574f26; Fri, 29 Jan 2021 10:45:58 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3b96bfaa-ea30-43aa-90ae-e97994574f26 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1611917158; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Tv4cBwEAsvdZj0In03grL69EwnWIPYvdtYFViraxlCw=; b=heEQui6/Hcfj0UIueMzpsIIFbSwgt0e9pkTGgs5g2REpiN2iueTvlQGg 6yU0PW+qupV9SszUu1xIqcpsDv0EYDB1R4Y2uONJzrVisisPOAqE2x1X0 /FZ+A9WHGriyXCcN4VWGVTQTEXjoln8pk/OoIZVyk3BL+Hq4xyYZ8e8pc 0=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: SpVlrpEWSl+OnuyIfttemGZHhAdmBXE1B+7P38b1dZKAIwUHZvGBw22W5LaN7rN4AujlHEIiV8 zqnPznKbd4FpO3Z5lRJltfOu4WiRCEXJ5T2X1EPodAGQoSoE2eEjwHybffJ4d26DLbwNd1hhKf Jp86OoGqClOGEWFSR0/a9hk5SwDQpg1Rfpw0/ZSUSuUTFMNGTEoEUJQbAeewwes1SiBwkSqKcm f8x0JRowtothAAFRIlVqkaJ5z2lPR8cYfSjCJhfB1HFMrKxQ8mAmCysS+O99BIDttj/8QVz1Kj 4OE= X-SBRS: 5.1 X-MesageID: 36517224 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.79,385,1602561600"; d="scan'208";a="36517224" From: Andrew Cooper To: Xen-devel CC: Andrew Cooper , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu Subject: [PATCH 2.5/3] x86/svm: Reimplement VMRUN/STGI/CLGI with new asm-defns.h infrastructure Date: Fri, 29 Jan 2021 10:45:40 +0000 Message-ID: <20210129104540.32137-1-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210115231046.31785-1-andrew.cooper3@citrix.com> References: <20210115231046.31785-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 ... in order to reuse stgi elsewhere. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Roger Pau Monné CC: Wei Liu --- xen/arch/x86/hvm/svm/entry.S | 10 +++------- xen/include/asm-x86/asm-defns.h | 12 ++++++++++++ 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/xen/arch/x86/hvm/svm/entry.S b/xen/arch/x86/hvm/svm/entry.S index 1d2df08e89..e208a4b32a 100644 --- a/xen/arch/x86/hvm/svm/entry.S +++ b/xen/arch/x86/hvm/svm/entry.S @@ -22,10 +22,6 @@ #include #include -#define VMRUN .byte 0x0F,0x01,0xD8 -#define STGI .byte 0x0F,0x01,0xDC -#define CLGI .byte 0x0F,0x01,0xDD - ENTRY(svm_asm_do_resume) GET_CURRENT(bx) .Lsvm_do_resume: @@ -82,9 +78,9 @@ __UNLIKELY_END(nsvm_hap) pop %rsi pop %rdi - CLGI + clgi sti - VMRUN + vmrun SAVE_ALL @@ -93,7 +89,7 @@ __UNLIKELY_END(nsvm_hap) SPEC_CTRL_ENTRY_FROM_HVM /* Req: b=curr %rsp=regs/cpuinfo, Clob: acd */ /* WARNING! `ret`, `call *`, `jmp *` not safe before this point. */ - STGI + stgi GLOBAL(svm_stgi_label) mov %rsp,%rdi call svm_vmexit_handler diff --git a/xen/include/asm-x86/asm-defns.h b/xen/include/asm-x86/asm-defns.h index 43f4868d40..2e3ec0ac01 100644 --- a/xen/include/asm-x86/asm-defns.h +++ b/xen/include/asm-x86/asm-defns.h @@ -8,6 +8,18 @@ .endm #endif +.macro vmrun + .byte 0x0f, 0x01, 0xd8 +.endm + +.macro stgi + .byte 0x0f, 0x01, 0xdc +.endm + +.macro clgi + .byte 0x0f, 0x01, 0xdd +.endm + .macro INDIRECT_BRANCH insn:req arg:req /* * Create an indirect branch. insn is one of call/jmp, arg is a single