From patchwork Tue Apr 15 15:46:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: D Scott Phillips X-Patchwork-Id: 14052427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 501FDC369AB for ; Tue, 15 Apr 2025 16:22:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:To: From:Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=G+rmAVxdnox/DDoFjtFbW1zT8e4hPhS5Q7ZU8ubJ6Tc=; b=lRnZ0h4jnomdmaolMM5tUSrd58 1w7aQGfFp6eDd9KVMSCo2htqa11I5LR7eSX8C74VSSKXJxkT12O9f7dCGeTrdimkUjK9kkgWIw8Vt box6Xa+dGXgCZ9jhKk2z9+jutWCXFgaS6QNlmq1vBQ0r9naaBA+a2USVgVQnss8aYCXuNv1xwlv7j CM5XYQdlPUvs1HBfEfz3obj/gtD4nDf89phEdMdl8W8A81Uf6V9A7g6uxTxixDYAeG17VdAHg/KqQ FQr4wIYzThE2f082BGa9Fo+P89tu1IQg8v/QK+/KZyebyUM4sv2gDGTeniKwuYZfODxLS33lXH1Lp dqRZjJ+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u4j30-00000006Ogp-2vmS; Tue, 15 Apr 2025 16:22:06 +0000 Received: from mail-mw2nam12on20726.outbound.protection.outlook.com ([2a01:111:f403:200a::726] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u4iVD-00000006Ive-0n8L for linux-arm-kernel@lists.infradead.org; Tue, 15 Apr 2025 15:47:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NsqZifO6+tJRjDW1K/wBeE6pp1wMJWtJQGmQjpLXhBWV2gYXoHC0Hsi7LUsO/91SG1+kVE5izhj6N3lfLBASsElNVpMRE3IeSPsV7eYFcPxrSjojcBk39L7Yk3soGan0ssHfmiXhXmZ4xoBtdQF/PIysXHGXkSOU1OYdPsJlsaV3z9KP69yKZj7qdA2aPP+uynJ+YhGmLHWpJJDHrNhoFpEw+qwrRMn83NFO0m+mZR+/kGwRtDIVAHDgFWZjkf/YRgJJM4EuNp9ttEjiJaUVzxbO8LtVLOj+l1XdSmzNpUebObsRMdtOz3tLzv7mhVANCIsQ/TjJ6efp6kQP62KmsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G+rmAVxdnox/DDoFjtFbW1zT8e4hPhS5Q7ZU8ubJ6Tc=; b=YnY1S4BLR8tmnlJ7+dhFUDmZw1Lt5uglg5/bH+krH2vROTcpyfVGrOkXAntHOnCYpWFTichiJU1vVg7PhNEA4A9l9a9rCrzdGbOusduBK87guZsgCdxKDTuInlfOn6df47nzsb/MleVjvZl326Z9R4EDpRVVT4SJfWOj+xcFS3430sDHqcs6doez58prSguG72WueWjVI7zZ3ER1pGry3xQRO5uXgcMtkGLi1KjP5YgcKpNfVjAW9BgMKePkNjyk6zkIjMevoaT7y7DS2l62sbbcMHR757INpnR7CGhJvUtnbrs+fN06gS8JMPIE1WTm/ZRC0ikjpqZGn1F0RYLcgw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G+rmAVxdnox/DDoFjtFbW1zT8e4hPhS5Q7ZU8ubJ6Tc=; b=oq+FBJkdxDFlrSu7tjrBQ3ZDREDMa/DlV3qOxhwlPKWG4qsgU5fXtslru684mMmXZPr6A9DuPb3Hop5y8Jzn5kH7kiS7njslftnvvuV7xr+yBhMTfW6v1wqxICRVnQfj3sCJv5D9ZIrop64wuywpl9SyLyYbBBc1KrQLwTPY/7M= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from LV2PR01MB7792.prod.exchangelabs.com (2603:10b6:408:14f::10) by BL3PR01MB7099.prod.exchangelabs.com (2603:10b6:208:33a::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.35; Tue, 15 Apr 2025 15:47:04 +0000 Received: from LV2PR01MB7792.prod.exchangelabs.com ([fe80::2349:ebe6:2948:adb9]) by LV2PR01MB7792.prod.exchangelabs.com ([fe80::2349:ebe6:2948:adb9%5]) with mapi id 15.20.8632.030; Tue, 15 Apr 2025 15:47:04 +0000 From: D Scott Phillips To: Catalin Marinas , Joey Gouly , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/2] KVM: arm64: Avoid blocking irqs when tlb flushing/ATing with HCR.TGE=0 Date: Tue, 15 Apr 2025 08:46:56 -0700 Message-ID: <20250415154656.1698522-2-scott@os.amperecomputing.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250415154656.1698522-1-scott@os.amperecomputing.com> References: <20250415154656.1698522-1-scott@os.amperecomputing.com> X-ClientProxiedBy: CH2PR17CA0029.namprd17.prod.outlook.com (2603:10b6:610:53::39) To LV2PR01MB7792.prod.exchangelabs.com (2603:10b6:408:14f::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR01MB7792:EE_|BL3PR01MB7099:EE_ X-MS-Office365-Filtering-Correlation-Id: 0f510695-38f8-4fe0-b21c-08dd7c34c502 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|52116014|366016|921020|38350700014; X-Microsoft-Antispam-Message-Info: mcp+U42Xa1VvYSEsX6xh8GJAg5CT/cOxeVNuTkS36kX5yqP098NYqzkvGcmJzQkw1ViIK2xoago1D6TcHatyjhgCiyEpy2dfgA6nkhmX2lpVfbAKRSvNpIqk6A16pKY6BStB1oZIRMPxiMURYqetcyss6fcnnJ+XNDwpv36LeZTj/lykaEI6OUJBCvSVmmYAtb4RwOOa2mG4aYhYnb7WIv9NhqyDMV2ETpjwlyqi/+y0fv7NqZ6EWlWf6Sxgc2N3b8eM2mTdUdP+VjlXWXsHb1c4HA/6wAkQfGQwND8LzAfBpGgI0GLqGvFbz0+bRFliF/kxcJvXAA+yfzoWUdB32HhciZ0Rl6SZp8OWb8sbGzOuLiHKOWsJICM+EXsejWSVkxacH1z9Jue64/ZQ57zUvXcQSNTaIYNLP4FGvhM2Qdk/qhFkCI2yCRCHJAzQlDfY8uLxU/G007V0ytVjwpHeelx6fMpREwqtX0nvhGuu5FVam4JuLeyrgZuELX9cIc7YML3Y8O/nl926M93Rk8p9BmhR8QRZS331N4Ny1257ix5OOUZfk3zuR+QtGix1NXJpdcjhzSQ15TbcLbGxZseY5zAb5KVwQOMJ2WWcEcfknsgFWVdPct+yChu960EHKARo4YdOkRwbTOnjwSShui8rS4CFJQUioutZRW1G3AQnn0GDjayUQpiIS7C7LtRkBObk15y5Zr4jYOmLQWIjTTKwu/7O5FbGjvCHvrH7CMKbc30YNi95Go4kxNGiKIjRD99pcRSffIBgjk7Omm4ChHQ1IjosuGXeMkgPuyLBcidsHg5veiEnFEvKjDo6wq3WMEGz6WIZ2rA5HFZ7EbDryiE4x9pf+iYT2zqszacREvTVd1qeiCaq9HuEn0UTZyuIoFS3ZdH351sISan/Lhm/bunq6wjy8uua792K46fA3JB6t7xaP98Q+VF7ugPe+87GpRgC+8Afo5rbJ1LOTWN6qeFzXRwOtx7uQPXFmHmyFRx9qNAOA7uBcaznDqPz6dI3QC/mfP9kZe3s90nuBOoj41/z/Z1S8WOfs6E2WBU1ExXXw8ZpO4K3lXNTScpjiOtpwKK6OWdBvWqV4k1K3vPCkV/4l+4Ls+IB4VJscoMbbmgtcIJ9cQhmkXgIDt8CKXKcrXHY9X23/DLyuL7FR1SJ6DxyayIgeH3jvbTW4jMvABhIz1St0dje6RsKQm4yu49SZyv41YwxHm0Wx5EEdBvVn30mDbnonkTZQkmAXgv/ZgBBtE1QM9f9K2BUjxc2tzGUofDOikxQAbOK65nuZ5w/UDJA9HQElzJOIZaFgex+idYJWFUJOVHjIzdg4AVzNw4GCDsYqrdBJppCra36rNAeJ4mq+/wfPn9aSXhsN/27p9He4LTo7IfxNTvjkaH9JhjIzNSKCDWhsytC2QY5rW4v6UB8Et3X0WgorFEWz9naRDIlIRCbKPhspgV1bzQVx1lHzSpUZkNTumzItVy+twdIoPh5L4hz6MeKexCQC4i4eLh6MsE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR01MB7792.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(52116014)(366016)(921020)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Obz5u96aXH2tAdCsXHs81UDfl6dq+cyUwLipqRvoJPisTb8R3Gfp1odmrqVeUi/d2dvCqSdblGHGmk1UXKWhl8nrb/B3yNJGHMTJjfTuu2xCxvgqXyeH9YvJ+dNprtxd2j0vsy3brCl5E4E9FqMM8OcPs0ARLqkvFxt8QHvW3KbeRzu9sVGr6BQOOLVsT74nX497YVgp9LxjalujpZSafpf0bT7Wae1asMvCxTOe8PVTQm+Rd5oiVtY5/UBBqc33JRwl9dHg03+KxMj9nD45Gs8Y9rzQn4XBEqN5+JjI+uuxjU0CJnSPwBB/I8q95RLV3W2uT9veABv8WrzXGvmz3xYVvpQPidnsjRSg91SkUkNXcwmNNOIF0sqyhZslTaZlAIUuUDms8PpbkKZmkbwKw8OzrNPXuuvdyGptcK7kBqeoh7BI9qelO7+ba5MHguf/J0NhWMc9tL9R7Trn2pe1vGsdCBPdNJKHpVAUJOA9iViSMsgK2js9xTRfr7yOeNUGnRTDn9nHXve/YxUwI5Slx5TilR3EVeAGvF3DvrlB+StW6a6PsydzE1WqWdVOjcfL7FidjIiVwndLRlE5abv+pwEHdEy1letZrXwUyDo6LOqzVlPmekANCPU4JF5Mm0Dez+9cPAKA92mFLv2ngcSnfaoapwm7WDFny+ZZyv4XqxrxPkwcZEY1L+GJ+bNh9LzCeMKarleONMX6m/m4yd5yLUspSCSV2AHVrn5p7+d4cuqfQgrebD3qH+xpYleFiRLFpEpzPUJaXNFMBbDoqo6Q6UBiEWXUE5tSOxlEschLMxQuNNm1Js2WOBkw0hsiBWKawXI2BtuYjI/OrnRts8aGt+iP32Z9gnXmduDefXz+aZT0G6VARDJCaajX+2006XHeydnqewpfQVShtVpeZTaa8kSBLqDaOZMAv/rXOnEw6MhA6D8lxyfO3hw2ijKZyls0s/+0R6v47Wkty8FFCPXlxyJ3o5H9/BmY7zI6oUTQjFeVTUX+N6LmmbNrRTbktelh8OiY4KQDL3yo9KUUETkL0weEQzxTWSq0GiCtwejjhAueBi68vCZyYmL1qxjC5tBWO144Zwou92popnFsmZvaUMUWhoJNFiqoL8BUehBn2Wxks/FCY4rqcx/M7mwWxCnuY3Uw0Z2LDxpZ5ndKQemYY6caRL+XycHwiVVwdYc42NPSDHx0QuUnU4xGd2EWGfjn0rKK3yQUhmA3hrKoaXq6GsmIeAOpTlI4TcSY4X76YsCeZFUFYGGRdnrH2X5wKDMj5Ol/FIHYQ41f26D8F80DQHazzQzouDGTcm3qVt1ted4E8I3tNpEN7hpK5ZVdg0Vm7709kJGEV6pgkUpMlJtMaFLrGmtY9V9eYJYKMjy7xtxWlXOcM82TPFDXjIJ1qatS8fDGI6A41cOUsJ+cbM4psAJmd1VJK8w2m3C3dIfebwRIioQt+ySE/QaZ4+A7n0o40Qfho+/qxq2hWmOv0630d54IQkXcCMweX80IBa7fLWgCBR322ncuxILFsfOf3FxaiVwdeZfGdH+UQ4YTE/FYKsg684FGH+1G/8ZCLkRxaw8gsrJJgYtZszwo2SDMGBdoFE/mUF1tPC34g2z+WcXy/JHNEzj60lCPTmaEOydZqrY= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0f510695-38f8-4fe0-b21c-08dd7c34c502 X-MS-Exchange-CrossTenant-AuthSource: LV2PR01MB7792.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2025 15:47:04.3067 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WfivTAjZBtoZD1LET26CrIhrxp9b5W/7YPqiP3liPbimFVW2yCHXjWIAalMm7BvE52inbSqjsN8BD/jZHXwhBgJLvsB2FRErw4HvLtHo16nLxGWCVIqtUUY1ycz8gGq+ X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR01MB7099 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250415_084711_230491_B8BB0FE4 X-CRM114-Status: GOOD ( 14.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org pNMIs are intended to be deliverable during operations like guest tlb flushing or nested AT, however the setting of HCR_EL2 controls are accidentally blocking async exceptions. You can see this by doing: # perf record -e cycles:Hk -g ./dirty_log_perf_test -m 3 \ -i 4 -s anonymous -b 4G -v 32 Where no samples will be collected in __kvm_tlb_flush_vmid_ipa_nsh() between enter_vmid_context() and exit_vmid_context() then many samples are collected right after the write to HCR_EL2 in exit_vmid_context(), where pNMIs actually get unmasked. Set HCR_EL2.IMO so that pNMIs are not blocked during guest tlb flushing or nested AT. Signed-off-by: D Scott Phillips --- arch/arm64/include/asm/hardirq.h | 3 ++- arch/arm64/kvm/at.c | 4 +++- arch/arm64/kvm/hyp/vhe/tlb.c | 10 ++++++++++ 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/hardirq.h b/arch/arm64/include/asm/hardirq.h index cbfa7b6f2e098..6eb3f93851023 100644 --- a/arch/arm64/include/asm/hardirq.h +++ b/arch/arm64/include/asm/hardirq.h @@ -41,7 +41,8 @@ do { \ \ ___hcr = read_sysreg(hcr_el2); \ if (!(___hcr & HCR_TGE)) { \ - write_sysreg(___hcr | HCR_TGE, hcr_el2); \ + write_sysreg((___hcr & ~(HCR_AMO | HCR_IMO | HCR_FMO)) |\ + HCR_TGE, hcr_el2); \ isb(); \ } \ /* \ diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c index ff4b06ce661af..f31f0d78c5813 100644 --- a/arch/arm64/kvm/at.c +++ b/arch/arm64/kvm/at.c @@ -1269,7 +1269,9 @@ static u64 __kvm_at_s1e01_fast(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) skip_mmu_switch: /* Clear TGE, enable S2 translation, we're rolling */ - write_sysreg((config.hcr & ~HCR_TGE) | HCR_VM, hcr_el2); + write_sysreg((config.hcr & ~HCR_TGE) | + HCR_AMO | HCR_IMO | HCR_FMO | HCR_VM, + hcr_el2); isb(); switch (op) { diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 3d50a1bd2bdbc..ecb700bab3b8f 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -55,6 +55,15 @@ static void enter_vmid_context(struct kvm_s2_mmu *mmu, * bits. Changing E2H is impossible (goodbye TTBR1_EL2), so * let's flip TGE before executing the TLB operation. * + * One other fun complication to consider is the target EL for + * asynchronous exceptions. We want to allow NMIs during tlb flushing, + * so we need to ensure that the target EL for IRQs remains as EL2. + * HCR_EL2.{E2H,TGE,IMO} = {1,0,0} would set the target EL for IRQs as + * EL1, and IRQs at EL2 would be "C" (Interrupts not taken regardless + * of the value of interrupt masks). So we need to set + * HCR_EL2.{E2H,TGE,IMO} = {1,0,1} so that NMIs will still be + * delivered. + * * ARM erratum 1165522 requires some special handling (again), * as we need to make sure both stages of translation are in * place before clearing TGE. __load_stage2() already @@ -63,6 +72,7 @@ static void enter_vmid_context(struct kvm_s2_mmu *mmu, __load_stage2(mmu, mmu->arch); val = read_sysreg(hcr_el2); val &= ~HCR_TGE; + val |= HCR_AMO | HCR_IMO | HCR_FMO; write_sysreg(val, hcr_el2); isb(); }