From patchwork Sat Mar 15 03:06:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017699 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B086318A6DF for ; Sat, 15 Mar 2025 03:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007996; cv=none; b=cCWoGs2LOXPNjbkEB7/R5PZlaAoIO7cp8uQKnYwxnsoAoFTJbfS2T4wvAJOOxf+zirJhpbANgwW09XESzvnN8tZ9WOeqt8W3dwgBPBIjmYKW0x6l++zOwTrhk1GFZ0eardDNzncWN0cPfKHZzvGHdwhhtikzh463cxhSeLAvYws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007996; c=relaxed/simple; bh=ORxMRxanum7dJ4QhbbOoW6MS6dV40SegF61qcn5cQK8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YWyUurtlCfZT2BoWRVolhoodbNSibC5lEIn366zqFPDwYzMntxo4aM9ldV8pLOk1B3MEyRtMZ/7Zpf4/ixpJCR4LOZrpT28+7eAOonxGVkwSFoDt/AzvEHuFNZQbnqiVlWlPLdxinehNxyg85ewav1fhjVq1WXaTfxgi8xZXdIA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UAMT/Ldj; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UAMT/Ldj" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3011c150130so499063a91.2 for ; Fri, 14 Mar 2025 20:06:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007994; x=1742612794; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=A+D7/CZc6x+lcAi6xbF5IFcWIU9h8go1MiK4EHRJXP4=; b=UAMT/LdjeX0edyYremYgTvkrkszCPGCpjVyAYX62V4ktJUi/Z/M1s/Qewwj03Gvphb 2H4ZickLHWjxa5xLDisv8t4jq7XBY5DL0zFG28iyXQduvGGaiYRSUJz8PCPav5XOQZqW YEBYZMlqBkAzLI4SQPS4b7QYouAKWh+8laX7hFkwSCjyp+f2XgR8dTuHLeGztbx9r7zo OMfHDtzBE+fj4iYBAdd/PvUeILJy+kD/YFbt4YNzL5LuHQMJjA+TcnOzFIGuX5iL8Un5 6HbE1FnABlkL9nZ8+uURlYasmbds3ssIkqV+0UF+xpdNZFZev7Bi17V9BkDo+kK54cw9 e1iA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007994; x=1742612794; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=A+D7/CZc6x+lcAi6xbF5IFcWIU9h8go1MiK4EHRJXP4=; b=A3MHD8N5IU5vof16oUIR+wyYAUCknLtRXiEJRGQrw6htp24lAGjhXNMZrXbBF8kn8E HvAI+Q3ubZMD3SxVMaCs4VYUrwieI6UxJ+rkiMGWxXI8KvJaLKC6DKC3Pq+EZIEzR/0C G52xMzRgeLytkGrSFC+tX7Z6TBlpwyNrJ4pjg/wZ849ROz782U1Pn70bYuUYsgw445PV 4UBlU7+Dg0z4OEfhWmhrNH59X2SQBxY3ju3MIJSwXFkmy3j/OZTsCpKvEEVXL0/bOW9O aZGE2C3S6MnuNpaGqtrB4QaoNeJk2xKvyLbIuiqXaU33q0uQfiE2u0sEb3Lg/v/8vhm9 R28g== X-Forwarded-Encrypted: i=1; AJvYcCXYSXmO8h4QnCjen0AC0ZbTWv5YAbe7QLH2Aps2Sv67y+buoCEQZYJ6NiDrai+9lNdm/lg=@vger.kernel.org X-Gm-Message-State: AOJu0Yzx1ajhWlzbbI5GGn/m/zFFVfFB/xztXXUJXlUznkIY9zQrXr0E IdcstFDx0LcL/UnqAM8kD53UamcbBaRo/D3ez4gMsxEC6qL10C0u2hfbGGNd5zIEuHZORgjks0V iHA== X-Google-Smtp-Source: AGHT+IH3X+L+I/21qttxgDTkKxHuhAtYSP/nASSYaa/gcSlKKoQ0V6Lk918oh7tBqn4FZDeDooNPZD4EXAU= X-Received: from pjl6.prod.google.com ([2002:a17:90b:2f86:b0:2ee:3128:390f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:fc4c:b0:2f4:4500:bb4d with SMTP id 98e67ed59e1d1-30151cc8c11mr5807708a91.20.1742007993991; Fri, 14 Mar 2025 20:06:33 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:22 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-2-seanjc@google.com> Subject: [PATCH 1/8] x86/irq: Ensure initial PIR loads are performed exactly once From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Ensure the PIR is read exactly once at the start of handle_pending_pir(), to guarantee that checking for an outstanding posted interrupt in a given chuck doesn't reload the chunk from the "real" PIR. Functionally, a reload is benign, but it would defeat the purpose of pre-loading into a copy. Fixes: 1b03d82ba15e ("x86/irq: Install posted MSI notification handler") Cc: Jacob Pan Signed-off-by: Sean Christopherson --- arch/x86/kernel/irq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 385e3a5fc304..9e5263887ff6 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -412,7 +412,7 @@ static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) bool handled = false; for (i = 0; i < 4; i++) - pir_copy[i] = pir[i]; + pir_copy[i] = READ_ONCE(pir[i]); for (i = 0; i < 4; i++) { if (!pir_copy[i]) From patchwork Sat Mar 15 03:06:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017700 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 558CD18E023 for ; Sat, 15 Mar 2025 03:06:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007997; cv=none; b=imROPvga3n4LLgDz4fZjRt9Mn2qKhrsN6gab7i4O8pqbHZvdpjkHStSW4LezlSERZ1zoU6jRxGmBBoJ/1Sw2LOX8ueclNYFmg1z1mSHh+V89uFHuFFARZv+9CpMNAxqt47++4dWswgcSBk6GYIo4dPOxtRu7RM+apjn10XI8bJo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007997; c=relaxed/simple; bh=+WK1bltbWDTrbagBZEBAiy7h0kT2VEa0FdxHN4yhE9Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NegGDO3hofPz7FvBsBQzsCA9als5aEzHB30EeLBTM1LROcBDmKnjPPtSaf/MG5lLI0kHm3adAI/e4SgeQ74lCjNcgnzHmFkCmctDZnzydpiTkNSILO/vV2u4ySA6HBptwvKSDGVUhA6vZcMjf4dNv7r4M9VGbN8h8/0x7MxQIz4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rkaKDIb5; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rkaKDIb5" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ff7cf599beso712122a91.0 for ; Fri, 14 Mar 2025 20:06:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007996; x=1742612796; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=vHe6yayu6Pa5fekV1jp+JV4SQqE+N7492v9RWvirBCU=; b=rkaKDIb5IkP2/N6o1yzoZwGPAerPT2LrFdFzI6HhSTjTCFDQ7TLxX1y9HjPlTNhSUk 2UZU4RIiQiugLzYpehug2+xrSxYHZb4JIVLgCd/hmE2SbTZC/lIEKTPsdx7PfRr/iis6 YCtiSH7EKRXmstg1DMbiy9/i2ynO5xOjkxqqOLm560glBJXGSp301U1kOuv6EwX+J76n SVECsJc66XIj5MXLZgeGvFfUaRlA0WfMzmTKXiPFxgcJXm+2qwCeId/ZJZClkMCEja7Q GK5PYr9bxWsxMgWInBILKkXSgIiFcHnSMvc9kXIWfjQlb4XQkJcNMev4SS6mAe6qzlyT /2jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007996; x=1742612796; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vHe6yayu6Pa5fekV1jp+JV4SQqE+N7492v9RWvirBCU=; b=SiW+wYUJSujbpQVMjwWRbayYoPGKp/bDSf0q5lhzBr4mC03Q1OKG3c8vlyoTv1b8Af Os8s6QwPUhsB8qIDv40A1ZXh8qYb7gS6+SkSIy3yORTbs0Q3F3nr/ql6ocVVbz9W9I9q sB3VV+NVSiNx3P3q3d7B5VGHhwoCrRcwtSnpo08u7ibQsqM0O4KpWRmHzNiRY5hfX001 hzKTx3wOqAwGvQVIA6u5RVxZ6SCM0xLCjF0wZTgx28ef33A0im56qBmBjuNJhOS5myO4 SCXgEV8fzzJGlX3VGv0KgG8D0pga2a1XJr74pW8t+ZJa+YVZCMrxPwBpKOmifHMkf9IF pASQ== X-Forwarded-Encrypted: i=1; AJvYcCWGLbFVQqdZmKG1Hjz2FeltH5P2I4IE7XjOEJGsl/061fPmp8Ky4CpJYGPRGjU1p1Pn62M=@vger.kernel.org X-Gm-Message-State: AOJu0Yw5I3NMIvy4DA1n8og6Rkwc37BOggGo230mgBf429Fu35Oh/oXJ YpAjB9Hxv2aLuNJoQAFR27CYivFsVHU4dL5AusQp6YML15U37Z1XlOI20Tiitupo4jy9OCBSLJS QOQ== X-Google-Smtp-Source: AGHT+IHm3ikpa45S+MzJ9ChgrVNIn//2DPUl62WT86s6estUNWscdmElpdksr3RkSuK4oMZ2hPanzmR+FmA= X-Received: from pjbsc17.prod.google.com ([2002:a17:90b:5111:b0:2ea:aa56:49c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5387:b0:2ff:4a8d:74f8 with SMTP id 98e67ed59e1d1-30151c9a341mr5024590a91.6.1742007995777; Fri, 14 Mar 2025 20:06:35 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:23 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-3-seanjc@google.com> Subject: [PATCH 2/8] x86/irq: Track if IRQ was found in PIR during initial loop (to load PIR vals) From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Track whether or not at least one IRQ was found in PIR during the initial loop to load PIR chunks from memory. Doing so generates slightly better code (arguably), especially for the case where there are no pending IRQs. Note, while PIR can be modified between the initial load and the XCHG, it can only _gain_ new IRQs, i.e. there is no danger of a false positive due to the final version of pir_copy[] being empty. Opportunistically rename the boolean in anticipation of moving the PIR accesses to a common helper that can be shared by posted MSIs and KVM. Old: <+74>: test %rdx,%rdx <+77>: je 0xffffffff812bbeb0 <+88>: mov $0x1,%dl> <+90>: test %rsi,%rsi <+93>: je 0xffffffff812bbe8c <+106>: mov $0x1,%dl <+108>: test %rcx,%rcx <+111>: je 0xffffffff812bbe9e <+124>: mov $0x1,%dl <+126>: test %rax,%rax <+129>: je 0xffffffff812bbeb9 <+142>: jmp 0xffffffff812bbec1 <+144>: xor %edx,%edx <+146>: test %rsi,%rsi <+149>: jne 0xffffffff812bbe7f <+151>: jmp 0xffffffff812bbe8c <+153>: test %dl,%dl <+155>: je 0xffffffff812bbf8e New: <+74>: mov %rax,%r8 <+77>: or %rcx,%r8 <+80>: or %rdx,%r8 <+83>: or %rsi,%r8 <+86>: setne %bl <+89>: je 0xffffffff812bbf88 <+95>: test %rsi,%rsi <+98>: je 0xffffffff812bbe8d <+109>: test %rdx,%rdx <+112>: je 0xffffffff812bbe9d <+125>: test %rcx,%rcx <+128>: je 0xffffffff812bbead <+141>: test %rax,%rax <+144>: je 0xffffffff812bbebd Signed-off-by: Sean Christopherson --- arch/x86/kernel/irq.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 9e5263887ff6..3f95b00ccd7f 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -409,25 +409,28 @@ static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) { int i, vec = FIRST_EXTERNAL_VECTOR; unsigned long pir_copy[4]; - bool handled = false; + bool found_irq = false; - for (i = 0; i < 4; i++) + for (i = 0; i < 4; i++) { pir_copy[i] = READ_ONCE(pir[i]); + if (pir_copy[i]) + found_irq = true; + } + + if (!found_irq) + return false; for (i = 0; i < 4; i++) { if (!pir_copy[i]) continue; pir_copy[i] = arch_xchg(&pir[i], 0); - handled = true; } - if (handled) { - for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) - call_irq_handler(vec, regs); - } + for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) + call_irq_handler(vec, regs); - return handled; + return true; } /* From patchwork Sat Mar 15 03:06:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017701 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D49AC1917E4 for ; Sat, 15 Mar 2025 03:06:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007999; cv=none; b=M4pjlyu3950BSvaOjUTTwZzvA5V+u3IYkeDXARAJoumcGVe4ZqxEexl1ac7SjynCdV6dNssKIkwWop9lZqEb3lcPWk0xRIKww4RnLbEm+kLwEoVc2wzXEatRHc8Yb/DnAGAKTOag54DbGEYc4oWTbc72XxdNlPUZ4mpoKRsAUJY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742007999; c=relaxed/simple; bh=3oNkXBgr8EIWda3cEEvkfRlryoj/MBVyRM1tKseboAk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XPOS3KIJ/IMGFb9j/FQ45U0pf5I1+H075szp6ZcOAjIAP/EroXxHdo6E8a9cRQmTqt9gBOYA7oVr1VmS+4cs7wmpuLsZ48iULuHQhXi1vgGB6VMWEogarI5a1/SPp9Oy7fxxHUBweB+5HgQSQ3VYxlU269y7JqKp6nsGJK4/zIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=DNe9Xnby; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DNe9Xnby" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ff8119b436so512008a91.0 for ; Fri, 14 Mar 2025 20:06:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007997; x=1742612797; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=+rGaV7EYztjwjHpumFCPp4cbhOrFaDcpQIEQlQEwrkg=; b=DNe9XnbyHkEO2O9917aoGeXFKUsA0CwhCFQts+2Z6d16hTD0cDo2z+tcdm3vi6pGRA mSGxOg1adjI3cwv98CcSuEaL5G0xzWwlGXooeAmp/Kie0SsGcV2w2Iys7hSBy4sErbd1 kI6ZiY2Auut6Ljo+4KkQO/n7p33prOYE5h+XYJ+PeN1bgpZn4fCCAzCF5ELhN+n9qICp lTT6vYBM/dXcCUfjbY8C223SFc3P9ddVos8YA+9jYVDjbaIKLpXDUo3NcrIvF/s9Bjx4 x6dU9nfkSvIzLt/dsUvWRJP2mFBMeuEKxPZMmW9k/OhNTYXaiU/bwxCdGAOL4dnOSqDq 5Yhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007997; x=1742612797; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+rGaV7EYztjwjHpumFCPp4cbhOrFaDcpQIEQlQEwrkg=; b=EAKR/zJU2/2Wf/7ZVmfyqg/9BiOiThUzkIPR7GDicuFpW3ZSpTNS9/akFUJrvqnxbU r9OR4Jz+9DIrNxUql594MTNX/GfwmCVjo2NDxYKS8ChxRDjTcS7+vP3o2Idr53VfprKo PBIyo1TRQeCDyHUqBz0ioDR5QKn9w0Jxkgavj5MQcN852DTpHJTdBOTQg1G6lJZCKlmV lkKyIcoRWOVB/CWj5dkySjLF/6zArdmHcFAWD04rtsscz6G3vP+4CJxVcVHPqbjEebfT T+WnVJEUHkfjQ4WdPSgiKKqC1r/5UMohKhTj4wiSauU+D2Cx1b/WEx3NL0BlUcqTNl9L j4VA== X-Forwarded-Encrypted: i=1; AJvYcCX6yzUHaT+GIz4EINw3Ne7oCiMOLaG2bLCdt0hroHDXT4+5GVVwlIulxHkbInf6AOXb5AY=@vger.kernel.org X-Gm-Message-State: AOJu0YzvK51fR/MkJIyP7TGz998yQCMJslA7e7z10kG2oCjgtlkgloam 51pFNPu00uj0ConZ2Z5T/IaEZHf0yLKNoQ8eP3nVo9ysRm0xEg7CyEQaE/3d+T/4XVYGtS+HhAs sZA== X-Google-Smtp-Source: AGHT+IHToyeqUaA2FjyTnWkl7XfPw0+f99lKhBPLNQjpZdYLGzY/BKFwEs/MhlWIFqJRpb5ryLcdl7TDFNM= X-Received: from pjbrr7.prod.google.com ([2002:a17:90b:2b47:b0:2ee:4b69:50e1]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4ec3:b0:2ee:d7d3:3019 with SMTP id 98e67ed59e1d1-30151c7a384mr7020001a91.12.1742007997207; Fri, 14 Mar 2025 20:06:37 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:24 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-4-seanjc@google.com> Subject: [PATCH 3/8] KVM: VMX: Ensure vIRR isn't reloaded at odd times when sync'ing PIR From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Read each vIRR exactly once when shuffling IRQs from the PIR to the vAPIC to ensure getting the highest priority IRQ from the chunk doesn't reload from the vIRR. In practice, a reload is functionally benign as vcpu->mutex is held and so IRQs can be consumed, i.e. new IRQs can appear, but existing IRQs can't disappear. Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 9dbc0f5d9865..cb4aeab914eb 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -667,7 +667,7 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) for (i = vec = 0; i <= 7; i++, vec += 32) { u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10); - irr_val = *p_irr; + irr_val = READ_ONCE(*p_irr); pir_val = READ_ONCE(pir[i]); if (pir_val) { From patchwork Sat Mar 15 03:06:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017702 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 900E9194C75 for ; Sat, 15 Mar 2025 03:06:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008002; cv=none; b=DPeVFbEpFgvQKtiviQM4L5rXaoFx5ddHkcZQKZK5WAqc9RQMkuUfUfvVpnjmwT04p2jiQj9sWmNLQJnvIOVUSLxQZvZZhPr7XLja87H79UDt4aF7kcjEpVG3uNYVJ11XA0pQH4ZSMT0p6UV/bFTV0GgiBoXAtQXFD/V6v2hbvVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008002; c=relaxed/simple; bh=Uta9a/3o/kiGmJmq8wNCcF+U80EDf0HSkBBuI1bpzK4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jJif3Nnr4T4h3vr9cc8nP6HbrV329nC5CFTeyQFO8KxbG8Hlc6Z18b1XKgU4yGMd9qY6LdITQZylUwUIjecLsvVvPnnRp1oyfsqH7VsftwMfilGSpCo3zExRG46ZFn2Q5QdQ+iThe+OJ/U7LljH38Oqtga3nmTj8887qwKCPz50= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GyccclzN; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GyccclzN" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ff82dd6de0so454763a91.0 for ; Fri, 14 Mar 2025 20:06:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742007999; x=1742612799; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=eXE4hRNdIiKSnjJQabn9+9Z7l4STEDkxZZthIWAaxQM=; b=GyccclzNZKflDwYfwsuNwTREjeT4y5XYdu85Irz+7wHGQgoBvF1CVb0oTBO79nelqj vY9ywEWq6u7ugedqR+nkgGzFK+nPh+UWKBa8lMiVzPg3ey7hVDscObqm8+Q81menAr+c Fxprb0qardBlT0onT1OHmHNBIxy5WpOEUCp02HLNeZ303Y25olnvxzJqdUkOKa0IX2O/ ARz0agITwnJNH9/Z07NuC8dU+Q8AcHB5KpQvOUNNt10qxexeVJo3s3w+HZHEGC8M5DZx arVTByS4z03GtR+nWUFn2aL2WOYxb3sGp9SHg2skwSiR+ClTQDCf3tusF+2PrSC5ZCg8 ns5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742007999; x=1742612799; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eXE4hRNdIiKSnjJQabn9+9Z7l4STEDkxZZthIWAaxQM=; b=WWFdfH3aVtPTqmTfyWXLSq3n538SgwSvUdslqGlVq2hB9Ki4PclhuB7RG1zR25A1Hv 42ntvuLqsAO+RmiMUC36ee0J3mYgiIREJA/rkiCjGzgSn3P7bs2GhV+p6BwFKBHycav/ DlhUmstXGkDZbLx1ASvLKQ/+NizNzFgcUuLO0YAtL1dTiBv2I+H2pAP0fABICE2HplAO epZMhEfdOLLQam2ZGOyhgk1rs7mM5NVnxSUL2ctgfmuhXsxYpbXX+ngtRHW0WacsdTa9 yG/shB6JRdeaeDXNe4bQ56bSUXNP8n/e6Nt+QQglAvktpjeY2t509+EHMpkicEOq4+0g ew/w== X-Forwarded-Encrypted: i=1; AJvYcCU0H+MqECKyqw66oEZwGApOxHYImf/GiD3kdLL6zuahsIxHkYlTVex7iq1XjU3p0A+W09o=@vger.kernel.org X-Gm-Message-State: AOJu0YwrngVbcOAQRHJHUUQkz5WFPDD0ubxNhCniydWo+nt20gqWJAn3 mu/Z8AVj1U2LRZ+9we+/X8gMP7c8DooloJAtIkyXJHwx8MAIay7DGq9OlCxziRHxdJw+7+Bdtfi Qqw== X-Google-Smtp-Source: AGHT+IF1o7Gq8dxb5C/i4W3LaVHx9Xt9tEF6sWZDA8tMLg9rmMb0LEnExUDVXCga37ha7MMcuP+qjT1SPsA= X-Received: from pjbtc11.prod.google.com ([2002:a17:90b:540b:b0:2ff:6e58:8a03]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2690:b0:2ee:af31:a7bd with SMTP id 98e67ed59e1d1-30151cb59afmr5565985a91.5.1742007998857; Fri, 14 Mar 2025 20:06:38 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:25 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-5-seanjc@google.com> Subject: [PATCH 4/8] x86/irq: KVM: Track PIR bitmap as an "unsigned long" array From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Track the PIR bitmap in posted interrupt descriptor structures as an array of unsigned longs instead of using unionized arrays for KVM (u32s) versus IRQ management (u64s). In practice, because the non-KVM usage is (sanely) restricted to 64-bit kernels, all existing usage of the u64 variant is already working with unsigned longs. Using "unsigned long" for the array will allow reworking KVM's processing of the bitmap to read/write in 64-bit chunks on 64-bit kernels, i.e. will allow optimizing KVM by reducing the number of atomic accesses to PIR. Opportunstically replace the open coded literals in the posted MSIs code with the appropriate macro. Deliberately don't use ARRAY_SIZE() in the for-loops, even though it would be cleaner from a certain perspective, in anticipation of decoupling the processing from the array declaration. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/posted_intr.h | 14 +++++++------- arch/x86/kernel/irq.c | 12 ++++++------ arch/x86/kvm/lapic.c | 9 +++++---- arch/x86/kvm/lapic.h | 4 ++-- arch/x86/kvm/vmx/posted_intr.h | 2 +- 5 files changed, 21 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index de788b400fba..c3e6e4221a5b 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -8,12 +8,12 @@ #define PID_TABLE_ENTRY_VALID 1 +#define NR_PIR_VECTORS 256 +#define NR_PIR_WORDS (NR_PIR_VECTORS / BITS_PER_LONG) + /* Posted-Interrupt Descriptor */ struct pi_desc { - union { - u32 pir[8]; /* Posted interrupt requested */ - u64 pir64[4]; - }; + unsigned long pir[NR_PIR_WORDS]; /* Posted interrupt requested */ union { struct { u16 notifications; /* Suppress and outstanding bits */ @@ -43,12 +43,12 @@ static inline bool pi_test_and_clear_sn(struct pi_desc *pi_desc) static inline bool pi_test_and_set_pir(int vector, struct pi_desc *pi_desc) { - return test_and_set_bit(vector, (unsigned long *)pi_desc->pir); + return test_and_set_bit(vector, pi_desc->pir); } static inline bool pi_is_pir_empty(struct pi_desc *pi_desc) { - return bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS); + return bitmap_empty(pi_desc->pir, NR_VECTORS); } static inline void pi_set_sn(struct pi_desc *pi_desc) @@ -105,7 +105,7 @@ static inline bool pi_pending_this_cpu(unsigned int vector) if (WARN_ON_ONCE(vector > NR_VECTORS || vector < FIRST_EXTERNAL_VECTOR)) return false; - return test_bit(vector, (unsigned long *)pid->pir); + return test_bit(vector, pid->pir); } extern void intel_posted_msi_init(void); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 3f95b00ccd7f..704c104ff7a4 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -405,13 +405,13 @@ void intel_posted_msi_init(void) * instead of: * read, xchg, read, xchg, read, xchg, read, xchg */ -static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) +static __always_inline bool handle_pending_pir(unsigned long *pir, struct pt_regs *regs) { int i, vec = FIRST_EXTERNAL_VECTOR; - unsigned long pir_copy[4]; + unsigned long pir_copy[NR_PIR_WORDS]; bool found_irq = false; - for (i = 0; i < 4; i++) { + for (i = 0; i < NR_PIR_WORDS; i++) { pir_copy[i] = READ_ONCE(pir[i]); if (pir_copy[i]) found_irq = true; @@ -420,7 +420,7 @@ static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) if (!found_irq) return false; - for (i = 0; i < 4; i++) { + for (i = 0; i < NR_PIR_WORDS; i++) { if (!pir_copy[i]) continue; @@ -460,7 +460,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) * MAX_POSTED_MSI_COALESCING_LOOP - 1 loops are executed here. */ while (++i < MAX_POSTED_MSI_COALESCING_LOOP) { - if (!handle_pending_pir(pid->pir64, regs)) + if (!handle_pending_pir(pid->pir, regs)) break; } @@ -475,7 +475,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) * process PIR bits one last time such that handling the new interrupts * are not delayed until the next IRQ. */ - handle_pending_pir(pid->pir64, regs); + handle_pending_pir(pid->pir, regs); apic_eoi(); irq_exit(); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index cb4aeab914eb..893e7d06e0e6 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -655,8 +655,9 @@ static u8 count_vectors(void *bitmap) return count; } -bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) +bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { + u32 *__pir = (void *)pir; u32 i, vec; u32 pir_val, irr_val, prev_irr_val; int max_updated_irr; @@ -668,10 +669,10 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10); irr_val = READ_ONCE(*p_irr); - pir_val = READ_ONCE(pir[i]); + pir_val = READ_ONCE(__pir[i]); if (pir_val) { - pir_val = xchg(&pir[i], 0); + pir_val = xchg(&__pir[i], 0); prev_irr_val = irr_val; do { @@ -691,7 +692,7 @@ bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr) } EXPORT_SYMBOL_GPL(__kvm_apic_update_irr); -bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr) +bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, unsigned long *pir, int *max_irr) { struct kvm_lapic *apic = vcpu->arch.apic; bool irr_updated = __kvm_apic_update_irr(pir, apic->regs, max_irr); diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 1a8553ebdb42..0d41780852e4 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -101,8 +101,8 @@ bool kvm_apic_match_dest(struct kvm_vcpu *vcpu, struct kvm_lapic *source, int shorthand, unsigned int dest, int dest_mode); int kvm_apic_compare_prio(struct kvm_vcpu *vcpu1, struct kvm_vcpu *vcpu2); void kvm_apic_clear_irr(struct kvm_vcpu *vcpu, int vec); -bool __kvm_apic_update_irr(u32 *pir, void *regs, int *max_irr); -bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, u32 *pir, int *max_irr); +bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr); +bool kvm_apic_update_irr(struct kvm_vcpu *vcpu, unsigned long *pir, int *max_irr); void kvm_apic_update_ppr(struct kvm_vcpu *vcpu); int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq, struct dest_map *dest_map); diff --git a/arch/x86/kvm/vmx/posted_intr.h b/arch/x86/kvm/vmx/posted_intr.h index ad9116a99bcc..4ff9d720dec0 100644 --- a/arch/x86/kvm/vmx/posted_intr.h +++ b/arch/x86/kvm/vmx/posted_intr.h @@ -18,7 +18,7 @@ static inline int pi_find_highest_vector(struct pi_desc *pi_desc) { int vec; - vec = find_last_bit((unsigned long *)pi_desc->pir, 256); + vec = find_last_bit(pi_desc->pir, 256); return vec < 256 ? vec : -1; } From patchwork Sat Mar 15 03:06:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017703 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C9A11991D2 for ; Sat, 15 Mar 2025 03:06:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008002; cv=none; b=fQDCUq0Eh7fVMYkrJZ4dgWKfqOQEBxJQ7tTvO22ZE3HLAFRHBFRC/7MUbF9brqecz0titXGH1eRUcjCfvwXeVRD0yJ/45skJntzjZAlaD1ObXvifsqYv+nsABvmBjTmWDgQqrEpCMh1XOnOPvtw0KYnux1s9OwpA+datoBVQD4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008002; c=relaxed/simple; bh=YPH0PHJw1tpa8l8R9QQeeIkiVyk20jHhX+FQL8dEQKY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dB88zZ37F+R1Cguk6c+JLkqvm2CQhfRnTi7uW6K/NRW5z5b061Qo7aau/NQUSVO6k0GH+Sk3Aid+p7+G7k3t/4l3ltIdnH7glADEYlN/oO0vCSZUK9KaFyVPaqeHpyCmiTaoOOpkCR34UPzaXO4dgCI5qerxVJQAO21Nkm1d1KA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ojr76eZ7; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ojr76eZ7" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3011bee1751so522688a91.1 for ; Fri, 14 Mar 2025 20:06:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008000; x=1742612800; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=yig8PS2Kt8/pb1c+Gkh8z0ntzMG9R4wz5XbuI5FRVMQ=; b=Ojr76eZ7/Cnv0xzFfKrdSeyZ72bzbBnuOeyFAMXAU9GWiWoKoOFRVz/QbIhLzMxrX6 ImRZBcHvQg6dEabtMGjRYoqvjou2z8+JD8X70Vw+cswDHnHNQx1PlQ50ikcNqSOQkTd9 KehFp2sdWkkuNSEZOxGQff+6Uclkmkxq62ngovRrGu8uXfBZUdlGxK5E39dDAqOW2J2A tRu/3gxr4sTpa/6oBCCA6KZz7YtakVlvkil7BIPd7wG8q6MeWVUA5vf7Xq/sp2e30uO9 hFPM9wl4AiNo1UnOM563eaqhXYCt9QBITMzrN/IpOSUOyQSf4q8/WgSIM09D8MApxvnl XyDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008000; x=1742612800; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yig8PS2Kt8/pb1c+Gkh8z0ntzMG9R4wz5XbuI5FRVMQ=; b=USm1LKu3Nngj0noL5ZmYRMkiORj3are+cou1VvNhLyL3bOLE63k0HnJMnEHiYgZWn7 i17C5ZSmX8EvCMFxhySLs6mT1bJBD+3bbDfTdpa2sh4sAKPak1s9s09ca9wuzY0kS03m z7y1fIAPAJuEmwezT4WVRxJY5X1WTVkUpdBUcCWmdYE4UUZaP3UYPpAqGnQSm3ggiQBc mv+KdJl5PtZQFCDn7io4WaGsBg0wcx8koXWTnFz9H6AnbNCVimBMAQG+XCOqUjln+7En 1ajckbR56C04pO0/BLNMyMT0PIcGZyo9omEvDySVBjbrGJv9loq3d/EQ74SmXfRVSF5d FWYw== X-Forwarded-Encrypted: i=1; AJvYcCUszt/VwSoI3Mq0dI1IluDRKncpcnJZkYShGAhgfCh1u7+hEd6ADrZVTby2spBXCcrJAag=@vger.kernel.org X-Gm-Message-State: AOJu0YwZpioEpy2og7BBj6weHEmbdAuGoiGYYP504/thRRTE9u4vruL0 QOBqirZPvmZc2WU21fhQQBNS6tj7fRpFUSuNo6kAORpvaTUdk3KlLaMO0OQrcjWi8J6pPLI59Gu McA== X-Google-Smtp-Source: AGHT+IE+tLpYYn+7SIxiMQbvt6BNA1OW+IsM1LNLKbpBuX9tC4yl9q4XDM8kwytvK/R01RnOYlCyet2fS44= X-Received: from pjl6.prod.google.com ([2002:a17:90b:2f86:b0:2ee:3128:390f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:e18f:b0:2ee:b6c5:1def with SMTP id 98e67ed59e1d1-30151ce1224mr6192064a91.8.1742008000654; Fri, 14 Mar 2025 20:06:40 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:26 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-6-seanjc@google.com> Subject: [PATCH 5/8] KVM: VMX: Process PIR using 64-bit accesses on 64-bit kernels From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Process the PIR at the natural kernel width, i.e. in 64-bit chunks on 64-bit kernels, so that the worst case of having a posted IRQ in each chunk of the vIRR only requires 4 loads and xchgs from/to the PIR, not 8. Deliberately use a "continue" to skip empty entries so that the code is a carbon copy of handle_pending_pir(), in anticipation of deduplicating KVM and posted MSI logic. Suggested-by: Jim Mattson Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 893e7d06e0e6..e4f182ee9340 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -657,26 +657,32 @@ static u8 count_vectors(void *bitmap) bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { - u32 *__pir = (void *)pir; + unsigned long pir_vals[NR_PIR_WORDS]; + u32 *__pir = (void *)pir_vals; u32 i, vec; - u32 pir_val, irr_val, prev_irr_val; + u32 irr_val, prev_irr_val; int max_updated_irr; max_updated_irr = -1; *max_irr = -1; + for (i = 0; i < NR_PIR_WORDS; i++) { + pir_vals[i] = READ_ONCE(pir[i]); + if (!pir_vals[i]) + continue; + + pir_vals[i] = xchg(&pir[i], 0); + } + for (i = vec = 0; i <= 7; i++, vec += 32) { u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10); irr_val = READ_ONCE(*p_irr); - pir_val = READ_ONCE(__pir[i]); - - if (pir_val) { - pir_val = xchg(&__pir[i], 0); + if (__pir[i]) { prev_irr_val = irr_val; do { - irr_val = prev_irr_val | pir_val; + irr_val = prev_irr_val | __pir[i]; } while (prev_irr_val != irr_val && !try_cmpxchg(p_irr, &prev_irr_val, irr_val)); From patchwork Sat Mar 15 03:06:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017704 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1EB07199EA2 for ; Sat, 15 Mar 2025 03:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008004; cv=none; b=enXHBlOHYp/e34mxq2NK1b9tgUXJvKdO0QL5qbYmxn5CFMyEW2fWxO0alVTtBeGBS3A2s2ROwhLHxagOoLd3pQmQqLh7c12TqXTbHn4Qhbuvga4k7buRERKxA9fkX8vzVvOvcMylb3OPyEyGh6FzYQd34JZx3hrtWlmaxNpRxtY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008004; c=relaxed/simple; bh=DIDA37JL8eeKhWGYjcN5PxZYRrpmJV0kRjJacIWUfhQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ri8tFKuWTKx7vP9js73kOTfmsX4XC93OrmqLMu4cs7+9ut5kPPbXxBjHScOXk9wHG7ynQdNt/3JCiSbi1M+pAalHkva+h7VhHwU9QHlsls5pzLcc7dyV2UtdtY5o6GprPI3v+Amfc2Kv3T3scmsI6OLxzz3FM1GQsNIQLdXjZkM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=1dv00/Uc; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="1dv00/Uc" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ff7f9a0b9bso439482a91.0 for ; Fri, 14 Mar 2025 20:06:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008002; x=1742612802; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=hv39/O/PnMVFihpfgZT9SpR99lFAgaV130aLFM9qjm4=; b=1dv00/UcK4oER/YLiOMaxFVrKEWirmRovwQWwX6yItUbKsCfoh7wT1wHH0k4wITgBC Q6Ad4AnqU8nJ1xIQevzuGzQPCrCNeX2URXUpPBUAhxbcMWgpvmhMGNWL+g0rVIrXDA/Y Qc7/pDe+k2f4vAd485redXeir59iSCOD32B/ZIgtrIhn12koiU2VyMqZFD1YD+SUjLy0 dB6Ua0QKW1juHyHTktBsqcg/ChpXS6LYe0CmX5YG5z1oK66ZE1CIVetlExdo47Y2gU5q U7sc03bYLwFU6qjIDpp5Ga4Tu1xJBgoemKAdxVfWx3kG8zjK8892umX+6JJZ/M0yMZUi ACqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008002; x=1742612802; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hv39/O/PnMVFihpfgZT9SpR99lFAgaV130aLFM9qjm4=; b=r77OAOVBzZm2EVD8OSoCalepO3z9tZB08oXU9MvpGo8hntjJn7MzQdDD6id1faTcrN Hz2R6xbQiQoo9Uhm89XB1ENLxoGi/5U7eJ9uyhFd4jjECaxd8MKrcpR6J9b+MJh8UP8r lL6UjRkXshG2/xTAspqOIVEsTlfh6g0/B9goTyTvjFUzO6sUcSbmqG5ivhth5/m4R47j TiGUV9foEyKYf67jr3hRMFP/L28kAnLloiKJzW0yU3BfTmpnnhJkadaUHPxk0W0Ybe0z Gz2voocvScd1XWeGaFBhIwb+jY/R8Ambkq8hyQWSJ9tb9Pxx8Jow0MAhbnFODwlMY/vM dWAg== X-Forwarded-Encrypted: i=1; AJvYcCW0jZ+V4Yb+Ih5qHFr7cCquzQMqWV+gBcls88o1jN4vH94MnHz40txU/e+apJurzdS3vHY=@vger.kernel.org X-Gm-Message-State: AOJu0Yy/ONfO1Q+wVptqd13zYcTrYHRd7hK1Z0+Jyoq2+XGZgFCKZO5w cHezxr+7ftmc4wEF9bmOy/kVtJ/WnCM1P/Cli5JM5Cavm9+OvfE/aAqkFMFG6oqB+/4vW7KxGLJ O2A== X-Google-Smtp-Source: AGHT+IHWjt/BJeVYOLv32GnaoqzmTC0s6P3U8PoNtZJT15p0guIAJtE+fq85gGZBlDG7yQuICOSFHSPkotI= X-Received: from pjbph6.prod.google.com ([2002:a17:90b:3bc6:b0:2fc:11a0:c549]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:514e:b0:2fa:157e:c78e with SMTP id 98e67ed59e1d1-30151cab344mr6435643a91.7.1742008002348; Fri, 14 Mar 2025 20:06:42 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:27 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-7-seanjc@google.com> Subject: [PATCH 6/8] KVM: VMX: Isolate pure loads from atomic XCHG when processing PIR From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Rework KVM's processing of the PIR to use the same algorithm as posted MSIs, i.e. to do READ(x4) => XCHG(x4) instead of (READ+XCHG)(x4). Given KVM's long-standing, sub-optimal use of 32-bit accesses to the PIR, it's safe to say far more thought and investigation was put into handling the PIR for posted MSIs, i.e. there's no reason to assume KVM's existing logic is meaningful, let alone superior. Matching the processing done by posted MSIs will also allow deduplicating the code between KVM and posted MSIs. See the comment for handle_pending_pir() added by commit 1b03d82ba15e ("x86/irq: Install posted MSI notification handler") for details on why isolating loads from XCHG is desirable. Suggested-by: Jim Mattson Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index e4f182ee9340..d7e36faffc72 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -659,6 +659,7 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { unsigned long pir_vals[NR_PIR_WORDS]; u32 *__pir = (void *)pir_vals; + bool found_irq = false; u32 i, vec; u32 irr_val, prev_irr_val; int max_updated_irr; @@ -668,6 +669,14 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) for (i = 0; i < NR_PIR_WORDS; i++) { pir_vals[i] = READ_ONCE(pir[i]); + if (pir_vals[i]) + found_irq = true; + } + + if (!found_irq) + return false; + + for (i = 0; i < NR_PIR_WORDS; i++) { if (!pir_vals[i]) continue; From patchwork Sat Mar 15 03:06:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017705 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A16BA19C574 for ; Sat, 15 Mar 2025 03:06:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008006; cv=none; b=jo8yIrtt4yq8swQXkFef5ujuJGcGYBBGrLk4iGLovcWnXNp6u2fUBZ0GdfpO3QaY23bR0DJRSDXszHMZZVGEGCPJylklhD9J9obUbmR5CYu+v8HgwsxdZYUxY7m2SZdyO020GM+Tw7Vo/H9n7Y/Z/halx5Ev8LhKhmge04v9zWk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008006; c=relaxed/simple; bh=kyZPgKDP+G4NiFhQkapY7Z40kUAzSIF8X2Em11vw2i8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KX1HO5pWA09y6gt7zXNFptU6HxR0amllbiBviCbXhF88PkrwBL+cQN1sFso/rK+wW3OYwhoXoLbCIp11thrTW3l0CYOqVMg6O0YQk3qvgNwJvd0XbcMshTDuaTJ6uhBJyRe4RUjj2uY24n6ly/wyYsA6qXkqppjBDELoXdqPxqQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gdiikDAQ; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gdiikDAQ" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2254e500a73so35826365ad.0 for ; Fri, 14 Mar 2025 20:06:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008004; x=1742612804; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=G6aJw5/ES1UOfEaKxfAACWN89ZnVy5JuxHZXx05VAEo=; b=gdiikDAQmhkZVpccgyQyzXAoXEyPBvXk7uxwtMnsIEoCMG/6J/BXs7ntlpXlvhaXA1 qQ2LV5Hub8fmwTAxiZrRsS7Mz8b6GrBvz68AtJVzAugBDu0ah73wDHoDEgJ+CapP0m/q EGXDQM7H2T47tzcJmExK1A9Dy1gMDJXJm7AhnGDP8O47sqB5Lvj993ozXe4kca2xFmOI s6yvHWTJZZd1PmuroBrrlZCg6S08dtxaU7fWrgmxz+ZUGaSFKZN90ThcSj33hQzpLw6X lrO6ftmct1pjCMTkIvTNYuTStn+DYlbCrnlyatbV33ODq3YtPYzHRtmIIR49Tkp8ttWZ CSeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008004; x=1742612804; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=G6aJw5/ES1UOfEaKxfAACWN89ZnVy5JuxHZXx05VAEo=; b=krNo8fAobZhcpG8OMMmeHQfsMtsN+H1MsOvxBH0eLFWhDdbWL+5Od6aQ3JgyB2ODuU vrvfHuoWaHvrnBxdz3OMbvNuMEfbz/B7TrTvZw1otF49nLv055IAZqdXRY69O3EvEhOB OLOqy5CuGuvXNmDLXibSUizBlFRoRnFyDU7ltxyEgCLACK3R65Q79MjwYQxpf8Jz0BCI St0O73fYrnS9ENGlBHNBrbtHluF2jiN+R2V4lBGzggL2H6QWi9P9WhLrkr2hniFIbSNS VoHp+hPPCa5lyUzBPU+g/fw7p0FbcDsDL/UZqAR/oNYwjfuhW1Lut9DmOBz7pN4Kt5fZ YBew== X-Forwarded-Encrypted: i=1; AJvYcCUApgvjldCprqhEhQBUN9xTCeVE4UT5bEfhF3NU+oRfSotzKKjcI7OJe7Xr23Y1nnATYXg=@vger.kernel.org X-Gm-Message-State: AOJu0YwXr7IoIKBe08bZ3I1IacLEA47P1Ivh+ThRwd/dgJzOd5i8jisA qWLAp6GrSiwUSuYeJoi/FM8z0F/aZyy56sT1f+QM6y7YPZQvpnNWsAlyJ0Q0jgdRs/ylQiqRWu9 zKw== X-Google-Smtp-Source: AGHT+IESdoU5JvhAzuOhjydI2CjAPYyqye5HZ3O+0f0UA0gPpp7LXZq02t7B33tZ8tgvi8g0NZmeMCZy9I0= X-Received: from pjbee15.prod.google.com ([2002:a17:90a:fc4f:b0:2fe:d556:ec6e]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e548:b0:223:653e:eb09 with SMTP id d9443c01a7336-225e0a369c4mr48322735ad.7.1742008004023; Fri, 14 Mar 2025 20:06:44 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:28 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-8-seanjc@google.com> Subject: [PATCH 7/8] KVM: VMX: Use arch_xchg() when processing PIR to avoid instrumentation From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Use arch_xchg() when moving IRQs from the PIR to the vIRR, purely to avoid instrumentation so that KVM is compatible with the needs of posted MSI. This will allow extracting the core PIR logic to common code and sharing it between KVM and posted MSI handling. Signed-off-by: Sean Christopherson --- arch/x86/kvm/lapic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index d7e36faffc72..b65e0f7223fe 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -680,7 +680,7 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) if (!pir_vals[i]) continue; - pir_vals[i] = xchg(&pir[i], 0); + pir_vals[i] = arch_xchg(&pir[i], 0); } for (i = vec = 0; i <= 7; i++, vec += 32) { From patchwork Sat Mar 15 03:06:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 14017706 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D44819D897 for ; Sat, 15 Mar 2025 03:06:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008007; cv=none; b=Cv+ZRZ2KgvxSwL3nuaA9zD7xh7EIeB1yK/6AJXEqevs+i0o8M2ItIPUoMWn+saoCnq1z81s68aTDiVf1PfJM24ox7kfqmfaJbdfts9jM3DA7twDMRsMr/PprQ1vfHTPsrhjDIf/VPRHNV5ydjGeYU3IoR+my7HPSov5wO2pEScU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1742008007; c=relaxed/simple; bh=6Ct4ueqTACT+40mgDDmV2LJlgiyUgvMZWTdwVdt+2AI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Usxiy15zxwZqi2QLdCHUQWYDJRcdh56B4EevMroELsLZQKySyEL09TG7g+UdQ/LjtPgKhnSZVg7l1f0TEenB7/WBZUviaZ9p6UGVDWEVAYmdsW/k98naPzF81+mcFtZdbaNrBNmQoiQMfZ3EySBf3mRee9yuqayOtO3yg1iS3Ok= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZnE+4jTn; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZnE+4jTn" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ff68033070so473402a91.2 for ; Fri, 14 Mar 2025 20:06:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742008005; x=1742612805; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Rqh78WRgDr7mgiGuvPqk9U1mgup3ydPWLYkuR7DIMcg=; b=ZnE+4jTnlbpy00U4j0HY28sde+eEOjoiEsddr8wce1yFa3en54ot9o407cEAIIoB5F qBnLJBUllfU6zfE6AAE0HuOBVVctF9EcfV+mz1a1b7sjkre0Se+Hrq0pJlM8YYKzldGI nC4p7MjI/YIT29mE1IKONotGxwL/hJS1DSNvW5N/LMoO6Pg6IUg5pYiuqWLUHUbmiTEm alGFNDOP+bY0Uzeab7rSiT9jFAthbxnx94DQrNb6sBSaOrQXenCegqLs+xYdfJDHE0gu NW/SE0XQhu6GdMroMvM8VgOOjfmIDLsc64gxmG50+gFZSk9BbWAb455oq5pFn7UExP9E 4hRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742008005; x=1742612805; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Rqh78WRgDr7mgiGuvPqk9U1mgup3ydPWLYkuR7DIMcg=; b=GC134bcTdglWmXeegCjZX6wCCSjDQUBIrZKdgw2dgG/r7eEMhW7mQG7zjAqJdN+WA+ fpkjw3yyeJBcfuzT7KVx8uLWIyU13YPPAOJ0ZLtrh4PN3/i4GBsKO9VrlaPzN4y+IoGR tVTfpQZ6yeYFeEOrMTqx6KUcu/zADp+5kUdeAPm9UbdUllXG+9dRIdpkkOc91SDDlzH1 YLSpJ1K1NQhxDUhFLF+1ndmqX1OHKquZArctrnLUBR2EoMjtfm9ln40t+5waVh8XO39M n4hfn1YXgg3kfkJcFr/6D2KVhLczqm14y7OmgeFzur/2Kejvrra/Ojm2kAY9GuZ1ipJd msIw== X-Forwarded-Encrypted: i=1; AJvYcCX1uAR1sLuAjxOjFv5LrukeqyDbkGB7f2eDUD+rBqNwQCt+m1obe6bEdZ+6OggUvcW+1yE=@vger.kernel.org X-Gm-Message-State: AOJu0YwkggTmeNa/uK4l/Hsqs7wbGyoaxr8YDJRN7Wx9dfeeejM8C/Tk mf5TSo5GCr0n5lRIczcT7ENOAsXT54f6dt5OiSqJBx2fD769wv/2hDoijBY3OWgtHc+iLPepOGI V/A== X-Google-Smtp-Source: AGHT+IFf+R/IdkB9dMtQz0tp3pLrCC/oKPbyU6izvyqg+RZIyQsLWC+utkXYHnwPh9OG4z6WE2OE7PdpDaY= X-Received: from pjbdy6.prod.google.com ([2002:a17:90b:6c6:b0:2f8:49ad:406c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4d10:b0:2fe:b470:dde4 with SMTP id 98e67ed59e1d1-30151cd796dmr7085921a91.12.1742008005673; Fri, 14 Mar 2025 20:06:45 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 14 Mar 2025 20:06:29 -0700 In-Reply-To: <20250315030630.2371712-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250315030630.2371712-1-seanjc@google.com> X-Mailer: git-send-email 2.49.0.rc1.451.g8f38331e32-goog Message-ID: <20250315030630.2371712-9-seanjc@google.com> Subject: [PATCH 8/8] x86/irq: KVM: Add helper for harvesting PIR to deduplicate KVM and posted MSIs From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Sean Christopherson , Paolo Bonzini Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jacob Pan , Jim Mattson Now that posted MSI and KVM harvesting of PIR is identical, extract the code (and posted MSI's wonderful comment) to a common helper. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/posted_intr.h | 65 ++++++++++++++++++++++++++++++ arch/x86/kernel/irq.c | 50 +---------------------- arch/x86/kvm/lapic.c | 16 +------- 3 files changed, 68 insertions(+), 63 deletions(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index c3e6e4221a5b..6b1ddebbf06a 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -1,8 +1,13 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _X86_POSTED_INTR_H #define _X86_POSTED_INTR_H + +#include +#include #include +#include + #define POSTED_INTR_ON 0 #define POSTED_INTR_SN 1 @@ -26,6 +31,66 @@ struct pi_desc { u32 rsvd[6]; } __aligned(64); +/* + * De-multiplexing posted interrupts is on the performance path, the code + * below is written to optimize the cache performance based on the following + * considerations: + * 1.Posted interrupt descriptor (PID) fits in a cache line that is frequently + * accessed by both CPU and IOMMU. + * 2.During software processing of posted interrupts, the CPU needs to do + * natural width read and xchg for checking and clearing posted interrupt + * request (PIR), a 256 bit field within the PID. + * 3.On the other side, the IOMMU does atomic swaps of the entire PID cache + * line when posting interrupts and setting control bits. + * 4.The CPU can access the cache line a magnitude faster than the IOMMU. + * 5.Each time the IOMMU does interrupt posting to the PIR will evict the PID + * cache line. The cache line states after each operation are as follows, + * assuming a 64-bit kernel: + * CPU IOMMU PID Cache line state + * --------------------------------------------------------------- + *...read64 exclusive + *...lock xchg64 modified + *... post/atomic swap invalid + *...------------------------------------------------------------- + * + * To reduce L1 data cache miss, it is important to avoid contention with + * IOMMU's interrupt posting/atomic swap. Therefore, a copy of PIR is used + * when processing posted interrupts in software, e.g. to dispatch interrupt + * handlers for posted MSIs, or to move interrupts from the PIR to the vIRR + * in KVM. + * + * In addition, the code is trying to keep the cache line state consistent + * as much as possible. e.g. when making a copy and clearing the PIR + * (assuming non-zero PIR bits are present in the entire PIR), it does: + * read, read, read, read, xchg, xchg, xchg, xchg + * instead of: + * read, xchg, read, xchg, read, xchg, read, xchg + */ +static __always_inline bool pi_harvest_pir(unsigned long *pir, + unsigned long *pir_vals) +{ + bool found_irq = false; + int i; + + for (i = 0; i < NR_PIR_WORDS; i++) { + pir_vals[i] = READ_ONCE(pir[i]); + if (pir_vals[i]) + found_irq = true; + } + + if (!found_irq) + return false; + + for (i = 0; i < NR_PIR_WORDS; i++) { + if (!pir_vals[i]) + continue; + + pir_vals[i] = arch_xchg(&pir[i], 0); + } + + return true; +} + static inline bool pi_test_and_set_on(struct pi_desc *pi_desc) { return test_and_set_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->control); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 704c104ff7a4..b98a5abdeaec 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -373,60 +373,14 @@ void intel_posted_msi_init(void) this_cpu_write(posted_msi_pi_desc.ndst, destination); } -/* - * De-multiplexing posted interrupts is on the performance path, the code - * below is written to optimize the cache performance based on the following - * considerations: - * 1.Posted interrupt descriptor (PID) fits in a cache line that is frequently - * accessed by both CPU and IOMMU. - * 2.During posted MSI processing, the CPU needs to do 64-bit read and xchg - * for checking and clearing posted interrupt request (PIR), a 256 bit field - * within the PID. - * 3.On the other side, the IOMMU does atomic swaps of the entire PID cache - * line when posting interrupts and setting control bits. - * 4.The CPU can access the cache line a magnitude faster than the IOMMU. - * 5.Each time the IOMMU does interrupt posting to the PIR will evict the PID - * cache line. The cache line states after each operation are as follows: - * CPU IOMMU PID Cache line state - * --------------------------------------------------------------- - *...read64 exclusive - *...lock xchg64 modified - *... post/atomic swap invalid - *...------------------------------------------------------------- - * - * To reduce L1 data cache miss, it is important to avoid contention with - * IOMMU's interrupt posting/atomic swap. Therefore, a copy of PIR is used - * to dispatch interrupt handlers. - * - * In addition, the code is trying to keep the cache line state consistent - * as much as possible. e.g. when making a copy and clearing the PIR - * (assuming non-zero PIR bits are present in the entire PIR), it does: - * read, read, read, read, xchg, xchg, xchg, xchg - * instead of: - * read, xchg, read, xchg, read, xchg, read, xchg - */ static __always_inline bool handle_pending_pir(unsigned long *pir, struct pt_regs *regs) { - int i, vec = FIRST_EXTERNAL_VECTOR; + int vec = FIRST_EXTERNAL_VECTOR; unsigned long pir_copy[NR_PIR_WORDS]; - bool found_irq = false; - for (i = 0; i < NR_PIR_WORDS; i++) { - pir_copy[i] = READ_ONCE(pir[i]); - if (pir_copy[i]) - found_irq = true; - } - - if (!found_irq) + if (!pi_harvest_pir(pir, pir_copy)) return false; - for (i = 0; i < NR_PIR_WORDS; i++) { - if (!pir_copy[i]) - continue; - - pir_copy[i] = arch_xchg(&pir[i], 0); - } - for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) call_irq_handler(vec, regs); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index b65e0f7223fe..1c611b84b8ab 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -659,7 +659,6 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) { unsigned long pir_vals[NR_PIR_WORDS]; u32 *__pir = (void *)pir_vals; - bool found_irq = false; u32 i, vec; u32 irr_val, prev_irr_val; int max_updated_irr; @@ -667,22 +666,9 @@ bool __kvm_apic_update_irr(unsigned long *pir, void *regs, int *max_irr) max_updated_irr = -1; *max_irr = -1; - for (i = 0; i < NR_PIR_WORDS; i++) { - pir_vals[i] = READ_ONCE(pir[i]); - if (pir_vals[i]) - found_irq = true; - } - - if (!found_irq) + if (!pi_harvest_pir(pir, pir_vals)) return false; - for (i = 0; i < NR_PIR_WORDS; i++) { - if (!pir_vals[i]) - continue; - - pir_vals[i] = arch_xchg(&pir[i], 0); - } - for (i = vec = 0; i <= 7; i++, vec += 32) { u32 *p_irr = (u32 *)(regs + APIC_IRR + i * 0x10);