From patchwork Fri Apr 5 22:30:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619547 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7D58174EF6; Fri, 5 Apr 2024 22:26:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356008; cv=none; b=r9t6nxOxgyqEXB8UvK1TijLH+IeDduj4JmIS1YAW6DPvLzIuo3DbV8uyArIB2uezOqb4aALt53xKLBoqTwpFa+3LWJpHbn8YoqEgclWNex4FlpipfGnn0wZAq3lpcD8QnbLYTSflryBlStQdq/zQLNpAPu3LRPwPQIx4Hi8r0Bw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356008; c=relaxed/simple; bh=Bh8qAq471IuQbwA8oP4ru9wTknH7iKls9mGK6YcKMpg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XJxNMfu7Ns5WvN23yDi831au4MUupMC8+v8GPszZMBS3SJMSCwgdOU19zROFowumWDERahQRM79LfDnFvFJi0HiFJEvScDQ8dMtIHBhk4T2VbPRS8oI6pT120JcXTBmkXlB5u7wfr59CMGdqm2q37lt8sC0Nab+p8g1N5E9ze4A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XokvInM3; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XokvInM3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356007; x=1743892007; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Bh8qAq471IuQbwA8oP4ru9wTknH7iKls9mGK6YcKMpg=; b=XokvInM3RIiZ1dEeTQuqOZzUL1b12Wcxlq7cvJjDHqfROV6SRgV8GWJM iV9ccygx8BOSsH0hYWfC7i17b+1c84QuCozKsM7PuuvY/ICB3qtjn3+bS PTBDU7SCFIqcp30NwovbEnuAHFEbIzrXyYyZyCUcpVl2FyUaUG1GyGVhq yBwv8TQ6CU7pd2eGdrMVW4YdjqMmG8EH5xqhrfpdGGA8oCly/vsQ2NN3P N6oo0Gmx7GhHtXHARC6VMGFESGS8V9Akks8kGrW6D/H3SRGIyA4nh78c5 Gwf3lwrABEz9gWCNSLNSI62qNd1HAs4pRu/U6vXuc1sY2SLHrfWtnUbt9 g==; X-CSE-ConnectionGUID: 1CfHmgnzRKWhUTCBWNuuww== X-CSE-MsgGUID: ZS9969qVRk6mQmqmdJfP5g== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062669" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062669" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:45 -0700 X-CSE-ConnectionGUID: cPYrfhevQh+UdDZtipl+Cg== X-CSE-MsgGUID: jc3lcFRGSWyLv6aWkzfreA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928308" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:44 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 01/13] x86/irq: Move posted interrupt descriptor out of vmx code Date: Fri, 5 Apr 2024 15:30:58 -0700 Message-Id: <20240405223110.1609888-2-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To prepare native usage of posted interrupt, move PID declaration out of VMX code such that they can be shared. Signed-off-by: Jacob Pan Acked-by: Sean Christopherson --- arch/x86/include/asm/posted_intr.h | 88 ++++++++++++++++++++++++++++ arch/x86/kvm/vmx/posted_intr.h | 93 +----------------------------- arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/kvm/vmx/vmx.h | 2 +- 4 files changed, 91 insertions(+), 93 deletions(-) create mode 100644 arch/x86/include/asm/posted_intr.h diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h new file mode 100644 index 000000000000..f0324c56f7af --- /dev/null +++ b/arch/x86/include/asm/posted_intr.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _X86_POSTED_INTR_H +#define _X86_POSTED_INTR_H + +#define POSTED_INTR_ON 0 +#define POSTED_INTR_SN 1 + +#define PID_TABLE_ENTRY_VALID 1 + +/* Posted-Interrupt Descriptor */ +struct pi_desc { + u32 pir[8]; /* Posted interrupt requested */ + union { + struct { + /* bit 256 - Outstanding Notification */ + u16 on : 1, + /* bit 257 - Suppress Notification */ + sn : 1, + /* bit 271:258 - Reserved */ + rsvd_1 : 14; + /* bit 279:272 - Notification Vector */ + u8 nv; + /* bit 287:280 - Reserved */ + u8 rsvd_2; + /* bit 319:288 - Notification Destination */ + u32 ndst; + }; + u64 control; + }; + u32 rsvd[6]; +} __aligned(64); + +static inline bool pi_test_and_set_on(struct pi_desc *pi_desc) +{ + return test_and_set_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_clear_on(struct pi_desc *pi_desc) +{ + return test_and_clear_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_clear_sn(struct pi_desc *pi_desc) +{ + return test_and_clear_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_and_set_pir(int vector, struct pi_desc *pi_desc) +{ + return test_and_set_bit(vector, (unsigned long *)pi_desc->pir); +} + +static inline bool pi_is_pir_empty(struct pi_desc *pi_desc) +{ + return bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS); +} + +static inline void pi_set_sn(struct pi_desc *pi_desc) +{ + set_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); +} + +static inline void pi_set_on(struct pi_desc *pi_desc) +{ + set_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->control); +} + +static inline void pi_clear_on(struct pi_desc *pi_desc) +{ + clear_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->control); +} + +static inline void pi_clear_sn(struct pi_desc *pi_desc) +{ + clear_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_on(struct pi_desc *pi_desc) +{ + return test_bit(POSTED_INTR_ON, (unsigned long *)&pi_desc->control); +} + +static inline bool pi_test_sn(struct pi_desc *pi_desc) +{ + return test_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); +} + +#endif /* _X86_POSTED_INTR_H */ diff --git a/arch/x86/kvm/vmx/posted_intr.h b/arch/x86/kvm/vmx/posted_intr.h index 26992076552e..6b2a0226257e 100644 --- a/arch/x86/kvm/vmx/posted_intr.h +++ b/arch/x86/kvm/vmx/posted_intr.h @@ -1,98 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef __KVM_X86_VMX_POSTED_INTR_H #define __KVM_X86_VMX_POSTED_INTR_H - -#define POSTED_INTR_ON 0 -#define POSTED_INTR_SN 1 - -#define PID_TABLE_ENTRY_VALID 1 - -/* Posted-Interrupt Descriptor */ -struct pi_desc { - u32 pir[8]; /* Posted interrupt requested */ - union { - struct { - /* bit 256 - Outstanding Notification */ - u16 on : 1, - /* bit 257 - Suppress Notification */ - sn : 1, - /* bit 271:258 - Reserved */ - rsvd_1 : 14; - /* bit 279:272 - Notification Vector */ - u8 nv; - /* bit 287:280 - Reserved */ - u8 rsvd_2; - /* bit 319:288 - Notification Destination */ - u32 ndst; - }; - u64 control; - }; - u32 rsvd[6]; -} __aligned(64); - -static inline bool pi_test_and_set_on(struct pi_desc *pi_desc) -{ - return test_and_set_bit(POSTED_INTR_ON, - (unsigned long *)&pi_desc->control); -} - -static inline bool pi_test_and_clear_on(struct pi_desc *pi_desc) -{ - return test_and_clear_bit(POSTED_INTR_ON, - (unsigned long *)&pi_desc->control); -} - -static inline bool pi_test_and_clear_sn(struct pi_desc *pi_desc) -{ - return test_and_clear_bit(POSTED_INTR_SN, - (unsigned long *)&pi_desc->control); -} - -static inline bool pi_test_and_set_pir(int vector, struct pi_desc *pi_desc) -{ - return test_and_set_bit(vector, (unsigned long *)pi_desc->pir); -} - -static inline bool pi_is_pir_empty(struct pi_desc *pi_desc) -{ - return bitmap_empty((unsigned long *)pi_desc->pir, NR_VECTORS); -} - -static inline void pi_set_sn(struct pi_desc *pi_desc) -{ - set_bit(POSTED_INTR_SN, - (unsigned long *)&pi_desc->control); -} - -static inline void pi_set_on(struct pi_desc *pi_desc) -{ - set_bit(POSTED_INTR_ON, - (unsigned long *)&pi_desc->control); -} - -static inline void pi_clear_on(struct pi_desc *pi_desc) -{ - clear_bit(POSTED_INTR_ON, - (unsigned long *)&pi_desc->control); -} - -static inline void pi_clear_sn(struct pi_desc *pi_desc) -{ - clear_bit(POSTED_INTR_SN, - (unsigned long *)&pi_desc->control); -} - -static inline bool pi_test_on(struct pi_desc *pi_desc) -{ - return test_bit(POSTED_INTR_ON, - (unsigned long *)&pi_desc->control); -} - -static inline bool pi_test_sn(struct pi_desc *pi_desc) -{ - return test_bit(POSTED_INTR_SN, - (unsigned long *)&pi_desc->control); -} +#include void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu); void vmx_vcpu_pi_put(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c37a89eda90f..d94bb069bac9 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -70,6 +70,7 @@ #include "x86.h" #include "smm.h" #include "vmx_onhyperv.h" +#include "posted_intr.h" MODULE_AUTHOR("Qumranet"); MODULE_LICENSE("GPL"); diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 65786dbe7d60..e133e8077e6d 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -7,10 +7,10 @@ #include #include #include +#include #include "capabilities.h" #include "../kvm_cache_regs.h" -#include "posted_intr.h" #include "vmcs.h" #include "vmx_ops.h" #include "../cpuid.h" From patchwork Fri Apr 5 22:30:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619549 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33B5717555B; Fri, 5 Apr 2024 22:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356009; cv=none; b=q+RyzbgJM/bfqf+0uKWVLLxOp6yo15jh71HFgLbVwCVV9Ljll1DMC3e6KwyjOyYIZvsVtxoIAdcUqvIhaI/6q7e/KVd5Aadn5LPYZzNhQCgSRwVw3kfVyKRTyUQVCCQej4260tBoIOR2ZTWteLbDoZRxTa8Qg3Y9BGS/y3aY79A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356009; c=relaxed/simple; bh=0GX8HhoSt+Bowm5iZvkwyHIzLD8tLG+XNrU90AlqK/Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=g+miIwR3EVi7afTLA/MXfNisdILhGWzhkpx6bN8GWngh/p9fbNlKGreoo1PcTaUmHsvaoIU93ONxXchHg3vrznEuO/78fV4NUDcID0YY/bFJAqPR17XTU33s7mZBqcHFMvn/FhjR9N6/QdP9kwBss8tg4UV0enmjab3FXgFuURE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=IvbgW2Lp; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="IvbgW2Lp" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356008; x=1743892008; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0GX8HhoSt+Bowm5iZvkwyHIzLD8tLG+XNrU90AlqK/Q=; b=IvbgW2LpmEB0sRGfKRTG2zfSi3ceSoQKRgK22yTUdWsrxiX22Z6gURcg Vyh4xPbZAoVazQaLSW+WoZPpxqL8vpKchBs3YLTYJZxA4Dn394xIHgZau nehysJJcBLlZGTJKw2FjVnhH1YMjRkF+pAU8g86fuQBTYblJo5ih/tixO RVScWMnl1H99snuKwaLKocZxMGaU9GNgDzltS/qqPB+mS5iazqA9OHuzw 3v4pFxnSdYFVXvDx8Ze2OOzM2VtWEFzE/5N31mAMNT8lXrmrYPzbKPBvY 5ECr2cFj20dhwnACsZT03nd4zWSEZTLMbQGgSLZYXJ1f7cD35Fkra72xL Q==; X-CSE-ConnectionGUID: 7Yhc0KWzTtWstoOCJbDCMw== X-CSE-MsgGUID: RSAoS6FIQxeMa74FDlKMGw== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062684" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062684" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:46 -0700 X-CSE-ConnectionGUID: 0z4oK8SVSReyiQmluJIm4g== X-CSE-MsgGUID: bs4BYb9qQQSpBWes3fTEKA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928311" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:45 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 02/13] x86/irq: Unionize PID.PIR for 64bit access w/o casting Date: Fri, 5 Apr 2024 15:30:59 -0700 Message-Id: <20240405223110.1609888-3-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Make PIR field into u64 such that atomic xchg64 can be used without ugly casting. Suggested-by: Thomas Gleixner Signed-off-by: Jacob Pan --- arch/x86/include/asm/posted_intr.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index f0324c56f7af..acf237b2882e 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -9,7 +9,10 @@ /* Posted-Interrupt Descriptor */ struct pi_desc { - u32 pir[8]; /* Posted interrupt requested */ + union { + u32 pir[8]; /* Posted interrupt requested */ + u64 pir64[4]; + }; union { struct { /* bit 256 - Outstanding Notification */ From patchwork Fri Apr 5 22:31:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619550 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 229B1175571; Fri, 5 Apr 2024 22:26:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356010; cv=none; b=C70Bm2wHgJPChmxKm7yKsRh10FXooRisMGSWiV6XHm4A6jZNRL6FJXkqPJKfQrKV77vJM5/eEetU1TYChpUs9l2AIiTt1xJk+K5N9EB6WAQqLixiNfh2+cbj6wT5AXPDluhvCI+Ik/hbgwe/QlBKH5uDeIZYwu5zmuxmrEtnVFY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356010; c=relaxed/simple; bh=Okro4Kem0N9Ie4cgbPOty9xrwMq2Uzf6Zhnd6S3vbE4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qvu09/oPyU5TfpzqSqDjGcsT3oQ8LqRjWmwBO+9QIn7+xE9o1SNLX0kEZmhbV0G5ntRXqbUgDyg7A/jtXQFN4kfkFEoVxTq+kA7xk5JIjS6p56Keai9DwTgDhJrwlTQFSBYT4RSmluKGwjjfZ9DSMy/onkyxl4+VMtaUznGGLxY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Obop4Ugm; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Obop4Ugm" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356009; x=1743892009; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Okro4Kem0N9Ie4cgbPOty9xrwMq2Uzf6Zhnd6S3vbE4=; b=Obop4UgmoyuCCkt9SxoQAgSnEiP80nVl2lGiOn55W4uJkVD6cqTiLLvo Qd1bwYwhiX8zaWeezOIbUjDK71HzEOSwB2rRc3iP7PPPIKEAWPz7CGDMn ICxWn+91TK+bl/igKwuPiFEhBcOql3Fce4Ypi5XWpKmHOijv47IwIPAM8 SfCAPS4R2wRwlk4+PLPP1gz9cPkZf5DrT9ycoa9n453yGrLG/zy+hixYD jjCR4EGISJ8sP/y0zDXEzzUfGZC/C1ZaBuBI6o/gw8ti8Wrn+LZkBWkEO Lz/ZNgLh3917mq6QTgr7nDq998zP/JMO8F06UK7CQMcDpbsYJ2HT2JNJ9 A==; X-CSE-ConnectionGUID: l6Pnn+JBTlS6bBEc2rPKdw== X-CSE-MsgGUID: gdWwNhilRsev0DBlM/2O7w== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062702" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062702" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:47 -0700 X-CSE-ConnectionGUID: 1HXxumqzT42vJ5sjlMnWsg== X-CSE-MsgGUID: zUxrBdZ5ReSw612yNtf9Fg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928314" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:46 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 03/13] x86/irq: Remove bitfields in posted interrupt descriptor Date: Fri, 5 Apr 2024 15:31:00 -0700 Message-Id: <20240405223110.1609888-4-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Mixture of bitfields and types is weird and really not intuitive, remove bitfields and use typed data exclusively. Link: https://lore.kernel.org/all/20240404101735.402feec8@jacob-builder/T/#mf66e34a82a48f4d8e2926b5581eff59a122de53a Suggested-by: Sean Christopherson Suggested-by: Thomas Gleixner Signed-off-by: Jacob Pan --- v2: - Replace bitfields, no more mix. --- arch/x86/include/asm/posted_intr.h | 10 +--------- arch/x86/kvm/vmx/posted_intr.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 2 +- 3 files changed, 4 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index acf237b2882e..c682c41d4e44 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -15,17 +15,9 @@ struct pi_desc { }; union { struct { - /* bit 256 - Outstanding Notification */ - u16 on : 1, - /* bit 257 - Suppress Notification */ - sn : 1, - /* bit 271:258 - Reserved */ - rsvd_1 : 14; - /* bit 279:272 - Notification Vector */ + u16 notif_ctrl; /* Suppress and outstanding bits */ u8 nv; - /* bit 287:280 - Reserved */ u8 rsvd_2; - /* bit 319:288 - Notification Destination */ u32 ndst; }; u64 control; diff --git a/arch/x86/kvm/vmx/posted_intr.c b/arch/x86/kvm/vmx/posted_intr.c index af662312fd07..592dbb765675 100644 --- a/arch/x86/kvm/vmx/posted_intr.c +++ b/arch/x86/kvm/vmx/posted_intr.c @@ -107,7 +107,7 @@ void vmx_vcpu_pi_load(struct kvm_vcpu *vcpu, int cpu) * handle task migration (@cpu != vcpu->cpu). */ new.ndst = dest; - new.sn = 0; + new.notif_ctrl &= ~POSTED_INTR_SN; /* * Restore the notification vector; in the blocking case, the @@ -157,7 +157,7 @@ static void pi_enable_wakeup_handler(struct kvm_vcpu *vcpu) &per_cpu(wakeup_vcpus_on_cpu, vcpu->cpu)); raw_spin_unlock(&per_cpu(wakeup_vcpus_on_cpu_lock, vcpu->cpu)); - WARN(pi_desc->sn, "PI descriptor SN field set before blocking"); + WARN(pi_desc->notif_ctrl & POSTED_INTR_SN, "PI descriptor SN field set before blocking"); old.control = READ_ONCE(pi_desc->control); do { diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d94bb069bac9..50580bbfba5d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -4843,7 +4843,7 @@ static void __vmx_vcpu_reset(struct kvm_vcpu *vcpu) * or POSTED_INTR_WAKEUP_VECTOR. */ vmx->pi_desc.nv = POSTED_INTR_VECTOR; - vmx->pi_desc.sn = 1; + vmx->pi_desc.notif_ctrl |= POSTED_INTR_SN; } static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) From patchwork Fri Apr 5 22:31:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619551 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A235175578; Fri, 5 Apr 2024 22:26:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356010; cv=none; b=D/HSW2cA+l1IzzLOORvSsqLyuf3cONF/rifnm/ufymF2sKkAngVr2tHyWNNsm/BMEyuUGZH+f1txkM7O+7vQ2cwpOa+VIJedfOGb7ZfyNQBq/0ukEXEaRZgY7KuOuzZFIpg3Nz8cxya88gyUAgFB/PaJ7AM3G/32r1h7EGj8vCY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356010; c=relaxed/simple; bh=BpLqiaNgpSDbcOwdg5E2m/UG/ZOHZCK4TD1uJoCRw+g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QSIAbDk0yNf/J4qT9eoL6ETFdp9pnLscW34kK/H+TnuILT+D6mTxwZVQkecWGsydauHsDNmBj5PsJQhTU9afBPIVbX7RjLYnSXHUwMON3MvFqlSMK+d9TsM0wY/1gktE5FHEKlmYV+PcXo/VUG5+dPJGrs2QuDjkGJZtE10Z5To= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=PLtPgPC9; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="PLtPgPC9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356010; x=1743892010; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BpLqiaNgpSDbcOwdg5E2m/UG/ZOHZCK4TD1uJoCRw+g=; b=PLtPgPC9xIaiu1DdRgTXdhhZ11NBGsjO79wcZPY64j3DOuCSRUYN/NIT 8T8xg6B8+Ul4ZZ09g4ud9lsHMui8Dux3ETYo1FK52+z6CxmNoajrmvAa6 0olbukhECKM++n2pyWztdLRQvL130GxgozQlU3AJnzsjQSo+o0x3pMaeL 7AAlD+Xlfv1QYoFOCfMy0p9v309Tn/UwdJw/PQzI7CB6JsmaRikUFiYLf DV1br842rvviPJkRLJntQz1htRYTnsguPWSyD4qrsiaMjp9OM9Us5kCsB uF6rQNeVkso5PtIpnZlYboa+HwpJHotm1i2ZsFyaamisdGujV97NcuNS6 Q==; X-CSE-ConnectionGUID: nXNelvspQQKeNW7KQz3CPg== X-CSE-MsgGUID: PiFS6lxMRqWHzCS1C7FLnw== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062717" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062717" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:47 -0700 X-CSE-ConnectionGUID: 13qTgNPtQAOkBjpU3IPkgQ== X-CSE-MsgGUID: mpnrgrsBTUWcfeColep1OQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928317" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:46 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 04/13] x86/irq: Add a Kconfig option for posted MSI Date: Fri, 5 Apr 2024 15:31:01 -0700 Message-Id: <20240405223110.1609888-5-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This option will be used to support delivering MSIs as posted interrupts. Interrupt remapping is required. Signed-off-by: Jacob Pan --- v2: Remove x2apic dependency --- arch/x86/Kconfig | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 39886bab943a..f5688f4e299b 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -464,6 +464,17 @@ config X86_X2APIC If you don't know what to do here, say N. +config X86_POSTED_MSI + bool "Enable MSI and MSI-x delivery by posted interrupts" + depends on X86_64 && IRQ_REMAP + help + This enables MSIs that are under interrupt remapping to be delivered as + posted interrupts to the host kernel. Interrupt throughput can + potentially be improved by coalescing CPU notifications during high + frequency bursts. + + If you don't know what to do here, say N. + config X86_MPPARSE bool "Enable MPS table" if ACPI default y From patchwork Fri Apr 5 22:31:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619552 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0301176FB5; Fri, 5 Apr 2024 22:26:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356011; cv=none; b=lnnuGC1UpMwD7cq1yI5/6I9Td1MspHrXc2jD27oL4hKX8NRYlxZC9X5zSKmuCcxJxpjVlHqP3q9DwdaWa49/O99e3EdlXLxab50jMfpMJv0Asj4uQqR5m7wOWxrHLtxh3L2i/eXX+BJBTo6PbJbxkW2UbmUrwXBvJaAA8WP6IPo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356011; c=relaxed/simple; bh=0w0uJEzA4z9kskjV3ew8MBQxd2yM5o3G1Z9OFkj0BDg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZMLuTeSDHxG2qw85hYgZDPsVII1xZaOjaDWElyuH+h8BAurh75Er3dcSsYguIDg8ARnn190DXVt09WoEj8nkKScP+gXnwxdcp8tdtQV9Hv3hEDECjiJyLNUmfpyeUMvbpTfT3d5mlaWSQDWyuAqmDnO65ZYAwP0jK6Ql1DMfURw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=PVixcInh; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="PVixcInh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356010; x=1743892010; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0w0uJEzA4z9kskjV3ew8MBQxd2yM5o3G1Z9OFkj0BDg=; b=PVixcInhTvo7UD1kYgCSxxz3PMCX389W6QOtoepLADiuhMSrPPvJoosX y0zfH60FAJykVgdvSAQzY54t32oM/rGTaudZ/h7DgA5YYwkAASfpr3SUj Mz0boQrM4lMrj34gbxBG1qy+SKirDHttOzNH1rfxU/zN3CnGTK7wC6f/s FVDEUBOLFDYTa+gkO93BAJNAsWGy301iiT9uNYtxD8QNqBGxpV40Lf+Ty lSTTl7OqQHXxcvsEkbp4PXpwKF/Q+UaB5r1whBKZC+Y6QLwYEciDecnsv IysCD+Mv+7Xjtc9X29TLWLCQHAB2zih5+gUx0tbHP3bHsKqeNUepunc/A Q==; X-CSE-ConnectionGUID: a0RflBLhSiuG/eUi+5Nm9w== X-CSE-MsgGUID: oQW3hgfwR067H4faxH9cHw== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062734" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062734" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:48 -0700 X-CSE-ConnectionGUID: YpW3/WDlS+mgWCYtHA3Bjw== X-CSE-MsgGUID: WoQPFCO3SEuXJvtd4IBqrg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928320" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:47 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 05/13] x86/irq: Reserve a per CPU IDT vector for posted MSIs Date: Fri, 5 Apr 2024 15:31:02 -0700 Message-Id: <20240405223110.1609888-6-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When posted MSI is enabled, all device MSIs are multiplexed into a single notification vector. MSI handlers will be de-multiplexed at run-time by system software without IDT delivery. Signed-off-by: Jacob Pan --- v2: - Add missing CONFIG_ in #ifdef - Extend changes to x86 tools --- arch/x86/include/asm/irq_vectors.h | 9 ++++++++- tools/arch/x86/include/asm/irq_vectors.h | 9 ++++++++- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/irq_vectors.h b/arch/x86/include/asm/irq_vectors.h index d18bfb238f66..1ee00be8218d 100644 --- a/arch/x86/include/asm/irq_vectors.h +++ b/arch/x86/include/asm/irq_vectors.h @@ -97,9 +97,16 @@ #define LOCAL_TIMER_VECTOR 0xec +/* + * Posted interrupt notification vector for all device MSIs delivered to + * the host kernel. + */ +#define POSTED_MSI_NOTIFICATION_VECTOR 0xeb #define NR_VECTORS 256 -#ifdef CONFIG_X86_LOCAL_APIC +#ifdef CONFIG_X86_POSTED_MSI +#define FIRST_SYSTEM_VECTOR POSTED_MSI_NOTIFICATION_VECTOR +#elif defined(CONFIG_X86_LOCAL_APIC) #define FIRST_SYSTEM_VECTOR LOCAL_TIMER_VECTOR #else #define FIRST_SYSTEM_VECTOR NR_VECTORS diff --git a/tools/arch/x86/include/asm/irq_vectors.h b/tools/arch/x86/include/asm/irq_vectors.h index 3f73ac3ed3a0..989816ca7c9e 100644 --- a/tools/arch/x86/include/asm/irq_vectors.h +++ b/tools/arch/x86/include/asm/irq_vectors.h @@ -99,9 +99,16 @@ #define LOCAL_TIMER_VECTOR 0xec +/* + * Posted interrupt notification vector for all device MSIs delivered to + * the host kernel. + */ +#define POSTED_MSI_NOTIFICATION_VECTOR 0xeb #define NR_VECTORS 256 -#ifdef CONFIG_X86_LOCAL_APIC +#ifdef CONFIG_X86_POSTED_MSI +#define FIRST_SYSTEM_VECTOR POSTED_MSI_NOTIFICATION_VECTOR +#elif defined(CONFIG_X86_LOCAL_APIC) #define FIRST_SYSTEM_VECTOR LOCAL_TIMER_VECTOR #else #define FIRST_SYSTEM_VECTOR NR_VECTORS From patchwork Fri Apr 5 22:31:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619554 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C994C17798A; Fri, 5 Apr 2024 22:26:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356012; cv=none; b=rgqPFEaOdXVt12GrWrumzf/edA05bF7xwPMZO7S0REfOjl9ZN/vwCEgjzTk0oMsbtCfZCevONsFVZ/La0mTE1BHqGeLw6F+j/fL3EBXoV5wlFTQzVFmEs0GKHlSO8wShFhgBcpducesyrbkIt5U9TYUu+SS9HUYHoCURsT9iQ54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356012; c=relaxed/simple; bh=SZUjCrw1LUAiGX4RfPDzIUAm4sRy4Tpnthwda+0PrNE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=g5zAaGcrd66wBILX+LUa4Kni0MJrKP79ZDtuO2U351mJqYu2dJyki3vz0gyEaZKxNoSprb8HraIg/VoxHl1T/nOJW6bHvHADwUKgP3hq9ooeyHMJSSts+PuR0mzc/4JWd+pQC9ZkVWDUgbjVK1OWl1pvydbpoY2NfzheFLrh4AA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=j36vr0q8; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="j36vr0q8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356011; x=1743892011; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SZUjCrw1LUAiGX4RfPDzIUAm4sRy4Tpnthwda+0PrNE=; b=j36vr0q8zFUCTvwNm6J8ebuvCz4P0fungbfdjziywe9hesKnHFBzH0wV +zysY0hkaCCxsXIJbI0r0TRxkIkvRJPTt6RWqmg+K25ggBFssFOWt6P28 tyDHPuSRjxvplrKT+Hulxdq37unA+2DN0Lrtym8/feuKJH0cG5p1+ogrC nXbqRgd19P3CzTHl5IAkpIk7BhIXK5gpbgtdj5rel+U5/LfSRmlNWkNKv pNa6V3cubOXwzHfiafUhk3l/2LQ/wGUG73bf5jyDH4U9DK3V4N8dH29JD KwiTz2O2FOqyIi7bb1t6IiR5gCg7w1rtBFucG+3PBJRKaC7zt6neUnj6/ Q==; X-CSE-ConnectionGUID: Uf7U6SoAQh2pdlSs1KeB4g== X-CSE-MsgGUID: +k9FiiP/SNm0mjrHenEQZg== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062748" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062748" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:49 -0700 X-CSE-ConnectionGUID: c7lY9DQnQJahb0u+aMx9BA== X-CSE-MsgGUID: kqbc0R3dTH+A67aTYnJJeg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928323" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:48 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 06/13] x86/irq: Set up per host CPU posted interrupt descriptors Date: Fri, 5 Apr 2024 15:31:03 -0700 Message-Id: <20240405223110.1609888-7-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To support posted MSIs, create a posted interrupt descriptor (PID) for each host CPU. Later on, when setting up IRQ CPU affinity, IOMMU's interrupt remapping table entry (IRTE) will point to the physical address of the matching CPU's PID. Each PID is initialized with the owner CPU's physical APICID as the destination. Originally-by: Thomas Gleixner Signed-off-by: Jacob Pan --- v2: Fix xAPIC destination ID assignment, Oliver Sang reported failture on system with x2apic optout BIOS option. --- arch/x86/include/asm/hardirq.h | 3 +++ arch/x86/include/asm/posted_intr.h | 7 +++++++ arch/x86/kernel/cpu/common.c | 3 +++ arch/x86/kernel/irq.c | 23 +++++++++++++++++++++++ 4 files changed, 36 insertions(+) diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index fbc7722b87d1..01ea52859462 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -48,6 +48,9 @@ typedef struct { DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); +#ifdef CONFIG_X86_POSTED_MSI +DECLARE_PER_CPU_ALIGNED(struct pi_desc, posted_interrupt_desc); +#endif #define __ARCH_IRQ_STAT #define inc_irq_stat(member) this_cpu_inc(irq_stat.member) diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index c682c41d4e44..4e5eea2d20e0 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -80,4 +80,11 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) return test_bit(POSTED_INTR_SN, (unsigned long *)&pi_desc->control); } +#ifdef CONFIG_X86_POSTED_MSI +extern void intel_posted_msi_init(void); + +#else +static inline void intel_posted_msi_init(void) {}; + +#endif /* X86_POSTED_MSI */ #endif /* _X86_POSTED_INTR_H */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 5c1e6d6be267..1a5fb657e1b6 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -68,6 +68,7 @@ #include #include #include +#include #include "cpu.h" @@ -2219,6 +2220,8 @@ void cpu_init(void) barrier(); x2apic_setup(); + + intel_posted_msi_init(); } mmgrab(&init_mm); diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 35fde0107901..f39f6147104c 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -22,6 +22,8 @@ #include #include #include +#include +#include #define CREATE_TRACE_POINTS #include @@ -334,6 +336,27 @@ DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm_posted_intr_nested_ipi) } #endif +#ifdef CONFIG_X86_POSTED_MSI + +/* Posted Interrupt Descriptors for coalesced MSIs to be posted */ +DEFINE_PER_CPU_ALIGNED(struct pi_desc, posted_interrupt_desc); + +void intel_posted_msi_init(void) +{ + u32 destination; + u32 apic_id; + + this_cpu_write(posted_interrupt_desc.nv, POSTED_MSI_NOTIFICATION_VECTOR); + + /* + * APIC destination ID is stored in bit 8:15 while in XAPIC mode. + * VT-d spec. CH 9.11 + */ + apic_id = this_cpu_read(x86_cpu_to_apicid); + destination = x2apic_enabled() ? apic_id : apic_id << 8; + this_cpu_write(posted_interrupt_desc.ndst, destination); +} +#endif /* X86_POSTED_MSI */ #ifdef CONFIG_HOTPLUG_CPU /* A cpu has been removed from cpu_online_mask. Reset irq affinities. */ From patchwork Fri Apr 5 22:31:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619553 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21D3717799D; Fri, 5 Apr 2024 22:26:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356012; cv=none; b=HWTRAL80cUmV2gpAHybi5sWu/cAuyXDYsTXAT9mX35H1xDu/MJioSJqfegcHU7YQKqJbzO2W1Mjepgtmh6a2ZcPpnWUb340Q/hPuHq6BBEZxUr1lE7GpnQZJthfEIxjBXi24nqBYfoQZZ1yraEs9kX2yyO4jopM3BQQRVcUdoVQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356012; c=relaxed/simple; bh=aMX9sw5AlWfnuSsCSRnkGkG0cgUx+sH4S0oPTBTmTuo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hgT7YrcoypaPQGq9aIqoGTXq9gLx0XiflV+VAsLpH8Wll+42W5fansXaxyQfyPse/7HQuMFiZ7EE3sWBQVXvAtrtoUdJOfaykZSVMpZVAiR2PMXTsJYcOvaPzlhXy/+KbxGvcxoNrGhMIoF5XK7EhzJyHzizX4zPskqMsQ+HTwQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=YiO91EgE; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="YiO91EgE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356011; x=1743892011; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aMX9sw5AlWfnuSsCSRnkGkG0cgUx+sH4S0oPTBTmTuo=; b=YiO91EgENp/mq0TUFgdy5gCVa2pDXiz9IZe5WZMrS2au+UAWT1eB6H+K AtY4aJmkZcA+LkQeSvEPx6svC+99l+ayVmFwztNLgdKOpGPfMe9U2SCrC JoysUApXB2MxdyouFGpP87xucQ+XlH0zZ/QlSLOQgiEFyeBmoCsTP4SY3 8vkZSdMcAiREgIkDglRDz4MATWFxz60N/3ii3xsDOz3z1P4PYZn0mtch1 yK8sQCCKAnmM7AzCw0D0aKw27GuVDGgi6H2M196im9P92Y9eIaUkPjHW2 mn858elRPPLtVlyOAS/NTXVFulYBnjeyuQTJURFg6QGVqJnC0ix8qqeEu A==; X-CSE-ConnectionGUID: mUtZWD4JQ5uqx6vWDcfugA== X-CSE-MsgGUID: TBEFnWwkRsGP1Wr+EqKOwg== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062762" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062762" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:50 -0700 X-CSE-ConnectionGUID: un+Vo/c2SMSc7Y7UhwtFGg== X-CSE-MsgGUID: uqcZv7bYQ1qshj4GMGZmpQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928326" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:48 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 07/13] x86/irq: Factor out calling ISR from common_interrupt Date: Fri, 5 Apr 2024 15:31:04 -0700 Message-Id: <20240405223110.1609888-8-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Prepare for calling external IRQ handlers directly from the posted MSI demultiplexing loop. Extract the common code with common interrupt to avoid code duplication. Signed-off-by: Jacob Pan --- arch/x86/kernel/irq.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index f39f6147104c..c54de9378943 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -242,18 +242,10 @@ static __always_inline void handle_irq(struct irq_desc *desc, __handle_irq(desc, regs); } -/* - * common_interrupt() handles all normal device IRQ's (the special SMP - * cross-CPU interrupts have their own entry points). - */ -DEFINE_IDTENTRY_IRQ(common_interrupt) +static __always_inline void call_irq_handler(int vector, struct pt_regs *regs) { - struct pt_regs *old_regs = set_irq_regs(regs); struct irq_desc *desc; - /* entry code tells RCU that we're not quiescent. Check it. */ - RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU"); - desc = __this_cpu_read(vector_irq[vector]); if (likely(!IS_ERR_OR_NULL(desc))) { handle_irq(desc, regs); @@ -268,7 +260,20 @@ DEFINE_IDTENTRY_IRQ(common_interrupt) __this_cpu_write(vector_irq[vector], VECTOR_UNUSED); } } +} + +/* + * common_interrupt() handles all normal device IRQ's (the special SMP + * cross-CPU interrupts have their own entry points). + */ +DEFINE_IDTENTRY_IRQ(common_interrupt) +{ + struct pt_regs *old_regs = set_irq_regs(regs); + + /* entry code tells RCU that we're not quiescent. Check it. */ + RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU"); + call_irq_handler(vector, regs); set_irq_regs(old_regs); } From patchwork Fri Apr 5 22:31:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619555 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6B69178CC3; Fri, 5 Apr 2024 22:26:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356013; cv=none; b=O3ROSB7ZAaHdU2xi/H8qY1Hx47b6268WDlrFeX4dwISejW5wp8issxEn43nDWvIRgFfisJ1DAMMUWSjdeT1Nc7Rsl39SSA/sI93P7e2l9uBx+5/ai83/14avWscfpJ3qN9STBFrGT1AJE5rqffwheqbPCgt9T67CV4dS8Cbq8DA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356013; c=relaxed/simple; bh=UOx0iCctnoI4CxYoYH2BoshQQWoeZtr8Db2m4Zrt2wA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tqcjw86bxh2rJ64q5YJFn0zjYKuRs5Pt/Csx5sako14F3IYFO0gtdJcFpbHCC604+C6iGt4ZlkoHRkNFAC4IIq6cDIXzHFN8cziZzvefjaFCbp/MHcqV3Z6VPKKp1w/03o3yttdTFtsv6uEr4NoZKvhtuq6vR2MTMOOX+c96QvY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=AbdRyA1j; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="AbdRyA1j" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356012; x=1743892012; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UOx0iCctnoI4CxYoYH2BoshQQWoeZtr8Db2m4Zrt2wA=; b=AbdRyA1jcRswUiJ7jITxWQBqgL0zTa5IZhqSfDfTTIyYMtYNEh2GCfLO DLTTTQYClsJYCrWcCpuY/bblDJ5I2fuWSLfedYm0ksIWobMCQQnzF5K0k EvT7Dh23H3i7zneDEO5XLUR6PaTHaViIN/C3PYyQXZkEDeYVD3+XYzdLI fiyKH6O3ljdt5iL1dsuiHwwa64U5bkIniwvS7A/VQEraekn+BRgquuARW /qRGrYncFWIsTV+A9RZadAbxP7eIx8yS3gPx9FIXC4RWavNRxOm2juW1F ZUvQFr2MKuTKbNF8TGdjCBDb22Agb/3V+1FBb4sdZV+huvqV3GLhJbr+y Q==; X-CSE-ConnectionGUID: M4eX8FMpQimbIQSmtvymbQ== X-CSE-MsgGUID: isQ24W/MRgqvtdAWUESPcg== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062777" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062777" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:50 -0700 X-CSE-ConnectionGUID: zaJw613uTSe3BubvMvJJWw== X-CSE-MsgGUID: NnCsgckpQSq8xOEmxpEMuQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928329" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:49 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 08/13] x86/irq: Install posted MSI notification handler Date: Fri, 5 Apr 2024 15:31:05 -0700 Message-Id: <20240405223110.1609888-9-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 All MSI vectors are multiplexed into a single notification vector when posted MSI is enabled. It is the responsibility of the notification vector handler to demultiplex MSI vectors. In this handler, for each pending bit, MSI vector handlers are dispatched without IDT delivery. For example, the interrupt flow will change as follows: (3 MSIs of different vectors arrive in a a high frequency burst) BEFORE: interrupt(MSI) irq_enter() handler() /* EOI */ irq_exit() process_softirq() interrupt(MSI) irq_enter() handler() /* EOI */ irq_exit() process_softirq() interrupt(MSI) irq_enter() handler() /* EOI */ irq_exit() process_softirq() AFTER: interrupt /* Posted MSI notification vector */ irq_enter() atomic_xchg(PIR) handler() handler() handler() pi_clear_on() apic_eoi() irq_exit() process_softirq() Except for the leading MSI, CPU notifications are skipped/coalesced. For MSIs arrive at a low frequency, the demultiplexing loop does not wait for more interrupts to coalesce. Therefore, there's no additional latency other than the processing time. Signed-off-by: Jacob Pan --- v2: - Delete extra inline attribute - Fix pir pointer in xchg (Zeng Guang) --- arch/x86/include/asm/hardirq.h | 3 + arch/x86/include/asm/idtentry.h | 3 + arch/x86/kernel/idt.c | 3 + arch/x86/kernel/irq.c | 113 ++++++++++++++++++++++++++++++++ 4 files changed, 122 insertions(+) diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index 01ea52859462..ee0c05366dec 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -44,6 +44,9 @@ typedef struct { unsigned int irq_hv_reenlightenment_count; unsigned int hyperv_stimer0_count; #endif +#ifdef CONFIG_X86_POSTED_MSI + unsigned int posted_msi_notification_count; +#endif } ____cacheline_aligned irq_cpustat_t; DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h index 749c7411d2f1..5a6fc7593cfc 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -701,6 +701,9 @@ DECLARE_IDTENTRY_SYSVEC(ERROR_APIC_VECTOR, sysvec_error_interrupt); DECLARE_IDTENTRY_SYSVEC(SPURIOUS_APIC_VECTOR, sysvec_spurious_apic_interrupt); DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECTOR, sysvec_apic_timer_interrupt); DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR, sysvec_x86_platform_ipi); +# ifdef CONFIG_X86_POSTED_MSI +DECLARE_IDTENTRY_SYSVEC(POSTED_MSI_NOTIFICATION_VECTOR, sysvec_posted_msi_notification); +# endif #endif #ifdef CONFIG_SMP diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index fc37c8d83daf..f445bec516a0 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -163,6 +163,9 @@ static const __initconst struct idt_data apic_idts[] = { # endif INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), +# ifdef CONFIG_X86_POSTED_MSI + INTG(POSTED_MSI_NOTIFICATION_VECTOR, asm_sysvec_posted_msi_notification), +# endif #endif }; diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index c54de9378943..8772b0a3abf6 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -183,6 +183,13 @@ int arch_show_interrupts(struct seq_file *p, int prec) seq_printf(p, "%10u ", irq_stats(j)->kvm_posted_intr_wakeup_ipis); seq_puts(p, " Posted-interrupt wakeup event\n"); +#endif +#ifdef CONFIG_X86_POSTED_MSI + seq_printf(p, "%*s: ", prec, "PMN"); + for_each_online_cpu(j) + seq_printf(p, "%10u ", + irq_stats(j)->posted_msi_notification_count); + seq_puts(p, " Posted MSI notification event\n"); #endif return 0; } @@ -361,6 +368,112 @@ void intel_posted_msi_init(void) destination = x2apic_enabled() ? apic_id : apic_id << 8; this_cpu_write(posted_interrupt_desc.ndst, destination); } + +/* + * De-multiplexing posted interrupts is on the performance path, the code + * below is written to optimize the cache performance based on the following + * considerations: + * 1.Posted interrupt descriptor (PID) fits in a cache line that is frequently + * accessed by both CPU and IOMMU. + * 2.During posted MSI processing, the CPU needs to do 64-bit read and xchg + * for checking and clearing posted interrupt request (PIR), a 256 bit field + * within the PID. + * 3.On the other side, the IOMMU does atomic swaps of the entire PID cache + * line when posting interrupts and setting control bits. + * 4.The CPU can access the cache line a magnitude faster than the IOMMU. + * 5.Each time the IOMMU does interrupt posting to the PIR will evict the PID + * cache line. The cache line states after each operation are as follows: + * CPU IOMMU PID Cache line state + * --------------------------------------------------------------- + *...read64 exclusive + *...lock xchg64 modified + *... post/atomic swap invalid + *...------------------------------------------------------------- + * + * To reduce L1 data cache miss, it is important to avoid contention with + * IOMMU's interrupt posting/atomic swap. Therefore, a copy of PIR is used + * to dispatch interrupt handlers. + * + * In addition, the code is trying to keep the cache line state consistent + * as much as possible. e.g. when making a copy and clearing the PIR + * (assuming non-zero PIR bits are present in the entire PIR), it does: + * read, read, read, read, xchg, xchg, xchg, xchg + * instead of: + * read, xchg, read, xchg, read, xchg, read, xchg + */ +static __always_inline bool handle_pending_pir(u64 *pir, struct pt_regs *regs) +{ + int i, vec = FIRST_EXTERNAL_VECTOR; + unsigned long pir_copy[4]; + bool handled = false; + + for (i = 0; i < 4; i++) + pir_copy[i] = pir[i]; + + for (i = 0; i < 4; i++) { + if (!pir_copy[i]) + continue; + + pir_copy[i] = arch_xchg(&pir[i], 0); + handled = true; + } + + if (handled) { + for_each_set_bit_from(vec, pir_copy, FIRST_SYSTEM_VECTOR) + call_irq_handler(vec, regs); + } + + return handled; +} + +/* + * Performance data shows that 3 is good enough to harvest 90+% of the benefit + * on high IRQ rate workload. + */ +#define MAX_POSTED_MSI_COALESCING_LOOP 3 + +/* + * For MSIs that are delivered as posted interrupts, the CPU notifications + * can be coalesced if the MSIs arrive in high frequency bursts. + */ +DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) +{ + struct pt_regs *old_regs = set_irq_regs(regs); + struct pi_desc *pid; + int i = 0; + + pid = this_cpu_ptr(&posted_interrupt_desc); + + inc_irq_stat(posted_msi_notification_count); + irq_enter(); + + /* + * Max coalescing count includes the extra round of handle_pending_pir + * after clearing the outstanding notification bit. Hence, at most + * MAX_POSTED_MSI_COALESCING_LOOP - 1 loops are executed here. + */ + while (++i < MAX_POSTED_MSI_COALESCING_LOOP) { + if (!handle_pending_pir(pid->pir64, regs)) + break; + } + + /* + * Clear outstanding notification bit to allow new IRQ notifications, + * do this last to maximize the window of interrupt coalescing. + */ + pi_clear_on(pid); + + /* + * There could be a race of PI notification and the clearing of ON bit, + * process PIR bits one last time such that handling the new interrupts + * are not delayed until the next IRQ. + */ + handle_pending_pir(pid->pir64, regs); + + apic_eoi(); + irq_exit(); + set_irq_regs(old_regs); +} #endif /* X86_POSTED_MSI */ #ifdef CONFIG_HOTPLUG_CPU From patchwork Fri Apr 5 22:31:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619558 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A97C1791F8; Fri, 5 Apr 2024 22:26:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356015; cv=none; b=lxHLL383q+BV/6bVUk+6aq6bJc6P14xqZLjeHQ10PZcZxWaeAajd3UwKwgS6j0yrpwuoYO2SOmupTDjKE4KS6FtUHuLCUaJYRJqyWTxw5YLmbPHEx9buUNdcjZeZO/SHVk3ieGVsYVZFCEJsROvz1vj2GdUC+NCFjQd5uKvvl7k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356015; c=relaxed/simple; bh=LkTbi44XEjcmD+H9V0/r7+1X5xEeMnDv3K3o3ANyhmw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Tme879AQ8RGl/z3r3/RT9Nz21uJZ2H0mK/1sGPVwJSlkkFfhtU4SVkLwrF2elGRuW1qZqYtYTQJe1IsKAmPe9TU/G2yVbPJAXusB6IC1PYL11KvMrPXU9IiTonzA9c2ODgAklxTbAQG4sFcdM6b8zm0GvGXM7EacWM7VPGK3piw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=K/ohXARw; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="K/ohXARw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356013; x=1743892013; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LkTbi44XEjcmD+H9V0/r7+1X5xEeMnDv3K3o3ANyhmw=; b=K/ohXARwH+bNb79vSuF48fODgcX+5v5JsD6922Lz+ocp95W13kEJhwuu diqyiPAv+SGfYsmT/hLkzp4RoYgJ3Ap3sLFyhmO+xMRte4crOWqpPp2B7 PajfypQLKzuhwqezg4Tk6wVmBj+AGURh6BHgh2Snn4F3e3dU8vJPoCK7E LZOf6g7smibbZjQ+wnVKyZct3+2q3A7gdTeBPRcmf5ydG1BM1L55wA+KW VseGr7lYSD+rkXwdL5fdyINEyP0fZuQppY8IFwUNh3DtVogzxHPi1YAt8 9mMOoBkJ5Bsr+4KsKUspes+u70p1YLGP30hYvdA4MNC3YDVnGVeJD1Sis Q==; X-CSE-ConnectionGUID: oA5CSr+VQbq9XBlIxV/sww== X-CSE-MsgGUID: 7I0nbuLgSDWKfzrXVEuRnw== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062791" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062791" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:51 -0700 X-CSE-ConnectionGUID: cOkIIhzfSXisjOEM48bJGQ== X-CSE-MsgGUID: zH4owt0ZTBS8MP9bwG+JrQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928332" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:50 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 09/13] x86/irq: Factor out common code for checking pending interrupts Date: Fri, 5 Apr 2024 15:31:06 -0700 Message-Id: <20240405223110.1609888-10-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Use a common function for checking pending interrupt vector in APIC IRR instead of duplicated open coding them. Additional checks for posted MSI vectors can then be contained in this function. Signed-off-by: Jacob Pan --- arch/x86/include/asm/apic.h | 11 +++++++++++ arch/x86/kernel/apic/vector.c | 5 ++--- arch/x86/kernel/irq.c | 5 ++--- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h index 94ce0f7c9d3a..ebf80718912d 100644 --- a/arch/x86/include/asm/apic.h +++ b/arch/x86/include/asm/apic.h @@ -499,6 +499,17 @@ static inline bool lapic_vector_set_in_irr(unsigned int vector) return !!(irr & (1U << (vector % 32))); } +static inline bool is_vector_pending(unsigned int vector) +{ + unsigned int irr; + + irr = apic_read(APIC_IRR + (vector / 32 * 0x10)); + if (irr & (1 << (vector % 32))) + return true; + + return false; +} + /* * Warm reset vector position: */ diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c index 185738c72766..9eec52925fa3 100644 --- a/arch/x86/kernel/apic/vector.c +++ b/arch/x86/kernel/apic/vector.c @@ -965,7 +965,7 @@ static void __vector_cleanup(struct vector_cleanup *cl, bool check_irr) lockdep_assert_held(&vector_lock); hlist_for_each_entry_safe(apicd, tmp, &cl->head, clist) { - unsigned int irr, vector = apicd->prev_vector; + unsigned int vector = apicd->prev_vector; /* * Paranoia: Check if the vector that needs to be cleaned @@ -979,8 +979,7 @@ static void __vector_cleanup(struct vector_cleanup *cl, bool check_irr) * fixup_irqs() was just called to scan IRR for set bits and * forward them to new destination CPUs via IPIs. */ - irr = check_irr ? apic_read(APIC_IRR + (vector / 32 * 0x10)) : 0; - if (irr & (1U << (vector % 32))) { + if (check_irr && is_vector_pending(vector)) { pr_warn_once("Moved interrupt pending in old target APIC %u\n", apicd->irq); rearm = true; continue; diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 8772b0a3abf6..3206d48f380d 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -480,7 +480,7 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_posted_msi_notification) /* A cpu has been removed from cpu_online_mask. Reset irq affinities. */ void fixup_irqs(void) { - unsigned int irr, vector; + unsigned int vector; struct irq_desc *desc; struct irq_data *data; struct irq_chip *chip; @@ -507,8 +507,7 @@ void fixup_irqs(void) if (IS_ERR_OR_NULL(__this_cpu_read(vector_irq[vector]))) continue; - irr = apic_read(APIC_IRR + (vector / 32 * 0x10)); - if (irr & (1 << (vector % 32))) { + if (is_vector_pending(vector)) { desc = __this_cpu_read(vector_irq[vector]); raw_spin_lock(&desc->lock); From patchwork Fri Apr 5 22:31:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619556 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A90D1791F7; Fri, 5 Apr 2024 22:26:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356014; cv=none; b=LvgOHxLxVf2vkf8EBdNt+J2PUcjB4aWpiWqFu7mgE2qSOXJxgGP2KMYwlgf4PgQkytqKFZIhygZd7Z83geaN8zBxvLz8gQlkRg9l8qwsaK3WHlpOen5nyLFdjBzQLcugeQUTdc7FNJxz2soVGvfK0or1whBbIFgqPWzmLXwPJpc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356014; c=relaxed/simple; bh=H0TGjg1UoYwF5qFJJyPYA+APS27Sn0hasVdnhJ92ypQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oK6K6b4OkYZO/W4DU4xO+15EIY0LgbHH7yJKkfA098fSOofWQ289/U9nTdTazGd9DDaGiyNih0ICQo+ddPFZzpCosb+lRBIg/N2hEw4/htkhswi98pt/Q461uhsmkbq6xqY6rmJNr6uqYAudGDrAaGbRX960XxoJAqvLeLWTpRw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=QeavWFWh; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="QeavWFWh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356013; x=1743892013; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=H0TGjg1UoYwF5qFJJyPYA+APS27Sn0hasVdnhJ92ypQ=; b=QeavWFWhHaPj/eIADbbRVVKA04Q5MQRzVTBrctUalAaf8KqYBVWLS/R9 o00G2EdagukR8oLZAXhU/iY5vUM9lggWBzNRZDWjxKxoCOATMkv7diilQ P736B2chKmioLiBrjD9B52zDYD0yRfeIEnEg5fEeXeRHeUQvwY/EM332B r4AL1mlzfUA3rESHhZIMVbt4u7Xt+pAqeAMgpJIsJTF0b9r5WJIKXMDHQ Z900qvIR7z9/yBWK/wr3o1QqZJbRhpDeN15gLZusaj+ELd7PhKezPAizu avH924j+vGalA8+tDgI64ro7QC847cIcXQOfnG+AIEWOaiMIrW22EkHD3 g==; X-CSE-ConnectionGUID: isRSDQQzTqyXWvrNNpYGow== X-CSE-MsgGUID: ZZ03KpwnSL+0nE5pZJYBMw== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062806" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062806" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:52 -0700 X-CSE-ConnectionGUID: CcfzkElUSke4mF1BjCAwnw== X-CSE-MsgGUID: Zz+SdyguRFaEXm2dCFFyfA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928336" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:50 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 10/13] x86/irq: Extend checks for pending vectors to posted interrupts Date: Fri, 5 Apr 2024 15:31:07 -0700 Message-Id: <20240405223110.1609888-11-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 During interrupt affinity change, it is possible to have interrupts delivered to the old CPU after the affinity has changed to the new one. To prevent lost interrupts, local APIC IRR is checked on the old CPU. Similar checks must be done for posted MSIs given the same reason. Consider the following scenario: Device system agent iommu memory CPU/LAPIC 1 FEEX_XXXX 2 Interrupt request 3 Fetch IRTE -> 4 ->Atomic Swap PID.PIR(vec) Push to Global Observable(GO) 5 if (ON*) i done;* else 6 send a notification -> * ON: outstanding notification, 1 will suppress new notifications If the affinity change happens between 3 and 5 in IOMMU, the old CPU's posted interrupt request (PIR) could have pending bit set for the vector being moved. This patch adds a helper function to check individual vector status. Then use the helper to check for pending interrupts on the source CPU's PID. Signed-off-by: Jacob Pan --- v2: Fold in helper function patch. --- arch/x86/include/asm/apic.h | 3 ++- arch/x86/include/asm/posted_intr.h | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h index ebf80718912d..5bf0d6c2523b 100644 --- a/arch/x86/include/asm/apic.h +++ b/arch/x86/include/asm/apic.h @@ -13,6 +13,7 @@ #include #include #include +#include #define ARCH_APICTIMER_STOPS_ON_C3 1 @@ -507,7 +508,7 @@ static inline bool is_vector_pending(unsigned int vector) if (irr & (1 << (vector % 32))) return true; - return false; + return pi_pending_this_cpu(vector); } /* diff --git a/arch/x86/include/asm/posted_intr.h b/arch/x86/include/asm/posted_intr.h index 4e5eea2d20e0..8aaa15515490 100644 --- a/arch/x86/include/asm/posted_intr.h +++ b/arch/x86/include/asm/posted_intr.h @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #ifndef _X86_POSTED_INTR_H #define _X86_POSTED_INTR_H +#include #define POSTED_INTR_ON 0 #define POSTED_INTR_SN 1 @@ -81,9 +82,26 @@ static inline bool pi_test_sn(struct pi_desc *pi_desc) } #ifdef CONFIG_X86_POSTED_MSI +/* + * Not all external vectors are subject to interrupt remapping, e.g. IOMMU's + * own interrupts. Here we do not distinguish them since those vector bits in + * PIR will always be zero. + */ +static inline bool pi_pending_this_cpu(unsigned int vector) +{ + struct pi_desc *pid = this_cpu_ptr(&posted_interrupt_desc); + + if (WARN_ON_ONCE(vector > NR_VECTORS || vector < FIRST_EXTERNAL_VECTOR)) + return false; + + return test_bit(vector, (unsigned long *)pid->pir); +} + extern void intel_posted_msi_init(void); #else +static inline bool pi_pending_this_cpu(unsigned int vector) { return false; } + static inline void intel_posted_msi_init(void) {}; #endif /* X86_POSTED_MSI */ From patchwork Fri Apr 5 22:31:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619557 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E607D17921A; Fri, 5 Apr 2024 22:26:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356015; cv=none; b=MHRYe+gGTza7qckoXTUJaacjrPkr2Hv/rNefPpqocp2bamDliEU+1A9ipE9m8br6UQQW3Y+xNSpTAB1OXfeNDsRcNNLe5ajGMW/MDPRSMAEFf+SG+IgqQd/xBZGSjCwsaAySEwv+Eqp6qQ0nlRhucMZNPVgfUbS73daYxrVxV1Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356015; c=relaxed/simple; bh=ki4jrj4UXSWxXrhulX5LioleJEjgSCmUEPsdgUkrskk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Hn35xc+rQjIwchCAscoy1Fl33kCMHN7pRrRxVsBMSzbjCTxeeT7ujj/C7dC8Xv1uQaV6z8Vz3PYjUb0kDjefGvJhpfW81r9iCHJAash3yUuCimXtXLIUDgiLH7xh5GlyvoQobhZNgxLZRxbhnoX81qgxAti2deApIr9vVYBR4XA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZmaRQILT; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZmaRQILT" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356014; x=1743892014; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ki4jrj4UXSWxXrhulX5LioleJEjgSCmUEPsdgUkrskk=; b=ZmaRQILTndojmMC8iz7ZFpTo0yR49+M4j9dsSgykxS0slcQMFwcCS3zs 3AGzitEnAjkoxha8HHXc7UhnV+7QH6SNVaKDgLOnyB/0VisdX35nBOWZQ PJvh5EgqBTD5NR5pjpSKiurbEa8s2oWaZStBkKav6VY5FsWESolWqWqDU oXBB6FbzTncSmpBXS54kWaf2xfB5JGJS9GiPed3TQT4FcJC7dFgXO5vxK gOP8+ZrDzs8TxFOMXCyRIGfRZh1LcX9COVCLgH17PcDOvxH6h/PTAYaD4 w6bLeNg92xt9ZtOPIN41kT/MvAB7uyQW1RnOK55nqTs/fNrai5rge4d2T A==; X-CSE-ConnectionGUID: m0PDuNw5Rt2EEzaTzsmJzA== X-CSE-MsgGUID: i+hxoaDxSPW2N/Ujw3C2Zg== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062821" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062821" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:52 -0700 X-CSE-ConnectionGUID: l268QKSvQXOxZq6H118Ylg== X-CSE-MsgGUID: iNNQOOp0RZeFkQxr0/UccQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928343" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:51 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 11/13] iommu/vt-d: Make posted MSI an opt-in cmdline option Date: Fri, 5 Apr 2024 15:31:08 -0700 Message-Id: <20240405223110.1609888-12-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a command line opt-in option for posted MSI if CONFIG_X86_POSTED_MSI=y. Also introduce a helper function for testing if posted MSI is supported on the platform. Signed-off-by: Jacob Pan --- Documentation/admin-guide/kernel-parameters.txt | 1 + arch/x86/include/asm/irq_remapping.h | 11 +++++++++++ drivers/iommu/irq_remapping.c | 13 ++++++++++++- 3 files changed, 24 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index bb884c14b2f6..e5fd02423c4c 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2251,6 +2251,7 @@ no_x2apic_optout BIOS x2APIC opt-out request will be ignored nopost disable Interrupt Posting + posted_msi enable MSIs delivered as posted interrupts iomem= Disable strict checking of access to MMIO memory strict regions from userspace. diff --git a/arch/x86/include/asm/irq_remapping.h b/arch/x86/include/asm/irq_remapping.h index 7a2ed154a5e1..e46bde61029b 100644 --- a/arch/x86/include/asm/irq_remapping.h +++ b/arch/x86/include/asm/irq_remapping.h @@ -50,6 +50,17 @@ static inline struct irq_domain *arch_get_ir_parent_domain(void) return x86_vector_domain; } +#ifdef CONFIG_X86_POSTED_MSI +extern int enable_posted_msi; + +static inline bool posted_msi_supported(void) +{ + return enable_posted_msi && irq_remapping_cap(IRQ_POSTING_CAP); +} +#else +static inline bool posted_msi_supported(void) { return false; }; +#endif + #else /* CONFIG_IRQ_REMAP */ static inline bool irq_remapping_cap(enum irq_remap_cap cap) { return 0; } diff --git a/drivers/iommu/irq_remapping.c b/drivers/iommu/irq_remapping.c index ee59647c2050..5672783c9f1f 100644 --- a/drivers/iommu/irq_remapping.c +++ b/drivers/iommu/irq_remapping.c @@ -24,6 +24,10 @@ int no_x2apic_optout; int disable_irq_post = 0; +#ifdef CONFIG_X86_POSTED_MSI +int enable_posted_msi; +#endif + static int disable_irq_remap; static struct irq_remap_ops *remap_ops; @@ -70,7 +74,14 @@ static __init int setup_irqremap(char *str) no_x2apic_optout = 1; else if (!strncmp(str, "nopost", 6)) disable_irq_post = 1; - +#ifdef CONFIG_X86_POSTED_MSI + else if (!strncmp(str, "posted_msi", 10)) { + if (disable_irq_post || disable_irq_remap) + pr_warn("Posted MSI not enabled due to conflicting options!"); + else + enable_posted_msi = 1; + } +#endif str += strcspn(str, ","); while (*str == ',') str++; From patchwork Fri Apr 5 22:31:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619559 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2F32179677; Fri, 5 Apr 2024 22:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356016; cv=none; b=QclPwercVt7lUO9H3t7n6T1Xdi3HUzRESm5VcIrd71JDhpuswS+QBuR5mzz61VrSlLhpi/IdsKRpEhSvhjX5uFqcjt7JY2AF+/g9CDMEnsYveAPXGH5UWR6amQ+Tgh0iL4In5RB0+EuvU5OHfPxBpqDHq8EjDAcVSx2RgVk3gv0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356016; c=relaxed/simple; bh=tBMxqVMHu63G2rJ3nBVSuahDnZ+Xdu27rPnRP82vOHs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=naYfFojEkY/F/U30MgZJT9/Zq1mDd1OmiuMPhuDXXW36k+z6AxXNkeEsklSTlhnFcwXJNfWmO35WQBar/S+y3GEh/UEwUiAQJat0GuLfyCBv7I99H5wFajOMoNOqBa+UWsKb7T+T9rAL/+RfgPV/3M/NV6awl4bbsYDsTWm3muI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=b9Uv3vWs; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="b9Uv3vWs" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356015; x=1743892015; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tBMxqVMHu63G2rJ3nBVSuahDnZ+Xdu27rPnRP82vOHs=; b=b9Uv3vWs3X0SC5OwfO5veNXD6M7ewLPuKwxIubIFjzxyV11aYYPJmKeW ryLhMBQftMMMCubh+XVNO1HJeX7bGkSp1V5nWwANd7R8nDTocNe1lm50J MdK0HjOvkXSgpCyzEzm6MPSrqKAW/ShfYQXR5nD8vfYxcd/hA+gvPxr7P 6WTgSxhC5M0J+o+8hcYHb01fHnEnFtUpKKhQ2AJi34WQJcdkBNYqZlIT0 qPS50289c9CGsapqN5OuXN/gvso70oesD0JzE2Fq+5BnkIT266hV9OVYk NQMAQo1K51w4+9EloSC4qiUExl5+WnZSouz6aLMOZiGzp9ihVZVCj9mgO w==; X-CSE-ConnectionGUID: EC57mj7dQAixXK6WNCHH8A== X-CSE-MsgGUID: Pd9wTMPRTIuhvBtjkd6VlQ== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062837" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062837" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:53 -0700 X-CSE-ConnectionGUID: 7tGU/fwcSE+SKauQKx7scQ== X-CSE-MsgGUID: 5NYrU499TKyC9vkaIsUwdQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928350" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:52 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 12/13] iommu/vt-d: Add an irq_chip for posted MSIs Date: Fri, 5 Apr 2024 15:31:09 -0700 Message-Id: <20240405223110.1609888-13-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Introduce a new irq_chip for posted MSIs, the key difference is in irq_ack where EOI is performed by the notification handler. When posted MSI is enabled, MSI domain/chip hierarchy will look like this example: domain: IR-PCI-MSIX-0000:50:00.0-12 hwirq: 0x29 chip: IR-PCI-MSIX-0000:50:00.0 flags: 0x430 IRQCHIP_SKIP_SET_WAKE IRQCHIP_ONESHOT_SAFE parent: domain: INTEL-IR-10-13 hwirq: 0x2d0000 chip: INTEL-IR-POST flags: 0x0 parent: domain: VECTOR hwirq: 0x77 chip: APIC Signed-off-by: Jacob Pan --- drivers/iommu/intel/irq_remapping.c | 46 +++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c index 566297bc87dd..fa719936b44e 100644 --- a/drivers/iommu/intel/irq_remapping.c +++ b/drivers/iommu/intel/irq_remapping.c @@ -1233,6 +1233,52 @@ static struct irq_chip intel_ir_chip = { .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity, }; +static void dummy(struct irq_data *d) +{ +} + +/* + * With posted MSIs, all vectors are multiplexed into a single notification + * vector. Devices MSIs are then dispatched in a demux loop where + * EOIs can be coalesced as well. + * + * "INTEL-IR-POST" IRQ chip does not do EOI on ACK, thus the dummy irq_ack() + * function. Instead EOI is performed by the posted interrupt notification + * handler. + * + * For the example below, 3 MSIs are coalesced into one CPU notification. Only + * one apic_eoi() is needed. + * + * __sysvec_posted_msi_notification() + * irq_enter(); + * handle_edge_irq() + * irq_chip_ack_parent() + * dummy(); // No EOI + * handle_irq_event() + * driver_handler() + * irq_enter(); + * handle_edge_irq() + * irq_chip_ack_parent() + * dummy(); // No EOI + * handle_irq_event() + * driver_handler() + * irq_enter(); + * handle_edge_irq() + * irq_chip_ack_parent() + * dummy(); // No EOI + * handle_irq_event() + * driver_handler() + * apic_eoi() + * irq_exit() + */ +static struct irq_chip intel_ir_chip_post_msi = { + .name = "INTEL-IR-POST", + .irq_ack = dummy, + .irq_set_affinity = intel_ir_set_affinity, + .irq_compose_msi_msg = intel_ir_compose_msi_msg, + .irq_set_vcpu_affinity = intel_ir_set_vcpu_affinity, +}; + static void fill_msi_msg(struct msi_msg *msg, u32 index, u32 subhandle) { memset(msg, 0, sizeof(*msg)); From patchwork Fri Apr 5 22:31:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jacob Pan X-Patchwork-Id: 13619560 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2694179961; Fri, 5 Apr 2024 22:26:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356017; cv=none; b=QOETdDAbPKkz2KDw9OAQGOxPCNxMrqd7YU4VA32G0F6+3V2x7R2LwZfigt7VIqwiU7ChF4qaeGAlg9Q65+r+UaIjGQz1bzs3V/b/q51PM/jCy3xMtWc+524ktG0Cmn51ESHcBGKIzZL/dgmDEC4jwDcs0mPgSu1QaFm5x+mx4Wk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712356017; c=relaxed/simple; bh=ZGWBCczfujdRtFbtPKsDRo5XDo8fcTnRO3bkTJQexVw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q1AnuOLSod+XXPNUNAldrVt4aUJrAUHXF7Odi9as2yCq9ffJv3vEUiydUdcLK9fiki9EQxMaICsv6TgVTFSZHI45iWxA6pGvd1IKX4GdHIY4OvInhUjDWv9e/Z+/XD7gcrXAPEJEQ4Lplvns8AoLAd6s9/PgjksZnwJ9I/ClAjY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=TnJGte1Y; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="TnJGte1Y" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712356016; x=1743892016; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZGWBCczfujdRtFbtPKsDRo5XDo8fcTnRO3bkTJQexVw=; b=TnJGte1YL+nyCRoQLZcXNkdQ2AfgvE3IQg/foLXtWevw6xCX22aY/XQE xlMuHjz/Zqp04UmVke+k7W2p4p1g3ElEgC+15CqJIWoQRr9OsnzA6kFwB 5ZZCFh5+qQEWuZYwnuyShe/P48uVl29tQraCJXq6MIdXoRAWKia4bt73K qVz4W5BN8j1wh+h9pSgT5cYt9guxxTWOqcB4mtpa7962ag6zpS4hm2qAf mbAQlit4JQF6dKkZNYdCD3tTq4sZmTKUGURrfx8OtkcfhSiTTGrSVhyiS CS0yTqwUBIkW+WjES0489u0HYSGYHwGw0RMq218IceN24+BMZ0cdAMs1/ w==; X-CSE-ConnectionGUID: Run8SIX+SWiU9HpnW8Geyw== X-CSE-MsgGUID: NC5Ia1NgTe6WY+u2eFMCrA== X-IronPort-AV: E=McAfee;i="6600,9927,11035"; a="19062853" X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="19062853" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2024 15:26:54 -0700 X-CSE-ConnectionGUID: dqE0TdDNQ62K5gm9q66mFQ== X-CSE-MsgGUID: icfEMMlNRoyitei5C2xCcw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,182,1708416000"; d="scan'208";a="23928356" Received: from jacob-builder.jf.intel.com ([10.54.39.125]) by fmviesa004.fm.intel.com with ESMTP; 05 Apr 2024 15:26:52 -0700 From: Jacob Pan To: LKML , X86 Kernel , Peter Zijlstra , iommu@lists.linux.dev, Thomas Gleixner , "Lu Baolu" , kvm@vger.kernel.org, Dave Hansen , Joerg Roedel , "H. Peter Anvin" , "Borislav Petkov" , "Ingo Molnar" Cc: Paul Luse , Dan Williams , Jens Axboe , Raj Ashok , "Tian, Kevin" , maz@kernel.org, seanjc@google.com, "Robin Murphy" , jim.harris@samsung.com, a.manzanares@samsung.com, "Bjorn Helgaas" , guang.zeng@intel.com, robert.hoo.linux@gmail.com, Jacob Pan Subject: [PATCH v2 13/13] iommu/vt-d: Enable posted mode for device MSIs Date: Fri, 5 Apr 2024 15:31:10 -0700 Message-Id: <20240405223110.1609888-14-jacob.jun.pan@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> References: <20240405223110.1609888-1-jacob.jun.pan@linux.intel.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 With posted MSI feature enabled on the CPU side, iommu interrupt remapping table entries (IRTEs) for device MSI/x can be allocated, activated, and programed in posted mode. This means that IRTEs are linked with their respective PIDs of the target CPU. Handlers for the posted MSI notification vector will de-multiplex device MSI handlers. CPU notifications are coalesced if interrupts arrive at a high frequency. Excluding the following: - legacy devices IOAPIC, HPET (may be needed for booting, not a source of high MSIs) - VT-d's own IRQs (not remappable). Suggested-by: Thomas Gleixner Signed-off-by: Jacob Pan --- v2: Fold in helper function for retrieving PID address v1: Added a warning if the effective affinity mask is not set up --- drivers/iommu/intel/irq_remapping.c | 69 +++++++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c index fa719936b44e..ac5f9e83943b 100644 --- a/drivers/iommu/intel/irq_remapping.c +++ b/drivers/iommu/intel/irq_remapping.c @@ -19,6 +19,7 @@ #include #include #include +#include #include "iommu.h" #include "../irq_remapping.h" @@ -49,6 +50,7 @@ struct irq_2_iommu { u16 sub_handle; u8 irte_mask; enum irq_mode mode; + bool posted_msi; }; struct intel_ir_data { @@ -1118,6 +1120,14 @@ static void prepare_irte(struct irte *irte, int vector, unsigned int dest) irte->redir_hint = 1; } +static void prepare_irte_posted(struct irte *irte) +{ + memset(irte, 0, sizeof(*irte)); + + irte->present = 1; + irte->p_pst = 1; +} + struct irq_remap_ops intel_irq_remap_ops = { .prepare = intel_prepare_irq_remapping, .enable = intel_enable_irq_remapping, @@ -1126,6 +1136,47 @@ struct irq_remap_ops intel_irq_remap_ops = { .enable_faulting = enable_drhd_fault_handling, }; +#ifdef CONFIG_X86_POSTED_MSI + +static phys_addr_t get_pi_desc_addr(struct irq_data *irqd) +{ + int cpu = cpumask_first(irq_data_get_effective_affinity_mask(irqd)); + + if (WARN_ON(cpu >= nr_cpu_ids)) + return 0; + + return __pa(per_cpu_ptr(&posted_interrupt_desc, cpu)); +} + +static void intel_ir_reconfigure_irte_posted(struct irq_data *irqd) +{ + struct intel_ir_data *ir_data = irqd->chip_data; + struct irte *irte = &ir_data->irte_entry; + struct irte irte_pi; + u64 pid_addr; + + pid_addr = get_pi_desc_addr(irqd); + + if (!pid_addr) { + pr_warn("Failed to setup IRQ %d for posted mode", irqd->irq); + return; + } + + memset(&irte_pi, 0, sizeof(irte_pi)); + + /* The shared IRTE already be set up as posted during alloc_irte */ + dmar_copy_shared_irte(&irte_pi, irte); + + irte_pi.pda_l = (pid_addr >> (32 - PDA_LOW_BIT)) & ~(-1UL << PDA_LOW_BIT); + irte_pi.pda_h = (pid_addr >> 32) & ~(-1UL << PDA_HIGH_BIT); + + modify_irte(&ir_data->irq_2_iommu, &irte_pi); +} + +#else +static inline void intel_ir_reconfigure_irte_posted(struct irq_data *irqd) {} +#endif + static void intel_ir_reconfigure_irte(struct irq_data *irqd, bool force) { struct intel_ir_data *ir_data = irqd->chip_data; @@ -1139,8 +1190,9 @@ static void intel_ir_reconfigure_irte(struct irq_data *irqd, bool force) irte->vector = cfg->vector; irte->dest_id = IRTE_DEST(cfg->dest_apicid); - /* Update the hardware only if the interrupt is in remapped mode. */ - if (force || ir_data->irq_2_iommu.mode == IRQ_REMAPPING) + if (ir_data->irq_2_iommu.posted_msi) + intel_ir_reconfigure_irte_posted(irqd); + else if (force || ir_data->irq_2_iommu.mode == IRQ_REMAPPING) modify_irte(&ir_data->irq_2_iommu, irte); } @@ -1194,7 +1246,7 @@ static int intel_ir_set_vcpu_affinity(struct irq_data *data, void *info) struct intel_ir_data *ir_data = data->chip_data; struct vcpu_data *vcpu_pi_info = info; - /* stop posting interrupts, back to remapping mode */ + /* stop posting interrupts, back to the default mode */ if (!vcpu_pi_info) { modify_irte(&ir_data->irq_2_iommu, &ir_data->irte_entry); } else { @@ -1320,6 +1372,11 @@ static void intel_irq_remapping_prepare_irte(struct intel_ir_data *data, break; case X86_IRQ_ALLOC_TYPE_PCI_MSI: case X86_IRQ_ALLOC_TYPE_PCI_MSIX: + if (posted_msi_supported()) { + prepare_irte_posted(irte); + data->irq_2_iommu.posted_msi = 1; + } + set_msi_sid(irte, pci_real_dma_dev(msi_desc_to_pci_dev(info->desc))); break; @@ -1407,7 +1464,11 @@ static int intel_irq_remapping_alloc(struct irq_domain *domain, irq_data->hwirq = (index << 16) + i; irq_data->chip_data = ird; - irq_data->chip = &intel_ir_chip; + if (posted_msi_supported() && + ((info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI) || (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSIX))) + irq_data->chip = &intel_ir_chip_post_msi; + else + irq_data->chip = &intel_ir_chip; intel_irq_remapping_prepare_irte(ird, irq_cfg, info, index, i); irq_set_status_flags(virq + i, IRQ_MOVE_PCNTXT); }