From patchwork Thu Apr 6 03:57:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13202844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC2B4C76196 for ; Thu, 6 Apr 2023 03:58:04 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.518702.805488 (Exim 4.92) (envelope-from ) id 1pkGky-0008Te-Tb; Thu, 06 Apr 2023 03:57:52 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 518702.805488; Thu, 06 Apr 2023 03:57:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGky-0008TX-QJ; Thu, 06 Apr 2023 03:57:52 +0000 Received: by outflank-mailman (input) for mailman id 518702; Thu, 06 Apr 2023 03:57:52 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGkx-0008TM-OO for xen-devel@lists.xenproject.org; Thu, 06 Apr 2023 03:57:52 +0000 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 3244fcb9-d42f-11ed-85db-49a42c6b2330; Thu, 06 Apr 2023 05:57:49 +0200 (CEST) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id D583A5C016C; Wed, 5 Apr 2023 23:57:48 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 05 Apr 2023 23:57:48 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 5 Apr 2023 23:57:46 -0400 (EDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3244fcb9-d42f-11ed-85db-49a42c6b2330 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1680753468; x=1680839868; bh=Fv CSUD7PDGeiswNRgMg7zZom3rqpEr1RXA0kWfFNpvs=; b=IRBTl0BZo62ArcDldO x5wTInwPltu0N3kqFHPdwIr4rnKyHDkUHIoZsFCs1JoZbbnNdcERcr/YpwchvPRm ByRBlIHB32C0sspNmaoMEbN56XIOQ3i9aITpYQCvwqtWCExxvzS58TNapl/wrx8R JlL6z1op0nqFBlPhBfrO3ea7pUp4J5sCL13lAR2hrHXmXYO3sSyG/xoZds59CFpl GhUeiXLMKM1XHpe58mSAuAKS1DXelBO4Y1+rBMciD4TQHJ8PkJx5QBAgWhDurx84 ABvSzBULj8RGLoMdsBTs5QtFssgrHS3g8tBVqdtXuYdqvOHro9kYDknMnt5lTN3L AOtw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680753468; x=1680839868; bh=FvCSUD7PDGeiswNRgMg7zZom3rqpEr1RXA0 kWfFNpvs=; b=YYVk8sH4fMz6Huu3gzMg7puPpyyK0TDy0doCcaG+0OeGjI5O2BI HZtmjOT5fLWlfxIfFTK5DeUVDljFlaNts38BYk5T4ToxgLQvh85iETLm5L9jbcWf Sf4lcPOiH24srqsVZinLiIfYXvopyqiCh16eqZIelbMd+HtuonoSu3QgKSzGMv1p f2f/gKvZNguuMn80W0F5Uwsh2HbuxbilI33fZJb1LnhepLH9oTcPJxkcLCGJS2x4 w1a3A2DezelWLI4nX7qmIDX5IopCH3o6I4gcB816eB2B1oUsZeuXFdyR+tFalPPQ o1DXE1ScJi8chnueuNYJoOmpMavDbpr3ADw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdejvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghr vghkucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesih hnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucggtffrrghtthgvrhhnpefg ueduhefgvdefheehudejheefudevueeghfekhfehleegveduteeuiedugffgffenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmrghrmhgrrhgv khesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhm X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Jan Beulich , Andrew Cooper , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu , George Dunlap , Julien Grall , Stefano Stabellini , Paul Durrant Subject: [PATCH v3 1/4] x86/msi: passthrough all MSI-X vector ctrl writes to device model Date: Thu, 6 Apr 2023 05:57:23 +0200 Message-Id: X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 QEMU needs to know whether clearing maskbit of a vector is really clearing, or was already cleared before. Currently Xen sends only clearing that bit to the device model, but not setting it, so QEMU cannot detect it. Because of that, QEMU is working this around by checking via /dev/mem, but that isn't the proper approach. It's just a workaround which in fact is racy. Give all necessary information to QEMU by passing all ctrl writes, including masking a vector. While this commit doesn't move the whole maskbit handling to QEMU (as discussed on xen-devel as one of the possibilities), it is a necessary first step anyway. Including telling QEMU it will get all the required information to do so. The actual implementation would need to include: - a hypercall for QEMU to control just maskbit (without (re)binding the interrupt again - a methor for QEMU to tell Xen it will actually do the work Those are not part of this series. Signed-off-by: Marek Marczykowski-Górecki --- v3: - advertise changed behavior in XEN_DMOP_get_ioreq_server_info - make "flags" parameter IN/OUT - move len check back to msixtbl_write() - will be needed there anyway in a later patch v2: - passthrough quad writes to emulator too (Jan) - (ab)use len==0 for write len=4 completion (Jan), but add descriptive #define for this magic value Should flags on output include only "out" values (current version), or also include those passed in by the caller unchanged? --- xen/arch/x86/hvm/vmsi.c | 18 ++++++++++++++---- xen/common/ioreq.c | 9 +++++++-- xen/include/public/hvm/dm_op.h | 12 ++++++++---- 3 files changed, 29 insertions(+), 10 deletions(-) diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index 3cd4923060c8..231253a2cbd4 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -272,6 +272,15 @@ out: return r; } +/* + * This function returns X86EMUL_UNHANDLEABLE even if write is properly + * handled, to propagate it to the device model (so it can keep its internal + * state in sync). + * len==0 means really len==4, but as a write completion that will return + * X86EMUL_OKAY on successful processing. Use WRITE_LEN4_COMPLETION to make it + * less confusing. + */ +#define WRITE_LEN4_COMPLETION 0 static int msixtbl_write(struct vcpu *v, unsigned long address, unsigned int len, unsigned long val) { @@ -283,8 +292,9 @@ static int msixtbl_write(struct vcpu *v, unsigned long address, unsigned long flags; struct irq_desc *desc; - if ( (len != 4 && len != 8) || (address & (len - 1)) ) - return r; + if ( (len != 4 && len != 8 && len != WRITE_LEN4_COMPLETION) || + (len && (address & (len - 1))) ) + return X86EMUL_UNHANDLEABLE; rcu_read_lock(&msixtbl_rcu_lock); @@ -345,7 +355,7 @@ static int msixtbl_write(struct vcpu *v, unsigned long address, unlock: spin_unlock_irqrestore(&desc->lock, flags); - if ( len == 4 ) + if ( len == WRITE_LEN4_COMPLETION ) r = X86EMUL_OKAY; out: @@ -635,7 +645,7 @@ void msix_write_completion(struct vcpu *v) return; v->arch.hvm.hvm_io.msix_unmask_address = 0; - if ( msixtbl_write(v, ctrl_address, 4, 0) != X86EMUL_OKAY ) + if ( msixtbl_write(v, ctrl_address, WRITE_LEN4_COMPLETION, 0) != X86EMUL_OKAY ) gdprintk(XENLOG_WARNING, "MSI-X write completion failure\n"); } diff --git a/xen/common/ioreq.c b/xen/common/ioreq.c index ecb8f545e1c4..bd6f074c1e85 100644 --- a/xen/common/ioreq.c +++ b/xen/common/ioreq.c @@ -743,7 +743,8 @@ static int ioreq_server_destroy(struct domain *d, ioservid_t id) static int ioreq_server_get_info(struct domain *d, ioservid_t id, unsigned long *ioreq_gfn, unsigned long *bufioreq_gfn, - evtchn_port_t *bufioreq_port) + evtchn_port_t *bufioreq_port, + uint16_t *flags) { struct ioreq_server *s; int rc; @@ -779,6 +780,9 @@ static int ioreq_server_get_info(struct domain *d, ioservid_t id, *bufioreq_port = s->bufioreq_evtchn; } + /* Advertise supported features/behaviors. */ + *flags = XEN_DMOP_all_msix_writes; + rc = 0; out: @@ -1374,7 +1378,8 @@ int ioreq_server_dm_op(struct xen_dm_op *op, struct domain *d, bool *const_op) NULL : (unsigned long *)&data->ioreq_gfn, (data->flags & XEN_DMOP_no_gfns) ? NULL : (unsigned long *)&data->bufioreq_gfn, - &data->bufioreq_port); + &data->bufioreq_port, &data->flags); + break; } diff --git a/xen/include/public/hvm/dm_op.h b/xen/include/public/hvm/dm_op.h index acdf91693d0b..490b151c5dd7 100644 --- a/xen/include/public/hvm/dm_op.h +++ b/xen/include/public/hvm/dm_op.h @@ -70,7 +70,9 @@ typedef struct xen_dm_op_create_ioreq_server xen_dm_op_create_ioreq_server_t; * not contain XEN_DMOP_no_gfns then these pages will be made available and * the frame numbers passed back in gfns and * respectively. (If the IOREQ Server is not handling buffered emulation - * only will be valid). + * only will be valid). When Xen returns XEN_DMOP_all_msix_writes + * flag set, it will notify the IOREQ server about all writes to MSI-X table + * (if it's handled by this IOREQ server), not only those clearing a mask bit. * * NOTE: To access the synchronous ioreq structures and buffered ioreq * ring, it is preferable to use the XENMEM_acquire_resource memory @@ -81,11 +83,13 @@ typedef struct xen_dm_op_create_ioreq_server xen_dm_op_create_ioreq_server_t; struct xen_dm_op_get_ioreq_server_info { /* IN - server id */ ioservid_t id; - /* IN - flags */ + /* IN/OUT - flags */ uint16_t flags; -#define _XEN_DMOP_no_gfns 0 -#define XEN_DMOP_no_gfns (1u << _XEN_DMOP_no_gfns) +#define _XEN_DMOP_no_gfns 0 /* IN */ +#define _XEN_DMOP_all_msix_writes 1 /* OUT */ +#define XEN_DMOP_no_gfns (1u << _XEN_DMOP_no_gfns) +#define XEN_DMOP_all_msix_writes (1u << _XEN_DMOP_all_msix_writes) /* OUT - buffered ioreq port */ evtchn_port_t bufioreq_port; From patchwork Thu Apr 6 03:57:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13202845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2F4E2C761A6 for ; Thu, 6 Apr 2023 03:58:06 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.518704.805508 (Exim 4.92) (envelope-from ) id 1pkGl5-0000au-CR; Thu, 06 Apr 2023 03:57:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 518704.805508; Thu, 06 Apr 2023 03:57:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGl5-0000am-8G; Thu, 06 Apr 2023 03:57:59 +0000 Received: by outflank-mailman (input) for mailman id 518704; Thu, 06 Apr 2023 03:57:58 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGl4-0000Z1-Dt for xen-devel@lists.xenproject.org; Thu, 06 Apr 2023 03:57:58 +0000 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 3328af7d-d42f-11ed-b464-930f4c7d94ae; Thu, 06 Apr 2023 05:57:52 +0200 (CEST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 5D6E55C0176; Wed, 5 Apr 2023 23:57:50 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Wed, 05 Apr 2023 23:57:50 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 5 Apr 2023 23:57:49 -0400 (EDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3328af7d-d42f-11ed-b464-930f4c7d94ae DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1680753470; x=1680839870; bh=D/ T8tgnYwHUEAgL2AzdECT/ODhwEEHPrmRQMVyr6mbY=; b=hs/M75R9CPozgbKmPG ZW08qrDVQJHKNs+NsRXrC1+vytZM6C+uNBHlzFOiJzbRZcqkAUHkOubllB0BMo6J YDhRjpmig6EYPzKKpIpx084EJM1bj4LWL2YqwqZwIK67jcZKZkc51kOuPBBxnOnE kk4WjXG8cOeAg5aWO48M7w8TN6UVi8uQBwUeK8v/uXOXP1WoMOPq4spa04Zyi0PH S8gnP/f9UBIZx9Ryu25UD5ocXPi5rbALRYgFokBstGm5S4KvjriSpztjeBAo/MzI XO4rP2Pc00B0wpTx35O5DVf5/eIB3UMbDKmPr6KsCBSfQef6wexgdM2xRh+qptiE hKUQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680753470; x=1680839870; bh=D/T8tgnYwHUEAgL2AzdECT/ODhwEEHPrmRQ MVyr6mbY=; b=ktX93hm1t6Ke6dLk4j4QQ76cGOlhDm/ItUtXhq7etEB9b4c5AOQ TuzxDZceQlk6Qwcgn5hMJxR/pL1BpZnwpwDXKe1dJVFsUnanWwJ470j5fLoVijC0 ZIfFFPoZHXDy7Uy8H2YpDGYRmC4dktVGxdRTUYaSPAgsH42ljyKI3L7hpe/dzGv4 +pZNkNp+rAAPtGnp4BJU4mJrBCS4lRraZ37LBWNcQ0YOmGI0p0gWnyX6zAutbveo g7uH2l+31w9ldDheps02nXVGIgT+StN53MyXl5F5+Q7gpyp8M5CaTnlQzK25FZgF Lq0HCIUZiFVcPQr8Mk7H46u8DXBJEb7XHvg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdejvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghr vghkucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesih hnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucggtffrrghtthgvrhhnpefg ueduhefgvdefheehudejheefudevueeghfekhfehleegveduteeuiedugffgffenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmrghrmhgrrhgv khesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhm X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Wei Liu , Anthony PERARD , Juergen Gross Subject: [PATCH v3 2/4] tools/xendevicemodel: Introduce ..._get_ioreq_server_info_ext Date: Thu, 6 Apr 2023 05:57:24 +0200 Message-Id: <1f6dc87eebe5d1c27ae15ec8f5d8006e5aa1c36d.1680752649.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 Add xendevicemodel_get_ioreq_server_info_ext() which additionally returns output flags that XEN_DMOP_get_ioreq_server_info can now return. Do not change signature of existing xendevicemodel_get_ioreq_server_info() so existing users will not need to be changed. This advertises behavior change of "x86/msi: passthrough all MSI-X vector ctrl writes to device model" patch. Signed-off-by: Marek Marczykowski-Górecki --- v3: - new patch Should there be some HAVE_* #define in the header? Does this change require soname bump (I hope it doesn't...). --- tools/include/xendevicemodel.h | 23 +++++++++++++++++++++++ tools/libs/devicemodel/core.c | 16 ++++++++++++++-- 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/tools/include/xendevicemodel.h b/tools/include/xendevicemodel.h index 797e0c6b2961..77a99e670551 100644 --- a/tools/include/xendevicemodel.h +++ b/tools/include/xendevicemodel.h @@ -72,6 +72,29 @@ int xendevicemodel_get_ioreq_server_info( evtchn_port_t *bufioreq_port); /** + * This function retrieves the necessary information to allow an + * emulator to use an IOREQ Server, including feature flags. + * + * @parm dmod a handle to an open devicemodel interface. + * @parm domid the domain id to be serviced + * @parm id the IOREQ Server id. + * @parm ioreq_gfn pointer to a xen_pfn_t to receive the synchronous ioreq + * gfn. (May be NULL if not required) + * @parm bufioreq_gfn pointer to a xen_pfn_t to receive the buffered ioreq + * gfn. (May be NULL if not required) + * @parm bufioreq_port pointer to a evtchn_port_t to receive the buffered + * ioreq event channel. (May be NULL if not required) + * @parm flags pointer to receive flags bitmask, see hvm/dm_op.h for details. + * (May be NULL if not required) + * @return 0 on success, -1 on failure. + */ +int xendevicemodel_get_ioreq_server_info_ext( + xendevicemodel_handle *dmod, domid_t domid, ioservid_t id, + xen_pfn_t *ioreq_gfn, xen_pfn_t *bufioreq_gfn, + evtchn_port_t *bufioreq_port, + unsigned int *flags); + +/** * This function registers a range of memory or I/O ports for emulation. * * @parm dmod a handle to an open devicemodel interface. diff --git a/tools/libs/devicemodel/core.c b/tools/libs/devicemodel/core.c index 8e619eeb0a1f..337622e608c2 100644 --- a/tools/libs/devicemodel/core.c +++ b/tools/libs/devicemodel/core.c @@ -189,10 +189,10 @@ int xendevicemodel_create_ioreq_server( return 0; } -int xendevicemodel_get_ioreq_server_info( +int xendevicemodel_get_ioreq_server_info_ext( xendevicemodel_handle *dmod, domid_t domid, ioservid_t id, xen_pfn_t *ioreq_gfn, xen_pfn_t *bufioreq_gfn, - evtchn_port_t *bufioreq_port) + evtchn_port_t *bufioreq_port, unsigned int *flags) { struct xen_dm_op op; struct xen_dm_op_get_ioreq_server_info *data; @@ -226,9 +226,21 @@ int xendevicemodel_get_ioreq_server_info( if (bufioreq_port) *bufioreq_port = data->bufioreq_port; + if (flags) + *flags = data->flags; + return 0; } +int xendevicemodel_get_ioreq_server_info( + xendevicemodel_handle *dmod, domid_t domid, ioservid_t id, + xen_pfn_t *ioreq_gfn, xen_pfn_t *bufioreq_gfn, + evtchn_port_t *bufioreq_port) +{ + return xendevicemodel_get_ioreq_server_info_ext( + dmod, domid, id, ioreq_gfn, bufioreq_gfn, bufioreq_port, NULL); +} + int xendevicemodel_map_io_range_to_ioreq_server( xendevicemodel_handle *dmod, domid_t domid, ioservid_t id, int is_mmio, uint64_t start, uint64_t end) From patchwork Thu Apr 6 03:57:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13202847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 54AD0C76196 for ; Thu, 6 Apr 2023 03:58:08 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.518706.805522 (Exim 4.92) (envelope-from ) id 1pkGl7-0000vY-2n; Thu, 06 Apr 2023 03:58:01 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 518706.805522; Thu, 06 Apr 2023 03:58:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGl6-0000ue-Ur; Thu, 06 Apr 2023 03:58:00 +0000 Received: by outflank-mailman (input) for mailman id 518706; Thu, 06 Apr 2023 03:57:59 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGl5-0000Z1-3G for xen-devel@lists.xenproject.org; Thu, 06 Apr 2023 03:57:59 +0000 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 3428a853-d42f-11ed-b464-930f4c7d94ae; Thu, 06 Apr 2023 05:57:52 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 0D6C75C0178; Wed, 5 Apr 2023 23:57:52 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 05 Apr 2023 23:57:52 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 5 Apr 2023 23:57:50 -0400 (EDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3428a853-d42f-11ed-b464-930f4c7d94ae DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1680753472; x=1680839872; bh=+x aJZckXvX1k1Bf8G7KYguuLRWvLEosFkkfVW7RbnBc=; b=kJXc197cjVxU8s7j8L ClSe15MEgoDBr304tK+Dvx8Vu/B3o+r+ND+29+fBrtcmiZd9rtUNuUHNx7wvyNzi 928xmpQWPrY6hF5fJvKrWyokQFVbrhC0PqGQtwP4HCOg0WcnzX+yxtwhpMelDqxX 3yFPWKS2f5chuIamgCgfpqdZTXh/VW5l2/1N3bRkSHXf51coS2s8LoD9YnPqTZS8 GElwbf9qNNPIqiTfVIh+LEfUNkfBQ/3XqDoEm98MwINYOgiQYjBA1v7Eur9bX93f 5LKv6euFZx+xVin8gsVhG2QjJr4320TEdmRxXd7a1JMlT7PpAE3jwM80Pe1MHy7Z XeMg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680753472; x=1680839872; bh=+xaJZckXvX1k1Bf8G7KYguuLRWvLEosFkkf VW7RbnBc=; b=fBMwm7xs3ACD/y3xTi8IHfZCtBWI47c3FFWWbP+YTMJ3otiDb3W DIrsifgHeTM0BJ1ZshuUJ5H1GbmxliLMJLV2zIqOjopZjNWXOc/GVVE1ALLR4xmz a6lTn9xF1Pf3zem7szRxYS/7M2kvlfFpw98mhGRGGWCoyjH2DilDnq2Slj9nXWZE 5P7/giIpYJcyO3m0btql70vXmnfwgBUIKQ5KmqUai/atmtd0X1AWfSrGPOGB0Fsw QQ//Dk4ivg79EDCiN4oaJer8j1W3SpNtx+1U6TztpKjLvwZWcJ7PKvcmcj+4Q0aO aTpFCtzCJeQS9KrK1EIodxL+wJeo7MtHTIg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdejvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghr vghkucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesih hnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucggtffrrghtthgvrhhnpefg ueduhefgvdefheehudejheefudevueeghfekhfehleegveduteeuiedugffgffenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmrghrmhgrrhgv khesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhm X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Jan Beulich , Andrew Cooper , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu Subject: [PATCH v3 3/4] x86/hvm: Allow writes to registers on the same page as MSI-X table Date: Thu, 6 Apr 2023 05:57:25 +0200 Message-Id: <3a8f54cf631e0342b144935950c853d1884a7eac.1680752649.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 Some devices (notably Intel Wifi 6 AX210 card) keep auxiliary registers on the same page as MSI-X table. Device model (especially one in stubdomain) cannot really handle those, as direct writes to that page is refused (page is on the mmio_ro_ranges list). Instead, extend msixtbl_mmio_ops to handle such accesses too. Doing this, requires correlating write location with guest view of MSI-X table address. Since QEMU doesn't map MSI-X table to the guest, it requires msixtbl_entry->gtable, which is HVM-only. Similar feature for PV would need to be done separately. This will be also used to read Pending Bit Array, if it lives on the same page, making QEMU not needing /dev/mem access at all (especially helpful with lockdown enabled in dom0). If PBA lives on another page, QEMU will map it to the guest directly. If PBA lives on the same page, discard writes and log a message. Technically, writes outside of PBA could be allowed, but at this moment the precise location of PBA isn't saved, and also no known device abuses the spec in this way (at least yet). To access those registers, msixtbl_mmio_ops need the relevant page mapped. MSI handling already has infrastructure for that, using fixmap, so try to map first/last page of the MSI-X table (if necessary) and save their fixmap indexes. Note that msix_get_fixmap() does reference counting and reuses existing mapping, so just call it directly, even if the page was mapped before. Also, it uses a specific range of fixmap indexes which doesn't include 0, so use 0 as default ("not mapped") value - which simplifies code a bit. GCC gets confused about 'desc' variable: arch/x86/hvm/vmsi.c: In function ‘msixtbl_range’: arch/x86/hvm/vmsi.c:553:8: error: ‘desc’ may be used uninitialized [-Werror=maybe-uninitialized] 553 | if ( desc ) | ^ arch/x86/hvm/vmsi.c:537:28: note: ‘desc’ was declared here 537 | const struct msi_desc *desc; | ^~~~ It's conditional initialization is actually correct (in the case where it isn't initialized, function returns early), but to avoid build failure initialize it explicitly to NULL anyway. Signed-off-by: Marek Marczykowski-Górecki --- v3: - merge handling into msixtbl_mmio_ops - extend commit message v2: - adjust commit message - pass struct domain to msixtbl_page_handler_get_hwaddr() - reduce local variables used only once - log a warning if write is forbidden if MSI-X and PBA lives on the same page - do not passthrough unaligned accesses - handle accesses both before and after MSI-X table --- xen/arch/x86/hvm/vmsi.c | 199 ++++++++++++++++++++++++++++++++-- xen/arch/x86/include/asm/msi.h | 5 +- xen/arch/x86/msi.c | 38 ++++++- 3 files changed, 231 insertions(+), 11 deletions(-) diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c index 231253a2cbd4..6f49493d3f58 100644 --- a/xen/arch/x86/hvm/vmsi.c +++ b/xen/arch/x86/hvm/vmsi.c @@ -181,15 +181,21 @@ static bool msixtbl_initialised(const struct domain *d) } static struct msixtbl_entry *msixtbl_find_entry( - struct vcpu *v, unsigned long addr) + struct vcpu *v, unsigned long addr, bool same_page) { struct msixtbl_entry *entry; struct domain *d = v->domain; list_for_each_entry( entry, &d->arch.hvm.msixtbl_list, list ) + { + if ( same_page && + PFN_DOWN(addr) >= PFN_DOWN(entry->gtable) && + PFN_DOWN(addr) <= PFN_DOWN(entry->gtable + entry->table_len) ) + return entry; if ( addr >= entry->gtable && addr < entry->gtable + entry->table_len ) return entry; + } return NULL; } @@ -213,6 +219,144 @@ static struct msi_desc *msixtbl_addr_to_desc( return NULL; } +/* + * Returns: + * - UINT_MAX if no handling should be done + * - UINT_MAX-1 if write should be discarded + * - a fixmap idx to use for handling + */ +#define ADJACENT_DONT_HANDLE UINT_MAX +#define ADJACENT_DISCARD_WRITE (UINT_MAX - 1) +static unsigned int adjacent_handle( + const struct msixtbl_entry *entry, unsigned long addr, bool write) +{ + unsigned int adj_type; + + if ( !entry || !entry->pdev ) + return ADJACENT_DONT_HANDLE; + + if ( PFN_DOWN(addr) == PFN_DOWN(entry->gtable) && addr < entry->gtable ) + adj_type = ADJ_IDX_FIRST; + else if ( PFN_DOWN(addr) == PFN_DOWN(entry->gtable + entry->table_len - 1) && + addr >= entry->gtable + entry->table_len ) + adj_type = ADJ_IDX_LAST; + else + return ADJACENT_DONT_HANDLE; + + ASSERT(entry->pdev->msix); + + if ( !entry->pdev->msix->adj_access_table_idx[adj_type] ) + { + gprintk(XENLOG_WARNING, + "Page for adjacent(%d) MSI-X table access not initialized for %pp (addr %#lx, gtable %#lx\n", + adj_type, &entry->pdev->sbdf, addr, entry->gtable); + + return ADJACENT_DONT_HANDLE; + } + + /* If PBA lives on the same page too, discard writes. */ + if ( write && ( + (adj_type == ADJ_IDX_LAST && + entry->pdev->msix->table.last == entry->pdev->msix->pba.first) || + (adj_type == ADJ_IDX_FIRST && + entry->pdev->msix->table.first == entry->pdev->msix->pba.last)) ) + { + gprintk(XENLOG_WARNING, + "MSI-X table and PBA of %pp live on the same page, " + "writing to other registers there is not implemented\n", + &entry->pdev->sbdf); + return ADJACENT_DISCARD_WRITE; + } + + return entry->pdev->msix->adj_access_table_idx[adj_type]; +} + +static int adjacent_read( + unsigned int fixmap_idx, + uint64_t address, uint32_t len, uint64_t *pval) +{ + const void __iomem *hwaddr; + + *pval = ~0UL; + + if ( !IS_ALIGNED(address, len) ) + { + gdprintk(XENLOG_WARNING, + "Dropping unaligned read from MSI-X table page at %" PRIx64 "\n", + address); + return X86EMUL_OKAY; + } + + ASSERT(fixmap_idx != ADJACENT_DISCARD_WRITE); + + hwaddr = fix_to_virt(fixmap_idx) + PAGE_OFFSET(address); + + switch ( len ) + { + case 1: + *pval = readb(hwaddr); + break; + + case 2: + *pval = readw(hwaddr); + break; + + case 4: + *pval = readl(hwaddr); + break; + + case 8: + *pval = readq(hwaddr); + break; + + default: + ASSERT_UNREACHABLE(); + } + return X86EMUL_OKAY; +} + +static int adjacent_write( + unsigned int fixmap_idx, + uint64_t address, uint32_t len, uint64_t val) +{ + void __iomem *hwaddr; + + if ( !IS_ALIGNED(address, len) ) + { + gdprintk(XENLOG_WARNING, + "Dropping unaligned write to MSI-X table page at %" PRIx64 "\n", + address); + return X86EMUL_OKAY; + } + + if ( fixmap_idx == ADJACENT_DISCARD_WRITE ) + return X86EMUL_OKAY; + + hwaddr = fix_to_virt(fixmap_idx) + PAGE_OFFSET(address); + + switch ( len ) { + case 1: + writeb(val, hwaddr); + break; + + case 2: + writew(val, hwaddr); + break; + + case 4: + writel(val, hwaddr); + break; + + case 8: + writeq(val, hwaddr); + break; + + default: + ASSERT_UNREACHABLE(); + } + return X86EMUL_OKAY; +} + static int cf_check msixtbl_read( const struct hvm_io_handler *handler, uint64_t address, uint32_t len, uint64_t *pval) @@ -220,16 +364,27 @@ static int cf_check msixtbl_read( unsigned long offset; struct msixtbl_entry *entry; unsigned int nr_entry, index; + unsigned int adjacent_fixmap; int r = X86EMUL_UNHANDLEABLE; - if ( (len != 4 && len != 8) || (address & (len - 1)) ) + if ( !IS_ALIGNED(address, len) ) return r; rcu_read_lock(&msixtbl_rcu_lock); - - entry = msixtbl_find_entry(current, address); + entry = msixtbl_find_entry(current, address, true); if ( !entry ) goto out; + + adjacent_fixmap = adjacent_handle(entry, address, false); + if ( adjacent_fixmap != ADJACENT_DONT_HANDLE ) + { + r = adjacent_read(adjacent_fixmap, address, len, pval); + goto out; + } + + if ( len != 4 && len != 8 ) + goto out; + offset = address & (PCI_MSIX_ENTRY_SIZE - 1); if ( offset != PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET ) @@ -291,16 +446,29 @@ static int msixtbl_write(struct vcpu *v, unsigned long address, int r = X86EMUL_UNHANDLEABLE; unsigned long flags; struct irq_desc *desc; + unsigned int adjacent_fixmap; - if ( (len != 4 && len != 8 && len != WRITE_LEN4_COMPLETION) || - (len && (address & (len - 1))) ) - return X86EMUL_UNHANDLEABLE; + if ( len && !IS_ALIGNED(address, len) ) + return r; rcu_read_lock(&msixtbl_rcu_lock); - entry = msixtbl_find_entry(v, address); + entry = msixtbl_find_entry(v, address, true); if ( !entry ) goto out; + + if ( len != WRITE_LEN4_COMPLETION ) + { + adjacent_fixmap = adjacent_handle(entry, address, true); + if ( adjacent_fixmap != ADJACENT_DONT_HANDLE ) + { + r = adjacent_write(adjacent_fixmap, address, len, val); + goto out; + } + if ( len != 4 && len != 8 ) + goto out; + } + nr_entry = array_index_nospec(((address - entry->gtable) / PCI_MSIX_ENTRY_SIZE), MAX_MSIX_TABLE_ENTRIES); @@ -375,14 +543,23 @@ static bool cf_check msixtbl_range( { struct vcpu *curr = current; unsigned long addr = r->addr; - const struct msi_desc *desc; + struct msixtbl_entry *entry; + const struct msi_desc *desc = NULL; + unsigned int adjacent_fixmap; + ASSERT(r->type == IOREQ_TYPE_COPY); rcu_read_lock(&msixtbl_rcu_lock); - desc = msixtbl_addr_to_desc(msixtbl_find_entry(curr, addr), addr); + entry = msixtbl_find_entry(curr, addr, true); + adjacent_fixmap = adjacent_handle(entry, addr, false); + if ( adjacent_fixmap == ADJACENT_DONT_HANDLE ) + desc = msixtbl_addr_to_desc(entry, addr); rcu_read_unlock(&msixtbl_rcu_lock); + if ( adjacent_fixmap != ADJACENT_DONT_HANDLE ) + return 1; + if ( desc ) return 1; @@ -627,7 +804,7 @@ void msix_write_completion(struct vcpu *v) uint32_t data; rcu_read_lock(&msixtbl_rcu_lock); - desc = msixtbl_addr_to_desc(msixtbl_find_entry(v, snoop_addr), + desc = msixtbl_addr_to_desc(msixtbl_find_entry(v, snoop_addr, false), snoop_addr); rcu_read_unlock(&msixtbl_rcu_lock); diff --git a/xen/arch/x86/include/asm/msi.h b/xen/arch/x86/include/asm/msi.h index a53ade95c9ad..d13cf1c1f873 100644 --- a/xen/arch/x86/include/asm/msi.h +++ b/xen/arch/x86/include/asm/msi.h @@ -207,6 +207,10 @@ struct msg_address { PCI_MSIX_ENTRY_SIZE + \ (~PCI_MSIX_BIRMASK & (PAGE_SIZE - 1))) +/* indexes in adj_access_table_idx[] below */ +#define ADJ_IDX_FIRST 0 +#define ADJ_IDX_LAST 1 + struct arch_msix { unsigned int nr_entries, used_entries; struct { @@ -214,6 +218,7 @@ struct arch_msix { } table, pba; int table_refcnt[MAX_MSIX_TABLE_PAGES]; int table_idx[MAX_MSIX_TABLE_PAGES]; + int adj_access_table_idx[2]; spinlock_t table_lock; bool host_maskall, guest_maskall; domid_t warned; diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index d0bf63df1def..c216acbf0e5d 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -961,6 +961,34 @@ static int msix_capability_init(struct pci_dev *dev, domain_crash(d); /* XXX How to deal with existing mappings? */ } + + /* + * If the MSI-X table doesn't start at the page boundary, map the first page for + * passthrough accesses. + */ + if ( PAGE_OFFSET(table_paddr) ) + { + int idx = msix_get_fixmap(msix, table_paddr, table_paddr); + + if ( idx > 0 ) + msix->adj_access_table_idx[ADJ_IDX_FIRST] = idx; + else + gprintk(XENLOG_ERR, "Failed to map first MSI-X table page: %d\n", idx); + } + /* + * If the MSI-X table doesn't span full page(s), map the last page for + * passthrough accesses. + */ + if ( PAGE_OFFSET(table_paddr + msix->nr_entries * PCI_MSIX_ENTRY_SIZE) ) + { + uint64_t entry_paddr = table_paddr + msix->nr_entries * PCI_MSIX_ENTRY_SIZE; + int idx = msix_get_fixmap(msix, table_paddr, entry_paddr); + + if ( idx > 0 ) + msix->adj_access_table_idx[ADJ_IDX_LAST] = idx; + else + gprintk(XENLOG_ERR, "Failed to map last MSI-X table page: %d\n", idx); + } } WARN_ON(msix->table.first != (table_paddr >> PAGE_SHIFT)); ++msix->used_entries; @@ -1090,6 +1118,16 @@ static void _pci_cleanup_msix(struct arch_msix *msix) WARN(); msix->table.first = 0; msix->table.last = 0; + if ( msix->adj_access_table_idx[ADJ_IDX_FIRST] ) + { + msix_put_fixmap(msix, msix->adj_access_table_idx[ADJ_IDX_FIRST]); + msix->adj_access_table_idx[ADJ_IDX_FIRST] = 0; + } + if ( msix->adj_access_table_idx[ADJ_IDX_LAST] ) + { + msix_put_fixmap(msix, msix->adj_access_table_idx[ADJ_IDX_LAST]); + msix->adj_access_table_idx[ADJ_IDX_LAST] = 0; + } if ( rangeset_remove_range(mmio_ro_ranges, msix->pba.first, msix->pba.last) ) From patchwork Thu Apr 6 03:57:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 13202846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A54EC77B71 for ; Thu, 6 Apr 2023 03:58:07 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.518705.805518 (Exim 4.92) (envelope-from ) id 1pkGl6-0000ra-P4; Thu, 06 Apr 2023 03:58:00 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 518705.805518; Thu, 06 Apr 2023 03:58:00 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGl6-0000rP-LT; Thu, 06 Apr 2023 03:58:00 +0000 Received: by outflank-mailman (input) for mailman id 518705; Thu, 06 Apr 2023 03:57:58 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pkGl4-0000Z1-OO for xen-devel@lists.xenproject.org; Thu, 06 Apr 2023 03:57:58 +0000 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 350fda72-d42f-11ed-b464-930f4c7d94ae; Thu, 06 Apr 2023 05:57:54 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 8C1395C0165; Wed, 5 Apr 2023 23:57:53 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 05 Apr 2023 23:57:53 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 5 Apr 2023 23:57:52 -0400 (EDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 350fda72-d42f-11ed-b464-930f4c7d94ae DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1680753473; x=1680839873; bh=jp uAaW3PocLxTjAz2gz/L9CxKTd3JuRdv2jiYB1F7uE=; b=CMyh8xUt2/XeuWuWly QYaK3x37WG6/AWwaIZouaj79YATV2yngaZrUzknYh5QowZWBFh8jqXeuLVObvZsp EZE0DMqwRYJUkzmwYTClsVB1dg4iirEZfdyvq0mHZfCz7LNXg5CSauXqCHeUWhiV YE4lVs3VIouNSDtQawOyHbvr4APMvSpJXF/IafpEAFG7rFTsoiaODXyGRBT4wj1Z ghEhB/i5rGNoRC3hDHGh8QtTA/k51E0jqx7U0R2eLnzNYm5NHSdlkPABbEIBU1Nd eUV0aDGEbVyIBDpsZfYMzeCDa0ZA2cpKi5K/Hah2cDHEUdniq446RRdL1xeb2pGY iTpg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1680753473; x=1680839873; bh=jpuAaW3PocLxTjAz2gz/L9CxKTd3JuRdv2j iYB1F7uE=; b=U4DzV7PsTmqf1zaY1JwFd6kf2wa5MT3kKgH82/AHh332No6k4/d QPIvFWtiX3FUUpUkvM4ILD60sx/zRjWLPJPdcmNz0+FJW942jK6mi087jjplNb+C SJWxzeY8D6mGriqv6gLVPe+WYVG1nsckhKVUdvY89Rp3DGD82V5AZxvQyiNPABoF r3ITXR9HRxVWutswrFJA2h85BTCQoFbku9uLbETBbXxv5TPbD3ozYjWAbuu0onx6 p3qskz8xSy93PsRGgHXcEwQOOSpJNTjC/iAxbHzfKfZRJiQS7oz8d2XwXS4gqJ4c XGWQhfK5yfh+4qubnwk0zJoFTPeD+M3PVPw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdejvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghr vghkucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesih hnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucggtffrrghtthgvrhhnpefg ueduhefgvdefheehudejheefudevueeghfekhfehleegveduteeuiedugffgffenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmrghrmhgrrhgv khesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhm X-ME-Proxy: Feedback-ID: i1568416f:Fastmail From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Cc: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Jason Andryuk , Jan Beulich , Paul Durrant , =?utf-8?q?Rog?= =?utf-8?q?er_Pau_Monn=C3=A9?= Subject: [PATCH v3 4/4] x86/msi: clear initial MSI-X state on boot Date: Thu, 6 Apr 2023 05:57:26 +0200 Message-Id: <6984a8571dac35d04c85117834d99b00fe1c4184.1680752649.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 Some firmware/devices are found to not reset MSI-X properly, leaving MASKALL set. Jason reports on his machine MASKALL persists through a warm reboot, but is cleared on cold boot. Xen relies on initial state being MASKALL clear. Especially, pci_reset_msix_state() assumes if MASKALL is set, it was Xen setting it due to msix->host_maskall or msix->guest_maskall. Clearing just MASKALL might be unsafe if ENABLE is set, so clear them both. Reported-by: Jason Andryuk Signed-off-by: Marek Marczykowski-Górecki Reviewed-by: Jan Beulich Tested-by: Jason Andryuk --- v3: - update comment - clear bits only when they were set --- xen/drivers/passthrough/msi.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/xen/drivers/passthrough/msi.c b/xen/drivers/passthrough/msi.c index ce1a450f6f4a..c9f7eac29ebf 100644 --- a/xen/drivers/passthrough/msi.c +++ b/xen/drivers/passthrough/msi.c @@ -46,6 +46,23 @@ int pdev_msi_init(struct pci_dev *pdev) spin_lock_init(&msix->table_lock); ctrl = pci_conf_read16(pdev->sbdf, msix_control_reg(pos)); + + if ( ctrl & (PCI_MSIX_FLAGS_MASKALL|PCI_MSIX_FLAGS_ENABLE) ) + { + /* + * pci_reset_msix_state() relies on MASKALL not being set + * initially, clear it (and ENABLE too - for safety), to meet that + * expectation. + */ + printk(XENLOG_WARNING + "%pp: unexpected initial MSI-X state (MASKALL=%d, ENABLE=%d), fixing\n", + &pdev->sbdf, + (ctrl & PCI_MSIX_FLAGS_MASKALL) ? 1 : 0, + (ctrl & PCI_MSIX_FLAGS_ENABLE) ? 1 : 0); + ctrl &= ~(PCI_MSIX_FLAGS_ENABLE|PCI_MSIX_FLAGS_MASKALL); + pci_conf_write16(pdev->sbdf, msix_control_reg(pos), ctrl); + } + msix->nr_entries = msix_table_size(ctrl); pdev->msix = msix;