From patchwork Fri May 17 17:06:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stewart Hildebrand X-Patchwork-Id: 13667188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 21202C04FFE for ; Fri, 17 May 2024 17:07:03 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.724355.1129592 (Exim 4.92) (envelope-from ) id 1s812i-0005ts-4k; Fri, 17 May 2024 17:06:52 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 724355.1129592; Fri, 17 May 2024 17:06:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s812i-0005tf-24; Fri, 17 May 2024 17:06:52 +0000 Received: by outflank-mailman (input) for mailman id 724355; Fri, 17 May 2024 17:06:51 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s812h-0005cV-AO for xen-devel@lists.xenproject.org; Fri, 17 May 2024 17:06:51 +0000 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on20600.outbound.protection.outlook.com [2a01:111:f403:200a::600]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id d964181a-146f-11ef-909e-e314d9c70b13; Fri, 17 May 2024 19:06:50 +0200 (CEST) Received: from CH2PR14CA0060.namprd14.prod.outlook.com (2603:10b6:610:56::40) by DM3PR12MB9326.namprd12.prod.outlook.com (2603:10b6:0:3d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.30; Fri, 17 May 2024 17:06:44 +0000 Received: from CH1PEPF0000A346.namprd04.prod.outlook.com (2603:10b6:610:56:cafe::27) by CH2PR14CA0060.outlook.office365.com (2603:10b6:610:56::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.28 via Frontend Transport; Fri, 17 May 2024 17:06:44 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000A346.mail.protection.outlook.com (10.167.244.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7587.21 via Frontend Transport; Fri, 17 May 2024 17:06:43 +0000 Received: from SATLEXMB06.amd.com (10.181.40.147) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:06:43 -0500 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB06.amd.com (10.181.40.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:06:43 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.35 via Frontend Transport; Fri, 17 May 2024 12:06:37 -0500 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d964181a-146f-11ef-909e-e314d9c70b13 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P01g5uxPvru8qmcnL+KhXnJeJOJPzjku6XF3xFGF+Ophw8Wh2fGi5ZsaY6wMzAv4KW6LTsgu4sqpio53l18dbuFzRDYmxgMzBRbdxdzwEb9DCy2g7A/2B10N6gbfJKvJxVN7GpiS+4PFNXNJ8KSu+1jewe4/zPSGm27Wr4Q3Qs3VOn3IEmmu714XillsA8Q1fGLNdHvTGC9+5cI5rOQjf9pDah8wpS8F+nUmRggiMLrXMspuhBUwOaOqO+VZgkvDp4WmJ4yqmF81HTAJr5k4tgvKu72C1v9tNCfEGDJGM2VUyOaKzp5cC84CklMhMoh+yjrDOhUGeWtznNFIhA/u8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=k8+/xJpZRZsK6+RKr2c/38/ePPkO2ljT1mguCOagC4g=; b=lCkUIbMsxmPmVPwA8WQfZHMciSU1fQJRq3/I+G5ti8Z77a0tbNPCFdLkwt5zi2ps31jV8yXrgrm1T0lCobVyYzGFK2cdN7xIA47UVlr/nR4cr8U1dyR5PyEHew50zlyP/0N9X3lC8cge15Q/gpPY1wV1mAAscHN7uCrBHXrmEnXuDDiWpd8hf6fuYJlov4yXI9uBOj+fk1dlQJJ1vprLO7v1q4q6wPBtH3K4Hq3B6iFBxiH8NPoq/ff0N2Q9268HWqpNUc7e59cfhkuI9iGDZnJov/JJNm+5Fs+rDneCl0iOe5kXkBUYRmR7RFc+An3UzXcPOSd+c3wXUTyPf8GHwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k8+/xJpZRZsK6+RKr2c/38/ePPkO2ljT1mguCOagC4g=; b=p9EvjbHziTFxIK4yn069lzHVP/bSaJOUT945UxW09zLG8uHMFsPz/ttdU19+wEVtZm6jw7TS6b0AiD20pEOoJD5NIs7poLq4j8IiCxmiV30Y7cl40H26OqxWLRblxFLnxy9vCxO9ufXkc3DATtGn2xTEqoSiG2874Oi45jyuV+c= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Volodymyr Babchuk , Stewart Hildebrand , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk Subject: [PATCH v15 1/5] arm/vpci: honor access size when returning an error Date: Fri, 17 May 2024 13:06:11 -0400 Message-ID: <20240517170619.45088-2-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240517170619.45088-1-stewart.hildebrand@amd.com> References: <20240517170619.45088-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000A346:EE_|DM3PR12MB9326:EE_ X-MS-Office365-Filtering-Correlation-Id: ea2952d8-bd52-48b8-ce1b-08dc7693ba57 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|82310400017|36860700004|376005|1800799015; X-Microsoft-Antispam-Message-Info: nvzXThYk1gFpvov+RcY0hREqCRWQgPhOFrmAsPWXl3o0uLtyFP6GJmIlbtjZtukrzbVEA0eIF8OYcSp6L9hi6KaDNl4RsbxXe8sH780ZXmbKod6qrdhJHaWhsobOYwAl6yWaNoKNMJFjMHR2LINJmmd8JeDhw7lY4hSGjw+LUHW0N2so6gw+wHniDCEUniD+5obrGSpOB7Vk+42Mc0F/mqZy1PdDPr7r+slQrcKpYNtLON4QDLS2SBQ5t+ih2T4PCalCpGUzeg/tqnk2q2IkOwYtCVMwJimHzU0OgR2944/c3E2q3zTgK0OnzFNIheBHD3Ely3V0umsG7kfIMViDp9wM4DtEunLQwhDIPjaU1uf71Vb0kxd5531IjP4yFXh2/OTUQ7tCTeNzuME2QzI/gv67CPYIbq6A2FWIoR8UkaAGo+UYXdCHgDoDm9z7DJU+dME76TapgQq6wvfzjyBTH8OWQexMGT5XZXNMQxhJIfAsPoH+kawSiQCefQZ0KCEyvc1YuGINDdiEzDy/Q/+YpG/c2geeMsNExf+oqvcDWlNezAkR4s21L9jqg8ntohH/vhVrX5CIuGDkJh4zp6e/Ey4YrV4jBgzxku1WE5m0DCxNIXQCltt8mtConmJJXcNZm7q6OMAH+mihPoeO1gCRK781H9tyAzw9PrsXvepZWdrZKyvnXHH2FXNHI3OB0QIyRpZ8hFulstgKtAVhUhIgiY5VjxfVkHkbpWCg1Nty9t4tD8HfEF/+AeEOoBuSV6gkDxdp853LH2qNh9vsAwv12xpc71TXHOAYKvWyJj8yW6xvhm+8PEjisFZeY/qdek6lTimAZq5xqefJ99Oyz+fXybUrGOew1y3E9JnpHFZn2lsoy6O/3XB42RxlaszuvH0wiHE5U1ll9O8jpcPRUo4Txw3w26wX1lnRos4H6b6FKiDVFaqiNzOhsVAKHyQcLiqt6PhpXQ1bGiozPZYNzAj5py42S1SUOyPDCcLgqUtIQX4AarOHKcYV62AHdy8NUiEeJzbmlENAdYSUlK/lhOaqmVi+1VcwMxFP7pyF0wn6HG6fIJBJ7Cz8svlzALoJMJ7sWDXlUh0RsLb6BeeWw1d3K0516X4jaAWDnqCmB3aInJ9mCrOZTvl8Iqgf1y7H5W/jCxAmATpA86ye0nwhc+Kt9Nbo+ppS+z7MOWbdz6m9PQyFPtoxL3E9wa23v01USnvK7BdULkl5lO6Y3qfF4sgxkzPByDwqJ+qr/wTDoBUcqj2wFLQuTaZBevCqw+GpIgjS9QXUK8HvYwufGR2T+K8uzTOrZb+crAwXSMz73sjGRWSTfg6m+bksv82Y600H+D/XL2XtXsJFrt3A5igXSwAJ0APGH0cDmMHtzPOsyIaQQ4FSAkw7GB/GGbFTpzVjhthX X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(82310400017)(36860700004)(376005)(1800799015);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2024 17:06:43.7729 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ea2952d8-bd52-48b8-ce1b-08dc7693ba57 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000A346.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM3PR12MB9326 From: Volodymyr Babchuk Guest can try to read config space using different access sizes: 8, 16, 32, 64 bits. We need to take this into account when we are returning an error back to MMIO handler, otherwise it is possible to provide more data than requested: i.e. guest issues LDRB instruction to read one byte, but we are writing 0xFFFFFFFFFFFFFFFF in the target register. Signed-off-by: Volodymyr Babchuk Signed-off-by: Stewart Hildebrand Acked-by: Julien Grall --- v14->v15: * re-order so this patch comes before ("xen/arm: translate virtual PCI bus topology for guests") * s/access_mask/invalid/ * add U suffix to 1 * s/uint8_t/unsigned int/ * s/uint64_t/register_t/ * although Julien gave an Acked-by on v14, I omitted it due to the changes made in v15 v9->10: * New patch in v10. --- xen/arch/arm/vpci.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 3bc4bb55082a..b63a356bb4a8 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -29,6 +29,8 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, { struct pci_host_bridge *bridge = p; pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + const unsigned int access_size = (1U << info->dabt.size) * 8; + const register_t invalid = GENMASK_ULL(access_size - 1, 0); /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; @@ -39,7 +41,7 @@ static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, return 1; } - *r = ~0ul; + *r = invalid; return 0; } From patchwork Fri May 17 17:06:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stewart Hildebrand X-Patchwork-Id: 13667189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6FD73C25B79 for ; Fri, 17 May 2024 17:07:09 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.724356.1129603 (Exim 4.92) (envelope-from ) id 1s812p-0006DK-Fd; Fri, 17 May 2024 17:06:59 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 724356.1129603; Fri, 17 May 2024 17:06:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s812p-0006DB-B7; Fri, 17 May 2024 17:06:59 +0000 Received: by outflank-mailman (input) for mailman id 724356; Fri, 17 May 2024 17:06:58 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s812n-0006C6-Rw for xen-devel@lists.xenproject.org; Fri, 17 May 2024 17:06:57 +0000 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on20600.outbound.protection.outlook.com [2a01:111:f403:200a::600]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id dc2c6de2-146f-11ef-b4bb-af5377834399; Fri, 17 May 2024 19:06:55 +0200 (CEST) Received: from CH0PR03CA0197.namprd03.prod.outlook.com (2603:10b6:610:e4::22) by SA0PR12MB4430.namprd12.prod.outlook.com (2603:10b6:806:70::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.30; Fri, 17 May 2024 17:06:51 +0000 Received: from CH2PEPF00000099.namprd02.prod.outlook.com (2603:10b6:610:e4:cafe::af) by CH0PR03CA0197.outlook.office365.com (2603:10b6:610:e4::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.30 via Frontend Transport; Fri, 17 May 2024 17:06:51 +0000 Received: from SATLEXMB03.amd.com (165.204.84.17) by CH2PEPF00000099.mail.protection.outlook.com (10.167.244.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7587.21 via Frontend Transport; Fri, 17 May 2024 17:06:51 +0000 Received: from SATLEXMB06.amd.com (10.181.40.147) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:06:50 -0500 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB06.amd.com (10.181.40.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:06:50 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.35 via Frontend Transport; Fri, 17 May 2024 12:06:49 -0500 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: dc2c6de2-146f-11ef-b4bb-af5377834399 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dcbDXjw5RSNAYliHXmmL/eqO5RfXY9IbNuSe8qlKjtSoYq0y2fsiMvDOojx7O5RcPSeoD/wg8vmrA1qelY6AfgAevW8i/uaUXPK/GmYsSoYRwm6R+XH/mwKb/naLoBT74UKvcnKn8bd+fmSB97dpUXzHOzAT1nCgjY6ycnRTUYBoKzPFUMlx4mOJuzii4EYVlzw/z9ULkKHtYNuEQKoqbPk9Ne1gSQhgQXRGyRzb/fK39JAejuGTgcSC/JBsJOSuRhI82SIbgTWK2nWSmu5beSIVDXvrcpJoeTJ/eYbbBT5xC7SuDv0zX+Izhk8oiGanVfd810lAe6MMINgDgkU2Fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hUlhftlOCNqSzK4Q7JMuj42oNyZSiQtLeJCRs2LCTBA=; b=MWhKriMgg5Qwmjoe+zPyTTvDpvDlNxCfqaa2qjOvmA1oiDesjxZtLGttVbLQJaqN6SQ0j8YfSAnElcWv9FqYYPw4JjiSU2sRZ2C81fzF3dxUWzXMYsTqYcoaPZWry/h3qB8zHaNU2BUrvhU1DI6OJ5KCw/2zWQBTZ+A77fKCBTgo0odLAZ7wcPLeJegc88uFXSvp78Mm6ANfv2vC7gwcue7j+yBIiqF9aInTDefdxPXazATiKlt46RLllbmuxt7s37k1UJLAHKPS2jq/WfSQ0WuuMrjlUgBolPe/qGpyDnb48poCK6qnBnUX2uJGHit8r1iA0/74xapPvNUKaSs+yg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hUlhftlOCNqSzK4Q7JMuj42oNyZSiQtLeJCRs2LCTBA=; b=R42IKjJRy4Duzqziy8v6jDlUawoVM9bh6g7XCK0KKlBju54DUDbmNGuiFTklTqU3psmC+MubvW0d5ZSrIbcCZ3iiq31nAkn6BADEh2VZ5sktVgsb4x1jas3dYj5Q/uBf5aRahAbdhWS1nRqUHj9mVyTLmvauXz+as0jBOvuWwgQ= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C From: Stewart Hildebrand To: CC: Oleksandr Andrushchenko , =?utf-8?q?Ro?= =?utf-8?q?ger_Pau_Monn=C3=A9?= , "Stewart Hildebrand" , Andrew Cooper , George Dunlap , "Jan Beulich" , Julien Grall , "Stefano Stabellini" , Volodymyr Babchuk Subject: [PATCH v15 2/5] vpci/header: emulate PCI_COMMAND register for guests Date: Fri, 17 May 2024 13:06:12 -0400 Message-ID: <20240517170619.45088-3-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240517170619.45088-1-stewart.hildebrand@amd.com> References: <20240517170619.45088-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF00000099:EE_|SA0PR12MB4430:EE_ X-MS-Office365-Filtering-Correlation-Id: cc549463-26a4-4999-a970-08dc7693bee8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|36860700004|1800799015|82310400017|376005; X-Microsoft-Antispam-Message-Info: 0rRgFlFW7quJWwFnx8/5onH2wBWgvczo2wc3R79RSU+m/X2pMZsXWpfNpTMQ+zKKahfxiwL90yV0nm8TPegP2Hl2Q2fNYWdfPpYXkAPLXyx1D5+cl/8ysMSWxLOzPRHrnwM2719vdv1RiQXrsssfZXxGfYBR5ZCQjdyhrFt3eplOFhl19upE1EyFqUwfT+4l+o/uzNxM9IqRHv/t93cKcnt7SwXcMBwMZmbAZ2+B8smcXHf91FhVK0rNpcs/9/AiehMby1FPuUpfSizo32YRS8o7TWlo2d3xI0FhEM2m+9de0gGEzfIPmJFjwoVkTCvn7HHDVUyf7/u7bv7h7jSbtxv2Rdls/dJ0gHN4cHLWsv0KjPPvFS/SYTqTamK3BD/V56MmMT/dR93dnNRnUisyrVUyPN7oiRpnowZ5YJW0TK7mbTqvEmgTQjag+Ly8saEI40DnrPnoDwmCtgux99oR2wtz6l7UjIIxReHzuxPQYM+iPssrmfit1PivByOWuLqn3jiIzadFkFyJIc44rGWVbaZbfvheGX6JA4iYTwIr3tBJmrb1k+stZsvQW056BpWgZjs4wyH/xqBpKtPLUFN+KSqek+l5vcMhZhDQ9ph8t6gJKwjkRLSRdPUGtywuSoqExz/2O6RK8IQJqeMDqX4ojs7WOEfAy99Ac28sxcCsWdag0XBI06epNFtATXlPC1rwuqVISTUmhoCBP6oibAcKmSm5ADM0S3ygLVe1TI0S96EkSeCQvCJckUGeHlI4E59ymoh2MYwWpsm5LiQ/se9GhtP11jB0m9JgaKPjHOkUm4fDYAXZy3bokWvqclsjMykIjvvEB2ZR1g7LuqZe5Aq0LhtIv4mqTivBgsFKTHDiOoWAvwtW1dyaf6IICxaURIgs+HV0GOtoxx0g9j7q5EJx1LdDUtpke/MG9z1pnF8xITfVBsqykYSCl2B8NSNwwqFMhz384Ebx8srC+RZYtCDs4o5XlPAPp1UL890IIcgI6TPXFLlb1eiwUIWspumqWvmstMYde7/0zSQZ6KxYq3Rllsn4MUuGWrX8pA5KgOvc+/UkTw4ej0BfeonipTSDBIjSSN9XO7Rhi8zk7cu0yK76k9gJib557lAQBpTUfejc/r6F73fmlWE3Akb41n+ewzE5renkL7+wdQXBd8kaXdDjiZbmVDA/ZBL4EKy0Y29OHZ6O5Z9tPUppmBQIwhgdvabKPvzoFS+utXDr04W8zlb39/NN1jfuCWJlUo3Av0biqpemnAldOgz6pLUkmJzmipWeMWLF2kILty9AAhAoT7b+z7l4ph5+IGcca0EbH/TXBiQS3NzsBJcPZZtBj4ZmYaw0 X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(36860700004)(1800799015)(82310400017)(376005);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2024 17:06:51.4331 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cc549463-26a4-4999-a970-08dc7693bee8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF00000099.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4430 From: Oleksandr Andrushchenko Xen and/or Dom0 may have put values in PCI_COMMAND which they expect to remain unaltered. PCI_COMMAND_SERR bit is a good example: while the guest's (domU) view of this will want to be zero (for now), the host having set it to 1 should be preserved, or else we'd effectively be giving the domU control of the bit. Thus, PCI_COMMAND register needs proper emulation in order to honor host's settings. According to "PCI LOCAL BUS SPECIFICATION, REV. 3.0", section "6.2.2 Device Control" the reset state of the command register is typically 0, so when assigning a PCI device use 0 as the initial state for the guest's (domU) view of the command register. Here is the full list of command register bits with notes about PCI/PCIe specification, and how Xen handles the bit. QEMU's behavior is also documented here since that is our current reference implementation for PCI passthrough. PCI_COMMAND_IO (bit 0) PCIe 6.1: RW PCI LB 3.0: RW QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest writes do not propagate to hardware. QEMU sets this bit to 1 in hardware if an I/O BAR is exposed to the guest. Xen domU: (rsvdp_mask) We treat this bit as RsvdP for now since we don't yet support I/O BARs for domUs. Xen dom0: We allow dom0 to control this bit freely. PCI_COMMAND_MEMORY (bit 1) PCIe 6.1: RW PCI LB 3.0: RW QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest writes do not propagate to hardware. QEMU sets this bit to 1 in hardware if a Memory BAR is exposed to the guest. Xen domU/dom0: We handle writes to this bit by mapping/unmapping BAR regions. Xen domU: For devices assigned to DomUs, memory decoding will be disabled at the time of initialization. PCI_COMMAND_MASTER (bit 2) PCIe 6.1: RW PCI LB 3.0: RW QEMU: Pass through writes to hardware. Xen domU/dom0: Pass through writes to hardware. PCI_COMMAND_SPECIAL (bit 3) PCIe 6.1: RO, hardwire to 0 PCI LB 3.0: RW QEMU: Pass through writes to hardware. Xen domU/dom0: Pass through writes to hardware. PCI_COMMAND_INVALIDATE (bit 4) PCIe 6.1: RO, hardwire to 0 PCI LB 3.0: RW QEMU: Pass through writes to hardware. Xen domU/dom0: Pass through writes to hardware. PCI_COMMAND_VGA_PALETTE (bit 5) PCIe 6.1: RO, hardwire to 0 PCI LB 3.0: RW QEMU: Pass through writes to hardware. Xen domU/dom0: Pass through writes to hardware. PCI_COMMAND_PARITY (bit 6) PCIe 6.1: RW PCI LB 3.0: RW QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest writes do not propagate to hardware. Xen domU: (rsvdp_mask) We treat this bit as RsvdP. Xen dom0: We allow dom0 to control this bit freely. PCI_COMMAND_WAIT (bit 7) PCIe 6.1: RO, hardwire to 0 PCI LB 3.0: hardwire to 0 QEMU: res_mask Xen domU: (rsvdp_mask) We treat this bit as RsvdP. Xen dom0: We allow dom0 to control this bit freely. PCI_COMMAND_SERR (bit 8) PCIe 6.1: RW PCI LB 3.0: RW QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest writes do not propagate to hardware. Xen domU: (rsvdp_mask) We treat this bit as RsvdP. Xen dom0: We allow dom0 to control this bit freely. PCI_COMMAND_FAST_BACK (bit 9) PCIe 6.1: RO, hardwire to 0 PCI LB 3.0: RW QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest writes do not propagate to hardware. Xen domU: (rsvdp_mask) We treat this bit as RsvdP. Xen dom0: We allow dom0 to control this bit freely. PCI_COMMAND_INTX_DISABLE (bit 10) PCIe 6.1: RW PCI LB 3.0: RW QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest writes do not propagate to hardware. QEMU checks if INTx was mapped for a device. If it is not, then guest can't control PCI_COMMAND_INTX_DISABLE bit. Xen domU: We prohibit a guest from enabling INTx if MSI(X) is enabled. Xen dom0: We allow dom0 to control this bit freely. Bits 11-15 PCIe 6.1: RsvdP PCI LB 3.0: Reserved QEMU: res_mask Xen domU/dom0: rsvdp_mask Signed-off-by: Oleksandr Andrushchenko Signed-off-by: Volodymyr Babchuk Signed-off-by: Stewart Hildebrand Reviewed-by: Jan Beulich Reviewed-by: Roger Pau Monné --- RFC: There is an unaddressed question for Roger: should we update the guest view of the PCI_COMMAND_INTX_DISABLE bit in msi.c/msix.c:control_write()? See prior discussion at [1]. In my opinion, I think we should make sure that hardware state and the guest view are consistent (i.e. don't lie to the guest). [1] https://lore.kernel.org/xen-devel/86b25777-788c-4b9a-8166-a6f8174bedc9@suse.com/ In v15: - add Jan's R-b - add blank line after declaration in msi.c:control_write() In v14: - check for 0->1 transition in INTX_DISABLE-setting logic in msi.c:control_write() to match msix.c:control_write() - clear domU-controllable bits in header.c:init_header() In v13: - Update right away (don't defer) PCI_COMMAND_MEMORY bit in guest_cmd variable in cmd_write() - Make comment single line in xen/drivers/vpci/msi.c:control_write() - Rearrange memory decoding disabling snippet in init_header() In v12: - Rework patch using vpci_add_register_mask() - Add bitmask #define in pci_regs.h according to PCIe 6.1 spec, except don't add the RO bits because they were RW in PCI LB 3.0 spec. - Move and expand TODO comment about properly emulating bits - Update guest_cmd in msi.c/msix.c:control_write() - Simplify cmd_write(), thanks to rsvdp_mask - Update commit description In v11: - Fix copy-paste mistake: vpci->msi should be vpci->msix - Handle PCI_COMMAND_IO - Fix condition for disabling INTx in the MSI-X code - Show domU changes to only allowed bits - Show PCI_COMMAND_MEMORY write only after P2M was altered - Update comments in the code In v10: - Added cf_check attribute to guest_cmd_read - Removed warning about non-zero cmd - Updated comment MSI code regarding disabling INTX - Used ternary operator in vpci_add_register() call - Disable memory decoding for DomUs in init_bars() In v9: - Reworked guest_cmd_read - Added handling for more bits Since v6: - fold guest's logic into cmd_write - implement cmd_read, so we can report emulated INTx state to guests - introduce header->guest_cmd to hold the emulated state of the PCI_COMMAND register for guests Since v5: - add additional check for MSI-X enabled while altering INTX bit - make sure INTx disabled while guests enable MSI/MSI-X Since v3: - gate more code on CONFIG_HAS_MSI - removed logic for the case when MSI/MSI-X not enabled --- xen/drivers/vpci/header.c | 60 ++++++++++++++++++++++++++++++++++---- xen/drivers/vpci/msi.c | 9 ++++++ xen/drivers/vpci/msix.c | 7 +++++ xen/include/xen/pci_regs.h | 1 + xen/include/xen/vpci.h | 3 ++ 5 files changed, 74 insertions(+), 6 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 47648c395132..2491dbae8901 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -524,9 +524,21 @@ static void cf_check cmd_write( { struct vpci_header *header = data; + if ( !is_hardware_domain(pdev->domain) ) + { + const struct vpci *vpci = pdev->vpci; + + if ( (vpci->msi && vpci->msi->enabled) || + (vpci->msix && vpci->msix->enabled) ) + cmd |= PCI_COMMAND_INTX_DISABLE; + + header->guest_cmd = cmd; + } + /* * Let Dom0 play with all the bits directly except for the memory - * decoding one. + * decoding one. Bits that are not allowed for DomU are already + * handled above and by the rsvdp_mask. */ if ( header->bars_mapped != !!(cmd & PCI_COMMAND_MEMORY) ) /* @@ -540,6 +552,14 @@ static void cf_check cmd_write( pci_conf_write16(pdev->sbdf, reg, cmd); } +static uint32_t cf_check guest_cmd_read( + const struct pci_dev *pdev, unsigned int reg, void *data) +{ + const struct vpci_header *header = data; + + return header->guest_cmd; +} + static void cf_check bar_write( const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -754,9 +774,23 @@ static int cf_check init_header(struct pci_dev *pdev) return -EOPNOTSUPP; } - /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); + /* + * Setup a handler for the command register. + * + * TODO: If support for emulated bits is added, re-visit how to handle + * PCI_COMMAND_PARITY, PCI_COMMAND_SERR, and PCI_COMMAND_FAST_BACK. + */ + rc = vpci_add_register_mask(pdev->vpci, + is_hwdom ? vpci_hw_read16 : guest_cmd_read, + cmd_write, PCI_COMMAND, 2, header, 0, 0, + PCI_COMMAND_RSVDP_MASK | + (is_hwdom ? 0 + : PCI_COMMAND_IO | + PCI_COMMAND_PARITY | + PCI_COMMAND_WAIT | + PCI_COMMAND_SERR | + PCI_COMMAND_FAST_BACK), + 0); if ( rc ) return rc; @@ -836,9 +870,23 @@ static int cf_check init_header(struct pci_dev *pdev) if ( pdev->ignore_bars ) return 0; - /* Disable memory decoding before sizing. */ cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); - if ( cmd & PCI_COMMAND_MEMORY ) + + /* + * For DomUs, clear PCI_COMMAND_{MASTER,MEMORY,IO} and other + * DomU-controllable bits in PCI_COMMAND. Devices assigned to DomUs will + * start with memory decoding disabled, and modify_bars() will not be called + * at the end of this function. + */ + if ( !is_hwdom ) + cmd &= ~(PCI_COMMAND_VGA_PALETTE | PCI_COMMAND_INVALIDATE | + PCI_COMMAND_SPECIAL | PCI_COMMAND_MASTER | PCI_COMMAND_MEMORY | + PCI_COMMAND_IO); + + header->guest_cmd = cmd; + + /* Disable memory decoding before sizing. */ + if ( !is_hwdom || (cmd & PCI_COMMAND_MEMORY) ) pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd & ~PCI_COMMAND_MEMORY); for ( i = 0; i < num_bars; i++ ) diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index 30adcf7df05d..dd6620ec5674 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -57,6 +57,8 @@ static void cf_check control_write( if ( new_enabled ) { + bool old_enabled = msi->enabled; + /* * If the device is already enabled it means the number of * enabled messages has changed. Disable and re-enable the @@ -70,6 +72,13 @@ static void cf_check control_write( if ( vpci_msi_arch_enable(msi, pdev, vectors) ) return; + + /* Make sure domU doesn't enable INTx while enabling MSI. */ + if ( !old_enabled && !is_hardware_domain(pdev->domain) ) + { + pci_intx(pdev, false); + pdev->vpci->header.guest_cmd |= PCI_COMMAND_INTX_DISABLE; + } } else vpci_msi_arch_disable(msi, pdev); diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 58c16ebdf283..fbe710ab92ef 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -135,6 +135,13 @@ static void cf_check control_write( } } + /* Make sure domU doesn't enable INTx while enabling MSI-X. */ + if ( new_enabled && !msix->enabled && !is_hardware_domain(pdev->domain) ) + { + pci_intx(pdev, false); + pdev->vpci->header.guest_cmd |= PCI_COMMAND_INTX_DISABLE; + } + msix->masked = new_masked; msix->enabled = new_enabled; diff --git a/xen/include/xen/pci_regs.h b/xen/include/xen/pci_regs.h index 0bc18efabb74..250ba106dbd3 100644 --- a/xen/include/xen/pci_regs.h +++ b/xen/include/xen/pci_regs.h @@ -48,6 +48,7 @@ #define PCI_COMMAND_SERR 0x100 /* Enable SERR */ #define PCI_COMMAND_FAST_BACK 0x200 /* Enable back-to-back writes */ #define PCI_COMMAND_INTX_DISABLE 0x400 /* INTx Emulation Disable */ +#define PCI_COMMAND_RSVDP_MASK 0xf800 #define PCI_STATUS 0x06 /* 16 bits */ #define PCI_STATUS_IMM_READY 0x01 /* Immediate Readiness */ diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 6e4c972f35ed..2064d9354f5b 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -107,6 +107,9 @@ struct vpci { } bars[PCI_HEADER_NORMAL_NR_BARS + 1]; /* At most 6 BARS + 1 expansion ROM BAR. */ + /* Guest (domU only) view of the PCI_COMMAND register. */ + uint16_t guest_cmd; + /* * Store whether the ROM enable bit is set (doesn't imply ROM BAR * is mapped into guest p2m) if there's a ROM BAR on the device. From patchwork Fri May 17 17:06:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stewart Hildebrand X-Patchwork-Id: 13667191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5A145C04FFE for ; Fri, 17 May 2024 17:07:31 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.724364.1129623 (Exim 4.92) (envelope-from ) id 1s813C-0007Le-4f; Fri, 17 May 2024 17:07:22 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 724364.1129623; Fri, 17 May 2024 17:07:22 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s813C-0007LT-1A; Fri, 17 May 2024 17:07:22 +0000 Received: by outflank-mailman (input) for mailman id 724364; Fri, 17 May 2024 17:07:20 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s813A-0006C6-OQ for xen-devel@lists.xenproject.org; Fri, 17 May 2024 17:07:20 +0000 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on20600.outbound.protection.outlook.com [2a01:111:f403:2412::600]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id ea53e26b-146f-11ef-b4bb-af5377834399; Fri, 17 May 2024 19:07:19 +0200 (CEST) Received: from DM6PR13CA0061.namprd13.prod.outlook.com (2603:10b6:5:134::38) by PH0PR12MB7887.namprd12.prod.outlook.com (2603:10b6:510:26d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.28; Fri, 17 May 2024 17:07:13 +0000 Received: from CH1PEPF0000A349.namprd04.prod.outlook.com (2603:10b6:5:134:cafe::d3) by DM6PR13CA0061.outlook.office365.com (2603:10b6:5:134::38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7611.13 via Frontend Transport; Fri, 17 May 2024 17:07:13 +0000 Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000A349.mail.protection.outlook.com (10.167.244.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7587.21 via Frontend Transport; Fri, 17 May 2024 17:07:12 +0000 Received: from SATLEXMB05.amd.com (10.181.40.146) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:06:59 -0500 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB05.amd.com (10.181.40.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:06:58 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.35 via Frontend Transport; Fri, 17 May 2024 12:06:57 -0500 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ea53e26b-146f-11ef-b4bb-af5377834399 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FXlIt6aRLa/lQKTfM+WWsVar9wDtckkBisIw8/U4Arw0/P7gsnMk+eY6LuuouvOQc3nrUENeMy8QWbhF3BQnGHV7qeSuB6FALovQF70pnPgjkmTJY1Uhkk/4XT/6oTUmJ+p03Lm9EGHRQA0Lrt05E8RaRUgSx9hRDKfMDvFeQDlSH1O8DQVKjDqpK8Qvh45NJBM7ekphsKSwoNGHlj/u9kVRNx+lkZW67QHK0hmgEjAJd7oQ4SaqiJjy21xIE+/cIsaxm+yxOAFY01pl3zdcYnhEpytBECtJ5YWqYIlK3D+sKVz/XLYY+0tw1kLUTNxDSWDxNW9s+rYtYqQu8rAEQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bv7+nDlnnhjXbpYGAA9ivNDf2bKqLCBADTr28OUk9Vg=; b=aETuViYGb5htggsb9glyy2sZzlELADBNsd26btEK+QcbgRUYQ9UOK9HHyLlOW0s3xzazArgUcn279N4aeK1dOZCwS9Xel8QLfegtMJ7LOrAVT9GsFDvCJL/XlMKzRnqv4Zt2LDtB7IOWacfuiM9D1OWM7bdo8RQSK7YCT3hJJQGJ+pTSLp0lOtPSKkMdEkAZbmbivPq/5UjwUhtsYAL53TwBP0loOc3Kudy1PD64PQ8xZeN8UoSSUBUgsfWnjKpD5BnQzDhM6zJxpQuf59b0aRhQVjrkeTZ7O8/CkcAUmMS5/fjZmMroo5BYLTeDUF69IEyR3mvRwJ/2IyjZ7mBsgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bv7+nDlnnhjXbpYGAA9ivNDf2bKqLCBADTr28OUk9Vg=; b=AJ5xnYte03j+kOZ4Is6aWcOEyWtVqVhYOxflDYeXctbkZKM+CU7lnsQMJ262LiZhV5ldrO4+80kkvSWZkzais8O3eL3dqNyjDA6btvFATPnj0iuJq03HUJcvKlNe0Iv2S5H2P2LrYZowul/E3uuUVz1aSYm+oAM3ejJRPnngT+M= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C From: Stewart Hildebrand To: CC: Oleksandr Andrushchenko , Andrew Cooper , George Dunlap , "Jan Beulich" , Julien Grall , "Stefano Stabellini" , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Stewart Hildebrand , Volodymyr Babchuk Subject: [PATCH v15 3/5] vpci: add initial support for virtual PCI bus topology Date: Fri, 17 May 2024 13:06:13 -0400 Message-ID: <20240517170619.45088-4-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240517170619.45088-1-stewart.hildebrand@amd.com> References: <20240517170619.45088-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Received-SPF: None (SATLEXMB05.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000A349:EE_|PH0PR12MB7887:EE_ X-MS-Office365-Filtering-Correlation-Id: 472dc026-7d09-452b-9f67-08dc7693cb8a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|1800799015|82310400017|376005|36860700004; X-Microsoft-Antispam-Message-Info: /i+5S5SGy2bG6UpvQvWOsdrdytOEUQj/mqE+G1xy11UV6daVf3zfRrNg9qzfo7N/UVYNaMao2QpONcgAhgcP8AVfztautVjXy9Kl65EIUpIci7Pew3SjtxgxaKG9LMmyE9toE7COZUf9UDp/gGgVNABJV8vmz9Ji28Vg1a1uFDh/PTmEEIarijYyR1V9TDMOg+CL3e51w1x7Dwbcp4zp3Zul5YyNzX5qGVseQ/8ulb+JjiYlMSGy/i+WLqNLg3I15L/r3JG+C/uQIS9eCAO7JL2CLUe3bKQ8/Cdr1kRjKBzatG+4X61Zdw6VNoi9/Gv46xaGUk7Cn1q/ACJdYsgI+uS7sQAXWkXYi0VW2SjkVx/O4fndl6+JeC2+CYCippRk5wXfjRrGJFJE3l8Ns5+5Zza8AjjQ4s4kVbWmgprBgHqpDCssdi52Veq72sJFu1HyY7nQyig+uvpCLoIqUoK/QHEOfV/NIOM05qHqgb8iaHn0fRwNbDuCAJrA28yQ5h66YgepwM/84pswG5/SrXTCFF4Jn99n36lsfTPXqWonCIfYnX/48gFq7m25TSaHf3BQYNPudIq0XJY8qz+Eq/bfnGyye3RVfjMCnaOdPriIhD6iSQldTOnhJ0EeN2EoQP+PODQBWqAt/KDly/8L2dRTQw3rab3uanExQFuEHIB2hpSNsFcsBObRL/Kx6SG/Dp93F1nUJZ/LUAgMr5v3XwFU4hnPWhIiSuQczQA+t2AKnGvclLSYzwy8Pa09y3bKlQmTp8u0S5Y3R4RFDPAi06gAelGBrb10iwenSpsMXyfi0H7guJI71ivHN+LECEs5UUKBibdl3kDGFa6o90i2uPaO4HO2jIyRERoP+NiT1IvVJ7AgOAedYvy/uHqiF8LzkTrG+b4uc5MV4SpO2IvRlqQKNTa7vnGb9FPJ25e5CUc5BauayLF256LrHNhiUb+IPDgNM/JwU+MhwVybQp7qn0LfStz0AyLouIomBWnVbmQqfoAiDJ7ApY8seKS3fEqJ4D9mRKa8smVPErQKtrC5WUc1NT2wZsviFHkgShUihpUYtLZmgCNitwGwbI0/dw6x1M6bvvVi1p22l+oghRjVv4fLt7c8MupoxlrXHRBi+RWwA4KZtae1DdA+B2bft58yiptwdOr1d4I7h3rn4T1eGFaKddHvKR1JFyOSuJEa+Zf4qKOxsHLvV9zy2BjoEOeHruL1Lyh3N/teBlyBD/BRy2ddYRHjWrB9Zf+Zy2+8i+v3SdBzRJr1DPVooJ8FO7NAoL+rCVs/V2qoRKctAB7Qv24nP8NPNN3ElMu9X9Y92+vI3V6ZY7JKvBQ9D6BXrWDe0Ve/MTuhtn7QXcVIHYrH/AfDyzRzPi5NQOqLRjd4thDA/og= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(1800799015)(82310400017)(376005)(36860700004);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2024 17:07:12.6272 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 472dc026-7d09-452b-9f67-08dc7693cb8a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000A349.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7887 From: Oleksandr Andrushchenko Assign SBDF to the PCI devices being passed through with bus 0. The resulting topology is where PCIe devices reside on the bus 0 of the root complex itself (embedded endpoints). This implementation is limited to 32 devices which are allowed on a single PCI bus. Please note, that at the moment only function 0 of a multifunction device can be passed through. Signed-off-by: Oleksandr Andrushchenko Signed-off-by: Volodymyr Babchuk Signed-off-by: Stewart Hildebrand Acked-by: Jan Beulich --- In v15: - add Jan's A-b In v13: - s/depends on/select/ in Kconfig - check pdev->sbdf.fn instead of two booleans in add_virtual_device() - comment #endifs in sched.h - clarify comment about limits in vpci.h with seg/bus limit In v11: - Fixed code formatting - Removed bogus write_unlock() call - Fixed type for new_dev_number In v10: - Removed ASSERT(pcidevs_locked()) - Removed redundant code (local sbdf variable, clearing sbdf during device removal, etc) - Added __maybe_unused attribute to "out:" label - Introduced HAS_VPCI_GUEST_SUPPORT Kconfig option, as this is the first patch where it is used (previously was in "vpci: add hooks for PCI device assign/de-assign") In v9: - Lock in add_virtual_device() replaced with ASSERT (thanks, Stewart) In v8: - Added write lock in add_virtual_device Since v6: - re-work wrt new locking scheme - OT: add ASSERT(pcidevs_write_locked()); to add_virtual_device() Since v5: - s/vpci_add_virtual_device/add_virtual_device and make it static - call add_virtual_device from vpci_assign_device and do not use REGISTER_VPCI_INIT machinery - add pcidevs_locked ASSERT - use DECLARE_BITMAP for vpci_dev_assigned_map Since v4: - moved and re-worked guest sbdf initializers - s/set_bit/__set_bit - s/clear_bit/__clear_bit - minor comment fix s/Virtual/Guest/ - added VPCI_MAX_VIRT_DEV constant (PCI_SLOT(~0) + 1) which will be used later for counting the number of MMIO handlers required for a guest (Julien) Since v3: - make use of VPCI_INIT - moved all new code to vpci.c which belongs to it - changed open-coded 31 to PCI_SLOT(~0) - added comments and code to reject multifunction devices with functions other than 0 - updated comment about vpci_dev_next and made it unsigned int - implement roll back in case of error while assigning/deassigning devices - s/dom%pd/%pd Since v2: - remove casts that are (a) malformed and (b) unnecessary - add new line for better readability - remove CONFIG_HAS_VPCI_GUEST_SUPPORT ifdef's as the relevant vPCI functions are now completely gated with this config - gate common code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/drivers/Kconfig | 4 +++ xen/drivers/vpci/vpci.c | 57 +++++++++++++++++++++++++++++++++++++++++ xen/include/xen/sched.h | 10 +++++++- xen/include/xen/vpci.h | 12 +++++++++ 4 files changed, 82 insertions(+), 1 deletion(-) diff --git a/xen/drivers/Kconfig b/xen/drivers/Kconfig index db94393f47a6..20050e9bb8b3 100644 --- a/xen/drivers/Kconfig +++ b/xen/drivers/Kconfig @@ -15,4 +15,8 @@ source "drivers/video/Kconfig" config HAS_VPCI bool +config HAS_VPCI_GUEST_SUPPORT + bool + select HAS_VPCI + endmenu diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 97e115dc5798..23722634d50b 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -40,6 +40,49 @@ extern vpci_register_init_t *const __start_vpci_array[]; extern vpci_register_init_t *const __end_vpci_array[]; #define NUM_VPCI_INIT (__end_vpci_array - __start_vpci_array) +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +static int add_virtual_device(struct pci_dev *pdev) +{ + struct domain *d = pdev->domain; + unsigned int new_dev_number; + + if ( is_hardware_domain(d) ) + return 0; + + ASSERT(rw_is_write_locked(&pdev->domain->pci_lock)); + + /* + * Each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + */ + if ( pdev->sbdf.fn ) + { + gdprintk(XENLOG_ERR, "%pp: only function 0 passthrough supported\n", + &pdev->sbdf); + return -EOPNOTSUPP; + } + new_dev_number = find_first_zero_bit(d->vpci_dev_assigned_map, + VPCI_MAX_VIRT_DEV); + if ( new_dev_number == VPCI_MAX_VIRT_DEV ) + return -ENOSPC; + + __set_bit(new_dev_number, &d->vpci_dev_assigned_map); + + /* + * Both segment and bus number are 0: + * - we emulate a single host bridge for the guest, e.g. segment 0 + * - with bus 0 the virtual devices are seen as embedded + * endpoints behind the root complex + * + * TODO: add support for multi-function devices. + */ + pdev->vpci->guest_sbdf = PCI_SBDF(0, 0, new_dev_number, 0); + + return 0; +} + +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ + void vpci_deassign_device(struct pci_dev *pdev) { unsigned int i; @@ -49,6 +92,12 @@ void vpci_deassign_device(struct pci_dev *pdev) if ( !has_vpci(pdev->domain) || !pdev->vpci ) return; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + if ( pdev->vpci->guest_sbdf.sbdf != ~0 ) + __clear_bit(pdev->vpci->guest_sbdf.dev, + &pdev->domain->vpci_dev_assigned_map); +#endif + spin_lock(&pdev->vpci->lock); while ( !list_empty(&pdev->vpci->handlers) ) { @@ -103,6 +152,13 @@ int vpci_assign_device(struct pci_dev *pdev) INIT_LIST_HEAD(&pdev->vpci->handlers); spin_lock_init(&pdev->vpci->lock); +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + pdev->vpci->guest_sbdf.sbdf = ~0; + rc = add_virtual_device(pdev); + if ( rc ) + goto out; +#endif + for ( i = 0; i < NUM_VPCI_INIT; i++ ) { rc = __start_vpci_array[i](pdev); @@ -110,6 +166,7 @@ int vpci_assign_device(struct pci_dev *pdev) break; } + out: __maybe_unused; if ( rc ) vpci_deassign_device(pdev); diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 132b84199598..2dcd1d1a4f8a 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -486,7 +486,15 @@ struct domain * 2. pdev->vpci->lock */ rwlock_t pci_lock; -#endif +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* + * The bitmap which shows which device numbers are already used by the + * virtual PCI bus topology and is used to assign a unique SBDF to the + * next passed through virtual PCI device. + */ + DECLARE_BITMAP(vpci_dev_assigned_map, VPCI_MAX_VIRT_DEV); +#endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ +#endif /* CONFIG_HAS_PCI */ #ifdef CONFIG_HAS_PASSTHROUGH struct domain_iommu iommu; diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 2064d9354f5b..980aded26fc1 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -21,6 +21,14 @@ typedef int vpci_register_init_t(struct pci_dev *dev); #define VPCI_ECAM_BDF(addr) (((addr) & 0x0ffff000) >> 12) +/* + * Maximum number of devices supported by the virtual bus topology: + * each PCI bus supports 32 devices/slots at max or up to 256 when + * there are multi-function ones which are not yet supported. + * This limit implies only segment 0, bus 0 is supported. + */ +#define VPCI_MAX_VIRT_DEV (PCI_SLOT(~0) + 1) + #define REGISTER_VPCI_INIT(x, p) \ static vpci_register_init_t *const x##_entry \ __used_section(".data.vpci." p) = (x) @@ -175,6 +183,10 @@ struct vpci { struct vpci_arch_msix_entry arch; } entries[]; } *msix; +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT + /* Guest SBDF of the device. */ + pci_sbdf_t guest_sbdf; +#endif #endif }; From patchwork Fri May 17 17:06:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stewart Hildebrand X-Patchwork-Id: 13667190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72F53C04FFE for ; Fri, 17 May 2024 17:07:28 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.724363.1129612 (Exim 4.92) (envelope-from ) id 1s813A-00075f-R4; Fri, 17 May 2024 17:07:20 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 724363.1129612; Fri, 17 May 2024 17:07:20 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s813A-00075U-OX; Fri, 17 May 2024 17:07:20 +0000 Received: by outflank-mailman (input) for mailman id 724363; Fri, 17 May 2024 17:07:19 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s8139-0005cV-Ez for xen-devel@lists.xenproject.org; Fri, 17 May 2024 17:07:19 +0000 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on20601.outbound.protection.outlook.com [2a01:111:f403:2416::601]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id ea290aca-146f-11ef-909e-e314d9c70b13; Fri, 17 May 2024 19:07:18 +0200 (CEST) Received: from CH0PR07CA0014.namprd07.prod.outlook.com (2603:10b6:610:32::19) by SJ0PR12MB6781.namprd12.prod.outlook.com (2603:10b6:a03:44b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.26; Fri, 17 May 2024 17:07:13 +0000 Received: from CH2PEPF000000A0.namprd02.prod.outlook.com (2603:10b6:610:32:cafe::d6) by CH0PR07CA0014.outlook.office365.com (2603:10b6:610:32::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.30 via Frontend Transport; Fri, 17 May 2024 17:07:13 +0000 Received: from SATLEXMB03.amd.com (165.204.84.17) by CH2PEPF000000A0.mail.protection.outlook.com (10.167.244.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7587.21 via Frontend Transport; Fri, 17 May 2024 17:07:12 +0000 Received: from SATLEXMB05.amd.com (10.181.40.146) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:07:12 -0500 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB05.amd.com (10.181.40.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:07:12 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.35 via Frontend Transport; Fri, 17 May 2024 12:07:05 -0500 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ea290aca-146f-11ef-909e-e314d9c70b13 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f4zeHa5+mYcNejHDgbSebek8KjYBpkQK2u5AlNgP+8Ippj2O3xRRv7FciQrlt4yX2LMXjJPAEh6DByroDztdhjLbCGwNsGMM4WgeTlO75rSg4TzqDGA3mJu+oGBf4+ipBaf8IgTk3HQkV6CDToaxQJk9iWeJAzE6z1OCyKUXkEzyZ6PW+MDigjWw9is/76yeX5LSXkwJcbKzVBhc33N9E/ve4ZAgiAdz3vS8M5yoATI439kNYgtUmxcMvN0K0LReAd1KfIiHCe0+UHiYMwzP7UhAvlp8E1USR+5nqtL6Ywt+RXlqO0+8B9mAEVC0DQL/FAKlbyVOVDCtUxZOYO9Rsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zUM6dVyovYpjq8KjZD0l2PS8q/TOLp/Wit92KUplaCw=; b=JN00j14+XaNx7V/ype4xKqhW6VbtqyqIYd9MOe6QeHdpZIIykewkc+Jljghvc/BV3sJ+wLbpxaYhhd0CGmgo2KEP3qF2Iezn5lAEBtcPA/GFvfHcZ+c3hYt11WsoLxzXgs4ye4OqpUjIP0c+CvMTnIORJEJmAVcj6N541Xt/2c/rx5O/f+Lp8qtQNPoCoBI0CULrvT0W59uFqhRZ/DYgq3t+DcnOosKz59ag9pwuSgg3Yq1UpvZPiERXDTWt8PSLM1WGmORW9Le2FVnaBE8+EF7P9fCnb6mdRsPsbv2KrO2rbo9ZLdCpMFtlrYc5/gIzBiWyHGHP7sB1uZ9bMlr36Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zUM6dVyovYpjq8KjZD0l2PS8q/TOLp/Wit92KUplaCw=; b=gnayb4iCZKRb2sZgjkEyZQ8C4m3nAFCZz01hI92Ijyx0b5yX5A9/lBK5/Z6Aa+vmU9bCWGVbiHmV2xoDeWwSEkisGlhjBnO+vX6zuuv5asU4AArpxceNv8Jnsii2CZ7QCGXpPiiprFzLGoICldpZLcaJlDBKlDbAalEinqA88c8= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C From: Stewart Hildebrand To: CC: Oleksandr Andrushchenko , "Stewart Hildebrand" , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , "Volodymyr Babchuk" , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= , Volodymyr Babchuk Subject: [PATCH v15 4/5] xen/arm: translate virtual PCI bus topology for guests Date: Fri, 17 May 2024 13:06:14 -0400 Message-ID: <20240517170619.45088-5-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240517170619.45088-1-stewart.hildebrand@amd.com> References: <20240517170619.45088-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Received-SPF: None (SATLEXMB05.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF000000A0:EE_|SJ0PR12MB6781:EE_ X-MS-Office365-Filtering-Correlation-Id: f5c0efbd-de52-42a2-50cc-08dc7693cba1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|82310400017|376005|36860700004|1800799015; X-Microsoft-Antispam-Message-Info: CTwG2Q13s/um5srL3kk8yIvX9sf9IhUslUJiU0HBvoAqJu3S25dnTFvbPvSOTTwCJznSa8zFuftEpNXI+n19tRjUs/8BeEFcsacOVp89LG8QRrAL87W/1xgavodKMFDV9HJ+ajQt675qVgdhAfnZE0Qh+oQ3awMpxOv8iwIi5VqS6vCL2atDiYgMGRx/Yd/t6PC5VYRtiWVprcQNV77yWwBRP+kHS0RvLFOHgZBK1isElQcZpxLBT+7F2J2zoxNFaAJBGAO4dfLRD3pl46cNkF+oqn2VK65jCCZOtffndTcl5ZYa5E6Ae64uQMIsf4mE+g+Zz7YUjvAGVmuOveYWEMUhOaDlDk+SRrxlA2ts+FW6Hq9zTTTfcaZ39N+7vs0pMU6pxWAZmGHZzGsB7DMBc1hZR7cLCLaClDlOdCjYZE5tRX62bkvV0oMJNZ/JR0lU3tN9vDURC/A6EMmAufVlXgafhIRGWIqiXqkZYMSyLlgqE63UL7LqItik2CcNshy/4ud+Hxflv+0eXhgx0WtlxFy/a7Q6R3aEXyzk8cxozLmtI3XGhoOQIc53CIBBPSolnIffxzuWd2DiaT2xTLd8AsIoW2jcfDL/ZIjTaDYnAZ5tsyudr83kgGZo+n3prhgguMeYSm2NMAn9HNxy1TIPQ/Gzfi0AXDWaMRj7gS06Gp/jm+c7JSsBMkwUucMZfW9yBsKvNa/8AC81jog4SAGAvMFe6asI9Z0yzhQd0LQLR+vuPxJo4BqixKy9blkDIXm0w44w4Ou08P6OODHR5jLEsOI54uy2ykJjYcI/Z024H/uo87kSqWvt2BalyRRRaNAHe9UX3Hz0RhXG/OWQ8yqWBEYMD4VHfpmbqSO+rOgntjaujbyG6uggG7s5RljCKP0mC5N/LbXQ2WSXoeYKfhtj+7bKqjt795E7Ftvnn/zTas6SwJS4k12qWszxMap5sfmhC9IBaPmaTs001GgMknX/nd676pD7+GokngWNqcGKcq3HUQrseMHIWj+n/97yKp4xe6bQlzoIsRexd6UGlRIV2CFKhWopJ4YUgdu9oL+me1i092eT7nEfv7Rs/JZnSx5x+Hh7zXgnkLZi9oPyhTaxfX4PkqK6un87/9WnZ+3maoZLi9aGoVjmoDS67mylUJELWMFX3WvXS76yYfz3nF2QJYjtRbLr+q4Oi3vlKyHh73nkdUKp8ESC0hTCctC/tSkgqF4q9vmJc4FdWwSKSYal0aVK8QfFe7YJPChu/gFzK9Dd7gUcWOYFrO3jGAxCiDiVmCtprBzPlF7Kbl6a6FNuf1DWhK69Ma7LraqHjfnnjXBTF8XgZhTiSD+xHzyrakZQ5tw4Uo5E+8BaCWSyV+ERkVCWGIsuQZSL1mnOfh51sqVr4Ws3mhficJ2CcJ+oxSb+ X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(82310400017)(376005)(36860700004)(1800799015);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2024 17:07:12.7759 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f5c0efbd-de52-42a2-50cc-08dc7693cba1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF000000A0.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6781 From: Oleksandr Andrushchenko There are three originators for the PCI configuration space access: 1. The domain that owns physical host bridge: MMIO handlers are there so we can update vPCI register handlers with the values written by the hardware domain, e.g. physical view of the registers vs guest's view on the configuration space. 2. Guest access to the passed through PCI devices: we need to properly map virtual bus topology to the physical one, e.g. pass the configuration space access to the corresponding physical devices. 3. Emulated host PCI bridge access. It doesn't exist in the physical topology, e.g. it can't be mapped to some physical host bridge. So, all access to the host bridge itself needs to be trapped and emulated. Signed-off-by: Oleksandr Andrushchenko Signed-off-by: Volodymyr Babchuk Signed-off-by: Stewart Hildebrand Acked-by: Julien Grall --- In v15: - base on top of ("arm/vpci: honor access size when returning an error") In v11: - Fixed format issues - Added ASSERT_UNREACHABLE() to the dummy implementation of vpci_translate_virtual_device() - Moved variable in vpci_sbdf_from_gpa(), now it is easier to follow the logic in the function Since v9: - Commend about required lock replaced with ASSERT() - Style fixes - call to vpci_translate_virtual_device folded into vpci_sbdf_from_gpa Since v8: - locks moved out of vpci_translate_virtual_device() Since v6: - add pcidevs locking to vpci_translate_virtual_device - update wrt to the new locking scheme Since v5: - add vpci_translate_virtual_device for #ifndef CONFIG_HAS_VPCI_GUEST_SUPPORT case to simplify ifdefery - add ASSERT(!is_hardware_domain(d)); to vpci_translate_virtual_device - reset output register on failed virtual SBDF translation Since v4: - indentation fixes - constify struct domain - updated commit message - updates to the new locking scheme (pdev->vpci_lock) Since v3: - revisit locking - move code to vpci.c Since v2: - pass struct domain instead of struct vcpu - constify arguments where possible - gate relevant code with CONFIG_HAS_VPCI_GUEST_SUPPORT New in v2 --- xen/arch/arm/vpci.c | 45 ++++++++++++++++++++++++++++++++--------- xen/drivers/vpci/vpci.c | 24 ++++++++++++++++++++++ xen/include/xen/vpci.h | 12 +++++++++++ 3 files changed, 71 insertions(+), 10 deletions(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index b63a356bb4a8..516933bebfb3 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -7,33 +7,53 @@ #include -static pci_sbdf_t vpci_sbdf_from_gpa(const struct pci_host_bridge *bridge, - paddr_t gpa) +static bool vpci_sbdf_from_gpa(struct domain *d, + const struct pci_host_bridge *bridge, + paddr_t gpa, pci_sbdf_t *sbdf) { - pci_sbdf_t sbdf; + bool translated = true; + + ASSERT(sbdf); if ( bridge ) { - sbdf.sbdf = VPCI_ECAM_BDF(gpa - bridge->cfg->phys_addr); - sbdf.seg = bridge->segment; - sbdf.bus += bridge->cfg->busn_start; + sbdf->sbdf = VPCI_ECAM_BDF(gpa - bridge->cfg->phys_addr); + sbdf->seg = bridge->segment; + sbdf->bus += bridge->cfg->busn_start; } else - sbdf.sbdf = VPCI_ECAM_BDF(gpa - GUEST_VPCI_ECAM_BASE); + { + /* + * For the passed through devices we need to map their virtual SBDF + * to the physical PCI device being passed through. + */ + sbdf->sbdf = VPCI_ECAM_BDF(gpa - GUEST_VPCI_ECAM_BASE); + read_lock(&d->pci_lock); + translated = vpci_translate_virtual_device(d, sbdf); + read_unlock(&d->pci_lock); + } - return sbdf; + return translated; } static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info, register_t *r, void *p) { struct pci_host_bridge *bridge = p; - pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + pci_sbdf_t sbdf; const unsigned int access_size = (1U << info->dabt.size) * 8; const register_t invalid = GENMASK_ULL(access_size - 1, 0); /* data is needed to prevent a pointer cast on 32bit */ unsigned long data; + ASSERT(!bridge == !is_hardware_domain(v->domain)); + + if ( !vpci_sbdf_from_gpa(v->domain, bridge, info->gpa, &sbdf) ) + { + *r = invalid; + return 1; + } + if ( vpci_ecam_read(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, &data) ) { @@ -50,7 +70,12 @@ static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info, register_t r, void *p) { struct pci_host_bridge *bridge = p; - pci_sbdf_t sbdf = vpci_sbdf_from_gpa(bridge, info->gpa); + pci_sbdf_t sbdf; + + ASSERT(!bridge == !is_hardware_domain(v->domain)); + + if ( !vpci_sbdf_from_gpa(v->domain, bridge, info->gpa, &sbdf) ) + return 1; return vpci_ecam_write(sbdf, ECAM_REG_OFFSET(info->gpa), 1U << info->dabt.size, r); diff --git a/xen/drivers/vpci/vpci.c b/xen/drivers/vpci/vpci.c index 23722634d50b..98b294f09688 100644 --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -81,6 +81,30 @@ static int add_virtual_device(struct pci_dev *pdev) return 0; } +/* + * Find the physical device which is mapped to the virtual device + * and translate virtual SBDF to the physical one. + */ +bool vpci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf) +{ + const struct pci_dev *pdev; + + ASSERT(!is_hardware_domain(d)); + ASSERT(rw_is_locked(&d->pci_lock)); + + for_each_pdev ( d, pdev ) + { + if ( pdev->vpci && (pdev->vpci->guest_sbdf.sbdf == sbdf->sbdf) ) + { + /* Replace guest SBDF with the physical one. */ + *sbdf = pdev->sbdf; + return true; + } + } + + return false; +} + #endif /* CONFIG_HAS_VPCI_GUEST_SUPPORT */ void vpci_deassign_device(struct pci_dev *pdev) diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 980aded26fc1..7e5a0f0c50c1 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -303,6 +303,18 @@ static inline bool __must_check vpci_process_pending(struct vcpu *v) } #endif +#ifdef CONFIG_HAS_VPCI_GUEST_SUPPORT +bool vpci_translate_virtual_device(const struct domain *d, pci_sbdf_t *sbdf); +#else +static inline bool vpci_translate_virtual_device(const struct domain *d, + pci_sbdf_t *sbdf) +{ + ASSERT_UNREACHABLE(); + + return false; +} +#endif + #endif /* From patchwork Fri May 17 17:06:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stewart Hildebrand X-Patchwork-Id: 13667192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0A7CDC25B79 for ; Fri, 17 May 2024 17:07:42 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.724367.1129632 (Exim 4.92) (envelope-from ) id 1s813P-0007rn-I0; Fri, 17 May 2024 17:07:35 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 724367.1129632; Fri, 17 May 2024 17:07:35 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s813P-0007rg-F6; Fri, 17 May 2024 17:07:35 +0000 Received: by outflank-mailman (input) for mailman id 724367; Fri, 17 May 2024 17:07:34 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1s813O-0006C6-Ag for xen-devel@lists.xenproject.org; Fri, 17 May 2024 17:07:34 +0000 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on20600.outbound.protection.outlook.com [2a01:111:f403:200a::600]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id f2977c4c-146f-11ef-b4bb-af5377834399; Fri, 17 May 2024 19:07:32 +0200 (CEST) Received: from CH0PR04CA0018.namprd04.prod.outlook.com (2603:10b6:610:76::23) by BL1PR12MB5921.namprd12.prod.outlook.com (2603:10b6:208:398::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.30; Fri, 17 May 2024 17:07:26 +0000 Received: from CH2PEPF0000009C.namprd02.prod.outlook.com (2603:10b6:610:76:cafe::19) by CH0PR04CA0018.outlook.office365.com (2603:10b6:610:76::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7587.31 via Frontend Transport; Fri, 17 May 2024 17:07:26 +0000 Received: from SATLEXMB03.amd.com (165.204.84.17) by CH2PEPF0000009C.mail.protection.outlook.com (10.167.244.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7587.21 via Frontend Transport; Fri, 17 May 2024 17:07:25 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 17 May 2024 12:07:25 -0500 Received: from ubuntu.mshome.net (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2507.35 via Frontend Transport; Fri, 17 May 2024 12:07:19 -0500 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f2977c4c-146f-11ef-b4bb-af5377834399 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mu3r6ORU84PuX3HDQ7SwzWqlPti1SO9+IBG+NK7Lc6MXrIJH47Owd+6a7a9HtlhdifYm3RMsdUZV3CTt5NGAjjZghczZKzhg5bb5+E8LN1N1iaejzmsq7QPe8pFuFS7D0a//EFswBJ0N1cCndia4KN8n3rpr81N7dUkEugc4+Gg3519U3XvgxzkQkBz6SEtRlkppjhaIL08Y+6qiHsPg4NMtGyH7wQP+IDc9q44XT+SSH4sMuZ7xqK7FKE3ryLe9tXofjLQxI36skZ/70cqhciT7+BF68W6jMegb6ay6RqKCE3nMXIEzdS6RY1Nsf9hHha13I3iMBBU5ohIBsMrN9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wg1lIYFsxCkrbMSdoPQQCZQ7ozFKfKSY7zA6PHJCXCc=; b=c2QhoeBJ6EdKJXNGkRQGfCTo0BnA+L575RBbWFhAzsr2GuKMbAdQue7Noy5D/+AlH/HjDpr0Bh9bfGv6rIwU5WKcRNgmguTiZqKsnx+aZnArpYRycko5ZrFlc4/Lh5Gcv8RsxEmA/sJwz5C5DQVn8AoUwAbPu6lwt8/BE9sVYYhbYJzp5nXVv0Du0E3aolO7k7pqhJE6RBmIzoNdgFB05Roun1NW29M7R2XHuNfsbcDUAVc8lUqCEUktZs990uOWnuSMmC3Zsf0KASMnXbB4pMnepb5xO6yweIoWxdW7kdn7jPiio6Yvm7+q5Hg4109UxMrPC9Wkda+C0pePnVhx3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wg1lIYFsxCkrbMSdoPQQCZQ7ozFKfKSY7zA6PHJCXCc=; b=DNQ3978As1te2Q0tl4Hxt8zwQoa3PtLIAPQJE6z9JczMc0Zvm8MH0gAJ8H3OXYgPCEdAmNSGx5t0N1qwtgqXAxAeMvH/dPiVbxN/jXiebG+OAyt9Yv/dKbeN7ZLJt38ZK+Xud550jbx8qLLHBhJOwHmTRTwgP98uBREp+Lv7vC0= X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C From: Stewart Hildebrand To: CC: Oleksandr Andrushchenko , "Stewart Hildebrand" , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , "Volodymyr Babchuk" , Julien Grall , Volodymyr Babchuk Subject: [PATCH v15 5/5] xen/arm: account IO handlers for emulated PCI MSI-X Date: Fri, 17 May 2024 13:06:15 -0400 Message-ID: <20240517170619.45088-6-stewart.hildebrand@amd.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240517170619.45088-1-stewart.hildebrand@amd.com> References: <20240517170619.45088-1-stewart.hildebrand@amd.com> MIME-Version: 1.0 Received-SPF: None (SATLEXMB03.amd.com: stewart.hildebrand@amd.com does not designate permitted sender hosts) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000009C:EE_|BL1PR12MB5921:EE_ X-MS-Office365-Filtering-Correlation-Id: b40d0cd0-3024-40b6-df5d-08dc7693d386 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230031|36860700004|82310400017|1800799015|376005; X-Microsoft-Antispam-Message-Info: Y3kuB04l1s5iff60H3ERnvdonr9dL1NtYAX1rH6weVScjRX+6rAUKlRg08SuS+rBsVd1QigYKbhZ3DfXGO+59RNofwDwFPlPAkhEXpHhppXigKRDAGy0H1kWdnjDwzMHcRr+qVqUWAQ5B0/lINZFxPsr0BkMr+vgAyyunsDiA8pXYk0MSOZ2vi29aZbcjYj3G1bZ30GTlcfKZlamiRwutiIpWcMEBKF/ltmjJ2Wa5Sw8JE+l8GRkyjZrOKhpWFUcM/lDM64hRqo5jM46Aj89KTTpiljKcM0nwj1pihYxrtxhTMlQosszQveMhEEy6TlJR/fHmmoCJvQY8H/LAu01CNsTwPdWTSP8dNf+hQuTfvG9+yhXUOvZjKEBRYqbgp1Gv1lk4QtNw0ul8or9ZWp/Z8meW7UWuzfLctTs6LsP7uu1CJcVw8zcY6rx9TT9/yJUaiULz/C7tYc0M9m7DmoFJG/QHJyVxDki+c5G6Yl7VW0/am9jBpefRAR439Q4DmLXFBnjFJ1i9bUvpmpLYjXiOjyDT8awWlrkT504dKpC+cIgtkHfhgztMYA2dDR2PWkAOclBPGALQX3jVD+uaWM1sLN/99vSkrsJoT+1xKwcCrGq3YfjqaIV+xjfiQdW3pTkjoksPM6fFf66NwZji4j1iDMHzQ2cXOsHk5Tx9yPWH70LaLw0kZOuYDTlUC8yg9r/BGhk03LaaE5KkXVqHlMHwQRaZ6JaxRVPAl9loM4mJhhr4JENqvdIoMNiUAwEJlpgJOBC9GbJtxAcidClDpV61WA1QF57qfhf9SEnK2V69TL1nkdL2+RrhCI18I0rkIKjHi1RqktDsV2dimnfzuRXbLlEiZomqVDZvd2pFi4vj695JW6CJpacfaonMJUMHdcUxpE1iBi/Mp2lMr7ZgGiMhToHRuAejfVoOksGycdOERgx6yqloBnMmjSIwdq0D5HSmaWpKZmn5FKQMgj46DTCOK5xfEAJfGg0fUKUkrVfOSnnKpkiYgmaXZ+LQzxwxm3iztD/1p3UOozblh7k40PjP2Z0AnY134a55DGGaPwQYU7Rbr9n3PRPwJy4Ycf/JpWD2+FYBEH2dmbxH86WXMpVvuNfP03QWboVQxJjb9MHewxGUdwX8yF0eDkI7RGP6/HGTGqPmpQwNT3whh8Tp794uwsITNR5iQe79iOyeYz7lcexGqcFthNhfeRK5t8zSHk0eMZ/XmHyCCy3TFxvKYL5Vrd1gVTgHzxbeHdBoN8qJywaweQChhoynw6cKdWaxBnv1YQBDL3BdUur9tHiPFlpBHQHl5VjSbHCz00y9WG8jUBOSKVoCrpZG3qAvxAjtSwJwrLX3AOVozzQuGc/IHXrbdgKC1vjScBD8KQJc9HMfQXBIWEDPK9qkwKaVmEQudi2 X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(36860700004)(82310400017)(1800799015)(376005);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2024 17:07:25.8838 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b40d0cd0-3024-40b6-df5d-08dc7693d386 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000009C.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5921 From: Oleksandr Andrushchenko At the moment, we always allocate an extra 16 slots for IO handlers (see MAX_IO_HANDLER). So while adding IO trap handlers for the emulated MSI-X registers we need to explicitly tell that we have additional IO handlers, so those are accounted. Signed-off-by: Oleksandr Andrushchenko Acked-by: Julien Grall Signed-off-by: Volodymyr Babchuk Signed-off-by: Stewart Hildebrand --- This depends on a constant defined in ("vpci: add initial support for virtual PCI bus topology"), so cannot be committed without the dependency. Since v5: - optimize with IS_ENABLED(CONFIG_HAS_PCI_MSI) since VPCI_MAX_VIRT_DEV is defined unconditionally New in v5 --- xen/arch/arm/vpci.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/vpci.c b/xen/arch/arm/vpci.c index 516933bebfb3..4779bbfa9be3 100644 --- a/xen/arch/arm/vpci.c +++ b/xen/arch/arm/vpci.c @@ -132,6 +132,8 @@ static int vpci_get_num_handlers_cb(struct domain *d, unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) { + unsigned int count; + if ( !has_vpci(d) ) return 0; @@ -152,7 +154,17 @@ unsigned int domain_vpci_get_num_mmio_handlers(struct domain *d) * For guests each host bridge requires one region to cover the * configuration space. At the moment, we only expose a single host bridge. */ - return 1; + count = 1; + + /* + * There's a single MSI-X MMIO handler that deals with both PBA + * and MSI-X tables per each PCI device being passed through. + * Maximum number of emulated virtual devices is VPCI_MAX_VIRT_DEV. + */ + if ( IS_ENABLED(CONFIG_HAS_PCI_MSI) ) + count += VPCI_MAX_VIRT_DEV; + + return count; } /*