From patchwork Sat Dec 2 01:27:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13476717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DBFC2C4167B for ; Sat, 2 Dec 2023 01:27:44 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.645842.1008359 (Exim 4.92) (envelope-from ) id 1r9En5-0005R4-Mh; Sat, 02 Dec 2023 01:27:31 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 645842.1008359; Sat, 02 Dec 2023 01:27:31 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1r9En5-0005Mc-4U; Sat, 02 Dec 2023 01:27:31 +0000 Received: by outflank-mailman (input) for mailman id 645842; Sat, 02 Dec 2023 01:27:28 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1r9En2-000272-KO for xen-devel@lists.xenproject.org; Sat, 02 Dec 2023 01:27:28 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f25d04f0-90b1-11ee-98e5-6d05b1d4d9a1; Sat, 02 Dec 2023 02:27:25 +0100 (CET) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B1NLGut008549; Sat, 2 Dec 2023 01:27:08 GMT Received: from eur03-dba-obe.outbound.protection.outlook.com (mail-dbaeur03lp2169.outbound.protection.outlook.com [104.47.51.169]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3uqfjgj05q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 02 Dec 2023 01:27:07 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by DU0PR03MB8599.eurprd03.prod.outlook.com (2603:10a6:10:3e6::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.27; Sat, 2 Dec 2023 01:27:02 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::8e03:368:1fd7:1822]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::8e03:368:1fd7:1822%6]) with mapi id 15.20.7046.027; Sat, 2 Dec 2023 01:27:02 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f25d04f0-90b1-11ee-98e5-6d05b1d4d9a1 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IIqBCiNYoGhG/BlDqcA6mrjYkL019QzzNWF/XwrWndnDqavoj8v6LF0RBbNxoNytTsHKeZ0J9QwBFNXuFtVEpWTt/TxnX26ps9UefBMzVKLjy+znJEzZWaRvr6QO31bwy3BO/XmaIUJLXy+uJrJ3AQHywj7jPFqhGHUE8ESdzq7u7PyCC7Ybvq/2kY/NPT+6OSlontyXylEBqgljTJrs+SwDNzsTCm5/347Zt2XmC3WpLY2RB/EkVK+IH8Il+0eP4efPG1H9Swm8FU9ryhtkTk82oClmIAjtSxFaQ1ojrYGmgbwDQIQiThF9kf+6xCMlQECEgH+kyTUHNl1LRkZ2hA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/8REnoszUXW5wdC5BqFWoMkuaC9Yq6ifN4ZAzhjkY6U=; b=ayvET9LAl41JH6VNmuztqeCLydjMI3OuZDs61K+IP5ufREXvdsVuZW7cnKRZS0DYUQXksDYK1Mhguk7hAV0eovqgiKMurmaQ0WdgVmyQjN818FszpDf8uOTV1mgEhuzWQPx6BBS4yl9IEJkFPtrDtQ/kIrKDxP1Xj72enzZx+qLO7QWkmEB7ZQAR2P3b9NMkE7u+NKIOgCPvcD0mNblheJZ3LmoqBypZKtujCa4znisFnXGJYbDxYZ3lBGTyqwCwvv9ubK8PPfUtNtHwsNUjMRZmeyCMv/lFtBeXELnmxJ3cJXjfIvOkHkrJX7sZLiKM2rSQxAxwSZUhEBUnddizkw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/8REnoszUXW5wdC5BqFWoMkuaC9Yq6ifN4ZAzhjkY6U=; b=m3UP1Jk7+MAZo88/BJftbssBvUlEKkXny74VF8bTLE9MRzarL/+lvJjBpWxF/M+TmbsU8m0pvTgojjNalNdjruf4yewj7e5RaVtKdEzFl5kLLbMttBMS1Bky7owLCYUS3WzUodsjk/DOcUOtnathqBIuYrIWe/nsrJS9AiEJT1w6tMn3PRN22YGVg0+DSWwLdS6BK/qedEKXUy1jpv92qHLfOjxx4kgsMR5VfI9IwcSPpf98jEUW3m0Lc2l7KhGIfOAERxpRWD50Bn0pIruJnwnGEB7TVNr6FSNk32Fg0ZSoyXs2DZZ+peep8MuNfzJsJY6YABTk0+lAmh8Yyl55aQ== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Stewart Hildebrand , Volodymyr Babchuk , Jan Beulich , Andrew Cooper , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Wei Liu , George Dunlap , Julien Grall , Stefano Stabellini , Paul Durrant , Kevin Tian , Jun Nakajima , "Daniel P. Smith" , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk Subject: [PATCH v11 00/17] PCI devices passthrough on Arm, part 3 Thread-Topic: [PATCH v11 00/17] PCI devices passthrough on Arm, part 3 Thread-Index: AQHaJL6myOgOEzRKR0O0HH5CmIGjcw== Date: Sat, 2 Dec 2023 01:27:02 +0000 Message-ID: <20231202012556.2012281-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.42.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|DU0PR03MB8599:EE_ x-ms-office365-filtering-correlation-id: 912774a1-66b4-4982-4653-08dbf2d5c93f x-ld-processed: b41b72d0-4e9f-4c26-8a69-f949f367c91d,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: H2NgLWBk3jq92rGbZMnfcC0nK4F07GEbnlaQIT7yMkqzy+j/K09U2pun+578v6KWpaWLQOi/xZ+SBn0U9JPp8MGixfF1hwrgaSb4ughpoliRTT6jS4kr3/g3dWaQJMm75NIny/NhO+ecomvsNNJExjeG3GQ6u4WRuvh2zljmUpnn0uQkWhwrxkRBG2bX6viY5b1AhlQR5vmYDXSENkfqM7JeNfPUud8cKoqL/jqIPEi6y26R1jmDHTuFkeArapG0jQMIKq9C0alaURJfRjv/6/O4ys0FaY+f1rP/GXGmwRVgRqrcjlazi/Tx6Hcb7B8XkaABwjV801UvBZk0g0GJD64O659Or4D36IOrwI9s6HJwnYeo+nHZDi5kwjdTRZ1Bwh7n7tgxRsgBrM/W2F6M8Zsy7foXPJL5DojrfbNHHlbr+5GVKPWX62IWhBWSjdz3d2wJsVmv0Y1uK5U5r7f8YqglXoMxeSBEv0ynG7r5T9iEp1vYlc6vRcSDHc7WDSindLCUorn3iw1KYUSEnpVmLWwLM2+rH+3TD3DEiZC5u7a0Epq0KaH+saZ/pR2pbYE99U7JNCWMlkYFnxXsp/Crqd/DoqNtLKCLzoO6f0KixQqNmYcse/MUo7sS+0wGqNlx x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(396003)(39860400002)(346002)(376002)(366004)(230922051799003)(186009)(64100799003)(451199024)(1800799012)(5660300002)(7416002)(86362001)(4326008)(8676002)(8936002)(2906002)(38070700009)(41300700001)(36756003)(2616005)(107886003)(1076003)(6512007)(6506007)(55236004)(83380400001)(6486002)(478600001)(26005)(71200400001)(38100700002)(122000001)(316002)(91956017)(54906003)(64756008)(66446008)(6916009)(66476007)(66556008)(76116006)(66946007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?+nX1C/S9ymCSbase49/cRVLQ2B+x?= =?utf-8?q?qa5cB6+EkjKAFm+smEQVQZaGJTgxBezkNVqDMiibh0+NXj6DnaTkL0PVCoO26vtK4?= =?utf-8?q?Ev+od8+QQkQmjyS39h9c0IKsdG2tVpSaHDDOjGMU5qD3B0Iyfv+shYBpyC45S+VgT?= =?utf-8?q?QDcmcnPxHjaX2RwdJyQiZRjyoyq8Tqr1MyflMNwhEwSHXKmjkPUdRLY86Q0mOubzn?= =?utf-8?q?RDW2OnR7S8aZM7Pn4ZzH9vn8kggE8FjkzqHwF0nB7gZIsCHVBcPCxlWjtSzt1wD25?= =?utf-8?q?BiGMNi5isAdkuMlGGtq5QU9N6O51XevHX4VoafeeuhQBvPOzaENWy47MVPM65dHuH?= =?utf-8?q?py2w02Gm+cqWR2ApoQD7/INxPKt1/ZhI+SoGRoLXALHKdA5aQ/OPVXt/H5yJESesq?= =?utf-8?q?zin3a1Eyp0ocbcnWyzPGLzE7I/kudSGxkHzDUMS6FSL8lfSu3Y8Wox+54/ZOczWiy?= =?utf-8?q?ktfnzBaVLXFr5KRh4h1QmstJTIrkQHvZJCwOgrvEj0dKm94e42rK+cdqxTh6LJuv5?= =?utf-8?q?vDOC94Drwq/kk/5CvpBDa/tlL5kamLO46LQFG/skHJ8QYFCfG7JwIJFR/t9L92Rwk?= =?utf-8?q?EPnGt2Z9TYUTwHmqzD14i9LqlG0GHXtnxujQMd0+cqCvDs0iOPossvgN0z9hQNzg0?= =?utf-8?q?3EApiCaqZJ7wwpq9pfMo0Io7EN6VQJQpYvgy0amgDkCCmI0dsPd8KCsJYBMopuXR5?= =?utf-8?q?LGx18JCbyDiWpCXIgLVFzi0ldtjx1Qq9Pf7QqGRTcUqy0areToTxEcgZqTOBVrjzq?= =?utf-8?q?52S3r6ek5Do+72dZGvaku71m3BX4YQS4tPhSjouVD8Cl+UzLDiX94dhxWT5jc5rK9?= =?utf-8?q?2cN/VR9hV3zfTbz6Vo1sPsKcU3Dy0hM/zTVILL/D4U9jlGK93iX3dVPjoiMi+LVP6?= =?utf-8?q?YYBHtFl2QHVuT607EF4DRNsUiqQQ5sp1gfimtQX7XH7eb8LOi8TzXM1e18qOIKRXf?= =?utf-8?q?OxuytFcPZSUKCPBvo4GyMTa39xUHRHjyzkCg/QzPiKqqT6jOnUAXr0EB9g2zpMWds?= =?utf-8?q?bN/gsWc1djBxy6pysVa0Yu9UI/EKs3Sw4VPOihKFJ73IR0KS0uxI7I1bGI66ZAltV?= =?utf-8?q?//SbLk4TFJp/vrAT931BNn+AD8UNBXyfWbymqWFPydC+WDTrKhVFfKaW3KSHXcxUW?= =?utf-8?q?asraeXstoIhfX8J3iU2qA0Ly7ajPaHj/D6R9mdQKc1a/zd20i/jF1KNa57/eZpfE1?= =?utf-8?q?E5YQjFH3jqORvVWOSlXlag4GpWYH/Cu0HA1o7k9vVztQe4CHx7Z1Q4wCkq1kshjw4?= =?utf-8?q?vtdB1anjJslg+cmSNrREHwtHW8O3kpcRuucqjCKdXff+ebcp5AN/qCRh0/aly8ZAG?= =?utf-8?q?V433p7fY9IRORGSnAeDr3YxmhWCjV2mCm7b/15Iryq2kw8euztLbkNo9VqiScGHk1?= =?utf-8?q?vS5wZpK4PG4SOJSW58lhUoXsQbK6TOb9Sv/L4e9QrBaAgyNPrfNARw/WIaumi8cIZ?= =?utf-8?q?xC/jyngbghzSSTd2Ijl+Lv7s61pKWhIiawLojrP1WqIf47U4LIjSwG1tYOORX/r75?= =?utf-8?q?qCfxXYK/yBEity6pvQxIeYkvgEMbQd1IFw=3D=3D?= Content-ID: <6BF1220A4C9618489C236B32BB61BDCC@eurprd03.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 912774a1-66b4-4982-4653-08dbf2d5c93f X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Dec 2023 01:27:02.1276 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ExMgDjOZIDHt/Wau8GTG2MXZUtaPoJo37TeYL8X4n+C9lZjBzvwzUYuQcMO5y7juYfDismga5irDfv6060WKno1nkZPUR1B/5h/iu8fm4kQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR03MB8599 X-Proofpoint-GUID: v6lXct2_OViYMeufBpgk_jBtWo6kngv- X-Proofpoint-ORIG-GUID: v6lXct2_OViYMeufBpgk_jBtWo6kngv- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-01_24,2023-11-30_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 phishscore=0 lowpriorityscore=0 spamscore=0 clxscore=1011 suspectscore=0 impostorscore=0 adultscore=0 bulkscore=0 priorityscore=1501 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000 definitions=main-2312020008 This is next version of vPCI rework. Aim of this series is to prepare ground for introducing PCI support on ARM platform. in v11: - Added my Signed-off-by tag to all patches - Patch "vpci/header: emulate PCI_COMMAND register for guests" is in intermediate state, because it was agreed to rework it once Stewart's series on register handling are in. - Addressed comments, please see patch descriptions for details. in v10: - Removed patch ("xen/arm: vpci: check guest range"), proper fix for the issue is part of ("vpci/header: emulate PCI_COMMAND register for guests") - Removed patch ("pci/header: reset the command register when adding devices") - Added patch ("rangeset: add rangeset_empty() function") because this function is needed in ("vpci/header: handle p2m range sets per BAR") - Added ("vpci/header: handle p2m range sets per BAR") which addressed an issue discovered by Andrii Chepurnyi during virtio integration - Added ("pci: msi: pass pdev to pci_enable_msi() function"), which is prereq for ("pci: introduce per-domain PCI rwlock") - Fixed "Since v9/v8/... " comments in changelogs to reduce confusion. I left "Since" entries for older versions, because they were added by original author of the patches. in v9: v9 includes addressed commentes from a previous one. Also it introduces a couple patches from Stewart. This patches are related to vPCI use on ARM. Patch "vpci/header: rework exit path in init_bars" was factored-out from "vpci/header: handle p2m range sets per BAR". in v8: The biggest change from previous, mistakenly named, v7 series is how locking is implemented. Instead of d->vpci_rwlock we introduce d->pci_lock which has broader scope, as it protects not only domain's vpci state, but domain's list of PCI devices as well. As we discussed in IRC with Roger, it is not feasible to rework all the existing code to use the new lock right away. It was agreed that any write access to d->pdev_list will be protected by **both** d->pci_lock in write mode and pcidevs_lock(). Read access on other hand should be protected by either d->pci_lock in read mode or pcidevs_lock(). It is expected that existing code will use pcidevs_lock() and new users will use new rw lock. Of course, this does not mean that new users shall not use pcidevs_lock() when it is appropriate. Changes from previous versions are described in each separate patch. Hello all, This is next version of vPCI rework. Aim of this series is to prepare ground for introducing PCI support on ARM platform. in v10: - Removed patch ("xen/arm: vpci: check guest range"), proper fix for the issue is part of ("vpci/header: emulate PCI_COMMAND register for guests") - Removed patch ("pci/header: reset the command register when adding devices") - Added patch ("rangeset: add rangeset_empty() function") because this function is needed in ("vpci/header: handle p2m range sets per BAR") - Added ("vpci/header: handle p2m range sets per BAR") which addressed an issue discovered by Andrii Chepurnyi during virtio integration - Added ("pci: msi: pass pdev to pci_enable_msi() function"), which is prereq for ("pci: introduce per-domain PCI rwlock") - Fixed "Since v9/v8/... " comments in changelogs to reduce confusion. I left "Since" entries for older versions, because they were added by original author of the patches. in v9: v9 includes addressed commentes from a previous one. Also it introduces a couple patches from Stewart. This patches are related to vPCI use on ARM. Patch "vpci/header: rework exit path in init_bars" was factored-out from "vpci/header: handle p2m range sets per BAR". in v8: The biggest change from previous, mistakenly named, v7 series is how locking is implemented. Instead of d->vpci_rwlock we introduce d->pci_lock which has broader scope, as it protects not only domain's vpci state, but domain's list of PCI devices as well. As we discussed in IRC with Roger, it is not feasible to rework all the existing code to use the new lock right away. It was agreed that any write access to d->pdev_list will be protected by **both** d->pci_lock in write mode and pcidevs_lock(). Read access on other hand should be protected by either d->pci_lock in read mode or pcidevs_lock(). It is expected that existing code will use pcidevs_lock() and new users will use new rw lock. Of course, this does not mean that new users shall not use pcidevs_lock() when it is appropriate. Changes from previous versions are described in each separate patch. Oleksandr Andrushchenko (11): vpci: use per-domain PCI lock to protect vpci structure vpci: restrict unhandled read/write operations for guests vpci: add hooks for PCI device assign/de-assign vpci/header: implement guest BAR register handlers rangeset: add RANGESETF_no_print flag vpci/header: handle p2m range sets per BAR vpci/header: program p2m with guest BAR view vpci/header: emulate PCI_COMMAND register for guests vpci: add initial support for virtual PCI bus topology xen/arm: translate virtual PCI bus topology for guests xen/arm: account IO handlers for emulated PCI MSI-X Stewart Hildebrand (1): xen/arm: vpci: permit access to guest vpci space Volodymyr Babchuk (5): pci: msi: pass pdev to pci_enable_msi() function pci: introduce per-domain PCI rwlock vpci/header: rework exit path in init_bars rangeset: add rangeset_empty() function arm/vpci: honor access size when returning an error xen/arch/arm/vpci.c | 72 ++- xen/arch/x86/hvm/vmsi.c | 22 +- xen/arch/x86/hvm/vmx/vmx.c | 2 - xen/arch/x86/include/asm/msi.h | 5 +- xen/arch/x86/irq.c | 10 +- xen/arch/x86/msi.c | 25 +- xen/arch/x86/physdev.c | 2 + xen/common/domain.c | 5 +- xen/common/rangeset.c | 21 +- xen/drivers/Kconfig | 4 + xen/drivers/char/ns16550.c | 28 +- xen/drivers/passthrough/amd/pci_amd_iommu.c | 9 +- xen/drivers/passthrough/pci.c | 98 +++- xen/drivers/passthrough/vtd/iommu.c | 9 +- xen/drivers/vpci/header.c | 513 ++++++++++++++++---- xen/drivers/vpci/msi.c | 34 +- xen/drivers/vpci/msix.c | 57 ++- xen/drivers/vpci/vpci.c | 151 +++++- xen/include/xen/rangeset.h | 8 +- xen/include/xen/sched.h | 9 + xen/include/xen/vpci.h | 54 ++- 21 files changed, 928 insertions(+), 210 deletions(-)