From patchwork Thu Feb 24 16:37:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 12758860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D438EC433F5 for ; Thu, 24 Feb 2022 16:37:46 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.278490.475761 (Exim 4.92) (envelope-from ) id 1nNH7M-0006hA-IL; Thu, 24 Feb 2022 16:37:24 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 278490.475761; Thu, 24 Feb 2022 16:37:24 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nNH7M-0006h3-ED; Thu, 24 Feb 2022 16:37:24 +0000 Received: by outflank-mailman (input) for mailman id 278490; Thu, 24 Feb 2022 16:37:22 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1nNH7K-0006gx-MJ for xen-devel@lists.xenproject.org; Thu, 24 Feb 2022 16:37:22 +0000 Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 08315fc4-9590-11ec-8eb8-a37418f5ba1a; Thu, 24 Feb 2022 17:37:20 +0100 (CET) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 08315fc4-9590-11ec-8eb8-a37418f5ba1a DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1645720640; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=t+8rG6Fm+mpvRCzCQQn9JDgF4eVQf/agejDrWs9oP8Y=; b=AqKoe3QxBV3xsVbNaTxwCa8Pto3FWR3TzxB17CO5zLHxvH8FvgZDe1yk +3zIjYXTQOecSwSbtCR7Vbi15ydjCRis5U07eK/du1KZEAs5hjjMXtptY HOUeTr8ZbgJRHxSTnlZuQ35nZZAWIwfrNjb8s24704JSAgGHFF8frAvG7 0=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com X-SBRS: 5.1 X-MesageID: 67183607 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.156.83 X-Policy: $RELAYED IronPort-Data: A9a23:jdxGlqKOs49YKtNoFE+RJpUlxSXFcZb7ZxGr2PjKsXjdYENSgjdWy 2IdWjvTbq7cM2OkL49yPovjpB9SvMfcxoUwGlBlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s q3yv/GZdJhcokf0/0vrav67xZVF/fngqoDUUYYoAQgsA148IMsdoUg7wbRh2NQ02YPR7z6l4 rseneWOYDdJ5BYsWo4kw/rrRMRH5amaVJsw5zTSVNgT1LPsvyB94KE3fMldG0DQUIhMdtNWc s6YpF2PEsE1yD92Yj+tuu6TnkTn2dc+NyDW4pZdc/DKbhSvOkXee0v0XRYRQR4/ttmHozx+4 NQRvMCXZgMLApePt+oYehB7MA1kJLITrdcrIVDn2SCS50jPcn+qyPRyFkAme4Yf/46bA0kXq 6ZecmpUKEne2aTmm9pXScE17ignBNPsM44F/Glp0BnSDOo8QICFSKLPjTNd9Glv35AVQ66BD yYfQWVvTB+eQAdDBk4GKJIGkuyFr3fEdCIN/Tp5ooJoujOOnWSdyoPFL979atGMA8JPkS6wv Xna9m70BhUbMt23yjef9H+owOjVkkvTUYQeDufgrqZCj1iax2hVAxoTPXO4rOe4jAiiWttZA 00S5icq66M18SSDTNPwQhm5q36spQMHVpxbFOhSwBGAzO/Y7hiUAkAATyVdc5o2uckuXzso2 1SV2dTzClRHsqCRSH+b3qeZq3W1Iyd9BXcLeCssXQYDpd75r+kOYgnnF4g5VvTv15usRG+2k 2viQDUCa6s7jJICi4KU+XL7uXGDqJ3gRy0Wyh/OQTfwhu9mX7KNa4ut4FndyP9PKoeFU1WM1 EQ5d9iiAPMmVs/UynHUKAkZNPTwvqvebmWA6bJ6N8R5r1yQF2ifkZe8Cd2UDGNgKY46dDDge yc/UisBtcYIbBNGgUKaCr9d6vjGL4C9TbwJtdiON7Kih6SdkifdokmCgmbKggjQfLAEy/1XB HtiWZ/E4YwmIapm1iGqYOwWzKUmwCszrUuKG8ymkUj9jeDFOybOIVvgDLdoRrpnhE9jiF+Im +uzyuPQk0kPOAEASnO/HXEvwaAiciFgWMGeRz1/fe+fOAt2cFzN+NeKqY7Nj7dNxvwP/s+Rp ynVchYBlDLX2C2WQS3XOysLQO6+Av5CQYcTYHVE0aCAgCN4P+5CLc43KvMKQFXQ3LY9laUtF 6FdIa1twJ1nE1z6xtjUVrGkxKRKfxW3nwOeeS2jZTk0ZZl7QALVvNTje2PSGOMmV0JbaeNWT 2Wc6z7m IronPort-HdrOrdr: A9a23:g9xeUaov9XioBRaDGViMpT4aV5oveYIsimQD101hICG9Ffbo8P xG/c5rsSMc7Qx7ZJhOo7y90cW7Lk80lqQU3WByB9mftWDd0QPDQb2KhrGC/xTQXwH46+5Bxe NBXsFFebjN5IFB/KXHCd+DYrQd/OU= X-IronPort-AV: E=Sophos;i="5.90,134,1643691600"; d="scan'208";a="67183607" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KXkrGviOt2pG4WfAHMClrh6Ik3H0DNSnOn99ENfvzmZ5F8vCJLW5AcdE9DcyyqqfHUlTAYQr7JXp7HEpldBW12AoNtS9VMDZhG0wofnBaeIw59lW9AqlyX+wGlOKHxQwFZoMXxrtpplarrRoptm9x0FMVR8VQ/PzE/Zguu9gDiGjqfGOORkjnUjfq3fBBk0/cEZD5MvCFQ5oZq0iuishErfGFa3u6ZD7boP4PRaq6y9ReUOq7GbWhkLsN3ZVe5sNVRAnISAo3QTYAKY05Ib2uXwK0M/ftUNXb15PgreyzYOgYL4D5O2lFCtjRtPHggS61WHz2DLzXJWq+L74Xh7N7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Yeyjqo6BAR6+mndrcxmXz+lkFiDgi+mh4YsAir/cbrI=; b=AzV9FTrRBv2o4E0UwqOI0GJVSqUH39Y3I5N9G/dVWjS+Vsjoe0zQL9aIefwf9JB/Ie23ijs5Z57Ts+A/5VbNMnVg6tuZXMXRK1kaJBPchQbBnWO3WtauiaIgatoxyvGz80cL6JATvKAlzQY++dDTw1Lvxjd5H8FEFg6Bbwosthf7BlhlgXw7yTYuxAh/GJaxJF/5YFUdKCnWJL4UEyCQWBUuamCh4OMimsHlRFX3wxHKSbWpiBxiPrmnF9n2VsuiQjVYCCME+4QIDCT9p3tRvwq0dGSEIw4jr5Mf01knYEBccPvOtym5WS0R3fizRKcO1TD7DEECsuNzddvnDfelGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yeyjqo6BAR6+mndrcxmXz+lkFiDgi+mh4YsAir/cbrI=; b=nQ0amT+JAV3uUBprJ/qt5ewDva9xJ2mpqZpgZKJH84j/ZmV3EnBHHlyXlt5wxxmm7dUaAi8q5w04NiYzo1vaig4nkw9Sa38H5JtHDtdzOqJdU9zIP9gzbCO8m5NAHvTGswTYdz5RAeenTbCp8H6ouq1Q3PjarhLtp/WrIZovr4Y= From: Roger Pau Monne To: CC: Roger Pau Monne , Jan Beulich , Paul Durrant , Andrew Cooper , George Dunlap , Julien Grall , Stefano Stabellini , Wei Liu , Kevin Tian , Oleksandr Andrushchenko Subject: [PATCH v2] pci/ats: do not allow broken devices to be assigned to guests Date: Thu, 24 Feb 2022 17:37:01 +0100 Message-ID: <20220224163701.89404-1-roger.pau@citrix.com> X-Mailer: git-send-email 2.34.1 X-ClientProxiedBy: LO2P265CA0087.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::27) To DS7PR03MB5608.namprd03.prod.outlook.com (2603:10b6:5:2c9::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8f69a873-98b6-486a-01ab-08d9f7b3e9ec X-MS-TrafficTypeDiagnostic: CH2PR03MB5350:EE_ X-LD-Processed: 335836de-42ef-43a2-b145-348c2ee9ca5b,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kLDltzc0XdWzDjWuE7gbGIdWTWOWeK15NCtLPiLMQbHbSHv/2vw8QL1O0Mk1BRgDb59UHCgQpbJcLPWJp4s9PmQB6p+N96Si7+gWwfmrSckmpWU7zIoy+Yzs+gpBdEYhjCrURkgGmNW0LWaPIYAoTTrpWRaicIk5Pgxtf2RsQxjh7UK+1y5zmldfCMx18+9kLwVk7iurArTOaXTa/583xVGAjU9rbL7jK3YD9yrUyebTyEcv8idIuE1fNE9rerVRehLz7bJy1c8CsVceMD7wpQk6ievZkPekl0W2GpA3VpT8rqyCQASfSyuWHVXRDkDqgChp60IxCMyzYxSuP7LmZNM1eqA12CIQ1vAoxF28mJM/EW4POIh5uLO2Yh+nzeUiMWcBIpgPGozHQV4BSUuPLYX+6q9039mnKxlDaYr0wRoOVLNiBFICZi27JmK+y1Ma9d8Ol4EssMfLiUry33QCBoKQg0mW5ThMSEGJcPGeGk6qPKFuGn0quT6Rhp4Cq/3N4OdgZ32Z/JoSt/uT/dAgApqaLb78a8akX7ok2gOnb4R2x81sienH9FdgeQOn4DBak3kQk9c2ICoJ+UWNKTMXEykA0pMm8wXQ7DTj6Uewgz76ETZgMUphOrm1AkLaqErpQQHneGMytVUXjckjuOwbjA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR03MB5608.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(2616005)(54906003)(66476007)(1076003)(316002)(4326008)(8936002)(2906002)(6916009)(66556008)(66946007)(186003)(26005)(86362001)(6486002)(8676002)(5660300002)(82960400001)(83380400001)(6666004)(6512007)(508600001)(36756003)(38100700002)(6506007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?efd3sliiO28sNf9Ix76yxwQ4a2Kj?= =?utf-8?q?rDdGJMOxpr1BlTSkCZJ4Rg/yKKNb2alhPUhflEaiVD08DC7T7TFLN9FDuquOYrb/J?= =?utf-8?q?F0uCqhf/ho/Z4cjNDTmc+NcZPzG41GmlkQuA/fpaueThtLIvgtweFulMkG31y0DF/?= =?utf-8?q?hqbr2A7xWP/n8VLh+i+cnzABaJlw1WX+7hK59iF7Ljk+Epe2LQ4OGJ4wNvgkmVNGq?= =?utf-8?q?2Da7F23EpS6CjCPAy7SuHq1wesAGTARgVcdXkecMbDR9UdwQoWxd7FR9/VD4vxF4o?= =?utf-8?q?3R5FfaP4aWACYVoXmlrJ2oJd5er0rnNpD0z1dk0aSLSmiOoAs2EJxh0mNOStQO/uz?= =?utf-8?q?y5GD3vlLTzAdUBsU92punNpZjGjopkOka6pslDNFPWszgwF9f1udZic1EGsVr/hji?= =?utf-8?q?pV4OsfXvwmuudPhVT9OTnNECkcC4SBokWjbCI/tSMbTiWzDfa1OGRWhskeqZqrxdi?= =?utf-8?q?EP8mIBToMbt9P/HjccEFN5AFlEq9Jnn6d9GBMfyODGayiKXF+yHt8NQuP1ssBMFBL?= =?utf-8?q?7aZDKfcbTeasa0/6DrDSMYwVUyfAJ2TBmtGO+Fo9Hih8kwDo1t69xNN2Q6leAx+7b?= =?utf-8?q?XSXuQEjtFABxoxRTF0OnnLtsHBH23V5kxGg9kUBCWBQ1FE+0GwqYdioFJ9i4EJa4g?= =?utf-8?q?wLlSmchgpX7BYcEBmlE7Q3fH/410iUhxxW3ak1iz1/VnKaYzvKPNdp0G/1B4p9AMP?= =?utf-8?q?JAC4sN5HexweF0R52Yn8HkdpSdxFzHnjrMHpIxNJYSZMm6L2FbEqEx7TbZFW6+zCZ?= =?utf-8?q?Yrv1OXe/8nmGoq2NwMCGbTuDdu6ycr5Vy7fk8kSQuROpd5dJP4kPMAGU3vuJIQr0U?= =?utf-8?q?7FuUf1rOMPOw0UPICK02quDy+gHWJBAxFYqAf71rDjqIiy+l0L3AHxsl2ksUuk59Y?= =?utf-8?q?fgDCz03nBKbHPgHPNYv7D4XmA3RbqC4xKa/+mElG6dqW2aXiJ8GvDHHdVhlqv6IdG?= =?utf-8?q?5oSTouNXCSNrgZWJhh+D1UNS8r1nZkVFcYQE9h3a6+rWN8kTwgUhGSJ/6ttmAJf/+?= =?utf-8?q?ZP0aKVrvx0n+2g7wVr/ITQWdCqbD34clcNlMuVa7vNi2eOJ42GYZ5OgHqv6N3IiUN?= =?utf-8?q?T0qMuWMZ+vBg7gY3UMHD+DdXKjnssy9Yi4WYOkjciKVB/TusHwfdwkaWCLKj5LDXQ?= =?utf-8?q?Q9lINqHIcWf5h4ray9qSh+LyYf/20WbZk77QfT2Bx/4ARBdferVtw8Dxd5ZYp0ocW?= =?utf-8?q?ADRJxTKZofndHUuCrQVm2EpzhksbwFAQHXnOISSd72MPjYSa6IbFa/48ffx6DCL/D?= =?utf-8?q?1yQK0XkIyUJzkQfcdSJezxGv1IDlzPMASH4rNfvrG2llxQbXC4rZNu4UajO2UdTsb?= =?utf-8?q?XIvVTNGHNz5YffKEQumVxMDTtNxhE9uRH5d0Vov5Jog67cIa0B58x+0JlKf4QU3sE?= =?utf-8?q?Ae3hmcMcavlFxVbAFCRCWQxLqa6dtiVtlMhXJeyk5BKEsiNG/RSaUAc2du2BgJqo2?= =?utf-8?q?wcrdS21JxoD+OX5lZEvS6j76g9M9Ofm+MUm7f69DeN/xBW0yqEI5Zykt4eYYYSROE?= =?utf-8?q?KyeN8qROPyrAClfNQn2zR+wPV71RQElGDcdKTNx2lCUVWvlHz+c4TCs=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 8f69a873-98b6-486a-01ab-08d9f7b3e9ec X-MS-Exchange-CrossTenant-AuthSource: DS7PR03MB5608.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 16:37:14.7587 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wrBEr1TFGMw0+A26x7+EiQr/hlFi+SPcxByCEgnwO+DrAwfqsyFqX3eoEmZC5t+1LWDlnwpjvbq25Byk80+6NQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR03MB5350 X-OriginatorOrg: citrix.com Introduce a new field to mark devices as broken: having it set prevents the device from being assigned to guests. Use the field in order to mark ATS devices that have failed a flush as broken, thus preventing them to be assigned to any guest. This allows the device IOMMU context entry to be cleaned up properly, as calling _pci_hide_device will just change the ownership of the device, but the IOMMU context entry of the device would be left as-is. It would also leak a Domain ID, as removing the device from it's previous owner will allow releasing the DID used by the device without having cleaned up the context entry. Signed-off-by: Roger Pau Monné Reviewed-by: Jan Beulich Reviewed-by: Kevin Tian --- Cc: Oleksandr Andrushchenko --- Changes since v1: - Allow assigning broken devices to dom_io or the hardware domain. --- xen/drivers/passthrough/pci.c | 11 +++++++---- xen/drivers/passthrough/vtd/qinval.c | 8 +++++++- xen/include/xen/pci.h | 3 +++ 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 70b6684981..91b43a3f04 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -501,7 +501,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev *pdev) xfree(pdev); } -static void _pci_hide_device(struct pci_dev *pdev) +static void __init _pci_hide_device(struct pci_dev *pdev) { if ( pdev->domain ) return; @@ -1487,6 +1487,11 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) ASSERT(pdev && (pdev->domain == hardware_domain || pdev->domain == dom_io)); + /* Do not allow broken devices to be assigned to guests. */ + rc = -EBADF; + if ( pdev->broken && d != hardware_domain && d != dom_io ) + goto done; + rc = pdev_msix_assign(d, pdev); if ( rc ) goto done; @@ -1585,9 +1590,7 @@ void iommu_dev_iotlb_flush_timeout(struct domain *d, struct pci_dev *pdev) return; } - list_del(&pdev->domain_list); - pdev->domain = NULL; - _pci_hide_device(pdev); + pdev->broken = true; if ( !d->is_shutting_down && printk_ratelimit() ) printk(XENLOG_ERR "dom%d: ATS device %pp flush failed\n", diff --git a/xen/drivers/passthrough/vtd/qinval.c b/xen/drivers/passthrough/vtd/qinval.c index 9f291f47e5..510961a203 100644 --- a/xen/drivers/passthrough/vtd/qinval.c +++ b/xen/drivers/passthrough/vtd/qinval.c @@ -227,7 +227,7 @@ static int __must_check dev_invalidate_sync(struct vtd_iommu *iommu, ASSERT(iommu->qinval_maddr); rc = queue_invalidate_wait(iommu, 0, 1, 1, 1); - if ( rc == -ETIMEDOUT ) + if ( rc == -ETIMEDOUT && !pdev->broken ) { struct domain *d = rcu_lock_domain_by_id(did_to_domain_id(iommu, did)); @@ -241,6 +241,12 @@ static int __must_check dev_invalidate_sync(struct vtd_iommu *iommu, iommu_dev_iotlb_flush_timeout(d, pdev); rcu_unlock_domain(d); } + else if ( rc == -ETIMEDOUT ) + /* + * The device is already marked as broken, ignore the error in order to + * allow {de,}assign to succeed. + */ + rc = 0; return rc; } diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index b6d7e454f8..02b31f7259 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -108,6 +108,9 @@ struct pci_dev { /* Device with errata, ignore the BARs. */ bool ignore_bars; + /* Device misbehaving, prevent assigning it to guests. */ + bool broken; + enum pdev_type { DEV_TYPE_PCI_UNKNOWN, DEV_TYPE_PCIe_ENDPOINT,