From patchwork Tue May 30 12:38:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13259960 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 28FB1C7EE23 for ; Tue, 30 May 2023 12:39:24 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.541157.843600 (Exim 4.92) (envelope-from ) id 1q3ycz-0001QH-Rb; Tue, 30 May 2023 12:39:05 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 541157.843600; Tue, 30 May 2023 12:39:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1q3ycz-0001QA-Nw; Tue, 30 May 2023 12:39:05 +0000 Received: by outflank-mailman (input) for mailman id 541157; Tue, 30 May 2023 12:39:03 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1q3ycx-0001Q4-NF for xen-devel@lists.xenproject.org; Tue, 30 May 2023 12:39:03 +0000 Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on20617.outbound.protection.outlook.com [2a01:111:f400:7d00::617]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id f4b5edef-fee6-11ed-b231-6b7b168915f2; Tue, 30 May 2023 14:39:02 +0200 (CEST) Received: from VE1PR04MB6560.eurprd04.prod.outlook.com (2603:10a6:803:122::25) by AM9PR04MB8276.eurprd04.prod.outlook.com (2603:10a6:20b:3e7::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.23; Tue, 30 May 2023 12:38:58 +0000 Received: from VE1PR04MB6560.eurprd04.prod.outlook.com ([fe80::e442:306f:7711:e24c]) by VE1PR04MB6560.eurprd04.prod.outlook.com ([fe80::e442:306f:7711:e24c%5]) with mapi id 15.20.6433.022; Tue, 30 May 2023 12:38:58 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f4b5edef-fee6-11ed-b231-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UvAGqtyn56T87B0hutB68vkVzkAc6aQkwNEdsWAuJjmwXJa9os8ISyNsZDLF4S6cXfoEg7+pYwsG3bg9BoJzlCDROUVlytctUCfwZYDyHFIPug+S/LYiGyedj+ZQTxRF3oG7N8cwlmCYrzBBzwTBW38GXDymj5h6cidByFr1sC7F1QUCVt14mWWsonMYgZ7boo3W5PFG1sEvWIF018Y5eC2Hh76viJPwqEbhX82Zh6W9fRC15TzxOcLiJz3jkBovmzc73PJzmyiOal+A7GygHn87TihBQRwE6a7V3CTXPo8C1LspWlKtqHwQwqVyY/4oaCgsjT2Ztl4TzcQVQ1RdyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9sdhBVPmtL1e1+Kh9sGgFXp6ChoVMN6PauBpRwmz8hg=; b=Opw8qOrBox5dvxlHv6gkUsV+dwwjCO0rNwV2i3e1DePZNn+cY/KkFlX2q5paTNCLw+pq4/Gxgxn/GLN5s2oAocdzmkhhIbb4hoihYvBZv+Jv0L4nX0sI5C240QJb4cPYyAG7b6LMnoCEDIgX9DqZMtJKUSLzMBElA9fWfAPJ8nB7NFsDrzMwCryc5BdrTs9mDdX5MiQEOI2qzJXM6hwYbJijUuc013jj/hyFYpAZ8meWeHBS8tzISs5aS1SdT1fhUupL57978iBVswiwxt+cFmFi4LssZ5CAu9zA2232t5p9H8E0l6sSavLFcb9+Hke94/oDaXNC/xBh5tuCgipZzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=suse.com; dmarc=pass action=none header.from=suse.com; dkim=pass header.d=suse.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9sdhBVPmtL1e1+Kh9sGgFXp6ChoVMN6PauBpRwmz8hg=; b=nsl/fpf5S/L3e+fwUumWP4TvDdaIidpHGVY2WhiJOn4znTJCjtgOH4hum4IWrCEccE+/Jzk7bqUO0h5/FuTaI4jBguh8Re//lTB+ffNTaUsZSPRFCjw+4gaJ5IEVqvXMa8U43288+pA3AKibDKxuxjq/Y9wyNI206/Uhj5DkLdTkF0/MQmvR51QU49XQv6pDf/8NAXW6dFTrwD44q79TWyQyaYmoiMZGThVrhC3wmvFoCLDbk/5+7SlLq3KOfjFMOLzBOZrIhF27XhJs0rPXsu+v60bRVL9O+0b9jDm+Sw0BC3ROyFG8ny66I7tgSHcqYSVqHM8Y58jFTSLRHCw0bg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=suse.com; Message-ID: Date: Tue, 30 May 2023 14:38:56 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 From: Jan Beulich Subject: [PATCH v3] vPCI: account for hidden devices To: "xen-devel@lists.xenproject.org" Cc: =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Stefano Stabellini Content-Language: en-US X-ClientProxiedBy: FR0P281CA0053.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:48::17) To VE1PR04MB6560.eurprd04.prod.outlook.com (2603:10a6:803:122::25) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VE1PR04MB6560:EE_|AM9PR04MB8276:EE_ X-MS-Office365-Filtering-Correlation-Id: 4da559de-489f-411c-5cfa-08db610ad672 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FHh7WfLOu2hiAI3P8oNMVwsx0g8tEKLT72n8hrvWJO/9BKvGOkXeINo+5WLMng4BvjUOLi93ilFtjmQ+lZJMl1sZx88QCHoMme5+8Gez/j7YaldsQVWGBYtRu5Y4E7LYuTiimtkffyWnds9H6T0PtPNzyinoR75F8HuYbzUXvPDKthq8RM86ze7Gs8jGE6Tl533AA1IDlILGW1dbNZRr0STrLkVaWLwlu7+aK5c+ZjvmoYqUZf9YNJacnRtPVq6azRKEYNedwKxZIM/XZHaqKLMwrFV59tU44+1ScBv20bXFTeJYuMmUOUdUiMW5swk7XyN6ZW8lPmJKnbzeIKrrFy5lqC42RumYc9p+gSV7S1rAgfwh/AuUXNJ2DD1NeOb4Wa6X4KA8/9p+aulnxoHLZnL04/aoCs2T6Oh+JBjsKrVf4y2P+nO121U1e45wo+1V5K61iZ0gFu+OXzvYeMKnXiHa616bH6Ex4ugxOG+v3s98eYmhqGOCT0E4vGLSh6Kd9L12eRFu5vVYEsYkAerY/s7h9zMfB8wQdPhGz9MhjuHFn518Av5w8OzaS7r+1mP/VfHhwODh6Up3BiXRFr8O5zHFNi6ruGZu61FiEo3TLjlIyXXgpvtp4HiAo+ngDufM3P1XUo8OAsqu5GBmzPpJwQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VE1PR04MB6560.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(39860400002)(376002)(451199021)(26005)(6486002)(6506007)(6512007)(2906002)(316002)(186003)(5660300002)(41300700001)(36756003)(8936002)(8676002)(478600001)(38100700002)(31686004)(54906003)(6916009)(86362001)(31696002)(4326008)(2616005)(66946007)(66476007)(66556008)(83380400001)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?41zokOVokBINoD4g4+sXeondqqUF?= =?utf-8?q?mla8ryenzzaOAKh4zJmaHYnpaIIO/kCyLhb72CffzgPF8QWbLJL3J8KV07q+1VWVB?= =?utf-8?q?ZnxAEKO+7Xj5izg6xyuRumAU5pI14P32XrA6l6N2OfjREYnG1xHs8b9CZ4Jahv8yf?= =?utf-8?q?kCdEryM8wrCZ1kHAT/1ZtB6V2F2C86xGsP3QkYb6DtZ4rmR15LwN8LJzbXv8bsFbj?= =?utf-8?q?2WOWxUzADII6fwHnYv0BhF+egqfzDWetxs91ffB7o5pw6cTtleIaCRkN1IZj1yTlj?= =?utf-8?q?LHusWVHaOIzqFB6BbBhJMJEsB7HbDm2cPVT+LLiW+RQ/1StsoMtFjqsl81pp40ITQ?= =?utf-8?q?BrWkDuSgwg7HVHxXGJZ4u1MNPSm0A6WKomZi9aHolHtZkkZ3+/uwmd9mqYf6DYBjg?= =?utf-8?q?k9vZ59fsiwwR1LchZMCXvyyERsVcvsfHUeg8xAONN+1KW8EY66hOGC6rwPURJit63?= =?utf-8?q?fJWamItpF1kUfqC3zgrnIc/Rw8jz/ufdzEa5n8VpCuVF1kdys/c74J5s9T+vj5bB9?= =?utf-8?q?ZYV0nHMdYgw9U8ijtL/lnQ4XtPmGAkSumRIIeMtziY24fTJyPoMWCyKGBdYeMDZL3?= =?utf-8?q?KHRoM6GIomWERZLC7Oav0QjJgFH9scdD7Ou78SJi5Yg/OZAH5+UucCblfF7nuTOxz?= =?utf-8?q?1xS0QmA61Y5XnIY/voPlywlQkU+YW44Z61KuXbwtIF7fVTxHd8JjfB1AxxVYi5alJ?= =?utf-8?q?ijRHvqUwAiSI2E9WMmVZtMmKynZaYW0FymS9UhXC03omppCmLMOl1A4Hv0+C3uhub?= =?utf-8?q?FP+SS6h4OvwVyMURxMp1Uac1XqATqN/YxHPAp806ddzAPvlTS/fR8xZc+d8Ip9ACU?= =?utf-8?q?m4+XYXcnJfowzxmiztmSi0XGpVJCWg7KZncBLpkhAfwfR6eXl92BchIlnaHyFkrgZ?= =?utf-8?q?a5BoWnHVIW+Jn8kOLoCgaPxA1p9wEGwABGN7zfE4jtDr1/gkS65YVrMvpHJZ/ze7W?= =?utf-8?q?5zbCy8qOS2fLyC7HTSrCdcuVECi3m3vFOC10FTSRbg8oXlNnWd6Y8h5/hMsK20Xe+?= =?utf-8?q?yzPLb59TjyUhCs13wBGDrM/aLMaM54O4rtnuPb+Txw7bZw9kakdDmyrG3NcJDRyQy?= =?utf-8?q?YmmmNy5LBagsdtkFjKqKjRhDhGFbXld0tw7YZQRZOnqQk4VDD+zW3m1v7Xf+V/kIb?= =?utf-8?q?hQJuew1seWgtocotQUXydVrmqrx3EKh1o4HZO7M8EjTrbAgK4JKfJ0ciJgxRrkV5F?= =?utf-8?q?rNyWOKggzhVjI29m/0r8mWC1CbBUW9ZMNt93E4VLOfNpZBBTULs8NHOtjlu/GGytT?= =?utf-8?q?XpsqD6FmSpeONNzmBeKVgz52oJ9TNU2F9VShhwBLQ3c4ToT7lsHkoBam47v6fAoTn?= =?utf-8?q?tFCt6lT/USYI+X3eCeTjY9zamSFl+BD5JVUtiXU09JFuYNx26W2OG5TCHtWAXjKXO?= =?utf-8?q?OqySl55yM8+7wCJu14z7I3qriP9QUlZagMxLtMU2wxULIW4dFMJpTtWA0Iukg5EOl?= =?utf-8?q?jysBwP9zKWJwLVpeWbep1E/xIjifOWNhUb7xQLyOyZU5eljnC1nduENQ64sDIE8x+?= =?utf-8?q?UdSeiPZ98llG?= X-OriginatorOrg: suse.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4da559de-489f-411c-5cfa-08db610ad672 X-MS-Exchange-CrossTenant-AuthSource: VE1PR04MB6560.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 May 2023 12:38:58.2352 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JD8DoXlL+yDTdYIZ5EQK+WYzXSVWkFfhORv//XYfWPyPJBzFwJIw1ps8MtbcIUDVIG2fuuEsP7nt6enG+qMRPg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR04MB8276 Hidden devices (e.g. an add-in PCI serial card used for Xen's serial console) are associated with DomXEN, not Dom0. This means that while looking for overlapping BARs such devices cannot be found on Dom0's list of devices; DomXEN's list also needs to be scanned. Suppress vPCI init altogether for r/o devices (which constitute a subset of hidden ones). Signed-off-by: Jan Beulich Reviewed-by: Roger Pau Monné Tested-by: Stefano Stabellini --- v3: Also consider pdev being DomXEN's in modify_bars(). Also consult DomXEN in vpci_{read,write}(). Move vpci_write()'s check of the r/o map out of mainline code. Re-base over the standalone addition of the loop continuation in modify_bars(), and finally make the code change there well-formed. v2: Extend existing comment. Relax assertion. Don't initialize vPCI for r/o devices. --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -218,6 +218,7 @@ static int modify_bars(const struct pci_ struct vpci_header *header = &pdev->vpci->header; struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; + const struct domain *d; const struct vpci_msix *msix = pdev->vpci->msix; unsigned int i; int rc; @@ -285,58 +286,69 @@ static int modify_bars(const struct pci_ /* * Check for overlaps with other BARs. Note that only BARs that are - * currently mapped (enabled) are checked for overlaps. + * currently mapped (enabled) are checked for overlaps. Note also that + * for hwdom we also need to include hidden, i.e. DomXEN's, devices. */ - for_each_pdev ( pdev->domain, tmp ) + for ( d = pdev->domain != dom_xen ? pdev->domain : hardware_domain; ; ) { - if ( !tmp->vpci ) - /* - * For the hardware domain it's possible to have devices assigned - * to it that are not handled by vPCI, either because those are - * read-only devices, or because vPCI setup has failed. - */ - continue; - - if ( tmp == pdev ) + for_each_pdev ( d, tmp ) { - /* - * Need to store the device so it's not constified and defer_map - * can modify it in case of error. - */ - dev = tmp; - if ( !rom_only ) + if ( !tmp->vpci ) /* - * If memory decoding is toggled avoid checking against the - * same device, or else all regions will be removed from the - * memory map in the unmap case. + * For the hardware domain it's possible to have devices + * assigned to it that are not handled by vPCI, either because + * those are read-only devices, or because vPCI setup has + * failed. */ continue; - } - for ( i = 0; i < ARRAY_SIZE(tmp->vpci->header.bars); i++ ) - { - const struct vpci_bar *bar = &tmp->vpci->header.bars[i]; - unsigned long start = PFN_DOWN(bar->addr); - unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); - - if ( !bar->enabled || !rangeset_overlaps_range(mem, start, end) || - /* - * If only the ROM enable bit is toggled check against other - * BARs in the same device for overlaps, but not against the - * same ROM BAR. - */ - (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) - continue; + if ( tmp == pdev ) + { + /* + * Need to store the device so it's not constified and defer_map + * can modify it in case of error. + */ + dev = tmp; + if ( !rom_only ) + /* + * If memory decoding is toggled avoid checking against the + * same device, or else all regions will be removed from the + * memory map in the unmap case. + */ + continue; + } - rc = rangeset_remove_range(mem, start, end); - if ( rc ) + for ( i = 0; i < ARRAY_SIZE(tmp->vpci->header.bars); i++ ) { - printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", - start, end, rc); - rangeset_destroy(mem); - return rc; + const struct vpci_bar *bar = &tmp->vpci->header.bars[i]; + unsigned long start = PFN_DOWN(bar->addr); + unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + + if ( !bar->enabled || + !rangeset_overlaps_range(mem, start, end) || + /* + * If only the ROM enable bit is toggled check against + * other BARs in the same device for overlaps, but not + * against the same ROM BAR. + */ + (rom_only && tmp == pdev && bar->type == VPCI_BAR_ROM) ) + continue; + + rc = rangeset_remove_range(mem, start, end); + if ( rc ) + { + printk(XENLOG_G_WARNING "Failed to remove [%lx, %lx]: %d\n", + start, end, rc); + rangeset_destroy(mem); + return rc; + } } } + + if ( !is_hardware_domain(d) ) + break; + + d = dom_xen; } ASSERT(dev); --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -70,6 +70,7 @@ void vpci_remove_device(struct pci_dev * int vpci_add_handlers(struct pci_dev *pdev) { unsigned int i; + const unsigned long *ro_map; int rc = 0; if ( !has_vpci(pdev->domain) ) @@ -78,6 +79,11 @@ int vpci_add_handlers(struct pci_dev *pd /* We should not get here twice for the same device. */ ASSERT(!pdev->vpci); + /* No vPCI for r/o devices. */ + ro_map = pci_get_ro_map(pdev->sbdf.seg); + if ( ro_map && test_bit(pdev->sbdf.bdf, ro_map) ) + return 0; + pdev->vpci = xzalloc(struct vpci); if ( !pdev->vpci ) return -ENOMEM; @@ -332,8 +338,13 @@ uint32_t vpci_read(pci_sbdf_t sbdf, unsi return data; } - /* Find the PCI dev matching the address. */ + /* + * Find the PCI dev matching the address, which for hwdom also requires + * consulting DomXEN. Passthrough everything that's not trapped. + */ pdev = pci_get_pdev(d, sbdf); + if ( !pdev && is_hardware_domain(d) ) + pdev = pci_get_pdev(dom_xen, sbdf); if ( !pdev || !pdev->vpci ) return vpci_read_hw(sbdf, reg, size); @@ -427,7 +438,6 @@ void vpci_write(pci_sbdf_t sbdf, unsigne const struct pci_dev *pdev; const struct vpci_register *r; unsigned int data_offset = 0; - const unsigned long *ro_map = pci_get_ro_map(sbdf.seg); if ( !size ) { @@ -435,18 +445,20 @@ void vpci_write(pci_sbdf_t sbdf, unsigne return; } - if ( ro_map && test_bit(sbdf.bdf, ro_map) ) - /* Ignore writes to read-only devices. */ - return; - /* - * Find the PCI dev matching the address. - * Passthrough everything that's not trapped. + * Find the PCI dev matching the address, which for hwdom also requires + * consulting DomXEN. Passthrough everything that's not trapped. */ pdev = pci_get_pdev(d, sbdf); + if ( !pdev && is_hardware_domain(d) ) + pdev = pci_get_pdev(dom_xen, sbdf); if ( !pdev || !pdev->vpci ) { - vpci_write_hw(sbdf, reg, size, data); + /* Ignore writes to read-only devices, which have no ->vpci. */ + const unsigned long *ro_map = pci_get_ro_map(sbdf.seg); + + if ( !ro_map || !test_bit(sbdf.bdf, ro_map) ) + vpci_write_hw(sbdf, reg, size, data); return; }