From patchwork Wed May 24 13:45:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 13254023 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A665BC77B7A for ; Wed, 24 May 2023 13:46:34 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.539050.839537 (Exim 4.92) (envelope-from ) id 1q1oob-00039H-1N; Wed, 24 May 2023 13:46:09 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 539050.839537; Wed, 24 May 2023 13:46:09 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1q1ooa-00039A-T0; Wed, 24 May 2023 13:46:08 +0000 Received: by outflank-mailman (input) for mailman id 539050; Wed, 24 May 2023 13:46:08 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1q1ooa-000394-14 for xen-devel@lists.xenproject.org; Wed, 24 May 2023 13:46:08 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2060a.outbound.protection.outlook.com [2a01:111:f400:7e1a::60a]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 53e72a47-fa39-11ed-b230-6b7b168915f2; Wed, 24 May 2023 15:46:05 +0200 (CEST) Received: from VE1PR04MB6560.eurprd04.prod.outlook.com (2603:10a6:803:122::25) by VE1PR04MB7280.eurprd04.prod.outlook.com (2603:10a6:800:1af::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.28; Wed, 24 May 2023 13:46:01 +0000 Received: from VE1PR04MB6560.eurprd04.prod.outlook.com ([fe80::e442:306f:7711:e24c]) by VE1PR04MB6560.eurprd04.prod.outlook.com ([fe80::e442:306f:7711:e24c%5]) with mapi id 15.20.6411.029; Wed, 24 May 2023 13:46:01 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 53e72a47-fa39-11ed-b230-6b7b168915f2 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TIOAP64M9OGKguvqRc+5WRZsESRSVtyHh38/m7PcT2hmbXCKyOIk3AAKMQ9wZgLElyCP1o4lWJZtzEQ2y8eQVLIW3rnutFFxx8NO/hL/U9rTU9Wid/Vr1+qJCAb7JBzaPrQA9L+WuORFWZ9FkdMwcjCfUMfJiLeWL7pvx7hFBjbKxBlTjdRg8vay0xDlMg0+EuzCFZn0wn+tzS0YbdQf9XHknOXc1dOFjZ7/Tj+U7v2DPXHcZIozIk1UcpBOPI5AMWL98OYUBOHiqUZisaz97+miN7Ndt2X/OcQlCzFvDZJ5/ZzwThfPSfmO6T0fJSQaSIskohya795oXpHibTWnKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ujJbCJNF4sWBy6RwEm197UJw1XLCnTru4XS4yhruDZ0=; b=ebAesyQyxg+5qM625uEsl3qLHH7NFr25eksRavZwasUbvBuBNjsHa9Kyp6IGSKnw0Lf53Q/tGYNQ47lbsLVn3dwB8yT1X/vRu90hFGLc2gLK4P0hAgRINMtzhjiccUD4PXF+PoJ/vrD+Kugzo9CYfOrn7BAQrFoyi3Ebz2jKWU24DsPEWfTW9YFgk+OClEQtWEqdfWNypEQmSGOPpycw80LC0FMwdK/EGV9ynsvP3NGq+r91FXVwR+tR6SSt9bmg9FN2YVh2i1NAvc3hm5rQA+12yJvqg45eKLwGba4FSc+/5PB5r5ne2ipaifCo5rF2JaUG7V7Q7QEjWmvu8GMuMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=suse.com; dmarc=pass action=none header.from=suse.com; dkim=pass header.d=suse.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ujJbCJNF4sWBy6RwEm197UJw1XLCnTru4XS4yhruDZ0=; b=PLz5dl1AvYSZapm8SJDp2Rd94YY9f0BxKG9mkKdfdGvZY6Dc7mMboFGpS1WRYYzPolODQ6mZvJxkAIEd63wxapwXzCVQnfF3eHNhuzkdLX2VWXJyrars0g5FC8UCb2jA2TCg9uAW0TGh78mlNq97cKjU/OMvWI/inS5vNvDsKiKDn13+hSqIvaGyT0MsTwN3kZuuobsCmTQVW84Bn3lITMRB7NRx77TTxi9vVnDJiOL0OwFh5LBybaaWFBTNdL8Z6PRVGvPxPU+OM4qVWSw7sOxggsdpiSJDlfmEh2VeT94FdwgXwEZkac3n97inbaHMpe03cQ+zHmS7H3gEzx4qiA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=suse.com; Message-ID: <7294a70c-0089-e375-bb5a-bf9544d4f251@suse.com> Date: Wed, 24 May 2023 15:45:58 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 From: Jan Beulich Subject: [PATCH RFC v2] vPCI: account for hidden devices To: "xen-devel@lists.xenproject.org" Cc: =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Stefano Stabellini Content-Language: en-US X-ClientProxiedBy: FRYP281CA0017.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10::27) To VE1PR04MB6560.eurprd04.prod.outlook.com (2603:10a6:803:122::25) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VE1PR04MB6560:EE_|VE1PR04MB7280:EE_ X-MS-Office365-Filtering-Correlation-Id: bf1a3f79-f118-4d7b-f40f-08db5c5d35b4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: V0F8gMfpdIxfLDD88Svo1Aes34fKQy3qpJFBfN3sEfug1y4zyBQUyQfWl1kNhe/BkfNZDdth6ddB2TnhzNumCFGEX0SfxOaTB7C8bASAMxOvpdho0vqeXA6ENWIpbV7u47sJTxNouAkAsDgph+EbbEYe2hesMNTbwWgtLzfUzgXlO5u1qD1ATLxBNpEb5l/ekBzjjWNZ9HX/Z91iS2DbNHDRnkb3N9SAn8eL3CsDABIj+aaL3XsKSnBQyUx7rBnRIxVj9KjwVgI+PTqrRPP7qRmAi/W2muzKTGGjrhWmCzZBR5eqCzrD37ubaz+tV8xcW03c6nVeGftH7KqM06a6KH6rb3MHEBwFvqrNNmfQ0QZxCKfqWQb+5yxWwAhwqpzv0rnU55Pre4I8eHi+TzDSPbp3Wd1SqUyKeACmwbPMtkwym8YxqAw0szNfjWdRloihZ22HcsIKuomY57D1S/ZWwtPFxA0WVXYp+KCe58pDmsi/SC5iNDtngT/TrUKaheklL15NhmsgFGA5nBttd4pVkSii0obFELB8NQYJk2MSwcRw8INq5/IUjxX7FlipKdFacUXFJDILwVjsJTJdqq8YhlusdEi3LLZoblgRVWRa4QBfm4FHo1F/nYFMMljlH1sZuAFp4LxteT2RjC9qw3PJpw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VE1PR04MB6560.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(376002)(346002)(396003)(366004)(136003)(451199021)(26005)(6512007)(6506007)(2906002)(186003)(2616005)(36756003)(83380400001)(38100700002)(316002)(6666004)(66946007)(66556008)(66476007)(6916009)(4326008)(86362001)(41300700001)(6486002)(54906003)(478600001)(31696002)(31686004)(8936002)(8676002)(5660300002)(66899021)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?y/1rxOZOpFfrSHYxPLCAafaBvlKD?= =?utf-8?q?ycW28qi6z9GkXtsmKJduoGmYMKYaZBOxyMSJaqVWCe+C09yPXOFOydys9B29RpHIp?= =?utf-8?q?dcYM7AIsjlWCYOIEC8xnYUltqi+uaijr8VqN4ddc5lT0b+aY1fBm098UZThCTn8TU?= =?utf-8?q?fiQqTnf/jF4e7+aPz466wj0psBGxxnbRq9ndTfSafDJCRcV8x6J1SHl95XI/st0D2?= =?utf-8?q?DynpUGECX1vI5Xd8d9XxitTA4Q8thxcduwm2h3bagK6LNUYYsop4e142n3RYCSxh9?= =?utf-8?q?eEuqtZGVLO0sV74vpu4MSBA2cOKtlqx43hNcCH7UhdF+NmVrb84ga8WorzTeH0Mf+?= =?utf-8?q?JWkGRI5fM6XTgwYluSCm5eiOpqLqZRRm/3GgO+YU4ejGN76rB/Db60UgRJewJICTf?= =?utf-8?q?hqXvRonQWyLZl6fcLIlJR4Oa+k/1gG3PAU6UVn6SG0z6TTEVXZiygHne5S8XtOT+o?= =?utf-8?q?x3770onviYtIE9inFUz9sHQ/nDh2ZQWLyn0bF43mcMUUrlWJRBrvPfqVdJWlwfgMZ?= =?utf-8?q?rAUn3VQ1ALeJVrB6FCG4h1Md5Hz40+qU78I/IQztEpKkikib/DBvg7dZjt+2P8nNN?= =?utf-8?q?q/FHccUjZF4Fd1IFVvApGFs9sSr7yVGgyvWNfY98JkC0HSdeMFNHNOOMlctOfQi0T?= =?utf-8?q?A4bvRnyuOeWmRmjOnTPb+Fpcrx5XOz7qhLij8/E1x6+LsIMYAz0LBMsKYJ58qGhEw?= =?utf-8?q?l/tFsDdyTAVoxucJmB0j0xOmWvcg7rfBqIR4/hLuzI7tNDpI0f0BUapw50q52ivbJ?= =?utf-8?q?NhRPbOSjpoG+y+utDIpnikjlirBBZ/A+M75OB9JVGK5koT+uDMS0zgv9I7vzHgNH2?= =?utf-8?q?dZ3SSqmDfhwFNc1PGEIk+GYAt49CcrP2oBPXVMTI2c9g5xK8LCWY+wip5E5K8ejiY?= =?utf-8?q?sgCsCfnPgKfaxfwNDYLb9pAJZtr/rj1ML5B6bbCcCakVJR07w2VdHF/ZEgLhqP036?= =?utf-8?q?YsZrk8LA671UlDAAqHWv6Mp8/jPYR1iEZ5vS7oCkvmiBHy2+U6RYsQdtX8vJ2pv36?= =?utf-8?q?7WJwvh6GP82pavWTpwBUnqB7aAMcjHIveUxi2kR+mEsh138SnAKHGHGyNAEggbm6H?= =?utf-8?q?CHtIu6P60SfPQvhjDPvPBhVMlLw8Pa5QRnzGawBdaqMeeBM0Q8h4qs6EaMNrE2LZX?= =?utf-8?q?oHfgrO7+wedQWto0RguYKyy/QNQuUZWIfI4+pgTsblFlVdGoPtiInbYr+50M9+w3u?= =?utf-8?q?Iwb7qw8MSTSBG4nNtFkE9DktXyOIZski1JwvDmxJ52hO1AkHnu0eM2qu0a1xvqw+v?= =?utf-8?q?5ONMq5SFgx8S2DUNCOyqvhPkifhkqGd9dR9nY0K0UDtRKF7xdRwBaNJttOD3Y2Z3z?= =?utf-8?q?FhxodHEPfRarlUedkL8KzQfwd0htwHWf23NMFmQu88/zUhGOHO/sEJ700HS40z1Z6?= =?utf-8?q?tkxf2QJdulIbXgSZvkbRn7UnwjiRUzoRjZpvdzxqLeX0uIRRngBSOzQdbzS2ebqer?= =?utf-8?q?U2pbFftTva/MuPxQUQtlkgM5W6pFgv11B+f2l8zYZL+bs/egPXLOs+ThaM0LMpeWu?= =?utf-8?q?gWVgSBWyms+H?= X-OriginatorOrg: suse.com X-MS-Exchange-CrossTenant-Network-Message-Id: bf1a3f79-f118-4d7b-f40f-08db5c5d35b4 X-MS-Exchange-CrossTenant-AuthSource: VE1PR04MB6560.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2023 13:46:00.9445 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4ySPWVK/vrMnr7qu+IzCka0SmU+qYzz5tmUHLw3zgKWAPWgQpdoxKokwUYL1p4uz7NVauh6lKCnXOV9fgPSfug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR04MB7280 Hidden devices (e.g. an add-in PCI serial card used for Xen's serial console) are associated with DomXEN, not Dom0. This means that while looking for overlapping BARs such devices cannot be found on Dom0's list of devices; DomXEN's list also needs to be scanned. Suppress vPCI init altogether for r/o devices (which constitute a subset of hidden ones). Signed-off-by: Jan Beulich --- RFC: The modify_bars() change is intentionally mis-formatted, as the necessary re-indentation would make the diff difficult to read. At this point I'd merely like to gather input towards possible better approaches to solve the issue (not the least because quite possibly there are further places needing changing). RFC: Whether to actually suppress vPCI init is up for debate; adding the extra logic is following Roger's suggestion (I'm not convinced it is useful to have). If we want to keep that, a 2nd question would be whether to keep it in vpci_add_handlers(): Both of its callers (can) have a struct pci_seg readily available, so checking ->ro_map at the call sites would be easier. RFC: _setup_hwdom_pci_devices()' loop may want splitting: For modify_bars() to consistently respect BARs of hidden devices while setting up "normal" ones (i.e. to avoid as much as possible the "continue" path introduced here), setting up of the former may want doing first. RFC: vpci_write()'s check of the r/o map may want moving out of mainline code, into the case dealing with !pdev->vpci. --- v2: Extend existing comment. Relax assertion. Don't initialize vPCI for r/o devices. --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -218,6 +218,7 @@ static int modify_bars(const struct pci_ struct vpci_header *header = &pdev->vpci->header; struct rangeset *mem = rangeset_new(NULL, NULL, 0); struct pci_dev *tmp, *dev = NULL; + const struct domain *d; const struct vpci_msix *msix = pdev->vpci->msix; unsigned int i; int rc; @@ -285,9 +286,11 @@ static int modify_bars(const struct pci_ /* * Check for overlaps with other BARs. Note that only BARs that are - * currently mapped (enabled) are checked for overlaps. + * currently mapped (enabled) are checked for overlaps. Note also that + * for Dom0 we also need to include hidden, i.e. DomXEN's, devices. */ - for_each_pdev ( pdev->domain, tmp ) +for ( d = pdev->domain; ; d = dom_xen ) {//todo + for_each_pdev ( d, tmp ) { if ( tmp == pdev ) { @@ -304,6 +307,7 @@ static int modify_bars(const struct pci_ */ continue; } +if ( !tmp->vpci ) { ASSERT(d == dom_xen); continue; }//todo for ( i = 0; i < ARRAY_SIZE(tmp->vpci->header.bars); i++ ) { @@ -330,6 +334,7 @@ static int modify_bars(const struct pci_ } } } +if ( !is_hardware_domain(d) ) break; }//todo ASSERT(dev); --- a/xen/drivers/vpci/vpci.c +++ b/xen/drivers/vpci/vpci.c @@ -70,6 +70,7 @@ void vpci_remove_device(struct pci_dev * int vpci_add_handlers(struct pci_dev *pdev) { unsigned int i; + const unsigned long *ro_map; int rc = 0; if ( !has_vpci(pdev->domain) ) @@ -78,6 +79,11 @@ int vpci_add_handlers(struct pci_dev *pd /* We should not get here twice for the same device. */ ASSERT(!pdev->vpci); + /* No vPCI for r/o devices. */ + ro_map = pci_get_ro_map(pdev->sbdf.seg); + if ( ro_map && test_bit(pdev->sbdf.bdf, ro_map) ) + return 0; + pdev->vpci = xzalloc(struct vpci); if ( !pdev->vpci ) return -ENOMEM;