From patchwork Sat Dec 2 01:27:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Volodymyr Babchuk X-Patchwork-Id: 13476712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7C3FCC07E97 for ; Sat, 2 Dec 2023 01:27:40 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.645832.1008257 (Exim 4.92) (envelope-from ) id 1r9Emv-0002ZH-1u; Sat, 02 Dec 2023 01:27:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 645832.1008257; Sat, 02 Dec 2023 01:27:20 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1r9Emu-0002Wk-TN; Sat, 02 Dec 2023 01:27:20 +0000 Received: by outflank-mailman (input) for mailman id 645832; Sat, 02 Dec 2023 01:27:19 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1r9Ems-000272-T2 for xen-devel@lists.xenproject.org; Sat, 02 Dec 2023 01:27:19 +0000 Received: from mx0a-0039f301.pphosted.com (mx0a-0039f301.pphosted.com [148.163.133.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id edc4a75c-90b1-11ee-98e5-6d05b1d4d9a1; Sat, 02 Dec 2023 02:27:17 +0100 (CET) Received: from pps.filterd (m0174678.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B1NLGv2008549; Sat, 2 Dec 2023 01:27:14 GMT Received: from eur03-dba-obe.outbound.protection.outlook.com (mail-dbaeur03lp2169.outbound.protection.outlook.com [104.47.51.169]) by mx0a-0039f301.pphosted.com (PPS) with ESMTPS id 3uqfjgj05q-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 02 Dec 2023 01:27:14 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com (2603:10a6:803:31::18) by DU0PR03MB8599.eurprd03.prod.outlook.com (2603:10a6:10:3e6::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.27; Sat, 2 Dec 2023 01:27:09 +0000 Received: from VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::8e03:368:1fd7:1822]) by VI1PR03MB3710.eurprd03.prod.outlook.com ([fe80::8e03:368:1fd7:1822%6]) with mapi id 15.20.7046.027; Sat, 2 Dec 2023 01:27:09 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: edc4a75c-90b1-11ee-98e5-6d05b1d4d9a1 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MJ8P1LR0sHZ0c63MAEMiae1onWOKXr68uT2ZXgD/7RSnk/3zwyF/DD/aBZH1CsGHEBJzdPPVwH9hdONRJ73lp/vgYO1GoZ0wylmwMq1axear86DTDrjaMBZ1CQv70imOC8OcXq4j6CAWzzUt1lc+n/XEW3HMpMHAUAqUNE7hNHoqW8K4YKp1PPiGtocp4a8PUiwSAEFXq7aG4aTZmHOAxnWurqcRu3JRK3U/q07MefbhMPIS/Hz7oX9VBDSMi4OatzdZDalI2NPLZqwkozOJZpfU64hDff5PKpLbQh0XR8aVt48DOtcnDPxIJycNQEL/1eDX1/C3h8RLxlgi8Ga2bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6JZJ4KNkIkIzlijRLVq+YijCSEjROoN8dREC9RVL4YU=; b=eU5RxjpI8ooCAk0V2nT0qfntZrlj9CFAcWJ1Fmp49Ffp7EoDVGs1+QqCX7rJtFkResp3oMRGXQ1HSLjz/5oyNrKuvHEXDlCcEFre5KquO5Iv/XGtCPWmgovRkuRTsUr9BnSDFUs+ER6ocpMsSbt4UY++GRKvmpQjz/LlLfoH5uT01wu9IJqlDu9I8yZTTzci9muE85WZTYT9uayjC5pOuRCO29wG2CWJK6//PNIHIyHgMNO97x8BoHuEqqXFGBbnvGoD/qh9pc9URcj8G3SFDbXvXA8T2o5T5wMx8VaFD5ZwJQoQIidLRlkJWn1luoUi2GZOHP5VgiWGvdb3plwWSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6JZJ4KNkIkIzlijRLVq+YijCSEjROoN8dREC9RVL4YU=; b=Gnrv63yCISztxCQ2XP19Dnfrz17fRPZWzYohEHGTfL/bbgEG0loSmBHO+U8/TABph3rSF9kAr5LSsZKNJ2hedCG2qzOWSwz8kZizaL+RTU5bAUWM9r+G4TStBCdgwnHfdg0e4QRzJZJ+3bfpfFGT4sDETnku1sFj4jMq2RXoFkLYTGEyUVgDM0XrSYFYGLbPqO7ZAB4gcOQs8d2iQKSi0SB7G6boZO3wki+S5rfcv8z33AWR/m1jSSCnxXsL0Etiw+yplrxJ5R6yPxJz9B01BtLoIgQkRwpFN591av7C23cB5RKZA1h63tyqskmq0vsCqI8S1OpWeQbiNpLU5TYZ+g== From: Volodymyr Babchuk To: "xen-devel@lists.xenproject.org" CC: Stewart Hildebrand , Oleksandr Andrushchenko , =?iso-8859-1?q?Roger_Pau_Monn=E9?= , "Daniel P. Smith" , Volodymyr Babchuk Subject: [PATCH v11 11/17] vpci/header: program p2m with guest BAR view Thread-Topic: [PATCH v11 11/17] vpci/header: program p2m with guest BAR view Thread-Index: AQHaJL6om72B3RYnP0aRq1inaCmShA== Date: Sat, 2 Dec 2023 01:27:05 +0000 Message-ID: <20231202012556.2012281-12-volodymyr_babchuk@epam.com> References: <20231202012556.2012281-1-volodymyr_babchuk@epam.com> In-Reply-To: <20231202012556.2012281-1-volodymyr_babchuk@epam.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.42.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3710:EE_|DU0PR03MB8599:EE_ x-ms-office365-filtering-correlation-id: 28c3a8a2-7ce8-453b-af25-08dbf2d5cd8e x-ld-processed: b41b72d0-4e9f-4c26-8a69-f949f367c91d,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: peh66Gl38lFbeQ7++fR9s6DvKkkNl42E8JKcVxQX55SpMMRCmyQys8KdRUNqIbJMgJW+mFJ41RTGl08XcXEzOpOnsp/hUYs4kTg75K7RlT9e3sROz6yedFVIhcQ2ONou4z8hAxQ20XadoqlytwobXees7G2mKwt0+IrKMcNQhXzpBbUR7EbnlhkEBwD/R9QL4SlU5k99QRiBuyTUa/RMncDmIaQxLw4ehrR9kSE1WAXkwYmFHAIFIhwGKQR4t6AefpmC8J75UfKBqvazEzy4uTVem1up437hk1okHEy8yeV8TvgOLAkW5JMqnG7E5SQ5yM0Oh2AgkuhTYMMLoZlDX6Fv6/EjQeGukGQ7TapZy+1bRiv3ITF/YMlr6eo5gTjII72vkinqbh43NcTiR//QxjOSn12ssgrTDNjyMvAsUrY6mD/COpodsMl5skRTnl6HuXHkQ92THfuYnvs406lTQCA15SzgnRHYbwScHrjQJZbCRy5b0CYg/DXAkoY3q5evaOpCiQrkpQo4yUR8X89ktUlj0QJXEUAj2thGVSKJy1yyTjKo74182kssPXLkT5XwkkeLSAmXBtkVhgs34v8kDjLrngCWT3XvduMsupTKWbbxhtm97EjMuTXqOl7+cmXxvz4S7hOS4snm7jEbGHvXBM2Kn6SFD+4sRj1swUkw0FA= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3710.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(396003)(39860400002)(346002)(376002)(366004)(230922051799003)(230273577357003)(230173577357003)(186009)(64100799003)(451199024)(1800799012)(5660300002)(86362001)(4326008)(8676002)(8936002)(2906002)(30864003)(38070700009)(41300700001)(36756003)(2616005)(107886003)(1076003)(6512007)(6506007)(55236004)(83380400001)(6486002)(478600001)(26005)(71200400001)(6666004)(38100700002)(122000001)(316002)(91956017)(54906003)(64756008)(66446008)(6916009)(66476007)(66556008)(76116006)(66946007);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?dF9536ehMnXKgtonKO0GR6m?= =?iso-8859-1?q?nGuk8C055hgxSQXsq/2mGy1biysrurt/UpWLWSc3EQxeTM/m9Wh3cfCjsOiq?= =?iso-8859-1?q?yKrz7kJ576zx5pX+jeLOfITZ/jIQAJG607DxsjaV9HVhCBAOT5IkxNRdI6f5?= =?iso-8859-1?q?NFUx9h2jY4Ld+wmKuv5GP8oNZ3D+pFfBlLudxoWPN3EMkvTcTiTt8et51gus?= =?iso-8859-1?q?STJwTdedX45glc8FzG7n3ZDTAet22FF6Mb5wFWiS7R7o8reRlcJfFzLgEfsI?= =?iso-8859-1?q?SO9Ddfix1ujuwJifb2IOoN51otVPKUrmyjvuiiRKOHLRZNluM6e09oYHyUVw?= =?iso-8859-1?q?RG43Ck1ZBPbJul/o7pf6w2XsIP2nWHN3bpDusKTTCc2XbqWXn6abEqk7AzNw?= =?iso-8859-1?q?ly8F2FOxy3AS/Ai0B8RWXFcA4Y2dLMUxQCob2NJxt79MlwHWGEHyMuIoFdwB?= =?iso-8859-1?q?8vV2dO0+mapafyuJWM3lnPwG1bYU3BjKIfA9ZGtuU5lg1DA6zvtbMib2EFPm?= =?iso-8859-1?q?rjSfkToqsmhSg3YwpdURD3OtXwwllR4J55epmdMQqEz9X4q2FZ1jKnvZunpa?= =?iso-8859-1?q?X8cwbKO4jvoD34q0WipU3z1kdI/mTDRZVDU4tGLfwcGu8UqiteI8zpd4KYri?= =?iso-8859-1?q?hdtJrPtUdyhemshTCKwxVZI9Taf0LZf11PFs0N3Fj5iIzZMSGCpLth02PK1B?= =?iso-8859-1?q?LnxPWduHH2RU9FUKh4uCm7jrvxCl1d9cHllHXT3Hnz49QWzlBBxmq214H8RV?= =?iso-8859-1?q?MUQaL7X+sUgE7tsUtO577JqsU/zQvx06jmJsXZX33EikrKmgR1WAUUZmGkdz?= =?iso-8859-1?q?Qe58gnz2Sz6Yig8InGi3YeYvbXCpjpPkU5zK09GxAQw/PeZJVvl0pZrvXz2v?= =?iso-8859-1?q?WUmqEoVzVNyvtyrsC8aY8byY7dpxdrzy7ENUQqoh81nm8FXTnK5JbxGENLwV?= =?iso-8859-1?q?/U9kDPzXgf1Xxom1ds/DhzVkjRsPz4sGOFlyEo0OHx+OMEoOOG1FrfGwTuJc?= =?iso-8859-1?q?PtN6GvYrG6EMbssbSx8oUbyl0W+tie6NZsuKGeSFXS9g4iEtKD0w+HTxGW0x?= =?iso-8859-1?q?BA08t68C0eMPfD7qRbHf5W4npX2TzNOHqeiwnJC6gXtOsLgq56B/4TATV9av?= =?iso-8859-1?q?I4U6jsasKfCwixql6/HU0sPmKnWybysSg23lUN4N0mwL4gbIVlloGusHTZCv?= =?iso-8859-1?q?XqVmMrzl8dITjxZ3LfQCslp8xKXgUOyaid2UkP/AOkFzfw+n7An0PCkoF+1D?= =?iso-8859-1?q?5MOsNywc35aXGfUeToIvoGnJiAF8BIwR50qT4Ci7hy8Kzh2+kpgNC1ugn5Qp?= =?iso-8859-1?q?JgtHXlMOlB/SO8Wk0eWaHtrkruNXN0OhHpME+vXj9B2OkM2j0rAWIo0ReCgE?= =?iso-8859-1?q?EKwtLrjcPwzLwWsQ9i5Rv4e0UmMMPAZGlmBSJnC1Zst1Ii4+YpNC1T7qFTZC?= =?iso-8859-1?q?GtSBAwQKH3HiCwJcaDUKbDQin77X8W1DG+AYH0MhkNdoJlkug10evq3vv7RX?= =?iso-8859-1?q?WuXJb7c7NyIJ/rH4XKTkBscO6ClxYmvsiyt30BnB2gt9bJpPx79QUnvQlW9h?= =?iso-8859-1?q?Q8paF+o9iNGiZ1hh8nr4R3qIsdIxc2o6/ZcyEQzzWt6jy4GgtNVelgm7mhy/?= =?iso-8859-1?q?ubGLq8zADg8SNBTnNZdXFrdVmozqPvOMXeqV0Hw=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3710.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 28c3a8a2-7ce8-453b-af25-08dbf2d5cd8e X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Dec 2023 01:27:05.6553 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: c45RotjQVgWwH7ivQzjwrjyQeVc+xLGpUI+d5Rc3CgKUgOwSHYeNbp9OvckWzAzoh8i4wGBRjbFKWtg1jUvuZfYLupRPQDSNxG556BMkK84= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR03MB8599 X-Proofpoint-GUID: zCpSr2_hKR2E1wjHDHosQKpIg6EeqP3W X-Proofpoint-ORIG-GUID: zCpSr2_hKR2E1wjHDHosQKpIg6EeqP3W X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-01_24,2023-11-30_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 phishscore=0 lowpriorityscore=0 spamscore=0 clxscore=1015 suspectscore=0 impostorscore=0 adultscore=0 bulkscore=0 priorityscore=1501 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000 definitions=main-2312020008 From: Oleksandr Andrushchenko Take into account guest's BAR view and program its p2m accordingly: gfn is guest's view of the BAR and mfn is the physical BAR value. This way hardware domain sees physical BAR values and guest sees emulated ones. Hardware domain continues getting the BARs identity mapped, while for domUs the BARs are mapped at the requested guest address without modifying the BAR address in the device PCI config space. Signed-off-by: Oleksandr Andrushchenko Signed-off-by: Volodymyr Babchuk --- In v11: - Add vmsix_guest_table_addr() and vmsix_guest_table_base() functions to access guest's view of the VMSIx tables. - Use MFN (not GFN) to check access permissions - Move page offset check to this patch - Call rangeset_remove_range() with correct parameters In v10: - Moved GFN variable definition outside the loop in map_range() - Updated printk error message in map_range() - Now BAR address is always stored in bar->guest_addr, even for HW dom, this removes bunch of ugly is_hwdom() checks in modify_bars() - vmsix_table_base() now uses .guest_addr instead of .addr In v9: - Extended the commit message - Use bar->guest_addr in modify_bars - Extended printk error message in map_range - Moved map_data initialization so .bar can be initialized during declaration Since v5: - remove debug print in map_range callback - remove "identity" from the debug print Since v4: - moved start_{gfn|mfn} calculation into map_range - pass vpci_bar in the map_data instead of start_{gfn|mfn} - s/guest_addr/guest_reg Since v3: - updated comment (Roger) - removed gfn_add(map->start_gfn, rc); which is wrong - use v->domain instead of v->vpci.pdev->domain - removed odd e.g. in comment - s/d%d/%pd in altered code - use gdprintk for map/unmap logs Since v2: - improve readability for data.start_gfn and restructure ?: construct Since v1: - s/MSI/MSI-X in comments --- xen/drivers/vpci/header.c | 79 +++++++++++++++++++++++++++++---------- xen/include/xen/vpci.h | 13 +++++++ 2 files changed, 73 insertions(+), 19 deletions(-) diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 7c84cee5d1..21b3fb5579 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -33,6 +33,7 @@ struct map_data { struct domain *d; + const struct vpci_bar *bar; bool map; }; @@ -40,13 +41,24 @@ static int cf_check map_range( unsigned long s, unsigned long e, void *data, unsigned long *c) { const struct map_data *map = data; + /* Start address of the BAR as seen by the guest. */ + unsigned long start_gfn = PFN_DOWN(map->bar->guest_addr); + /* Physical start address of the BAR. */ + mfn_t start_mfn = _mfn(PFN_DOWN(map->bar->addr)); int rc; for ( ; ; ) { unsigned long size = e - s + 1; + /* + * Ranges to be mapped don't always start at the BAR start address, as + * there can be holes or partially consumed ranges. Account for the + * offset of the current address from the BAR start. + */ + mfn_t map_mfn = mfn_add(start_mfn, s - start_gfn); + unsigned long m_end = mfn_x(map_mfn) + size - 1; - if ( !iomem_access_permitted(map->d, s, e) ) + if ( !iomem_access_permitted(map->d, mfn_x(map_mfn), m_end) ) { printk(XENLOG_G_WARNING "%pd denied access to MMIO range [%#lx, %#lx]\n", @@ -54,7 +66,8 @@ static int cf_check map_range( return -EPERM; } - rc = xsm_iomem_mapping(XSM_HOOK, map->d, s, e, map->map); + rc = xsm_iomem_mapping(XSM_HOOK, map->d, mfn_x(map_mfn), m_end, + map->map); if ( rc ) { printk(XENLOG_G_WARNING @@ -72,8 +85,8 @@ static int cf_check map_range( * - {un}map_mmio_regions doesn't support preemption. */ - rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, _mfn(s)) - : unmap_mmio_regions(map->d, _gfn(s), size, _mfn(s)); + rc = map->map ? map_mmio_regions(map->d, _gfn(s), size, map_mfn) + : unmap_mmio_regions(map->d, _gfn(s), size, map_mfn); if ( rc == 0 ) { *c += size; @@ -82,8 +95,9 @@ static int cf_check map_range( if ( rc < 0 ) { printk(XENLOG_G_WARNING - "Failed to identity %smap [%lx, %lx] for d%d: %d\n", - map->map ? "" : "un", s, e, map->d->domain_id, rc); + "Failed to %smap [%lx %lx] -> [%lx %lx] for %pd: %d\n", + map->map ? "" : "un", s, e, mfn_x(map_mfn), + mfn_x(map_mfn) + size, map->d, rc); break; } ASSERT(rc < size); @@ -162,10 +176,6 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd, bool vpci_process_pending(struct vcpu *v) { struct pci_dev *pdev = v->vpci.pdev; - struct map_data data = { - .d = v->domain, - .map = v->vpci.cmd & PCI_COMMAND_MEMORY, - }; struct vpci_header *header = NULL; unsigned int i; @@ -185,6 +195,11 @@ bool vpci_process_pending(struct vcpu *v) for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { struct vpci_bar *bar = &header->bars[i]; + struct map_data data = { + .d = v->domain, + .map = v->vpci.cmd & PCI_COMMAND_MEMORY, + .bar = bar, + }; int rc; if ( rangeset_is_empty(bar->mem) ) @@ -235,7 +250,6 @@ bool vpci_process_pending(struct vcpu *v) static int __init apply_map(struct domain *d, const struct pci_dev *pdev, uint16_t cmd) { - struct map_data data = { .d = d, .map = true }; struct vpci_header *header = &pdev->vpci->header; int rc = 0; unsigned int i; @@ -245,6 +259,7 @@ static int __init apply_map(struct domain *d, const struct pci_dev *pdev, for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { struct vpci_bar *bar = &header->bars[i]; + struct map_data data = { .d = d, .map = true, .bar = bar }; if ( rangeset_is_empty(bar->mem) ) continue; @@ -310,12 +325,16 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) * First fill the rangesets with the BAR of this device or with the ROM * BAR only, depending on whether the guest is toggling the memory decode * bit of the command register, or the enable bit of the ROM BAR register. + * + * For non-hardware domain we use guest physical addresses. */ for ( i = 0; i < ARRAY_SIZE(header->bars); i++ ) { struct vpci_bar *bar = &header->bars[i]; unsigned long start = PFN_DOWN(bar->addr); unsigned long end = PFN_DOWN(bar->addr + bar->size - 1); + unsigned long start_guest = PFN_DOWN(bar->guest_addr); + unsigned long end_guest = PFN_DOWN(bar->guest_addr + bar->size - 1); if ( !bar->mem ) continue; @@ -335,11 +354,25 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) continue; } - rc = rangeset_add_range(bar->mem, start, end); + /* + * Make sure that the guest set address has the same page offset + * as the physical address on the host or otherwise things won't work as + * expected. + */ + if ( PAGE_OFFSET(bar->guest_addr) != PAGE_OFFSET(bar->addr) ) + { + gprintk(XENLOG_G_WARNING, + "%pp: Can't map BAR%d because of page offset mismatch: %lx vs %lx\n", + &pdev->sbdf, i, PAGE_OFFSET(bar->guest_addr), + PAGE_OFFSET(bar->addr)); + return -EINVAL; + } + + rc = rangeset_add_range(bar->mem, start_guest, end_guest); if ( rc ) { printk(XENLOG_G_WARNING "Failed to add [%lx, %lx]: %d\n", - start, end, rc); + start_guest, end_guest, rc); return rc; } @@ -351,12 +384,12 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) if ( rangeset_is_empty(prev_bar->mem) ) continue; - rc = rangeset_remove_range(prev_bar->mem, start, end); + rc = rangeset_remove_range(prev_bar->mem, start_guest, end_guest); if ( rc ) { gprintk(XENLOG_WARNING, "%pp: failed to remove overlapping range [%lx, %lx]: %d\n", - &pdev->sbdf, start, end, rc); + &pdev->sbdf, start_guest, end_guest, rc); return rc; } } @@ -365,8 +398,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) /* Remove any MSIX regions if present. */ for ( i = 0; msix && i < ARRAY_SIZE(msix->tables); i++ ) { - unsigned long start = PFN_DOWN(vmsix_table_addr(pdev->vpci, i)); - unsigned long end = PFN_DOWN(vmsix_table_addr(pdev->vpci, i) + + unsigned long start = PFN_DOWN(vmsix_guest_table_addr(pdev->vpci, i)); + unsigned long end = PFN_DOWN(vmsix_guest_table_addr(pdev->vpci, i) + vmsix_table_size(pdev->vpci, i) - 1); for ( j = 0; j < ARRAY_SIZE(header->bars); j++ ) @@ -424,8 +457,8 @@ static int modify_bars(const struct pci_dev *pdev, uint16_t cmd, bool rom_only) for ( i = 0; i < ARRAY_SIZE(tmp->vpci->header.bars); i++ ) { const struct vpci_bar *remote_bar = &tmp->vpci->header.bars[i]; - unsigned long start = PFN_DOWN(remote_bar->addr); - unsigned long end = PFN_DOWN(remote_bar->addr + + unsigned long start = PFN_DOWN(remote_bar->guest_addr); + unsigned long end = PFN_DOWN(remote_bar->guest_addr + remote_bar->size - 1); if ( !remote_bar->enabled ) @@ -512,6 +545,8 @@ static void cf_check bar_write( struct vpci_bar *bar = data; bool hi = false; + ASSERT(is_hardware_domain(pdev->domain)); + if ( bar->type == VPCI_BAR_MEM64_HI ) { ASSERT(reg > PCI_BASE_ADDRESS_0); @@ -542,6 +577,10 @@ static void cf_check bar_write( */ bar->addr &= ~(0xffffffffULL << (hi ? 32 : 0)); bar->addr |= (uint64_t)val << (hi ? 32 : 0); + /* + * Update guest address as well, so hardware domain sees BAR identity mapped + */ + bar->guest_addr = bar->addr; /* Make sure Xen writes back the same value for the BAR RO bits. */ if ( !hi ) @@ -793,6 +832,7 @@ static int cf_check init_bars(struct pci_dev *pdev) } bars[i].addr = addr; + bars[i].guest_addr = addr; bars[i].size = size; bars[i].prefetchable = val & PCI_BASE_ADDRESS_MEM_PREFETCH; @@ -814,6 +854,7 @@ static int cf_check init_bars(struct pci_dev *pdev) rom->type = VPCI_BAR_ROM; rom->size = size; rom->addr = addr; + rom->guest_addr = addr; header->rom_enabled = pci_conf_read32(pdev->sbdf, rom_reg) & PCI_ROM_ADDRESS_ENABLE; diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 18a0eca3da..c39fab4832 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -205,6 +205,19 @@ static inline paddr_t vmsix_table_addr(const struct vpci *vpci, unsigned int nr) (vpci->msix->tables[nr] & ~PCI_MSIX_BIRMASK); } +static inline paddr_t vmsix_guest_table_base(const struct vpci *vpci, + unsigned int nr) +{ + return (vpci->header.bars[vpci->msix->tables[nr] & + PCI_MSIX_BIRMASK].guest_addr & PCI_BASE_ADDRESS_MEM_MASK); +} + +static inline paddr_t vmsix_guest_table_addr(const struct vpci *vpci, + unsigned int nr) +{ + return vmsix_guest_table_base(vpci, nr) + + (vpci->msix->tables[nr] & ~PCI_MSIX_BIRMASK); +} /* * Note regarding the size calculation of the PBA: the spec mentions "The last * QWORD will not necessarily be fully populated", so it implies that the PBA