From patchwork Fri Oct 25 21:50:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Micha=C5=82_Winiarski?= X-Patchwork-Id: 13851864 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0056220C31B; Fri, 25 Oct 2024 21:51:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=198.175.65.17 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729893074; cv=fail; b=PaYrKizAdxqDnlY0xOqT7ADTVjpU18CGm33QPa5lc5ieWVD2sDQ5nBVqJkQJxChnF5CNI9aOnpLT7w6lHmBSG80WRQxmkobKrvIH450UJG7duFMgdz706BaGOw2QchAu419tRhdxK3884AVp/h7TMttrSZnkYQzbAJiFBNSmwAQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729893074; c=relaxed/simple; bh=gO0LvjUua6S62XJgIlioVb9aEnjmlOSXeHNkenlw+X8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=bXeqp7yvkSnuhdNtzrdh7q2oiaJYDPaYQ6FVYFLA4Dh/LFpW61UCeeweu5GIUegW7D2Alk5dLxp1QSnneNaumTFv9DKhUQVeIKK8vOx4z6gqZ6wNQa2dkrnDRdWGSp/w4PdQj9usAvmu47YhS6ALxzOwpl/+SL6ReSEQKZEIjXc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=E2jEkX1v; arc=fail smtp.client-ip=198.175.65.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="E2jEkX1v" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1729893069; x=1761429069; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=gO0LvjUua6S62XJgIlioVb9aEnjmlOSXeHNkenlw+X8=; b=E2jEkX1vXd8Vl1Ci/Wwp397I5nKHQjKc5SMEqm4Np7jF9Cc4qvL/JDfK 8WAb6GCxYYSRQw5TD8yLFfdsGcans3bu9rdPHy3S2tXRtXAJNeWGfvh6V tNXfkaLM7Sjw35dW5+pMhEn7tjzhev2GjKdG9xXeDDNPkzDpaQWbKIl0f 4qdG7zqHpOUd4L7ybWDkMwu19MvbPmeQ9VT7wM98jrssnuTwpFzJG6pBv 1Okz+an63oDuHxnfpMYRoMNVjWNZITBCoVtCwGbIRfOReOprzrcHgNrnE vFWE8z4csq10w/Jpou+meqsb7FYfpKgqSZkcn9g1yQFqGJFanNT/wkVPJ A==; X-CSE-ConnectionGUID: lobV45HGQFWmYNAoCyOj/A== X-CSE-MsgGUID: ZKrhu+6RSlWXTqP71jKPUw== X-IronPort-AV: E=McAfee;i="6700,10204,11236"; a="29683388" X-IronPort-AV: E=Sophos;i="6.11,233,1725346800"; d="scan'208";a="29683388" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa109.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2024 14:51:08 -0700 X-CSE-ConnectionGUID: JpRVv2PYT1SEzJq6WMWfeA== X-CSE-MsgGUID: zyPzq2r1T2eatpfsZu6EJw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,233,1725346800"; d="scan'208";a="81492860" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orviesa007.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 25 Oct 2024 14:51:08 -0700 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 25 Oct 2024 14:51:07 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Fri, 25 Oct 2024 14:51:07 -0700 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.171) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 25 Oct 2024 14:51:07 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zIaPtggEJsoOgF2fCXpaop6HmO0aC4yCtSEsPyaCADCGQVfGfJ1FcUtoMSF3dEEYa+y4UJFIjeS8/fuolZpRZlJk9WN8sVdV9XDzknbpBWZjAsPBF2PcVv0i7qB0+j4aTX8pHMt7lR6T7VLCYLWDfSFCT7EsgmppEEkk3Lzcp7S/wF95jh/d3XdYQOFxdPY3AnOb2zHljyoxRybUkpC7hFVAZw9RPftGD6XYV4xZZBxC4pP7Mq4lVOyT0KoKcB8bvP2KiRYOV6T/bLZcLEpM1/2OR5geeF1XdJPguf7dymekEXjqNgNe3sD7AtyIpMufFHCiCvQsmZTBhF2+dKi6pQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rxHJttmjn7E6Myjzq3+TueUpYOqq6Ozmnw7CozqMA8Y=; b=upF1qqP2QW4dlSl0MyLIZGFNUQLwox1xvht6bLuyvOPSKFmW1FFaOmBJlY/LJn9L/L1B1/4Uq9ZMtQJCwdeQe9OJEaPuV6kZkYxNNpVf+SSZKFCXK1NN8zkI4s7V0oLI48nPJzQacngydX9FeBwa31xYWgsQSI77VtcJrH6V6OY+wydon9hfKfbc2Tt6jOVPPiQ4AQ33EroJCS9XY7zAeixfygPBQyoNxZXrm/SgsfiFuAy2f3nkvHMjuUp9kpbL2sRElG5Y+YWyC3boWIjhTWElYK77o2AtUwTTngHpKmBNCXhfVB5x7x62DHE1W8HwYJ3KMeUlJFQGarTgek3OFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) by DS0PR11MB7682.namprd11.prod.outlook.com (2603:10b6:8:dc::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.21; Fri, 25 Oct 2024 21:51:05 +0000 Received: from DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39]) by DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39%5]) with mapi id 15.20.8093.018; Fri, 25 Oct 2024 21:51:05 +0000 From: =?utf-8?q?Micha=C5=82_Winiarski?= To: , , , , "Bjorn Helgaas" , =?utf-8?q?Christian_K=C3=B6nig?= , =?utf-8?q?Krzy?= =?utf-8?q?sztof_Wilczy=C5=84ski?= , =?utf-8?q?Ilpo_J=C3=A4rvi?= =?utf-8?q?nen?= CC: Rodrigo Vivi , Michal Wajdeczko , Lucas De Marchi , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matt Roper , =?utf-8?q?Micha=C5=82_Winiarski?= Subject: [PATCH v4 1/7] PCI/IOV: Restore VF resizable BAR state after reset Date: Fri, 25 Oct 2024 23:50:32 +0200 Message-ID: <20241025215038.3125626-2-michal.winiarski@intel.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241025215038.3125626-1-michal.winiarski@intel.com> References: <20241025215038.3125626-1-michal.winiarski@intel.com> X-ClientProxiedBy: WA0P291CA0012.POLP291.PROD.OUTLOOK.COM (2603:10a6:1d0:1::7) To DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB5373:EE_|DS0PR11MB7682:EE_ X-MS-Office365-Filtering-Correlation-Id: 87a0a3e0-998c-449f-8c04-08dcf53f200c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016; X-Microsoft-Antispam-Message-Info: =?utf-8?q?c+fMr9zjjw99pVnJm6ht46KaXAhSOEA?= =?utf-8?q?si5FbyplM1l4mBp5DipaO/DQgXpzPW07UAQZr56yH9Se3IVxOKBG9MR4yEdOv4kIF?= =?utf-8?q?vyLYaGtNhg7JpChAr1UqJOQDItP4zdgAnsP8truOygZsj0xTnIYMQiREj2lcmct8u?= =?utf-8?q?wsQqGsMmh2y5eKwTP2DcMdKMM0gXinGRWXqxisAheuzi/HoMaWgf78nJb7kz6gueu?= =?utf-8?q?tGybz+u/PnOXY3icDuWVy+ghP34eXpmM2DW5S69/aep5LYUMLGkqzcZlQt/sOQXtd?= =?utf-8?q?ZAH51bqJ+17fiCqJvHUBNeME13Y9cdjBhR/ksz+GY7vXDACxI48Qi6k/0KeRwvDiC?= =?utf-8?q?CDxaCOaF7t6dHtR72i5T7ZpElNMCEjF8KsUKJhPraxo9gwK9rTIO4cW4YLt+u0R+I?= =?utf-8?q?wMa2fPwKlHAoJZCm/pOMoxfPdYII8+zxJmJMdF6JpDzui76rMRFZZcCQjA2xj4Seo?= =?utf-8?q?eiXg6mtHs0n9LWmPkP2mt/+ydmb/pA6mKD3iM9M5515AC5HFyb2ihDk+0+oYR9gyZ?= =?utf-8?q?aTSUxQUyfQtho2L2KoVG8kHgdiQIiu0k+TzMHE/QQ/z/ukJcF2awFGVAg33JlkQx1?= =?utf-8?q?2um4Zb3bWLoNDoZ5zhTzeQq6Bd/im+eaDjFdANeX+VX25CZCyrhfFvi2bpe1S+Uqy?= =?utf-8?q?0aTOEaFGvoL3Y/lkOGqlWU24FG9FeYxlGwlpurlzfsIASyXxAiTftQIyFtkAUSabW?= =?utf-8?q?u+MB8EexFdcjpWC2U4rfoOT4Ec8NMhz4uT4SX3YBjBEzTjYN34JPbxnee6bldDxRp?= =?utf-8?q?0yRBbCnZCZXJTpSKmkFWRcX+Qf9D+1pbO/nIF0qbuAlZUHpAZudg4PXE3a4WdAscy?= =?utf-8?q?CZmLCOPC5X2IrKFG6f89ShBdzrw48frL+9Uf4Sfy8HWBNDS06oPJWub70BTTgFU+a?= =?utf-8?q?9ycNs0e2gs5seo0kTG14ZIJ47eLotqpGz2zEczCppbx949NmGwiaoc3jDmtIBJvFz?= =?utf-8?q?sAdnMw0v/nuK7SwuxvFF4UUNU94izg6OdgoxXMWl4LDbtgQLfDLRGn/Lwt1NvfF9F?= =?utf-8?q?YTc+RGAVJqYaGmZkTcjNdCmmpLbxYE8bATgsTv1RGNC5XyiYp4FInfcXPOSqdOET9?= =?utf-8?q?203ts5Z9MIwjxagZYWBgu6kRzApBh+OaEFVdQ+FxE0YiTKLov60b00y2nGRa/2OWd?= =?utf-8?q?ZFoOcNDYWkMJiyeks4Fh3EMe0g7uJC4Y9kz5wttCjSIph/OS2dKlWVRD/MvQl8ps3?= =?utf-8?q?S3BCEgoISB7j5FaQPdBCmtfQnce1tRRksnODUyMBHJ5SxpCVmaPUJ6XpnL0cbADhT?= =?utf-8?q?pXa7YdEZdcbO2?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR11MB5373.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?MKv5hRyjm8HwdFCcrJyfBmaX5OaZ?= =?utf-8?q?qYAarzlrnypjEb9A/mc6ArEm8kCPtBle2sOxeMF3wdXjTZ1QMxUawXKVEpQway/3/?= =?utf-8?q?TuNAYAS3HiwT9dS4c6T++8uO7I4U9YKnUFgEhr84uHAhnc70ufWEsm4tcQWr5rX6y?= =?utf-8?q?P6HPTfuemp0dxTNsKeuXS8PqcZlzrhiLglu36aYxVjEQ5MJMVE2g4b7JGfjG2VXST?= =?utf-8?q?Cz/HbgFgv5M9MYmSQSLyO+M9jBFLf+FDrxt8S0YPnii4NbLN3QEovDjprCvzqO/A9?= =?utf-8?q?4JYEwHDgSkFSZAaWI0b1xt/k3FI4wp5B6Vm/x8FahefFUuhjBnhYP0+ycUKr2gV1L?= =?utf-8?q?cvymVXEJvO/J/H9tnqKzW/TYeypdBSxSe+yI6p+mJe0rYu476TJm89bilIBlu9xiN?= =?utf-8?q?6p29fn8CQ22eC2Uh4XYgAICH/aLN86Pel7uGTebCjhpqJ7pC00qjEGPIoDm59s3Nm?= =?utf-8?q?7XkKAQfCGy78iYu4Cj634lUR+V3VAR2VkVSfkWtVoyAJsrPdqs0+AwlqsjS/ywd7X?= =?utf-8?q?rfrdXGHelsrI51oqHVJmbdlbEbBZImWqy2WXm92OjJNBAYfE5aGfLjgCPdi260BpS?= =?utf-8?q?MAJMh297ZKnkzgG8EE6J+ZfRZpI+pPEL7FuiRJOkXE1E65SW4wOGGDH80pobXmIUG?= =?utf-8?q?hYV4pMrK7QPVfZkiHZ7dl14UC/TPDdGsUkUyFH8cBFOsOFNeQqBrkLqILKZ5hSD6M?= =?utf-8?q?qixJoLR75NkF5JJU0UkzAyDG19Tnv5fxQE/lnWAyPVIuVsiC0FKZCRl8vncJYO4Vk?= =?utf-8?q?/l1UbT8Wu4lbRhiIXrtdiHfhc14ua43GHU26hEIXovB1T3hzBUmLKczLAIwBhMj4C?= =?utf-8?q?MkQBf1Gu84YW3F5kxWCcwiKm2xqeotuNaeDt/xwFTxAa2bb3cVqXATGR892Recoj0?= =?utf-8?q?WGTCc5GOMFSlcge0oCQ9BQEN/UpxzaA3Qrxrbvvy/finzjV2iTyO+++vJKChE2nR0?= =?utf-8?q?NF/A5DViyAps8ZSJ5ngkvfVDsraKnnrg/tVGyNBBdYkfENTizBLD8o8QjMMDB5mPz?= =?utf-8?q?bX5n6ewExholJ3URbrLRaNOPUMN2665bjFaS+7Epo1uukaznu4zQ09On8BwPu0gxb?= =?utf-8?q?NiQDBgYIIe1OUkY7+aiE1jcox0GT/94MbKfpXTmlhcgtuDx2rviHX/8lt4kZduRjK?= =?utf-8?q?qdsN90r7dCNIIAAUg07d4Kjdg9tflFL2P88VTDP2nf/bMcr49lXVcc6Jdoqh54+j6?= =?utf-8?q?GOgGlkObBPiZD9P58vJx3FlXxZiVkHJdmLPbXfCckgie9c93AKU0JdIB2BGmVDYcW?= =?utf-8?q?eAxg+71uPENDe5d8/iTDphPn8aYlUY/L1iQbTwt/T1MiukKv0PrUNwQdrFNnO6xu7?= =?utf-8?q?z9zWQ9GCYM9VjT92EK8xSwxgFjIxRxNaeyOBrFEHO36dch7OTZb0uwsr2ontcByJp?= =?utf-8?q?nLO1Oxe9TE4XYAh6QZ3HfEoevQS7PERhOIbdNvbV4zxQS/d05YiGTVP/SA5cOd1BY?= =?utf-8?q?JNSV6zlqgSsMCWxpTxeckVzOloJBu7BbEN9rireSJfsK+AClQF70S+aanz6fpJ1eT?= =?utf-8?q?FOuAkiV4sdfqLjMsR5sJY6cn9i79hfQyfw=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 87a0a3e0-998c-449f-8c04-08dcf53f200c X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2024 21:51:05.0689 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Rlqrkr1UVhT8C748GdR5513UtX2aRGNX5daPLMK8YKgmVEBksqwH9pBHIAw4eBYXKy8SNHEwyOMHf0NeHjS3m61tJUJPdp7jZl7/Sc7EbE0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7682 X-OriginatorOrg: intel.com Similar to regular resizable BAR, VF BAR can also be resized, e.g. by the system firmware or the PCI subsystem itself. Add the capability ID and restore it as a part of IOV state. See PCIe r4.0, sec 9.3.7.4. Signed-off-by: Michał Winiarski Reviewed-by: Ilpo Järvinen --- drivers/pci/iov.c | 29 ++++++++++++++++++++++++++++- include/uapi/linux/pci_regs.h | 1 + 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index aaa33e8dc4c97..6bdc9950b9787 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -7,6 +7,7 @@ * Copyright (C) 2009 Intel Corporation, Yu Zhao */ +#include #include #include #include @@ -862,6 +863,30 @@ static void sriov_release(struct pci_dev *dev) dev->sriov = NULL; } +static void sriov_restore_vf_rebar_state(struct pci_dev *dev) +{ + unsigned int pos, nbars, i; + u32 ctrl; + + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VF_REBAR); + if (!pos) + return; + + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); + nbars = FIELD_GET(PCI_REBAR_CTRL_NBAR_MASK, ctrl); + + for (i = 0; i < nbars; i++, pos += 8) { + int bar_idx, size; + + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); + bar_idx = FIELD_GET(PCI_REBAR_CTRL_BAR_IDX, ctrl); + size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]); + ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE; + ctrl |= FIELD_PREP(PCI_REBAR_CTRL_BAR_SIZE, size); + pci_write_config_dword(dev, pos + PCI_REBAR_CTRL, ctrl); + } +} + static void sriov_restore_state(struct pci_dev *dev) { int i; @@ -1021,8 +1046,10 @@ resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno) */ void pci_restore_iov_state(struct pci_dev *dev) { - if (dev->is_physfn) + if (dev->is_physfn) { + sriov_restore_vf_rebar_state(dev); sriov_restore_state(dev); + } } /** diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 12323b3334a9c..a0cf701c4c3af 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -740,6 +740,7 @@ #define PCI_EXT_CAP_ID_L1SS 0x1E /* L1 PM Substates */ #define PCI_EXT_CAP_ID_PTM 0x1F /* Precision Time Measurement */ #define PCI_EXT_CAP_ID_DVSEC 0x23 /* Designated Vendor-Specific */ +#define PCI_EXT_CAP_ID_VF_REBAR 0x24 /* VF Resizable BAR */ #define PCI_EXT_CAP_ID_DLF 0x25 /* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26 /* Physical Layer 16.0 GT/s */ #define PCI_EXT_CAP_ID_NPEM 0x29 /* Native PCIe Enclosure Management */