From patchwork Thu Oct 10 10:31:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Micha=C5=82_Winiarski?= X-Patchwork-Id: 13829911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BC3EDCF11CB for ; Thu, 10 Oct 2024 10:32:23 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 39F4910E8C5; Thu, 10 Oct 2024 10:32:23 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="SQT19jZD"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 901F210E8C0; Thu, 10 Oct 2024 10:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1728556340; x=1760092340; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=Bs8Jn4giSEzArPZb//LTvOlHynnehCNqHp7FGu65Veg=; b=SQT19jZDlbSYP0+CfOCvbURo5F6ixeEM+nWbUO7VD1PSTUff8NjOEVVk /nOzH3T7sCX467P3W5uDmf0qMGVRNDNfzXyBOtLEvZAuDK6cg+vwP7zGD WJNREDLso09ZyOHZNMkSUq2ZLcF2tTfxCYz+TCfJ7bygnQOG3nBUSxU4I etgpTEkVOTRVomzWcaDgZIlDCHK7ws+CgyhWVNLWbbe/MnqdvpbLQlfQ1 A7tE3MORABFWHOFRo0IHW9siulEhp55Kvo6JNib8R1Kffk+972GWa6J2/ g/vqd264cAbES5wXBJ5MU9+tjZGi/eeE+YEBfiQ9OZOU7y7luNWGxEenH A==; X-CSE-ConnectionGUID: 54BuOuFMTsqQfrnm6pA++g== X-CSE-MsgGUID: CxdwnAsKRuKTyHXQcimQ2Q== X-IronPort-AV: E=McAfee;i="6700,10204,11220"; a="15528419" X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="15528419" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Oct 2024 03:32:20 -0700 X-CSE-ConnectionGUID: v55bOunjSt65Vs2U2noWuA== X-CSE-MsgGUID: Ispt2/E+T2e86earY0XKMg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="76464924" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orviesa010.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 10 Oct 2024 03:32:20 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:19 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 10 Oct 2024 03:32:19 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.43) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:19 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kxObkvnTz+CwG+LZrXmCHZzrTAMMIxAjYnGyVF38xc4UMAGEUqrx8KZu9Kc9sLjViPDfz/0VMn0FzmR+AhMJ0saI9s9NdLuEFyMeTKGwHF0m5vdwtcgBHLR+xzQlcO8OFR6alRQLl4utSPrjc5JDAb5mp9ScDoa7CJPNO1B+TUBrCwJuU+0enY3L0ZYwvgRIusVc+XEy+1p7PsVeRG/Rq17Z2owoIfNYMDnKqLUdSBUDuxAJ4MjRDWyHMmxIRjg7F52Kz0H7OXIVypBuqIW7Y57RGloH5JslrAiLzNx2sdBefVhZYS4kLcLNETJna92f9YIAmbI+hveQS8OkIuU8kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QyYEH2qAYR9ofrIMUSnRZ7+9N+tYnnjyVObjHaQr+A8=; b=JAhxDeiBdaGr0aspXtEA0yK+1nDrQUzQ6p84gKd5iJvjmjCkp+l8yrlVrkDaiEI0phNuEt6rm6UCqGxHDDNH83nuHBJwb5NXt0vFpHl+7GNdGH1vlkFUvi+GDt6A+85fvK6vzfBDgCLB9Ixnsi9LsWEEeSbSTWHJluAkl4zEGgm7d5DlBx8/V8gXlKgqa8YO/5KGzDwCeEh/nqVGrd5x/0wMmWRG4km/UWQPOfgyKQpjZaRE6KAWyX2jtKgCbsZstgWlq6GpBucB2a41OTs/YEUweN1W6/t+daVrvw1fD7D+2B7ua44e+J2AHSvz5WXXEOP0bETqMnzRwXErMDYXVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) by MN6PR11MB8103.namprd11.prod.outlook.com (2603:10b6:208:473::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.22; Thu, 10 Oct 2024 10:32:15 +0000 Received: from DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39]) by DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39%5]) with mapi id 15.20.8048.013; Thu, 10 Oct 2024 10:32:15 +0000 From: =?utf-8?q?Micha=C5=82_Winiarski?= To: , , , , "Bjorn Helgaas" , =?utf-8?q?Christian_K=C3=B6nig?= , =?utf-8?q?Krzy?= =?utf-8?q?sztof_Wilczy=C5=84ski?= , =?utf-8?q?Ilpo_J=C3=A4rvi?= =?utf-8?q?nen?= CC: Rodrigo Vivi , Michal Wajdeczko , Lucas De Marchi , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matt Roper , =?utf-8?q?Micha=C5=82_Winiarski?= Subject: [PATCH v3 1/5] PCI/IOV: Restore VF resizable BAR state after reset Date: Thu, 10 Oct 2024 12:31:59 +0200 Message-ID: <20241010103203.382898-2-michal.winiarski@intel.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241010103203.382898-1-michal.winiarski@intel.com> References: <20241010103203.382898-1-michal.winiarski@intel.com> X-ClientProxiedBy: WA2P291CA0029.POLP291.PROD.OUTLOOK.COM (2603:10a6:1d0:1f::29) To DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB5373:EE_|MN6PR11MB8103:EE_ X-MS-Office365-Filtering-Correlation-Id: 727fef6c-a28e-44ca-f976-08dce916ced5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?61vg0Rv0vmypUKBOYcff610viUTs3o2?= =?utf-8?q?LzppXpGm2YRbEjloReXxPXmlX4jAPQgB5uHji7UVax8S56TD3/4VylV0I5MTP58DM?= =?utf-8?q?n6H+2LQl74ekruG8TbJxDHOIxqdw+AEieDYAWcCRY4AGyj+jz/4IQFooojEU1AgkF?= =?utf-8?q?yNu0Wo7pB0g+K1EdN5lKauYnbCHCrjbnvhYvB5/Kmf7IJFcDcD9Fixk5h5EHGL+KU?= =?utf-8?q?tago9Pi2EuyO7s2rSe11xqOiQcWQ+X0FsQm2If04lENEeh0tuiZer+rhcb6PhV3mE?= =?utf-8?q?G2R9oNGr+jtc3KgLBzGb2Ut6GfUYELY7n+S/fs3hJMCuXC6NZNyR5G2YdIP8WjjYG?= =?utf-8?q?s9uYWzM2ebUeSSPTSZmlzwCZc7LQtwt0guM9ONqkBI/a1LrTgc4DbxsZD+Ec0xY3k?= =?utf-8?q?EbtoFTFBFT0KNcZeHBcL4C0MbeSXy7y4R+1X5Kbl4dUcQsAfAvcZUZrjfZh/wRzmk?= =?utf-8?q?JXlBXkEE8rjdk3yIxPvuCX5freVPWp/wkS6V5TT8a78bVQar4a/1o1XL/g3vO6SxC?= =?utf-8?q?sKV1QkzjyEgNTNxpZI9LzV+WSvzUOeqcLCoCs1CyGrZT0fUbvpL9Rb7xwS/CcloMA?= =?utf-8?q?Hs2e5rvblRKpPwhZ+oledhTLn5UnmrTWmOJVw8V5ECD158T9s6K1VMN1Z0QLicwn9?= =?utf-8?q?2AOPPAvke7kQiWEhb32tEUCqM+v/HxTZeZ62gFNjoshgn7BLL5tUVo5GCMj6AqrHo?= =?utf-8?q?ZeYI2EWxZkzjDv5aEssp+PZPYFnAUbY+5BgMHA7+G2KhUKQgPOE3GvMWG8TVK4iCy?= =?utf-8?q?E1RM84Si8QoQMwyVKJ3qU8m+lUiqTRO4tzl5rtQD7jscG6g7+uqXTdMl8+vMSGNlC?= =?utf-8?q?5A6h5UCj0EqS0uP2CGyOKHeVH3EfoM4asXuGp+gFkIaC1+BvquVPuBHfStg7MhzPl?= =?utf-8?q?ai+hs8LninjUPf6d1PfuBvFmhvZ/JsRoWrhR6jOlc8TzTyBhzSlbbcjE33tVpuoKD?= =?utf-8?q?x0+zi/hYdybppYddu/G47sT5MPiqkVTWGVRbW59TugW0wHEodPWFnINHc+1Twuaje?= =?utf-8?q?mmUf2sqgrpoUXpaOzUtSRhUvs4Ea0JCbb0VQR4KUYXVI3i7Ukv76Mx/eNo3ue2v08?= =?utf-8?q?tCwjqXu7YmRI6/kfRHu49pwOdutEzP7yJ22LlCwZQNLJBz1n5v+WnkH6VC75UKeES?= =?utf-8?q?vIWIR8elNn/dIaaaf5LRlhneNOJ08ssCMa2wNIzMzniIj+7v/qLK/MMKt+paXOTJ4?= =?utf-8?q?+mboCrV/Js1saF+DdtvDCyPehIqKnWesqD3A=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5373.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?L3qRzr08pgIzWyJIu1FNUI210Jn1?= =?utf-8?q?wsEj9d7TTsV5FhPqUr3pUgWPZ5vzUNCuIWI9wOL4BHNH4YcldLlHvEP/8xLwy06rl?= =?utf-8?q?r8YzrB/yWjO2imFUKFDcpE50f7yRObPgiwFgipwg8jdD9JuwbjssVknPep7GH78wF?= =?utf-8?q?faZa7GRWaAzdZUbNiwe0SaOawzXPfwYIHv09PVTzLnihY2wL59GNNLieiXKlpZIsw?= =?utf-8?q?ESkBG4Dmmx4AdKvqcWIigfIskS62tKGkMl+2gFGSqMxQt3CSU5zwporEJObSmYMOr?= =?utf-8?q?SnDx5A6rLbYNYklt4PmWxh/IBoBKlgRRiXBuHE/NUcH2BkyDhvR1X5s7j4nEWCw2p?= =?utf-8?q?352qxKvwbWogDmGJXWrGJTpTZQoyyxlYAG7fG44J0cqf//XMn4RxO7tCWR3yXgUCG?= =?utf-8?q?cFnvtaJYooeH3OqPZLVNIGyksH1It4Pss8bwsAs4a0Q4DqktaTuhOVflG9x47EwXV?= =?utf-8?q?xlCQco0lvuklQEKEA6KatIF22l5OjhvEQkdSoUizAiTOGEMIFqkG4wuaBQpGVZVj3?= =?utf-8?q?T4Rn7WeWYBNbbw2BOnIP/oJx84mMQccsigLFr7anYUHDPnq3zAItVfHyk3I1Lgcf1?= =?utf-8?q?Dgmw4CXYC3FhAe9xfUC+Dwf0UNjwp0N2IA5vBt8Uf6g/HVBYK6Ci83KHg+gnEVLhN?= =?utf-8?q?ecSbcl1xfttynAsFnn1uWBuUdkG7CGvgqSa8RwOf0ZVV6W5nz044wjv+BHPSzgPlJ?= =?utf-8?q?LbyYgrhT4JrzCHL5ClA9ZGvCAfLXrGsvuJxoX0KhR//BRc2eZ1VhxU+eeoHHMWNd5?= =?utf-8?q?6HbD/CiPXCoe8IaBQcaOLb0ZfxuCdYGe/50jmg3xoDwdJV6MNa2b6XUO6mNNWu+1/?= =?utf-8?q?1lTg94TBpnXaDuhKNjrXYFl5EXSS5jyPAn360cSGPWjeWgrxa5gIX7SaswlFA62Dd?= =?utf-8?q?1c/v2IlI0N1Icz6gM+gV3rUf3Ow4oSPagQtFnzFX/eax1H9ov3xb/Cdfn477gaFux?= =?utf-8?q?RmSVmREiBkQprEQ+LMXRtzcw55gKx+s7V/1fFqertyZNgPjitNAKJu9rkjXk5X6+k?= =?utf-8?q?TVpOzW9BL7nrC+75okGIFl3MkL5mKYt4XMZW8ng/3btJRRwL3h4n3P8rVdxfWaY5L?= =?utf-8?q?UjdOQBmhyS2OvDzuaJaWP6UTi9NCu2TFvA3RsThcpurhI+/Hd5GyU89Vb0/6BGsdh?= =?utf-8?q?M6f4Mpsjtrfa1HkDhk5w1qBt+hUGgCcCjm9ldjbBx+5aJdrmLZPwt4+WoyTJ6cQL0?= =?utf-8?q?8ydq90w2GJbwnHM4Tm8M8USYOo3vfa8QHh2wDBj+lQSmbz7PQW3cSNpHvNlpqPpfT?= =?utf-8?q?DOh5BAaDvqAVvBd/skZeRf35wyoB7Mc1tZA5yZp5WXcSY7skUnx0IEohTiVeHCLcV?= =?utf-8?q?lFQkTcY69dkBegweGil9JxZG+aWA6I8dRjLGS/dp7m8lkfhEwSn5v0NTxnkRMHkv7?= =?utf-8?q?KDx6y2QHc+U8EhnUzt+DJuSnJmWArv7zjXnJyuVnqzbyTJb50r8ihVXfW18rOmU+P?= =?utf-8?q?9BoamG0AlqKBeN3r49gRxSXCdEjJmcjtNwVh3DxJkA1duDHBUKPHAuHjLLwMeCfo/?= =?utf-8?q?ZPiQftKmKgEbPjz9hwUZiFUaGRz7n1oB3w=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 727fef6c-a28e-44ca-f976-08dce916ced5 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2024 10:32:14.9701 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vVXEKXCCClLRcvJhHTc5FqsYOxG5yD6sk7fa5GfU3VLM02bNLxQvHh2XuaMQFntp92y03M4nHCY+XHdhb4+S0xwY40qw0E5u6J2IuRwlSn8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8103 X-OriginatorOrg: intel.com X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Similar to regular resizable BAR, VF BAR can also be resized, e.g. by the system firmware, or the PCI subsystem itself. Add the capability ID and restore it as a part of IOV state. See PCIe r4.0, sec 9.3.7.4. Signed-off-by: Michał Winiarski Reviewed-by: Ilpo Järvinen --- drivers/pci/iov.c | 29 ++++++++++++++++++++++++++++- include/uapi/linux/pci_regs.h | 1 + 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index aaa33e8dc4c97..fd5c059b29c13 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -7,6 +7,7 @@ * Copyright (C) 2009 Intel Corporation, Yu Zhao */ +#include #include #include #include @@ -862,6 +863,30 @@ static void sriov_release(struct pci_dev *dev) dev->sriov = NULL; } +static void sriov_restore_vf_rebar_state(struct pci_dev *dev) +{ + unsigned int pos, nbars, i; + u32 ctrl; + + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VF_REBAR); + if (!pos) + return; + + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); + nbars = FIELD_GET(PCI_REBAR_CTRL_NBAR_MASK, ctrl); + + for (i = 0; i < nbars; i++, pos += 8) { + int bar_idx, size; + + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); + bar_idx = ctrl & PCI_REBAR_CTRL_BAR_IDX; + size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]); + ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE; + ctrl |= FIELD_PREP(PCI_REBAR_CTRL_BAR_SIZE, size); + pci_write_config_dword(dev, pos + PCI_REBAR_CTRL, ctrl); + } +} + static void sriov_restore_state(struct pci_dev *dev) { int i; @@ -1021,8 +1046,10 @@ resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno) */ void pci_restore_iov_state(struct pci_dev *dev) { - if (dev->is_physfn) + if (dev->is_physfn) { + sriov_restore_vf_rebar_state(dev); sriov_restore_state(dev); + } } /** diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 12323b3334a9c..a0cf701c4c3af 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -740,6 +740,7 @@ #define PCI_EXT_CAP_ID_L1SS 0x1E /* L1 PM Substates */ #define PCI_EXT_CAP_ID_PTM 0x1F /* Precision Time Measurement */ #define PCI_EXT_CAP_ID_DVSEC 0x23 /* Designated Vendor-Specific */ +#define PCI_EXT_CAP_ID_VF_REBAR 0x24 /* VF Resizable BAR */ #define PCI_EXT_CAP_ID_DLF 0x25 /* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26 /* Physical Layer 16.0 GT/s */ #define PCI_EXT_CAP_ID_NPEM 0x29 /* Native PCIe Enclosure Management */ From patchwork Thu Oct 10 10:32:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Micha=C5=82_Winiarski?= X-Patchwork-Id: 13829912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25565CF11C6 for ; Thu, 10 Oct 2024 10:32:27 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9D89510E8C9; Thu, 10 Oct 2024 10:32:26 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="Im//7s9Y"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 865EA10E8C6; Thu, 10 Oct 2024 10:32:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1728556343; x=1760092343; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=EnFCLTRhaSlbiiAj9WkS562z1cdHDnijHo6AeVHAiTw=; b=Im//7s9Y4ke/0xxJNt9tvAYSus2CRWoOGMNS7qTFXEFIiwmKJRb7h4pd WKtVE5XMNt4iuIsO0aS+pmK/bP2Z4eRbtjft/rJKODxAkNraCK7p450hZ kcxp+dc6Ajcp7qBnvVOd461UIzIYdcaYrb1m6Q/JWebpNOFSr6MlJFQ5W OeW3gLPBnSWQLtJE8SEGTcxYU+8ChznNmn8TtsFHyPmEkA0Ty4fO218lM X+MTzfA99DUIZJoD2L5kbiTArb3XqW1n6IDmDF+H2EbUoeCbYqJbn1V+K YyXxGtwsbkx5b8YSF6THBf3IdFCYLh6FrBZmf9LoiRCsjZ1tWPF8/DOC9 A==; X-CSE-ConnectionGUID: 6DcB/IR2SfOw21LLqLDILg== X-CSE-MsgGUID: 4Ne6yp2ISmiPDn9xM5DqCw== X-IronPort-AV: E=McAfee;i="6700,10204,11220"; a="15528438" X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="15528438" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Oct 2024 03:32:22 -0700 X-CSE-ConnectionGUID: GzonanFNRG+dmbLgm142ZQ== X-CSE-MsgGUID: cWaq87CCR16gnhX8GEWmfQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="76464938" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orviesa010.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 10 Oct 2024 03:32:22 -0700 Received: from fmsmsx601.amr.corp.intel.com (10.18.126.81) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:21 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:21 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 10 Oct 2024 03:32:21 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.43) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:21 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PTkKE+CM+SPD7StpZ8YKVCYqf42P6fpETw3uH8utQZaEWqDNHzVVvr9rcqDx/lmymaYgD/BIwi3Uz3YfddCbkwcp2HckAqiFW2F9ikDfFfKe+vK0/bSXMFU9D8Gmcgnr57rkNAaeL7mE+GbviYMAVMwAdLznXKgEwy/hr7BegJn9MIXyCrG174sv+/00z1vp8jLJlO/7gxUNkcTa08MCB4iXKu29yGxOLRVfep+zlE843GGOjCgF6WLTpCg0p+8W4el6CznCp45bSfqBMZ9raaGDHpvUh6+GwpiK5gHTa8loL2dgYmu86KtCboKZTzOrZpCxguRJddJ4ypog8uOXiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WHNZiwbsESwo2PZik+oXAQ+zuQFZTsGm7BehrubSLZc=; b=pxC0cIks/3P+pal+/Xzmnqg3+ViuD8zkqT590ciNHXDDMndoII907CwX9MBisyY1/6E5PTwpxR61VilIaxEUglZaIBIPt1hKXRfYuLaxGhexaOsGYbi4GYocdp4OOYWEhoUiXhQ80lobeyGL7U3RPzEONU8dG/kuPfl9W0+kIUwWpaZxBNMNDdjHtqinsGq/4UFlS7InnP3gGj93ZhMb2HgfvprgdB+kiKHJPFmzkOwC/uNLB6iaBmCqm9eTB6DeukvcUqfkdl8lbGvNdtNOvKpdTVa/KgRAX3EQSotKtN+4HpD69/EO3FZ8NJtmjAFF2aQ9PUeD6yjp6s4UpCSAwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) by MN6PR11MB8103.namprd11.prod.outlook.com (2603:10b6:208:473::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.22; Thu, 10 Oct 2024 10:32:19 +0000 Received: from DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39]) by DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39%5]) with mapi id 15.20.8048.013; Thu, 10 Oct 2024 10:32:19 +0000 From: =?utf-8?q?Micha=C5=82_Winiarski?= To: , , , , "Bjorn Helgaas" , =?utf-8?q?Christian_K=C3=B6nig?= , =?utf-8?q?Krzy?= =?utf-8?q?sztof_Wilczy=C5=84ski?= , =?utf-8?q?Ilpo_J=C3=A4rvi?= =?utf-8?q?nen?= CC: Rodrigo Vivi , Michal Wajdeczko , Lucas De Marchi , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matt Roper , =?utf-8?q?Micha=C5=82_Winiarski?= Subject: [PATCH v3 2/5] PCI: Add a helper to identify IOV resources Date: Thu, 10 Oct 2024 12:32:00 +0200 Message-ID: <20241010103203.382898-3-michal.winiarski@intel.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241010103203.382898-1-michal.winiarski@intel.com> References: <20241010103203.382898-1-michal.winiarski@intel.com> X-ClientProxiedBy: WA1P291CA0024.POLP291.PROD.OUTLOOK.COM (2603:10a6:1d0:19::18) To DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB5373:EE_|MN6PR11MB8103:EE_ X-MS-Office365-Filtering-Correlation-Id: 6031fa4b-da8e-48cd-dc63-08dce916d134 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?yEsIlWCjLF0hlr1uU68hDv/ESA3KX0W?= =?utf-8?q?PZ2YPNnC5fOazABPW+EI8W7SUCBcynUf34BOC4k8eGaCbUGRED3qU4zco73+EmAeN?= =?utf-8?q?EIs3fZhCSgjacxlNOoQCzCmRfA8yACApDkpAZbnL8NuGZkWQnYvu7hqzm8P6PJyBw?= =?utf-8?q?KW3mc4jUN1azO1Gng+NfC8SCdOZm85PcKO8A/haXUOHbhTfYmXCZyST8uX+Yrcv90?= =?utf-8?q?ZkMpyhec6E75odPscCaEWHpBvtuuIuErszCuC927oBhzEC+LQsURFt9RGxrMxJw5K?= =?utf-8?q?T02OYxtjvqZEoziYEuUI6otDLE37wNDMzQHrNAs5ZIBCJimlP28M1aP8o/31ofRIh?= =?utf-8?q?jLqD353ZHWvQ9ZivBWcEPmu2IcqJcwPxpOTPqfCYg5ZdhWRuljc7HIG+0cRg+qHRq?= =?utf-8?q?iPG9OrJhPVBGN21PG/PsCBafFw947VJBtxmG5Ek5kY4ABxo0EeScyy8rQXBABUHWD?= =?utf-8?q?lpZzP4wEA7V9TsjXH0M6WrCyWL1v74p4w8rgKMsgtAEBkNuCsPO5/DeihwSdeoorN?= =?utf-8?q?GBBcb3zqn5gntbFspx+wcRLNgpBvwo3AhxGW6r6r9Xe1ztKv8B8nABNiXO2Ls9K+m?= =?utf-8?q?/qdLbIy2MxuXAcbwf+oLEKrJ6lcocDcxjZiaATHhbIlyI+/C0vymFaVIPeSmtjBzy?= =?utf-8?q?iBG5kx+SJs0zrRWc6/0JoOQBNqBzVrLxqJd2Pf4rIfsI9dtX90pHU5nQJI1ynucgk?= =?utf-8?q?cuc9wa3M2ZH7sXSuAXoy3BoPZC/HxnK9zogyUmyIvJG0BD04fCmKuht4h/oqzFMbJ?= =?utf-8?q?cgQEHGik1+CsH0Z41821yqriT1841liheUylERG0t/GgVSiiRGp+HcDODowNaKfcT?= =?utf-8?q?WP1coVk+3eaZBrPQQ5h5FaD6BnlbARvlxIHrq7HqidlX77K/jvbWjvUIASHd27q0R?= =?utf-8?q?ymsDPUj/432DPbqekS7wJcDPAzG8yMzndfiCm0Jm1TlOv+6jVE/tErFI+GRB/+hiB?= =?utf-8?q?9rJe4wEkUazuQKuH0rKWlTq4ZX6t9BYPUdPTasjdwEkdTSLQdj9UKOEz42aEMZooY?= =?utf-8?q?M3VKT4lCwWATAEyM6eVH3jPCZg1jqdTS8VZfxq1FzsulJBZH016w/qdMgsh0ygaV1?= =?utf-8?q?+Z8C8B02eA89iW52cs+Ec0VjsQcr5rdAxcCrQJBhgMJf2WUdGvTIpwiixj2eTZ2A5?= =?utf-8?q?4w+H26NtHkWylXi3gmeH+21X/zlr5ipVDTjiAA2ZW3YxP4oHMvfAUXE/Pl9YTgQyq?= =?utf-8?q?3z/+5a/TnqElN/ZcGEHYmESr1GqhP8FcNluQ=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5373.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?7+mG90ySPLmBieRmx+lcbLkLUX1N?= =?utf-8?q?c7bivzHZKoAy5wDfDnIhUpGNdRikPX8koHvnTGxVPvUwOlK2fhR4fxHkJ0D6qK3jd?= =?utf-8?q?mwn7OmIO/kSSc6KD37Tj9J4qYL/jMDjJsKMmTOj+t61jmyzKD5vHpd8WCb0bTBi1J?= =?utf-8?q?yumdMkAvAJ9L++mZPlQrpiq1NmoZLDpAea/ro0Ft0x1FsFTqgs8u0k1dzdx4JEa8+?= =?utf-8?q?WypVQ5wngJ0sdRSxZj6h65JJ2mOnpulbAAWjJyfSriDPTa2xKBMPww/NaavbNX8Q3?= =?utf-8?q?6lFzXAJjWT8kbtncalI8VDdR1TPOHK4bwEwCNdC6yxAAYBokGF8rRaoOH72wOltO5?= =?utf-8?q?zqYfla8hzL9+/bVrKtC0o9HppFthM+Or0u0DHafuT1cHh3Q4xkDkoJ2mtzRj8gz/4?= =?utf-8?q?yWkT8V9oFVhZS1XSYPeiqOvuatVSPA9e31ULFq426nnhOnQEqkACIcpQ26vxJo5Qn?= =?utf-8?q?69kK6s/A4HM76DIZp4Gre6ZzHc93xO1lWh47jqMbRORc0PLx9mltySgY4MNf5bnYg?= =?utf-8?q?gHcY6/zxZvHKRmPVjl/I3BTsw1Jr4FnP2w7c1iuUoxhlVRsKaaFls3DLjF8yA9+Bi?= =?utf-8?q?KiY6GzrcSxSeqQbX4EpMFquKS2/r2SBXUC8ctBbMXzMOtPK8ba4Cmg4ZND69+amoH?= =?utf-8?q?Wt9zQDrgU563wi9ZrAwz6jDBEU6Vcq98qwTBUso8/c9hIORnsktplL348kNpkowra?= =?utf-8?q?ntMBypUBURVkXfn1TiBzV7+RkZrL27SIg/XJ5p5wZZPQ2u3ih+YXFvxcZ8Fjv9aRQ?= =?utf-8?q?vB4GOi4xdrfuO/H3m0JMSalxluWxKIZQzSQdAx5ZMQsu6UPPda9mfOF0X+n9mw1lj?= =?utf-8?q?FASebnMO6ane0uqdZIowMywtc2Tk2LXa4Din20474vfHRU/v/pFIcu/WmfIP0uG4e?= =?utf-8?q?uXzeSKs/aajH1vzmDchfman19c0gW/XnyvGBhJo5LxItlR97D2juCarC4Jt1xGDAd?= =?utf-8?q?eJcAuTUZIVdJxNHTcH860F3lYjKnNjoGzEbWANcYDZOemX8Le9E/kD680yZ79oejM?= =?utf-8?q?FNcHxyDTJEqMASprYerDyHu8OgBXln2nxFKAhhmCukmGTcWpVIJD9648fiSo2h23v?= =?utf-8?q?xz5pjsEyJQbEQYxvoZnWfu8QZIc8dyFnL/kJ4DMKCHO3VepnHKOqNcViVLDppkIt1?= =?utf-8?q?xmeHCGwjV0lauHpBelbe4w48iDSAyxAAUMFXQcbcdS9rXFr6z9mj7zarGwb0Md829?= =?utf-8?q?kifCh/DNZC7czOFdSMftzWhzM4sMavalKjlHSZY2y9uqTgXCb11TbrMeNalXBetUM?= =?utf-8?q?ZUUbOiVr4oYDAgp0JtQHU2mDGAbFtsD2i0Vt9l9XwzhORngOfDdoUkktmFzFYO4vw?= =?utf-8?q?yKQ3AfMMQi/Fag1L8TLoBmjFqErWEC/6oX9rkTSTCHaPJU/1mKSk/Zdz7idzSxiSD?= =?utf-8?q?c2jOjC9U0W4FM+yrk3T76SpB6lcx7tboWU5oZIpl75tUqN/xmFF/47eJlxjGpOaG/?= =?utf-8?q?pPhOdQY3at0N4IFv6CpeRYe+K63H2PHBGe/5qVUa4Mx5VFEEpV9BzVfl0LstfnlgR?= =?utf-8?q?cDM7D9ZXyD4ZagLBVZqmO8BEpdn41poFgA=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 6031fa4b-da8e-48cd-dc63-08dce916d134 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2024 10:32:18.9403 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: a1CMItMzJCuZTf5+fWMsnu2+z76wkLe8L05/aM7wRue+n9U12dztUBQsFOrO2kEjpGv0Y4AHxxby6qc6Gw5MjE3b7/WU/e542s+8AYdEs3o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8103 X-OriginatorOrg: intel.com X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" There are multiple places where special handling is required for IOV resources. Extract it to a helper and drop a few ifdefs. Signed-off-by: Michał Winiarski Reviewed-by: Ilpo Järvinen --- drivers/pci/pci.h | 18 ++++++++++++++---- drivers/pci/setup-bus.c | 5 +---- drivers/pci/setup-res.c | 4 +--- 3 files changed, 16 insertions(+), 11 deletions(-) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 14d00ce45bfa9..c55f2d7a4f37e 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -580,6 +580,10 @@ void pci_iov_update_resource(struct pci_dev *dev, int resno); resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno); void pci_restore_iov_state(struct pci_dev *dev); int pci_iov_bus_range(struct pci_bus *bus); +static inline bool pci_resource_is_iov(int resno) +{ + return resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END; +} extern const struct attribute_group sriov_pf_dev_attr_group; extern const struct attribute_group sriov_vf_dev_attr_group; #else @@ -589,12 +593,20 @@ static inline int pci_iov_init(struct pci_dev *dev) } static inline void pci_iov_release(struct pci_dev *dev) { } static inline void pci_iov_remove(struct pci_dev *dev) { } +static inline void pci_iov_update_resource(struct pci_dev *dev, int resno) { } +static inline resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno) +{ + return 0; +} static inline void pci_restore_iov_state(struct pci_dev *dev) { } static inline int pci_iov_bus_range(struct pci_bus *bus) { return 0; } - +static inline bool pci_resource_is_iov(int resno) +{ + return false; +} #endif /* CONFIG_PCI_IOV */ #ifdef CONFIG_PCIE_PTM @@ -616,12 +628,10 @@ unsigned long pci_cardbus_resource_alignment(struct resource *); static inline resource_size_t pci_resource_alignment(struct pci_dev *dev, struct resource *res) { -#ifdef CONFIG_PCI_IOV int resno = res - dev->resource; - if (resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END) + if (pci_resource_is_iov(resno)) return pci_sriov_resource_alignment(dev, resno); -#endif if (dev->class >> 8 == PCI_CLASS_BRIDGE_CARDBUS) return pci_cardbus_resource_alignment(res); return resource_alignment(res); diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 23082bc0ca37a..8909948bc9a9f 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -1093,17 +1093,14 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, (r->flags & mask) != type3)) continue; r_size = resource_size(r); -#ifdef CONFIG_PCI_IOV /* Put SRIOV requested res to the optional list */ - if (realloc_head && i >= PCI_IOV_RESOURCES && - i <= PCI_IOV_RESOURCE_END) { + if (realloc_head && pci_resource_is_iov(i)) { add_align = max(pci_resource_alignment(dev, r), add_align); r->end = r->start - 1; add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */); children_add_size += r_size; continue; } -#endif /* * aligns[0] is for 1MB (since bridge memory * windows are always at least 1MB aligned), so diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index c6d933ddfd464..e2cf79253ebda 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -127,10 +127,8 @@ void pci_update_resource(struct pci_dev *dev, int resno) { if (resno <= PCI_ROM_RESOURCE) pci_std_update_resource(dev, resno); -#ifdef CONFIG_PCI_IOV - else if (resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END) + else if (pci_resource_is_iov(resno)) pci_iov_update_resource(dev, resno); -#endif } int pci_claim_resource(struct pci_dev *dev, int resource) From patchwork Thu Oct 10 10:32:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Micha=C5=82_Winiarski?= X-Patchwork-Id: 13829913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B804ECF11CC for ; Thu, 10 Oct 2024 10:32:28 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 390C610E8CA; Thu, 10 Oct 2024 10:32:28 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="Jrd1pDWM"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8392F10E8C8; Thu, 10 Oct 2024 10:32:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1728556346; x=1760092346; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=ybaOv2ojJW7CwOeUTI7puwXRWFM1V1+uIvZzDLzNOR0=; b=Jrd1pDWMTwHTg+Ftf7dEcnZYoAHKJKusrsb9+hRd6m5WyF6MQq+SZmc5 VuzFCaQEcPSSqEEgxxi9AHTuKelOywJhCKUCMbLJQjjUUF5Vx8v1Vl4qf UmP/OqLjyEjcDJp5EuW+ZGRC2Wb3pNlTDbKX0CL7f5A9qG737yvDZ7f04 ueLIIfyQpiLwvo3je1SM3q3Zs79JGbRrOExxYuIwfBR9H684GpsTi2gXr LvRFFO3m7L6+UWeUteltmyZLlbyS0KJOm+mjaB+DfSzYXoMc0bZXIeAZ/ BeIid9c6DKT6nzZg8I8yjf8C8t34KVraqVTU62PdRUYlGo5rg4isJKlHq Q==; X-CSE-ConnectionGUID: P0UtIlmpSweMM0P+TR+RVA== X-CSE-MsgGUID: 1eywsvlKR92LR6MQ8357VA== X-IronPort-AV: E=McAfee;i="6700,10204,11220"; a="15528451" X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="15528451" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Oct 2024 03:32:26 -0700 X-CSE-ConnectionGUID: VeisuotSSoqWCy3d81VI8g== X-CSE-MsgGUID: qiOs39orTq2A4jEt31w1Fw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="76751659" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orviesa006.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 10 Oct 2024 03:32:25 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:25 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 10 Oct 2024 03:32:25 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.45) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:24 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FBzyVQkaH46NE0mUJ0zRMwbHAktKDr4xo4NmrHgA7Op97Sy4+zq2RnDm7iiUrcdVfJa7dPrhzuXGzaZKxQTgzG7FF6NBMhhxspEUm5dwQsNwFBKzVEWHi3t51dCnSZoQ1hffeQo5QdRZyLNqHn9HerF/Ce97qY7h5TYcNfrwnh/nrtb+RipOWro3ClysWJcQUWkUdigTMYTtJsAoebbI0oD+wrhK60d9x+TrsjIheV8TJlvSXPGAiKNshX/XHVy1Tt3uuLOpjxy4nryimhBM7ABAKUs3/teB+ztzuebtzQF6vCh3uO70ywRBgRJStWE3AQKMPND6jEr0LV16p/9r9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G8RdkHf2rYc9zV3jfSuykZ0/1sP4SE+Beit5J1B6fNE=; b=xrqEW7gfqrxP3LsNip7QudfHVB8erwkhLG5RHnlyrEx8UiTSBmtH4Pz3VOagrDJSiutzGCz4+AquDSoakikgW35o1FTLHLzeHPnr+vBnCwvbHEh7zmEDhtFoSixrCvXqfjI7nM+NRZ+OhVBeP6w3w8mECVRaCuzrGOFQuF/pxue81R7eslcqPcEW/rRxi4Po5TphwP7wnGK5UdLQFBtzGrpUvwtEokpgKKsiraQwhhR+tFlUNdZdKn0L6JneSCSyi7sOaev/wATbxix5mdG4ciJu84tI4/Rr+67yTkql7sDhTlhkVKWITZu+6uGqqCXH2wwMhYmuVPcD4xEOpbe2tw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) by MN6PR11MB8103.namprd11.prod.outlook.com (2603:10b6:208:473::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.22; Thu, 10 Oct 2024 10:32:23 +0000 Received: from DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39]) by DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39%5]) with mapi id 15.20.8048.013; Thu, 10 Oct 2024 10:32:22 +0000 From: =?utf-8?q?Micha=C5=82_Winiarski?= To: , , , , "Bjorn Helgaas" , =?utf-8?q?Christian_K=C3=B6nig?= , =?utf-8?q?Krzy?= =?utf-8?q?sztof_Wilczy=C5=84ski?= , =?utf-8?q?Ilpo_J=C3=A4rvi?= =?utf-8?q?nen?= CC: Rodrigo Vivi , Michal Wajdeczko , Lucas De Marchi , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matt Roper , =?utf-8?q?Micha=C5=82_Winiarski?= Subject: [PATCH v3 3/5] PCI: Allow IOV resources to be resized in pci_resize_resource Date: Thu, 10 Oct 2024 12:32:01 +0200 Message-ID: <20241010103203.382898-4-michal.winiarski@intel.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241010103203.382898-1-michal.winiarski@intel.com> References: <20241010103203.382898-1-michal.winiarski@intel.com> X-ClientProxiedBy: MI1P293CA0024.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:3::6) To DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB5373:EE_|MN6PR11MB8103:EE_ X-MS-Office365-Filtering-Correlation-Id: 97c693b3-3926-4deb-1322-08dce916d38c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?zmhztpgNoW8mpe/VzwN13aYm1PI1lBB?= =?utf-8?q?jxiqQ8ek0FJGM9FqHbX0FLWgyDV4P9XeLMo9bTKIkdKd3EgjzSQd+NyKEJPv0JLWB?= =?utf-8?q?rBtIRUX7crOl2bAmUsrGlF0v90SSLjMAf7sqFrvpm8KTZmMditX8PjZluDgHft3Ua?= =?utf-8?q?KRR1U8LK/+S2dGRuV0gepSgXM/dsRS2cNx9sakNWXpavEV3b52ANUN1c+wlfcKKDv?= =?utf-8?q?fHnlDj7sCWrtuU9mBxE/90k/atwfUrOi0y61QcG3n7IU5KxKig15vmkYWTUbOfIos?= =?utf-8?q?oU0UkH9OQ8wnyHcJM8/McvukVEEeMGMdDVpck1V/5/fSOPQItg3tgWKDvbAD7gl9X?= =?utf-8?q?N9H9nsCpY3NQB3E2zAb5oQCqZM8HdBfmqj0LmNkeXilJ0tVHafn4pZW4pIGPrVlFi?= =?utf-8?q?YDMfrfXqBXPDT9KoT0lJj0CyaSHTndFl6j2rmwvIQLJjvJZGDV1EgI74VCMc+nZvm?= =?utf-8?q?cH2laP3kfk9KI8+LPUb/r3PrHyo/2QuJYL8fFx+g0XaMf8Cn3/c6nccUEO3LIHMhM?= =?utf-8?q?Yb6QID2h3F3Gjvm21JP442laJwMLz1UM9KqBQuC0wkfRsuNJs53k3DsHLTEW9gNtf?= =?utf-8?q?8CUS6ifEQpnZVlyfHSPVs2KD6k9i7yEa053/EeS5E8b8Equ9HWR90n2ZwU4ypklNs?= =?utf-8?q?ZpUC9McE7985NhYT0+Eke0RdyjBwKKt6sVRux3YTIZ7B47SC9G/OfumOLYAw4bTJL?= =?utf-8?q?KCGdQJAB7c7DNT8HKznVgduUh8bKJL8qW7pA8njg/P4oyvG3p3uDf7JzpPngA2srx?= =?utf-8?q?PAbzfu3bHCXdy3cWz5cDDgAHw7kMLoRsFb55kxqgs1ji2jngMbrx3upS7fs5Gxgda?= =?utf-8?q?K0/5+QmI5vWbbqlb0U74EMlkyNN/ujwysJmziqggQsUBt7S0WFmZG6KrPxmYAZ2XL?= =?utf-8?q?i61bixMGlriUsYE+WbfjfY6PNYBZ6sdR6MvhrKXkbPijS+IvfAeALyBxcGObm6c63?= =?utf-8?q?N0BP8kCo0X6WwrK2tF1k7ohlMOj+kv1lxF+c/271ODx+UNN9AR93YDVl9DWt/4IqQ?= =?utf-8?q?hfha6jfvhCmKbKv0GeoRgBLZtB+4w0YYVXOsnjx0uPFCTesLr/TMUi9/LMIEAwJpN?= =?utf-8?q?2jA70tcnWrz/5dBHMVISuF7NSGGunf+6WZNUUVXiKqHtsAidAe/iQMHkFSv9oTt7C?= =?utf-8?q?bbpyuI+iKx4QrBnP3Ge34wqfBHCORkgKDYmCiy+X/GGrqMCjVOesmGKZteTJb4OcW?= =?utf-8?q?eX7GnENnxOpH4Bjanm+aM+1rkZSqQm4cznIg=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5373.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?yccBlx4OZQfgVWlsRGBvsCVgA1oU?= =?utf-8?q?GPHluyC7wu4DRRpWhyI/P1nll4wKr8wGC2HB2Cq25bTRRw7+Fu85tHHUZ9gGgtWSm?= =?utf-8?q?RIFwRd/zlmOUiHMTqZGwZAyznMwewyLfXJVgo3/dPOrT/kmnsYBVqHSkrEzG9e6TN?= =?utf-8?q?vLLf0uElxB9YHdaYyAEVsyRVWOolZLkoMidSGmnxm0Sra78UsNoheQjUkOygaYxB5?= =?utf-8?q?NTKJNwzJVjNBG0BS56jNecN0xkRPx1PiwJf8RlHwe3gYjN5IRhgBr0xpBM5IJo+Th?= =?utf-8?q?nZ9j0iGP0trSNEAj9G2qHgZv73Dbzc+Msc3O4u9DX7OIIVlgxCQU+wqp2D0riWGY8?= =?utf-8?q?qHsYA5H5hE2XFNw6SRPa5/No1McEpk2hJdNSJwCKDddxS+5mdF1AiKYNU3WbtSJ0C?= =?utf-8?q?Mb06ImMSm12W/+2fs8Ch3pGE7JDtNAOPZtV5vUObFBYDN1Gho1BFz9VSqoI5PByle?= =?utf-8?q?5/O9jHcpvLv1Fov7AlH1RWaKKZyYhna9yTO+6ZOF7bvl+eIueo8tJb10ur8WotcF1?= =?utf-8?q?vH7aGUQiDwtTQohAzjo1Z2EmNXYB1H4DymGlf3r9yjVYONiEV1vO2YtsCmye8KHH/?= =?utf-8?q?P/w4jau7L2fyjkMTajuXfR42zezxh7QWQU/dmCuSUsNT4GMEPLXRLhW36ZxphXylU?= =?utf-8?q?qkBYJXQCeqPdntDGlmwTmAO523kmVH7oo+T3snRqASbM2BOI2ycDxwo/m8v511qY5?= =?utf-8?q?Mt3hG407/HaC5BgpGGUSZ4BplsxMIwbYYHY4+b79qnq5Oly/lBSR0+UmI/qjoKRE8?= =?utf-8?q?4SIZub3sNm8rrYqPcP2ORFclhAYwOjMaxzq8wbn1H2B3ROmDNpm+gQ3M6sZlsUdB0?= =?utf-8?q?InS8Nlaaja2dhRBpfQI3d2OzfvhtgS5DOneLBtvNmmukw5IvKgCMegCIxoJXw0hMK?= =?utf-8?q?jRr/foeLq6MqvKlnTUaYuuR1t52fTmQwgAX5xTks4y+yQagFPotkonGD7mYXQM27b?= =?utf-8?q?PahlALtQBeWChJcmllwwS0pe3yz+qxPoe3teYoaeGLcpSt8DVRkt7JsLXCaItgZh8?= =?utf-8?q?BEDrpUSWtYCACEgmtSVVWjBLgwl4jHqxvER6RDS2AsuN9wc2hg+DgXWzEjLtxAB4W?= =?utf-8?q?Il+FRsdf4ivbNCDD2gYP1vlwLfYTMhIr6c7APfZgHvdOhr6yVGrcjYIEI0fhLy71T?= =?utf-8?q?TOjumvYJAyBcFZuXLFik/7vmPnwx1uK06nNAjmQQtxrI0UBxwqH5ZWiStv5ceid70?= =?utf-8?q?FjV/UAGuIB/rMw6Bg6PxdSQ0h+lBoRq4CYVeEOcj/GJWwpYVX411wEwV3BO9JYjxs?= =?utf-8?q?dUUMkCrCuYfS4vv3jmT/m85sOPWHgaV4sdvD8TsoayPOjFex8T9Uyxz9kfhMNOr7U?= =?utf-8?q?St/wCLnFpY8EkaRdtl5TWDPm+0K77MEjn8Jq3jaCG2Lm0iqT/d/Tklb47NcGfm2gB?= =?utf-8?q?KfWI2jdQJ8324yB0q5KV6R4wrixTu82DdfT2PyXQ55QtU1c/otvc6vO5HFUd1a126?= =?utf-8?q?BAneBgL145EptEwBAy0v35J74X4MnyBN4BnnqPN1fDqaScVAKIXzAt+STjHS+jRLx?= =?utf-8?q?sRYk3oiMvnESr/eMjtEx/lFXCurdby3Z0A=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 97c693b3-3926-4deb-1322-08dce916d38c X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2024 10:32:22.8536 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GQs655IqyfDqnVW72EGhn4fmOrcNfCLbm30t4ztwsyYSaH/Nd0LybMLiymkAjp4EgG991N+D9mapJjleq56bH5HZfAwg7mBK3s9S/3J1ioQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8103 X-OriginatorOrg: intel.com X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Similar to regular resizable BAR, VF BAR can also be resized. The structures are very similar, which means we can reuse most of the implementation. See PCIe r4.0, sec 9.3.7.4. Signed-off-by: Michał Winiarski --- drivers/pci/iov.c | 20 ++++++++++++++++++++ drivers/pci/pci.c | 9 ++++++++- drivers/pci/pci.h | 8 ++++++++ drivers/pci/setup-res.c | 33 ++++++++++++++++++++++++++++----- 4 files changed, 64 insertions(+), 6 deletions(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index fd5c059b29c13..591a3eae1618a 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -154,6 +154,26 @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno) return dev->sriov->barsz[resno - PCI_IOV_RESOURCES]; } +void pci_iov_resource_set_size(struct pci_dev *dev, int resno, resource_size_t size) +{ + if (!pci_resource_is_iov(resno)) { + pci_warn(dev, "%s is not an IOV resource\n", + pci_resource_name(dev, resno)); + return; + } + + dev->sriov->barsz[resno - PCI_IOV_RESOURCES] = size; +} + +bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev) +{ + u16 cmd; + + pci_read_config_word(dev, dev->sriov->pos + PCI_SRIOV_CTRL, &cmd); + + return cmd & PCI_SRIOV_CTRL_MSE; +} + static void pci_read_vf_config_common(struct pci_dev *virtfn) { struct pci_dev *physfn = virtfn->physfn; diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 7d85c04fbba2a..788ae61731213 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3718,10 +3718,17 @@ void pci_acs_init(struct pci_dev *dev) */ static int pci_rebar_find_pos(struct pci_dev *pdev, int bar) { + int cap = PCI_EXT_CAP_ID_REBAR; unsigned int pos, nbars, i; u32 ctrl; - pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR); +#ifdef CONFIG_PCI_IOV + if (pci_resource_is_iov(bar)) { + cap = PCI_EXT_CAP_ID_VF_REBAR; + bar -= PCI_IOV_RESOURCES; + } +#endif + pos = pci_find_ext_capability(pdev, cap); if (!pos) return -ENOTSUPP; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index c55f2d7a4f37e..e15fd8fe0f81f 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -584,6 +584,8 @@ static inline bool pci_resource_is_iov(int resno) { return resno >= PCI_IOV_RESOURCES && resno <= PCI_IOV_RESOURCE_END; } +void pci_iov_resource_set_size(struct pci_dev *dev, int resno, resource_size_t size); +bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev); extern const struct attribute_group sriov_pf_dev_attr_group; extern const struct attribute_group sriov_vf_dev_attr_group; #else @@ -607,6 +609,12 @@ static inline bool pci_resource_is_iov(int resno) { return false; } +static inline void pci_iov_resource_set_size(struct pci_dev *dev, int resno, + resource_size_t size) { } +static inline bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev) +{ + return false; +} #endif /* CONFIG_PCI_IOV */ #ifdef CONFIG_PCIE_PTM diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index e2cf79253ebda..95a13a5fa379c 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -425,13 +425,37 @@ void pci_release_resource(struct pci_dev *dev, int resno) } EXPORT_SYMBOL(pci_release_resource); +static bool pci_resize_is_memory_decoding_enabled(struct pci_dev *dev, int resno) +{ + u16 cmd; + + if (pci_resource_is_iov(resno)) + return pci_iov_is_memory_decoding_enabled(dev); + + pci_read_config_word(dev, PCI_COMMAND, &cmd); + + return cmd & PCI_COMMAND_MEMORY; +} + +static void pci_resize_resource_set_size(struct pci_dev *dev, int resno, int size) +{ + resource_size_t res_size = pci_rebar_size_to_bytes(size); + struct resource *res = dev->resource + resno; + + if (!pci_resource_is_iov(resno)) { + res->end = res->start + res_size - 1; + } else { + res->end = res->start + res_size * pci_sriov_get_totalvfs(dev) - 1; + pci_iov_resource_set_size(dev, resno, res_size); + } +} + int pci_resize_resource(struct pci_dev *dev, int resno, int size) { struct resource *res = dev->resource + resno; struct pci_host_bridge *host; int old, ret; u32 sizes; - u16 cmd; /* Check if we must preserve the firmware's resource assignment */ host = pci_find_host_bridge(dev->bus); @@ -442,8 +466,7 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) if (!(res->flags & IORESOURCE_UNSET)) return -EBUSY; - pci_read_config_word(dev, PCI_COMMAND, &cmd); - if (cmd & PCI_COMMAND_MEMORY) + if (pci_resize_is_memory_decoding_enabled(dev, resno)) return -EBUSY; sizes = pci_rebar_get_possible_sizes(dev, resno); @@ -461,7 +484,7 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) if (ret) return ret; - res->end = res->start + pci_rebar_size_to_bytes(size) - 1; + pci_resize_resource_set_size(dev, resno, size); /* Check if the new config works by trying to assign everything. */ if (dev->bus->self) { @@ -473,7 +496,7 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) error_resize: pci_rebar_set_size(dev, resno, old); - res->end = res->start + pci_rebar_size_to_bytes(old) - 1; + pci_resize_resource_set_size(dev, resno, old); return ret; } EXPORT_SYMBOL(pci_resize_resource); From patchwork Thu Oct 10 10:32:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Micha=C5=82_Winiarski?= X-Patchwork-Id: 13829914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3C58ECF11CA for ; Thu, 10 Oct 2024 10:32:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id AD1CD10E8C2; Thu, 10 Oct 2024 10:32:32 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="hNrhFGg4"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by gabe.freedesktop.org (Postfix) with ESMTPS id E60D210E8CB; Thu, 10 Oct 2024 10:32:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1728556351; x=1760092351; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=wLcAo1WhV3XHKAuXQH8tJQ1yh9wGo3PPw0FqfBgmVgg=; b=hNrhFGg4M60xhFqDzqwyLTCvW4UVx9jPefNoHmoULmCw2us0XiijZaN2 nZ/5UWHJ8iZsqWzYDE9vRD69A45lSoezcWivfpY0oPpdWWlP6ZXb+B05A XgCno2QlC5I+FA7ZFMBpMAdnQx8zBzO1r6nmXVfv+v9ztz8+M4yshRQ1x 6JufdGf90SoLhqNYh1jFFqwiPjv/ZQl9v1CAVyW6gsRX9wijZCmMZ5Opj PWliO+wPRsmE9/ikVsczVSYNEcWrF2+PfxxLaCptUA2yDi0Zg5o9mzr8K qkUy0RJXl65YJNGzMF/eVzvMs8X44WO97LtMXUAbBbA3owlLRRsqWR6dI Q==; X-CSE-ConnectionGUID: WnH4xYdyTEGgMyTdEtDP/w== X-CSE-MsgGUID: RRaTIIN5QnasQYIEU2KzLw== X-IronPort-AV: E=McAfee;i="6700,10204,11220"; a="15528480" X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="15528480" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Oct 2024 03:32:30 -0700 X-CSE-ConnectionGUID: nnlrErOJRmWMOTj/MwP17Q== X-CSE-MsgGUID: +GrUhSX4QsqxkK+YowOn0g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="76751669" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orviesa006.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 10 Oct 2024 03:32:30 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:29 -0700 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:29 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 10 Oct 2024 03:32:29 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.47) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:28 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yIAtwStontmQGMXV2b7+lIjnVS1toR+NRV9NlrJCA79su1Jffd5ua1cPP46GilFQTc60hXXhxvssQAgsK6f7gO5p5UgxkqeQBmpIS+GLsHrEtRg1yLsPUJpbXLc8KOjLY7jl8Gb7NBqw58NpupjSRiu1S/tReyr2ZNjvNY3boBp80RVB5rKVEGxn6yJ0sgmJz4y6cx+gYYqpwTwMUayTn/J8WcHTCQyLNiENUVCWp66LeghpR/l4tIKVON8552usiaYV73+su1hVkwJ2Wf1tV6dFvN0wPLwHf0NAUFc0QIpRfFJ199Fq7XiOSt47JPYeZUaqfDXadEuJb6VeNE2Rrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E+gdPjlhdRoa/icp7FwuL0+xbBgzz7H41qTyybjcdLY=; b=EjQlCoXc41qwCJ76l/LAwdOk9WM+5ru+24pJNOtiS9eNtNdN7peF94Skk1x4/yVShCwlY3KzDegbIhTpev3DsxzuvNlINQ9uk7EAAVbkoztTQaqEcr5nVW8KWDx0MaJutcCWulDrLj7KuFgzdfmOGXqIPNCYzTYGdNI+3FbNMSuL9Xh28s3yHdyoT/jlzJdDtT9zj/x0P79joPAhCTjnPImyxqweQZcsETHweQuiXUy0A6a+HsgqGvQ6yERziRBnw32m8Otr3dN/fWeF/7Vun0aRBvPmw9nWdzjJDE6VuWtKzSZRLt27eoNuv/O/DLU4ccZBM1+9nzEVB5fg3vfG7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) by MN6PR11MB8103.namprd11.prod.outlook.com (2603:10b6:208:473::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.22; Thu, 10 Oct 2024 10:32:27 +0000 Received: from DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39]) by DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39%5]) with mapi id 15.20.8048.013; Thu, 10 Oct 2024 10:32:27 +0000 From: =?utf-8?q?Micha=C5=82_Winiarski?= To: , , , , "Bjorn Helgaas" , =?utf-8?q?Christian_K=C3=B6nig?= , =?utf-8?q?Krzy?= =?utf-8?q?sztof_Wilczy=C5=84ski?= , =?utf-8?q?Ilpo_J=C3=A4rvi?= =?utf-8?q?nen?= CC: Rodrigo Vivi , Michal Wajdeczko , Lucas De Marchi , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matt Roper , =?utf-8?q?Micha=C5=82_Winiarski?= Subject: [PATCH v3 4/5] PCI/IOV: Allow extending VF BAR within original resource boundary Date: Thu, 10 Oct 2024 12:32:02 +0200 Message-ID: <20241010103203.382898-5-michal.winiarski@intel.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241010103203.382898-1-michal.winiarski@intel.com> References: <20241010103203.382898-1-michal.winiarski@intel.com> X-ClientProxiedBy: WA2P291CA0007.POLP291.PROD.OUTLOOK.COM (2603:10a6:1d0:1e::10) To DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB5373:EE_|MN6PR11MB8103:EE_ X-MS-Office365-Filtering-Correlation-Id: fd2b1505-1539-41b3-3a73-08dce916d5f8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?19eZPWq2c4Qp+nLbyRwHOPqFheM8C1+?= =?utf-8?q?x6jqRv7eR1YyypWJ5+WlN/DizeQRxRbCFMOD5SeybYAVw47Q8rsGlvw0xRlTBBYP5?= =?utf-8?q?M+Kb841hbskKtSizUPJBYeZl7Vsnqn/6ZtXlRg3UaO++gV562+3vLBAH9A3+kYNdv?= =?utf-8?q?W4LpBnGVKeTP/Q8utXxGOqcdPMDpRlxMT4/0ueIQ3VaRWAn70l4Sjaki2xQYPMz5m?= =?utf-8?q?QXaakpRhBtpC+8t5VcgM6fod3HKqLndNy97GPNLQh9quqMvCpOaqSr1iDMr+CiGgJ?= =?utf-8?q?3qfovwE2X63U2sLBDULN3p83Mx2rPZmb8ltyptuBGD4l2MD6U2KnCJHiNz/rS9ezk?= =?utf-8?q?QiLz8aCFiPQ+kZ09lrSI47DFP8zoarwSpaQPe75zC0MbXYc0h4NGoVkrDfw6kxB8g?= =?utf-8?q?P4TBBydADmBh8qVAdPqnwXHxoEOrVX/+9eo5vVP8gXx2cE5mIuPhnYzLDDdTehPBE?= =?utf-8?q?KJ48tVDWtDl4t/zOeG9/bPwrIPBFJNS5FO1/rOMSbZJf7hFLlYV61wYxBydVJ97Z1?= =?utf-8?q?paVviTNrJtvMMHLNWf8qAtpHYEcxSEFhLjonolJYZ9Zh1eTNB2u2QEoYYTN9Z97+w?= =?utf-8?q?S7SbPhIv47tIn17O0RSxlolQjJjolxqRWkU6yJ0/toFxu41pR+MJjPyzwWgLc5qi2?= =?utf-8?q?WW8hV5WJP0gXF1EtgO7U1fCpAPwJUaxhxgSesFNvV+CobXNulb1mZb4RabVtc7800?= =?utf-8?q?k7JnuISwNeQZQBlFR0snNoelBWl9xCQ8TUwDWNmBnYip1vOg8SI0rr1Syrd5SLUY0?= =?utf-8?q?hHsVUqfwLgxbIYOtP/U+2PkPd5ts2r9g0WU+Wo/zJO/ueH8tH+ve4S3UkuP1zB2NJ?= =?utf-8?q?WpNt9h3uxSlkJ2KcR4vVceRe3ouC0LZjRWhpKnDaNk/En2xdMT08wjIoOGb7+3Ktf?= =?utf-8?q?aXwiROVS8VTlRJGmhkCjX/Vw0BOV6onb2dZsqaKkBwwnS2FtYEFasdLsxq6GNPIQv?= =?utf-8?q?Ik5H+e2wznsCaNRRiN+rL7GGMrwVgxDno1+kvfbQeQRdXG9p+RzbvFEBCJXSwv3tS?= =?utf-8?q?EZbD2VWdDSsgsez8OLjiKsyuBJu04ucaOembFtN5u/Iwhaf/Gkcpg8cl9OM+h962t?= =?utf-8?q?kfNvP7kuOXAOB7GOcn6FUIVgV36/j30zev0AX0Yu9zh3nNbW/4v5FJElsIOGhL9oB?= =?utf-8?q?sFIg9rqNz1jBDr1oyTb61/92uuuBQggiqZtexOFjOZKLPi93d0dhg6hVHKdw+R53O?= =?utf-8?q?SsOvhTJc0/lDlTI9osTd/HjTqqmPjWKoDDyw=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5373.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?Flkx5Js6QAIc3PMzXHPt2nC7rmpk?= =?utf-8?q?tqRY41sehKk2gSh8fdzN5DDkz4iHuwe+mQqWHED5XdKzHvqYw8gcpDR8WfiOSdc3Y?= =?utf-8?q?4Xxsa6ZvMRT+yJz+fy91sMFacdPEO7aD6F/rkG22EYu5OG51aj0feerrtAKshp3FI?= =?utf-8?q?6PItyM+iMR942NUVI0cYvDz5efYqdmsHagWlR9h+NcWIcflSGJkvYqtApFP83kozG?= =?utf-8?q?T3QEn1DpuQKOPjupzZNtwBTfhhav1T0FuIx+42x9KLy0e0n2hE6IqMEOJViKkA4Ml?= =?utf-8?q?zN/NwAf3yChdF0l03G9n5yBOgV6owF4khlWLX5g34IwUeKA3KewKbnIJh3kx/zNuF?= =?utf-8?q?tj+tQcanG+CZ6fbt/E34OilcTx42Duyk/7ew6O1bNNMYslhtTnNog5nHnrcXFQngP?= =?utf-8?q?gpuCEwFAg79i1JbdIqswfmKS+Z1aeHWUL31Q8ceR93iaNizaLUjOliu2vilbJvj3K?= =?utf-8?q?1I3dBXQNQ4I3bJWsJ+j9XDM6P/iXPOSZS2L4/x8EerLCnrgTW7lnzeq148gB7JWmY?= =?utf-8?q?QWQ+69lyDiSh1gYJelumh1js6nM5rQN6aAG9rsXb2lemovyWKWeCu2KF6VAnq7yf6?= =?utf-8?q?jMrRvMnzJxocgskWPE2EKqVgBcV6wCLwBMTcEpGsj4FHO0lpwKDPnToFI4vE/Jy4E?= =?utf-8?q?TG8ayt1LgYO1GDbq24HMK/43gGr4qMnphuwdoa4O3Ccb4hqdhJzTOOqWvA860qyQ8?= =?utf-8?q?CKocVNWB4iqZ8lzHvs+n0CMVeNorifDyUtFDqt81dbjyaAgtDROj6LT6TO488cd+/?= =?utf-8?q?09iCTlrNrCkg0pPY34tVFb3Qx1Lo7OoBv2ToS7dj4Aw/QqzxiaLs+4dEg39KC3qSm?= =?utf-8?q?AyYmZvi1XDXCp9hHT4hIKi/2DK5F3GORyAhV5w+Ku7ExYtVyJt27tOuxVU/Z6z17n?= =?utf-8?q?w+GMXSFYvxks1FrdaLqmyjM6Fkj49DalQokL2FFQHQDRdDszSlaY0oeG9We7T7+xF?= =?utf-8?q?rYyqZyAkWP6Z4wir6NWL6S5gu60wbKXvDnTeA9ErVrCawKxtlCuLosQbiFRtoCNuJ?= =?utf-8?q?rHnLzEapVQBjfWwXHN3n2JipYJ+hFxXFn+O/RWt1w+ZMbYU1s7k4jgpuXfmzOhUYN?= =?utf-8?q?68PA0fP23aXscZWoZ2p/Y+JY64YXEbTYcz1GcWbSeUA7bRm+7r4IDz5g2aaGWGULl?= =?utf-8?q?cIBnnOAmXw6OQaVOssIu5N34DGSsSfp5ihSj+zDK06z69PQ8jFL5KQDVJD13yS3Nx?= =?utf-8?q?QDCWrG2fSGzeAzg0kM7QtaZKGtrwbqkv8cpojQL0f4byfEcWJvl7NNMmKVuCmQgWD?= =?utf-8?q?PdcDc9S3BKRWRX8OeZ9XOsoHZPu4qUcdA11RFuOLSc9m50b69CHdhi9VpONkuRO4v?= =?utf-8?q?sSgk7DFMpyqyCeAmEjQZJu77CWhLzFpjMnAf95fCgy0fKbPvgc2ZA5e2nr3d6ws3A?= =?utf-8?q?VGBsP26dmm9BfbwSIWt9P1vY8u+6PQr3SnpozX+cT56rD/QFTSfVT2SbFKwRlufJ/?= =?utf-8?q?zcu79vqKuBBWl0R8hQ/xJtXlMB9GGTKYPNVoY6wyhGrNkCnEVv1zzeChHJdkwRcBO?= =?utf-8?q?Rp5l13veCZS4hUGrEtTVYdct9YVy3lS3yw=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: fd2b1505-1539-41b3-3a73-08dce916d5f8 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2024 10:32:26.9304 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vQajWpZPgQEaOzB4W+4QVDJUsgSbCSqZKVEToILOHmRb6XhTswaKf3DRUmo3/ADMdKCtnu4dJ71m2UeUj9qhSKS2B9MYar/f51CfGblxMDA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8103 X-OriginatorOrg: intel.com X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" VF MMIO resource reservation, either created by system firmware and inherited by Linux PCI subsystem or created by the subsystem itself, contains enough space to fit the BAR of all SR-IOV Virtual Functions that can potentially be created (total VFs supported by the device). This can be leveraged when the device is exposing lower than optimal BAR size as a default, allowing access to the entire resource when lower number of VFs are created. It is achieved by dynamically resizing the BAR to largest possible value that allows to fit all newly created VFs within the original resource boundary. Signed-off-by: Michał Winiarski --- drivers/pci/iov.c | 92 ++++++++++++++++++++++++++++++++++++++++++++- drivers/pci/pci.h | 1 + include/linux/pci.h | 3 ++ 3 files changed, 95 insertions(+), 1 deletion(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 591a3eae1618a..f9071c1cfe9ee 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -174,6 +174,86 @@ bool pci_iov_is_memory_decoding_enabled(struct pci_dev *dev) return cmd & PCI_SRIOV_CTRL_MSE; } +static void pci_iov_resource_do_extend(struct pci_dev *dev, int resno, u16 num_vfs) +{ + resource_size_t size; + int ret, old, i; + u32 sizes; + + pci_config_pm_runtime_get(dev); + + if (pci_iov_is_memory_decoding_enabled(dev)) { + ret = -EBUSY; + goto err; + } + + sizes = pci_rebar_get_possible_sizes(dev, resno); + if (!sizes) { + ret = -ENOTSUPP; + goto err; + } + + old = pci_rebar_get_current_size(dev, resno); + if (old < 0) { + ret = old; + goto err; + } + + while (sizes > 0) { + i = __fls(sizes); + size = pci_rebar_size_to_bytes(i); + if (size * num_vfs <= pci_resource_len(dev, resno)) { + if (i != old) { + ret = pci_rebar_set_size(dev, resno, size); + if (ret) + goto err; + + pci_iov_resource_set_size(dev, resno, size); + pci_iov_update_resource(dev, resno); + } + break; + } + sizes &= ~BIT(i); + } + + pci_config_pm_runtime_put(dev); + + return; + +err: + pci_warn(dev, "Failed to extend %s: %d\n", + pci_resource_name(dev, resno), ret); + + pci_config_pm_runtime_put(dev); +} + +static void pci_iov_resource_do_restore(struct pci_dev *dev, int resno) +{ + if (dev->sriov->rebar_extend[resno - PCI_IOV_RESOURCES]) + pci_iov_resource_do_extend(dev, resno, dev->sriov->total_VFs); +} + +int pci_iov_resource_extend(struct pci_dev *dev, int resno, bool enable) +{ + if (!pci_resource_is_iov(resno)) { + pci_warn(dev, "%s is not an IOV resource\n", + pci_resource_name(dev, resno)); + + return -ENODEV; + } + + if (!pci_rebar_get_possible_sizes(dev, resno)) + return -ENOTSUPP; + + if (!enable) + pci_iov_resource_do_restore(dev, resno); + + dev->sriov->rebar_extend[resno - PCI_IOV_RESOURCES] = enable; + + return 0; +} +EXPORT_SYMBOL_GPL(pci_iov_resource_extend); + static void pci_read_vf_config_common(struct pci_dev *virtfn) { struct pci_dev *physfn = virtfn->physfn; @@ -438,7 +518,7 @@ static ssize_t sriov_numvfs_store(struct device *dev, const char *buf, size_t count) { struct pci_dev *pdev = to_pci_dev(dev); - int ret = 0; + int i, ret = 0; u16 num_vfs; if (kstrtou16(buf, 0, &num_vfs) < 0) @@ -480,6 +560,11 @@ static ssize_t sriov_numvfs_store(struct device *dev, goto exit; } + for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { + if (pdev->sriov->rebar_extend[i]) + pci_iov_resource_do_extend(pdev, i + PCI_IOV_RESOURCES, num_vfs); + } + ret = pdev->driver->sriov_configure(pdev, num_vfs); if (ret < 0) goto exit; @@ -874,8 +959,13 @@ static int sriov_init(struct pci_dev *dev, int pos) static void sriov_release(struct pci_dev *dev) { + int i; + BUG_ON(dev->sriov->num_VFs); + for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) + pci_iov_resource_do_restore(dev, i + PCI_IOV_RESOURCES); + if (dev != dev->sriov->dev) pci_dev_put(dev->sriov->dev); diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index e15fd8fe0f81f..57e79f75e4c8f 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -403,6 +403,7 @@ struct pci_sriov { u16 subsystem_vendor; /* VF subsystem vendor */ u16 subsystem_device; /* VF subsystem device */ resource_size_t barsz[PCI_SRIOV_NUM_BARS]; /* VF BAR size */ + bool rebar_extend[PCI_SRIOV_NUM_BARS]; /* Resize VF BAR */ bool drivers_autoprobe; /* Auto probing of VFs by driver */ }; diff --git a/include/linux/pci.h b/include/linux/pci.h index 573b4c4c2be61..023c0fa1dd9f2 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2371,6 +2371,7 @@ int pci_sriov_set_totalvfs(struct pci_dev *dev, u16 numvfs); int pci_sriov_get_totalvfs(struct pci_dev *dev); int pci_sriov_configure_simple(struct pci_dev *dev, int nr_virtfn); resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno); +int pci_iov_resource_extend(struct pci_dev *dev, int resno, bool enable); void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe); /* Arch may override these (weak) */ @@ -2423,6 +2424,8 @@ static inline int pci_sriov_get_totalvfs(struct pci_dev *dev) #define pci_sriov_configure_simple NULL static inline resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno) { return 0; } +static inline int pci_iov_resource_extend(struct pci_dev *dev, int resno, bool enable) +{ return -ENODEV; } static inline void pci_vf_drivers_autoprobe(struct pci_dev *dev, bool probe) { } #endif From patchwork Thu Oct 10 10:32:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Micha=C5=82_Winiarski?= X-Patchwork-Id: 13829915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03404CF11CA for ; Thu, 10 Oct 2024 10:32:39 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 896A310E8C3; Thu, 10 Oct 2024 10:32:38 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="P8TBiJML"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.18]) by gabe.freedesktop.org (Postfix) with ESMTPS id DC6F410E8C3; Thu, 10 Oct 2024 10:32:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1728556357; x=1760092357; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=doqr1dty0vN+uf31XtEqRtxzdxeUgEoyO4Rr7m7TfV0=; b=P8TBiJMLoFA4bTKK48jzhslY+BdCgXswx3kl8fuuYYKJohArE8Q2FXTo JT7nn/UJJT8ecfsId8kAyKkjAGUJSUyh1L457Hjldq6dknqguzvM+ceST 5Td8AyKJsHDzIG2t21AhTiJS0omgzt1y/w/LuiixC4agIxdMcNeVX5Fs6 RjmvjL002bI68U4qf/R0263+BLZB0N5FBsEQTp6r4AZ4sBY2uPDU5vgzd i88aFKAkXGK3jHXEuo/efhYpbkpTODy4lomeJSajpFs4h5EFAic49IvZn AvLyFfIqFVGiTuKGOxGdOR8vhmmiPZkpejwTqtN482sSdWgMSvVbzvYl0 A==; X-CSE-ConnectionGUID: rKNjOOPgQ0+84kHtzWtjqw== X-CSE-MsgGUID: qBTECrwiSsq3RLVyhBBCbw== X-IronPort-AV: E=McAfee;i="6700,10204,11220"; a="27351305" X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="27351305" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa112.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Oct 2024 03:32:36 -0700 X-CSE-ConnectionGUID: +4zwoDOwScSCV3JD/xSNrg== X-CSE-MsgGUID: T7i+5Ds5ThKAZZF9OaBviA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="80559343" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by fmviesa003.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 10 Oct 2024 03:32:34 -0700 Received: from orsmsx602.amr.corp.intel.com (10.22.229.15) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:33 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 10 Oct 2024 03:32:33 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.46) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:33 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=exTCYN/P+Bog60IrLiv0hAfZkj1EMBHMjpSZD4+s0d4fmXnyIRAu9wJPWtlw8RB9Nt17B6KyF94uNL8Mm1PMIA6/0o+HB5s9CNWvkaXk9TK+Bh0CtA+/MbOY07Bc1aECBb5oEB56qYXx+HirbliEmwS86AOfoMlGJwCuuvrvTmGi9Mvn1Fe/EKhtND3rFo6Wpdw7AFpocIm7roK0bHur3c+5Q6KgbtbnfguBa26/yyuwhVkm/XWYEgPxi1PYrF+AaYb6eDqM3QQ+zbF1wg45StDUc5D/XW8/w7BRrlWYeBYhlE4uQLISQW/0B2kSq9dJ0fW03LQt8Zl0IMiQJkruXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KdVn5Ub7EPPEMa3i3e7evc/RY/yZ3Kbngz9R7/bMOT4=; b=gZxGGWUeDGP6LN0/73Vvq4kdDctg0e3OtWV/wuUmiRHJ/esxaSMWW66TnjoFm303YaXL8xUZ6RifyVliz8AxpAIfyfn/kZqlI1Vo0ViOBCq8kSO3opD7uL1ifZCIGhYySwulyGZxjGjLyEG7jBg9L2hCDR9nVRHZFKK/yxvq6WujOqlqIp2eW7yUM8avGJvO/svvdGTEBk/8Ve4o/8Bnk5XcbdmSxA2W15O9YLBldb7fFNpbt3wgP6QJQOactwbMQP07g5EF4hybmPMD1hdHhyTD6wzFOdUdJQDXq1FRdrv0QQHv/zuO38q5d8vwkjFaIzkuDbN8PkJcBxxcUl2K6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) by MN6PR11MB8103.namprd11.prod.outlook.com (2603:10b6:208:473::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.22; Thu, 10 Oct 2024 10:32:31 +0000 Received: from DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39]) by DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39%5]) with mapi id 15.20.8048.013; Thu, 10 Oct 2024 10:32:30 +0000 From: =?utf-8?q?Micha=C5=82_Winiarski?= To: , , , , "Bjorn Helgaas" , =?utf-8?q?Christian_K=C3=B6nig?= , =?utf-8?q?Krzy?= =?utf-8?q?sztof_Wilczy=C5=84ski?= , =?utf-8?q?Ilpo_J=C3=A4rvi?= =?utf-8?q?nen?= CC: Rodrigo Vivi , Michal Wajdeczko , Lucas De Marchi , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matt Roper , =?utf-8?q?Micha=C5=82_Winiarski?= Subject: [PATCH v3 5/5] drm/xe/pf: Extend the VF LMEM BAR Date: Thu, 10 Oct 2024 12:32:03 +0200 Message-ID: <20241010103203.382898-6-michal.winiarski@intel.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241010103203.382898-1-michal.winiarski@intel.com> References: <20241010103203.382898-1-michal.winiarski@intel.com> X-ClientProxiedBy: WA0P291CA0006.POLP291.PROD.OUTLOOK.COM (2603:10a6:1d0:1::18) To DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB5373:EE_|MN6PR11MB8103:EE_ X-MS-Office365-Filtering-Correlation-Id: 312d2e32-a51e-4d8b-c513-08dce916d857 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?HF1fDWyhYk5m/9NIzum3HwQgsAxeOfK?= =?utf-8?q?1P8u+wKuWolAqGCap3N0b4bbA9CY+I5s+s0RgySfizymr+p1w0d6K9T48bn8KMZbo?= =?utf-8?q?Y+um3ZHOZTcIhblyCkdwgrXZUMR2abB0ANUzKjAIhXt59Yy0YJwaZysrc7hg1eAwZ?= =?utf-8?q?2St7Hvqarpr5UooCqS1K5ElNimfTCXSGng2/7/NRzwiByCT1NRz9pnaRxbjzRLWqq?= =?utf-8?q?JDeWN6xtkEYtF4lfEl4ZZtYoZoDkGls8uT/6CvEacxZ9o0068gyF/DWCGimQQBh95?= =?utf-8?q?ZAG9ZmR3cH78FPgmLWwvnI4Um+se7Jpcb/d7b2dAD8HvfmuJKmXn0Y+70iglw8lM2?= =?utf-8?q?u0Y+CXlwYoDUwRxikOP1BPCo/Y4xLupx3MRiXNBY+QSI8hOGhE6zN1SwVE/EMqQRN?= =?utf-8?q?7HHyhNFjoDQIYcOyJL7PpHumMDTQvWF7Ucax2Yynv9LYhUHGDsWR6Sn+Whe5eXHVi?= =?utf-8?q?VpFUexKlKTaSfeA4FjG4Gx064gwKg/75Pfc2J+ZpAZUDEs0v1uB9j+offc+SVbgvu?= =?utf-8?q?Z4nZR47Ce/kBLgOzENvXWyew04XoH3S3mYV3E1lj772ccwrLZhybOnfIaN0toF3E2?= =?utf-8?q?RfJZRoWWPx0d2qnKJ3OHQF5QD+KazOxBtk1RXEKI8JbXSSVVtgqSn+/5UfOazSaU6?= =?utf-8?q?/9lblvzc3l3juMyPr4I0jn3MxpLBBBxpCU7y4v4JPZk20yt8lecAPBS/uiPAdjXen?= =?utf-8?q?uCXdX+WgQ864XHrH9/zd/3AZr42SJL4wbcg8ksCf3DRzkKRTBH+wSDYXhREND0ogd?= =?utf-8?q?K9engNuZK4xC6b8JczPgUiNoiKb90aAHidMgCUoLTFB+g8r9H6TOf7liN62H9Jr4o?= =?utf-8?q?0AZJemh/+jS46adM0S77nWpEt5eQwWOHI4sXy5BAGrwnHlSZqH0B/eqrYSabV/PVx?= =?utf-8?q?jX8QS4ER9fW4wG4g1jy2VKvrJKwDcQw3tou5w2kku9yed15WTyAQoIWs4dL8icHZR?= =?utf-8?q?SExJSzPJMHeD6BCkCaNPMmqT2zF9pL+9lSrQf40Pc8uytVh+22aRzaEeFlWxOXYFA?= =?utf-8?q?EWb9Jj0hMGwEa4T341KDFx+zbs4dc+t5S1Tnv/6dtupoiWiNCu5yOplTBpv93UXhE?= =?utf-8?q?SKUOxq9n0b086ucb2dHT7MOe113hcVoT61azGyBy2U9ywu7GKrW93a2T3s4rmonXH?= =?utf-8?q?i35IbNYRfL3YTAu3SLzSrjfqdXCaXY3nZjZ5AlYuDMnELPS5L1L6mGeWJUsxzNjzi?= =?utf-8?q?J8uR2L4w1cpQb9nAE6ELoEpaspVTKXdv3qZw=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5373.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?lC2yiSIKKCunbzuv9MQcB5BgbFuV?= =?utf-8?q?gofeirmxw0+vMX2y3AeNlMd532DPW2Apz10t9aWuJlWuB129vh2vBfUN71QUEINP8?= =?utf-8?q?HmQW7gRCXr+pJ1phaSqxbHCzfEacd/ehAxKTFOh6wYD7zh4cB5vEFYaoD9M1AgjT/?= =?utf-8?q?BLoh16FHznYLlRXWIdhI+SquZTauV25eWha2bD0G32rnPGzMHtKZ76npWNkdvDhzU?= =?utf-8?q?Lbvp7EVzzMPPwq9Ph4TyvKIS27/TBx/EfLp4WLE/bssYr/GtLa2KhqopTuZhUSIUm?= =?utf-8?q?5KGQuTwBMBReeSARtTj6WGsUX6H2sg+2XEhpOUuzVxD8+VU3JmfN/fX6ja8N6t2cC?= =?utf-8?q?ZuHesy2hLkMV5QzsSo6bkyn0CKoErOly85uvLQho69j9iHAoVlD0WluehdGeBDj/F?= =?utf-8?q?jjQBr6DIMJRpqJbiwT4ELiXmjJu446XbmYE1g2J5OYbY5vMVSqniI88eN0QXTv85X?= =?utf-8?q?xIeJqE5gGsV3/xWI8HXMtCLawFUp7CRLgV4vThzAIqbP4TjPoUwvM/5tG5tFUVcOu?= =?utf-8?q?76l/ohvA6DuZUwUpLwB1oujRcPQ7YwrbFmd0guFZGqlsq1IV2mcCdfDEGLSMfwTq5?= =?utf-8?q?0IZZa2jEUOvKm+0BBX8nw6RdqY1avvc6uDfPo/7nSkVgT+NzjhPDmndUTfpK8mRX6?= =?utf-8?q?cQ9UruTdwjM9OEPKSCOXzoYpeMn2EQ+a+lF0CDiNiKFeNROokyHDAEb+CoolTckY2?= =?utf-8?q?+7TJFF6g/lPKzEjI//qd1LWJ7b6NVzFabxOegSqHoL/HVNmgx/DNt03ZUryUVEUtX?= =?utf-8?q?zBSDtMz5eViEYFk3Xh4bQZ60dKvG2R2ifYUlR/HytoV9denGO3iUtoafNtamhr5zc?= =?utf-8?q?rgzCx43WEB6yVGu8AeReqYUVzYvNcKHzbrGDuilUZy7CCZpEWLGG4Rn7jPTI11MF9?= =?utf-8?q?/JhaKU7yTIypQLdaED/udx3sux7cIcvDbfV0/cl3p9V9gh3/jAvxM3GeqfLLym07S?= =?utf-8?q?WnLrRH0I+XohYuNR1EEGu3TMZlJBonHFUPTzRf0q2Ngry9KuWakeuIs2d8b5/SRNv?= =?utf-8?q?wKS9c1IRUtY5PKNNlcGzW7NEE/jYelM3MmiskJwrgeuUDh03fkxMagOgCbFqAkeDu?= =?utf-8?q?h+1VQ0Q3yxdgJJ1nUnZSoFzk/6XXqyiO2wWPswePZoqcYaUulBl1bJq3jy/Zj/kiW?= =?utf-8?q?7GNpoiNZILrhcc7wD3ZHAZnB06jibmr+qgPXGSBl+P4Kf7Jtp0FtI0yIk0coVWZEv?= =?utf-8?q?m3q6Kdk/5TDD2j/S9EeTQbXEUUfEPu+Qz1yqyOVtqkCzLGF3Svysp2OCZ8T0+cXZP?= =?utf-8?q?uJXUep+yimhyDsRf9cHsrRUB6VBaD+1yOAL/afpEVguCmIVAHi1duYQbUNZO3M5NB?= =?utf-8?q?aBDEZNqSeuHrWF77xx1DnOIxu/iwC9gKB+KcAUwb30L6b90EFNpIuj7xbquzZWDtD?= =?utf-8?q?VjFaLEqkiZvXpNrPs62QeN38L/Qj4DZA2isc3jcrrCnZoT49fJR4vE1+7DgDIFHvg?= =?utf-8?q?Ko3+8gpGoWKw0IuGCAiliexqPw1IHDIpdfNUBOHZrntXJp4ExFxuZkRkoF9aV/nyb?= =?utf-8?q?dLXnsMmMW+P2yHdhM3ba7hQGtHcdpS+T4w=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 312d2e32-a51e-4d8b-c513-08dce916d857 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2024 10:32:30.8990 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cjHW+P7YzbGrckK7LL2/dLmZP5Ktfnv1m7sdrh4Q1edn/vkpvtfgqBRe/yAnxhLcpJdxlmfNg1xxe7gk0gYfE6cbTYwJKC/FtTnSU2/TA54= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8103 X-OriginatorOrg: intel.com X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Opt into extending the VF BAR. LMEM is partitioned between multiple VFs, and we expect that the more VFs we have, the less LMEM is assigned to each VF. This means that we can achieve full LMEM BAR access without the need to attempt full VF LMEM BAR resize via pci_resize_resource(). Signed-off-by: Michał Winiarski --- drivers/gpu/drm/xe/regs/xe_bars.h | 1 + drivers/gpu/drm/xe/xe_sriov_pf.c | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/drivers/gpu/drm/xe/regs/xe_bars.h b/drivers/gpu/drm/xe/regs/xe_bars.h index ce05b6ae832f1..880140d6ccdca 100644 --- a/drivers/gpu/drm/xe/regs/xe_bars.h +++ b/drivers/gpu/drm/xe/regs/xe_bars.h @@ -7,5 +7,6 @@ #define GTTMMADR_BAR 0 /* MMIO + GTT */ #define LMEM_BAR 2 /* VRAM */ +#define VF_LMEM_BAR 9 /* VF VRAM */ #endif diff --git a/drivers/gpu/drm/xe/xe_sriov_pf.c b/drivers/gpu/drm/xe/xe_sriov_pf.c index 0f721ae17b266..a26719b87ac1e 100644 --- a/drivers/gpu/drm/xe/xe_sriov_pf.c +++ b/drivers/gpu/drm/xe/xe_sriov_pf.c @@ -4,7 +4,9 @@ */ #include +#include +#include "regs/xe_bars.h" #include "xe_assert.h" #include "xe_device.h" #include "xe_module.h" @@ -80,8 +82,14 @@ bool xe_sriov_pf_readiness(struct xe_device *xe) */ int xe_sriov_pf_init_early(struct xe_device *xe) { + int err; + xe_assert(xe, IS_SRIOV_PF(xe)); + err = pci_iov_resource_extend(to_pci_dev(xe->drm.dev), VF_LMEM_BAR, true); + if (err) + xe_sriov_info(xe, "Failed to extend VF LMEM BAR: %d", err); + return drmm_mutex_init(&xe->drm, &xe->sriov.pf.master_lock); }