From patchwork Tue Aug 16 19:40:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 12945250 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF564C25B0E for ; Tue, 16 Aug 2022 19:40:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237224AbiHPTkI (ORCPT ); Tue, 16 Aug 2022 15:40:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237237AbiHPTkH (ORCPT ); Tue, 16 Aug 2022 15:40:07 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 684C685F90 for ; Tue, 16 Aug 2022 12:40:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660678806; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YJVt87I+RkFO3jUAhHrJxKBqiCZ7HeO0w2k1yZLam5o=; b=FNx7gBNrCuZKARxOy8qBuRAS4G8epEEe/HcdNyI3y671HCXjhTxVLSsKOyejLSWOTT8uFJ Jn1vNmaUBlbIXSlCIk2pPVxbic0pap7oPXkcXQg2ICqHUq3tiiLBOI5ddiD1V+geJh4Tbp CI3fyPYNpdpNaqBlfWRl/P0Sg1o5LdU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-275-uFEVRtW1MFW4FV7_cbTngg-1; Tue, 16 Aug 2022 15:40:05 -0400 X-MC-Unique: uFEVRtW1MFW4FV7_cbTngg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CDFBD18A6524; Tue, 16 Aug 2022 19:40:04 +0000 (UTC) Received: from [172.30.41.16] (unknown [10.22.18.188]) by smtp.corp.redhat.com (Postfix) with ESMTP id 84E761415128; Tue, 16 Aug 2022 19:40:04 +0000 (UTC) Subject: [PATCH 1/3] PCI: Fix double-free in resource attribute error path From: Alex Williamson To: linux-pci@vger.kernel.org, bhelgaas@google.com Cc: linux-kernel@vger.kernel.org Date: Tue, 16 Aug 2022 13:40:04 -0600 Message-ID: <166067879638.1885802.1955389738707186135.stgit@omen> In-Reply-To: <166067824399.1885802.12557332818208187324.stgit@omen> References: <166067824399.1885802.12557332818208187324.stgit@omen> User-Agent: StGit/1.5.dev2+g9ce680a52bd9 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org If pci_create_attr() fails the remaining resource bin attributes are removed and freed based on the pointer entries which are not cleared in the case that creation of the bin attribute fails. Signed-off-by: Alex Williamson --- drivers/pci/pci-sysfs.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c index fc804e08e3cb..9ac92e6a2397 100644 --- a/drivers/pci/pci-sysfs.c +++ b/drivers/pci/pci-sysfs.c @@ -1196,8 +1196,14 @@ static int pci_create_attr(struct pci_dev *pdev, int num, int write_combine) res_attr->size = pci_resource_len(pdev, num); res_attr->private = (void *)(unsigned long)num; retval = sysfs_create_bin_file(&pdev->dev.kobj, res_attr); - if (retval) + if (retval) { + if (write_combine) + pdev->res_attr_wc[num] = NULL; + else + pdev->res_attr[num] = NULL; + kfree(res_attr); + } return retval; } From patchwork Tue Aug 16 19:40:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 12945251 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3C93C25B0E for ; Tue, 16 Aug 2022 19:40:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237251AbiHPTkg (ORCPT ); Tue, 16 Aug 2022 15:40:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237239AbiHPTkg (ORCPT ); Tue, 16 Aug 2022 15:40:36 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 988ED86FFC for ; Tue, 16 Aug 2022 12:40:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660678834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ejPyL03exSUyv1sSmQS6RpycXfqdQ91zc3yteXbW3HE=; b=O5XTCm5/bAXxFP+eiw+/VyI4Qjc5OpE9W+w0ztKZH2rKufNnCg6A38TT3SBrw8OHQGdAYv oODiDZ1L2Gj/SunNJPdNTqKs8+dI0XaRD+A0oXC0eJw958iJzQnE7qxXRVS58b/RS/KQ87 pwqz0yFg2Z4aZBQuLFBdNQoz+XPqqqY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-540-h7lRGLixPaON3uAu49SmIw-1; Tue, 16 Aug 2022 15:40:31 -0400 X-MC-Unique: h7lRGLixPaON3uAu49SmIw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DDD301C04B63; Tue, 16 Aug 2022 19:40:30 +0000 (UTC) Received: from [172.30.41.16] (unknown [10.22.18.188]) by smtp.corp.redhat.com (Postfix) with ESMTP id 86B4B1121314; Tue, 16 Aug 2022 19:40:30 +0000 (UTC) Subject: [RFC PATCH 2/3] PCI: Skip reassigning bridge resources if reducing BAR size From: Alex Williamson To: linux-pci@vger.kernel.org, bhelgaas@google.com Cc: =?utf-8?q?Christian_K=C3=B6nig?= , linux-kernel@vger.kernel.org Date: Tue, 16 Aug 2022 13:40:30 -0600 Message-ID: <166067880991.1885802.3339740933334886723.stgit@omen> In-Reply-To: <166067824399.1885802.12557332818208187324.stgit@omen> References: <166067824399.1885802.12557332818208187324.stgit@omen> User-Agent: StGit/1.5.dev2+g9ce680a52bd9 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org More work is probably necessary here, pci_reassign_bridge_resources() has heavy weight requirements that the aperture windows are unused, without even testing if the requested resize is possible in the existing aperture. One case we can clearly skip is when reducing the size of a BAR. Cc: Christian König Signed-off-by: Alex Williamson --- drivers/pci/setup-res.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c index 439ac5f5907a..7a8e1e4f4d33 100644 --- a/drivers/pci/setup-res.c +++ b/drivers/pci/setup-res.c @@ -450,7 +450,7 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size) res->end = res->start + pci_rebar_size_to_bytes(size) - 1; /* Check if the new config works by trying to assign everything. */ - if (dev->bus->self) { + if (size > old && dev->bus->self) { ret = pci_reassign_bridge_resources(dev->bus->self, res->flags); if (ret) goto error_resize; From patchwork Tue Aug 16 19:41:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 12945252 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6641CC25B0E for ; Tue, 16 Aug 2022 19:41:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236907AbiHPTlL (ORCPT ); Tue, 16 Aug 2022 15:41:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236010AbiHPTlL (ORCPT ); Tue, 16 Aug 2022 15:41:11 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CC17876BD for ; Tue, 16 Aug 2022 12:41:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660678868; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iy9C67BR9Y8mzLSmrgLV3EkX5N4BTVdAX1TDUnMFU5U=; b=JS2MRAOHJywQWwCt9TOi4bovthqVRuuPDjqk+F8FBjRjD+pOtkY99m+A4kpoNhlkpoo9y/ wX1pgFlUv/hveYnLj6MyN32Qcig0DDdpjmygjJzdBENXzK0SkN8PHO/jo8HhUkcqop7shL 0S6MtIF08P4psjWFw01g85Jk1Z+MtTA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-213-uHBhXavFMo26heGh2Sk8OQ-1; Tue, 16 Aug 2022 15:41:05 -0400 X-MC-Unique: uHBhXavFMo26heGh2Sk8OQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1FA0D3C0D19D; Tue, 16 Aug 2022 19:41:05 +0000 (UTC) Received: from [172.30.41.16] (unknown [10.22.18.188]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7DD612026D4C; Tue, 16 Aug 2022 19:41:04 +0000 (UTC) Subject: [RFC PATCH 3/3] PCI: Expose PCIe Resizable BAR support via sysfs From: Alex Williamson To: linux-pci@vger.kernel.org, bhelgaas@google.com Cc: =?utf-8?q?Christian_K=C3=B6nig?= , linux-kernel@vger.kernel.org Date: Tue, 16 Aug 2022 13:41:04 -0600 Message-ID: <166067883598.1885802.7663904087127986133.stgit@omen> In-Reply-To: <166067824399.1885802.12557332818208187324.stgit@omen> References: <166067824399.1885802.12557332818208187324.stgit@omen> User-Agent: StGit/1.5.dev2+g9ce680a52bd9 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This proposes a simple sysfs interface to Resizable BAR support, largely for the purposes of assigning such devices to a VM through VFIO. Resizable BARs present a difficult feature to expose to a VM through emulation, as resizing a BAR is done on the host. It can fail, and often does, but we have no means via emulation of a PCIe REBAR capability to handle the error cases. A vfio-pci specific ioctl interface is also cumbersome as there are often multiple devices within the same bridge aperture and handling them is a challenge. In the interface proposed here, expanding a BAR potentially requires such devices to be soft-removed during the resize operation and rescanned after, in order for all the necessary resources to be released. A pci-sysfs interface is also more universal than a vfio specific interface. Please see the ABI documentation update for usage. Cc: Christian König Signed-off-by: Alex Williamson --- NB, I realize the read value of the syfs attribute provides two values, the bitmap of possible sizes and the current size. There are a number of ways to determine the current size, including stat(1) on the resourceN file, but I found this output to be useful while developing the interface and provides consistency with the store value to the attribute. Suggestions welcome for better semantics. Documentation/ABI/testing/sysfs-bus-pci | 27 +++++++ drivers/pci/pci-sysfs.c | 118 +++++++++++++++++++++++++++++++ include/linux/pci.h | 1 3 files changed, 146 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci index 6fc2c2efe8ab..5eea5d89c9f2 100644 --- a/Documentation/ABI/testing/sysfs-bus-pci +++ b/Documentation/ABI/testing/sysfs-bus-pci @@ -457,3 +457,30 @@ Description: The file is writable if the PF is bound to a driver that implements ->sriov_set_msix_vec_count(). + +What: /sys/bus/pci/devices/.../resourceN_resize +Date: August 2022 +Contact: Alex Williamson +Description: + These files provide an interface to PCIe Resizable BAR support. + A file is created for each BAR resource (N) supported by the + PCIe Resizable BAR extended capability of the device. Reading + each file exposes the capability and current setting for the + device, ex. + + # cat resource1_resize + 00000000000001c0:6 + + The first field provides the supported sizes bitmap, where + bit0 = 1MB, bit1 = 2MB, bit2 = 4MB, etc. In the above example + the devices supports 64MB, 128MB, and 256MB BAR sizes. The + second field provides the current setting, the value 6 + indicates bit6 is set, which corresponds to 64MB. + + When writing the file, only the latter is used, ex. + + # echo 7 > resource1_resize + + This indicates to set the size value corresponding to bit 7, + which is 128MB. The resulting size is 2 ^ (bit# + 20). This + definition matches the PCIe specification of this capability. diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c index 9ac92e6a2397..aa59a2de508f 100644 --- a/drivers/pci/pci-sysfs.c +++ b/drivers/pci/pci-sysfs.c @@ -1143,6 +1143,7 @@ static void pci_remove_resource_files(struct pci_dev *pdev) for (i = 0; i < PCI_STD_NUM_BARS; i++) { struct bin_attribute *res_attr; + struct dev_ext_attribute *resize_attr; res_attr = pdev->res_attr[i]; if (res_attr) { @@ -1155,6 +1156,13 @@ static void pci_remove_resource_files(struct pci_dev *pdev) sysfs_remove_bin_file(&pdev->dev.kobj, res_attr); kfree(res_attr); } + + resize_attr = pdev->res_attr_resize[i]; + if (resize_attr) { + sysfs_remove_file(&pdev->dev.kobj, + &resize_attr->attr.attr); + kfree(resize_attr); + } } } @@ -1208,6 +1216,108 @@ static int pci_create_attr(struct pci_dev *pdev, int num, int write_combine) return retval; } +static ssize_t pci_bar_resize_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct dev_ext_attribute *resize_attr = + container_of(attr, struct dev_ext_attribute, attr); + int bar = (int)(long)resize_attr->var; + ssize_t ret; + + pci_config_pm_runtime_get(pdev); + + /* + * pci_rebar_get_possible_sizes() only currently reads supported sizes + * from the capability register and therefore returns a u32. The spec + * allows additional supported bits in the control register, which + * then exceeds 32bit. Expose a u64 to userspace for future compat. + */ + ret = sysfs_emit(buf, "%016llx:%d\n", + (u64)pci_rebar_get_possible_sizes(pdev, bar), + pci_rebar_get_current_size(pdev, bar)); + + pci_config_pm_runtime_put(pdev); + + return ret; +} + +static ssize_t pci_bar_resize_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct dev_ext_attribute *resize_attr = + container_of(attr, struct dev_ext_attribute, attr); + int ret, i, bar = (int)(long)resize_attr->var; + unsigned long size, flags; + u16 cmd; + + if (kstrtoul(buf, 0, &size) < 0) + return -EINVAL; + + device_lock(dev); + if (dev->driver) { + ret = -EBUSY; + goto unlock; + } + + pci_config_pm_runtime_get(pdev); + + pci_read_config_word(pdev, PCI_COMMAND, &cmd); + pci_write_config_word(pdev, PCI_COMMAND, cmd & ~PCI_COMMAND_MEMORY); + + flags = pci_resource_flags(pdev, bar); + + for (i = 0; i < PCI_STD_NUM_BARS; i++) { + if (pci_resource_len(pdev, i) && + pci_resource_flags(pdev, i) == flags) + pci_release_resource(pdev, i); + } + + ret = pci_resize_resource(pdev, bar, size); + + pci_assign_unassigned_bus_resources(pdev->bus); + + pci_write_config_word(pdev, PCI_COMMAND, cmd); + + pci_config_pm_runtime_put(pdev); +unlock: + device_unlock(dev); + + return ret ? ret : count; +} + +static int pci_create_resize_attr(struct pci_dev *pdev, int num) +{ + struct dev_ext_attribute *resize_attr; + char *resize_attr_name; + int retval; + + resize_attr = kzalloc(sizeof(*resize_attr) + 17, GFP_ATOMIC); + if (!resize_attr) + return -ENOMEM; + + resize_attr_name = (char *)(resize_attr + 1); + + sysfs_attr_init(&resize_attr->attr.attr); + sprintf(resize_attr_name, "resource%d_resize", num); + resize_attr->attr.attr.name = resize_attr_name; + resize_attr->attr.attr.mode = 0600; + resize_attr->attr.show = pci_bar_resize_show; + resize_attr->attr.store = pci_bar_resize_store; + resize_attr->var = (void *)(long)num; + + retval = sysfs_create_file(&pdev->dev.kobj, &resize_attr->attr.attr); + if (retval) { + kfree(resize_attr); + return retval; + } + + pdev->res_attr_resize[num] = resize_attr; + return 0; +} + /** * pci_create_resource_files - create resource files in sysfs for @dev * @pdev: dev in question @@ -1235,6 +1345,14 @@ static int pci_create_resource_files(struct pci_dev *pdev) pci_remove_resource_files(pdev); return retval; } + + if (pci_rebar_get_current_size(pdev, i) >= 0) { + retval = pci_create_resize_attr(pdev, i); + if (retval) { + pci_remove_resource_files(pdev); + return retval; + } + } } return 0; } diff --git a/include/linux/pci.h b/include/linux/pci.h index 060af91bafcd..9c4db0c5f215 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -470,6 +470,7 @@ struct pci_dev { int rom_attr_enabled; /* Display of ROM attribute enabled? */ struct bin_attribute *res_attr[DEVICE_COUNT_RESOURCE]; /* sysfs file for resources */ struct bin_attribute *res_attr_wc[DEVICE_COUNT_RESOURCE]; /* sysfs file for WC mapping of resources */ + struct dev_ext_attribute *res_attr_resize[DEVICE_COUNT_RESOURCE]; /* sysfs file for resizing BAR resources */ #ifdef CONFIG_HOTPLUG_PCI_PCIE unsigned int broken_cmd_compl:1; /* No compl for some cmds */