From patchwork Thu Oct 14 09:57:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: yaozhenguo X-Patchwork-Id: 12558185 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8CADC433EF for ; Thu, 14 Oct 2021 09:58:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CE8496101E for ; Thu, 14 Oct 2021 09:58:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230145AbhJNKAb (ORCPT ); Thu, 14 Oct 2021 06:00:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230049AbhJNKAa (ORCPT ); Thu, 14 Oct 2021 06:00:30 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0202FC061570; Thu, 14 Oct 2021 02:58:26 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id g13-20020a17090a3c8d00b00196286963b9so6521623pjc.3; Thu, 14 Oct 2021 02:58:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7oaee+4oXg8HKCv9PzcF0QIAtHVH0NEu6qYXRmh9b7E=; b=SYT3ed+Kz4WckqM1sdzgCokwg940ETQPZayiRvNZNuxLmq9tIMS0KlpJHvzxURHvF8 SuGEOR8eZpBIkLiP1RzZHIAE8e4OHMHFaAKngIH/FZz/eogqpBBS7fGQsOeeakWVy402 hXVWXk3tlf+cxQpDFxSvoKCGGTMquePZU+fvNST57OE8D+kpHtMRlgqBo4mtR+em0lvH 4ki00TI11LY4vfRQkYQwtrw5LLXkg7HppWdxAoPVvUqbKPf2KuO+uwBesRxAP0WXljwc bOzFPAMxqkYpZNgSyUYVzmK4ElfmAoR5JHx29JLYkNeeqakmDvggJ/gIsRdSj9a7kppb THbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7oaee+4oXg8HKCv9PzcF0QIAtHVH0NEu6qYXRmh9b7E=; b=47Sub2eRAvsi033oh8bpdkwABV1hLC9ckxvXApTVjd6gnOy7Nk2awPr5Z5qOhXxH32 MQwCZmz+4taqH36nNerfkaPNKopw0uPEEViIDw87WjwXlyeUIQeGyBaHPyd3eibI3eiJ ykBJKBmuOyXi0K5sFA6yNR77PXRr02dM5EVucDWmTDqCYxXXvimrhBOsC89ye7afvXxq rryX5BnKmsbw+FEGaqKWEoTpBs5ikg/1dHu+OrZp2Qs5s7InqHXUbLhYxaVSiCXKMS2G 33Ir61Xng/x8DS0EUXit3bSBQCFLZV/6HKtl05haF5lJ2vdFx1Sr6Aw9DHr+Yv44ZOZV gaRA== X-Gm-Message-State: AOAM532fletkB2IXqsPvWQQbPVkMQQgXgBT+n0Mh/4k7fbjsP1xdtUEX HWGmzmyU2fMvwMWDS6Oh0u5oc+03ddbndg== X-Google-Smtp-Source: ABdhPJw72G2WanonNLSptbeZnKFwjyD1IiMloGP9Teb6H/jy+fy3maiOi+RWy6kdTz/K97d05wxcHA== X-Received: by 2002:a17:90b:1049:: with SMTP id gq9mr16087657pjb.180.1634205505581; Thu, 14 Oct 2021 02:58:25 -0700 (PDT) Received: from localhost.localdomain (5e.8a.38a9.ip4.static.sl-reverse.com. [169.56.138.94]) by smtp.gmail.com with ESMTPSA id k127sm2080664pfd.1.2021.10.14.02.58.22 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Oct 2021 02:58:25 -0700 (PDT) From: Zhenguo Yao To: bhelgaas@google.com, alex.williamson@redhat.com Cc: cohuck@redhat.com, jgg@ziepe.ca, mgurtovoy@nvidia.com, yishaih@nvidia.com, kvm@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, yaozhenguo@jd.com, Zhenguo Yao Subject: [PATCH v1 1/2] PCI: Add ignore_reset sysfs interface to control whether to do device reset in PCI drivers Date: Thu, 14 Oct 2021 17:57:47 +0800 Message-Id: <20211014095748.84604-2-yaozhenguo1@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211014095748.84604-1-yaozhenguo1@gmail.com> References: <20211014095748.84604-1-yaozhenguo1@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Some PCI devices can't do device reset in enable and disable operations. So, and ignore_reset sysfs interface to ignore reset in those operations. For example: echo 1 > /sys/bus/pci/xxxx/ignore_reset PCI drivers can ignore reset for this device based on ignore_reset. Signed-off-by: Zhenguo Yao --- drivers/pci/pci-sysfs.c | 25 +++++++++++++++++++++++++ include/linux/pci.h | 1 + 2 files changed, 26 insertions(+) diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c index 7fb5cd17cc98..c2fa2ed3ae55 100644 --- a/drivers/pci/pci-sysfs.c +++ b/drivers/pci/pci-sysfs.c @@ -255,6 +255,30 @@ static ssize_t ari_enabled_show(struct device *dev, } static DEVICE_ATTR_RO(ari_enabled); +static ssize_t ignore_reset_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct pci_dev *pci_dev = to_pci_dev(dev); + + return sprintf(buf, "%u\n", pci_dev->ignore_reset); +} +static ssize_t ignore_reset_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct pci_dev *pdev = to_pci_dev(dev); + unsigned long val; + + if (kstrtoul(buf, 0, &val) < 0) + return -EINVAL; + + pdev->ignore_reset = !!val; + + return count; +} +static DEVICE_ATTR_RW(ignore_reset); + static ssize_t modalias_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -618,6 +642,7 @@ static struct attribute *pci_dev_attrs[] = { #endif &dev_attr_driver_override.attr, &dev_attr_ari_enabled.attr, + &dev_attr_ignore_reset.attr, NULL, }; diff --git a/include/linux/pci.h b/include/linux/pci.h index cd8aa6fce204..ac026acd4572 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -508,6 +508,7 @@ struct pci_dev { /* These methods index pci_reset_fn_methods[] */ u8 reset_methods[PCI_NUM_RESET_METHODS]; /* In priority order */ + u8 ignore_reset; /* ignore reset control in driver */ }; static inline struct pci_dev *pci_physfn(struct pci_dev *dev) From patchwork Thu Oct 14 09:57:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: yaozhenguo X-Patchwork-Id: 12558187 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86129C433EF for ; Thu, 14 Oct 2021 09:58:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6AFEA6101E for ; Thu, 14 Oct 2021 09:58:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230150AbhJNKAn (ORCPT ); Thu, 14 Oct 2021 06:00:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230179AbhJNKAm (ORCPT ); Thu, 14 Oct 2021 06:00:42 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE4C9C061760; Thu, 14 Oct 2021 02:58:37 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id l6so3791240plh.9; Thu, 14 Oct 2021 02:58:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7pLIyNJD+d5mAa5PFcs4g1QmPkUK7kz7RJBNS5MvFNU=; b=Ah6yRqVAdD4LxJmM0L+oAgTEvbnLGQdgTDLVPsNv4h/MGGz4gP8NxmaQFWlLnlnnCV iCeC/m2bFkk2ynY0SAPHDJO7L+KjudbSaYQAwNrsWfRMqdnh/yftsB20y+tg8yCi+GZk cv2KGgSQAgK9f4l2yXlNlnvR0NKWMkC2wGuU5te8kHtdNvErrrH/vJEQSVSQbjpcx21X WZ0VwIvpbAZ0shhyRFxEriDi6CNi46ejgK4mM2N5oNZKlqokpGwyCurtfsxWNdEbV05u 0WrEd8qvFuUqPVLO/kjv1aCYD70loJVVGTh4tbU8wDG+FwS9xOcScT3r8Y0J8A2KEYQ0 Xtiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7pLIyNJD+d5mAa5PFcs4g1QmPkUK7kz7RJBNS5MvFNU=; b=p4L7GMuIm5c6/RJS50//6Mt+w2SaNO7l2EJ0Xn+HOFMIzplbvyOzt0XmzXIZwcgUA4 0alG7GDDHXUPjC5dRUeI0rsbxvln5RZinRg1xenXhOo/XDjEtxJfEKNZG+QNtV9i7iH7 LWpxAhnyCnhWIh91OAgkvE/abwLmYo8tqlTMhA+voq4EtAVYrLEfyBPcHyfPHEv/McPP rVBMoNkSp20ciNqNMt1vf3QDO2G6d3E8V5mTqdYmsKdR+i81i/1WsIx6JssdlExl0wsN yT77X5bSoAu8p+3mkDKyL2p4NKXKpnB5c8/gyHhDLb8jZLbDH5JWJqXU4cqbXngTrts6 fUAA== X-Gm-Message-State: AOAM531p0HkC+3qut4fej7CeKbNG9PxjjFCYCZWxeK9sY+XB15z8i+UV wrcX+3N95PP/02zRGljd+sk= X-Google-Smtp-Source: ABdhPJzXxvkWR+vqomWVBACjT7oXOvcLdFI29NK4NAXfXv87ebuNlyJybNcBQfZnAabrIZRuGoTl9g== X-Received: by 2002:a17:903:31cd:b0:134:5b6f:2ff8 with SMTP id v13-20020a17090331cd00b001345b6f2ff8mr4156279ple.46.1634205517429; Thu, 14 Oct 2021 02:58:37 -0700 (PDT) Received: from localhost.localdomain (5e.8a.38a9.ip4.static.sl-reverse.com. [169.56.138.94]) by smtp.gmail.com with ESMTPSA id k127sm2080664pfd.1.2021.10.14.02.58.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Oct 2021 02:58:37 -0700 (PDT) From: Zhenguo Yao To: bhelgaas@google.com, alex.williamson@redhat.com Cc: cohuck@redhat.com, jgg@ziepe.ca, mgurtovoy@nvidia.com, yishaih@nvidia.com, kvm@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, yaozhenguo@jd.com, Zhenguo Yao Subject: [PATCH v1 2/2] vfio-pci: Don't do device reset when ignore_reset is setting Date: Thu, 14 Oct 2021 17:57:48 +0800 Message-Id: <20211014095748.84604-3-yaozhenguo1@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211014095748.84604-1-yaozhenguo1@gmail.com> References: <20211014095748.84604-1-yaozhenguo1@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org In some scenarios, vfio device can't do any reset in initialization process. For example: Nvswitch and GPU A100 working in Shared NVSwitch Virtualization Model. In such mode, The GPUs can't do any reset when Guest VM is booting up. So, Using ignore_reset to control whether to do PCI reset in initialization. In Shared NVSwitch Virtualization Model, GPUs will ignore reset when Gust VM booting up. Signed-off-by: Zhenguo Yao --- drivers/vfio/pci/vfio_pci_core.c | 48 ++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 68198e0f2a63..83d3ef5d3a9c 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -254,11 +254,13 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev) if (ret) return ret; - /* If reset fails because of the device lock, fail this path entirely */ - ret = pci_try_reset_function(pdev); - if (ret == -EAGAIN) { - pci_disable_device(pdev); - return ret; + if (!pdev->ignore_reset) { + /* If reset fails because of the device lock, fail this path entirely */ + ret = pci_try_reset_function(pdev); + if (ret == -EAGAIN) { + pci_disable_device(pdev); + return ret; + } } vdev->reset_works = !ret; @@ -388,25 +390,30 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev) */ pci_write_config_word(pdev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE); - /* - * Try to get the locks ourselves to prevent a deadlock. The - * success of this is dependent on being able to lock the device, - * which is not always possible. - * We can not use the "try" reset interface here, which will - * overwrite the previously restored configuration information. - */ - if (vdev->reset_works && pci_dev_trylock(pdev)) { - if (!__pci_reset_function_locked(pdev)) - vdev->needs_reset = false; - pci_dev_unlock(pdev); + if (!pdev->ignore_reset) { + /* + * Try to get the locks ourselves to prevent a deadlock. The + * success of this is dependent on being able to lock the device, + * which is not always possible. + * We can not use the "try" reset interface here, which will + * overwrite the previously restored configuration information. + */ + if (vdev->reset_works && pci_dev_trylock(pdev)) { + if (!__pci_reset_function_locked(pdev)) + vdev->needs_reset = false; + pci_dev_unlock(pdev); + } } pci_restore_state(pdev); out: pci_disable_device(pdev); - if (!vfio_pci_dev_set_try_reset(vdev->vdev.dev_set) && !disable_idle_d3) - vfio_pci_set_power_state(vdev, PCI_D3hot); + if (!pdev->ignore_reset) { + if (!vfio_pci_dev_set_try_reset(vdev->vdev.dev_set) && + !disable_idle_d3) + vfio_pci_set_power_state(vdev, PCI_D3hot); + } } EXPORT_SYMBOL_GPL(vfio_pci_core_disable); @@ -919,6 +926,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, if (!vdev->reset_works) return -EINVAL; + if (vdev->pdev->ignore_reset) + return -EINVAL; vfio_pci_zap_and_down_write_memory_lock(vdev); ret = pci_try_reset_function(vdev->pdev); @@ -1007,6 +1016,9 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, bool slot = false; int group_idx, count = 0, ret = 0; + if (vdev->pdev->ignore_reset) + return -EINVAL; + minsz = offsetofend(struct vfio_pci_hot_reset, count); if (copy_from_user(&hdr, (void __user *)arg, minsz))