From patchwork Wed Aug 30 01:11:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bagas Sanjaya X-Patchwork-Id: 13369886 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61858C83F19 for ; Wed, 30 Aug 2023 01:12:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241508AbjH3BLm (ORCPT ); Tue, 29 Aug 2023 21:11:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241488AbjH3BLQ (ORCPT ); Tue, 29 Aug 2023 21:11:16 -0400 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27AF783; Tue, 29 Aug 2023 18:11:14 -0700 (PDT) Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-68bed286169so4309569b3a.1; Tue, 29 Aug 2023 18:11:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693357873; x=1693962673; darn=vger.kernel.org; h=subject:from:cc:to:content-language:user-agent:mime-version:date :message-id:from:to:cc:subject:date:message-id:reply-to; bh=am71R2qky8a6PNSlnd6gAGC+SFc1FXxSFKR0Xr/vvSc=; b=pMfInPpcsPSBTNiuG1N/vRUMWsymCo+7tQtn++J3pVRu0X1TfAkUX1ecjveCAtAz3I yeQu8SUnRVye//iAcuLyL3WpahedEp9N9PnTR+WUlbMnv862bH3zXyRLBWFv4U5ocROg AStz4vGCM2znOQri2lsASghhky3Mxi85BqZlwR6k0XrG/7vvEibVM5dCVIvCZE6HWRIA BjB5KgVVqg8Ct6L4N9hvOIVbKRsGlSn/7bzKuklcXs2RgwbMdWLBSBKgGz2l6ZN8T897 TFWA+OtFRA/q/Jq641quLF/TVjrNw4mZBUTuqE+Ujp1o8tkXDfFd7/WxRfZzgyFfU0Nv cgqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693357873; x=1693962673; h=subject:from:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=am71R2qky8a6PNSlnd6gAGC+SFc1FXxSFKR0Xr/vvSc=; b=Y9LTezBOQVwAOlKiErAxARNlr0t/NVbpa4xLasOt/iL2GYsPB79kbFX5nOkkxgU+8N h4E/wOnV01N2Ec1B7kBrBB8fCsC8vRPtHWtbB8qJ1y7XXm6z/go3LjUGrbZhzONDJ7Ce LRvxkZwlL97gkq3n643pXmnW/sJYN+Q8GQDn4k5zud/ItJfJ/1KP9RUDJPVNTkrlS+DZ 4ANr55C+LFcIgspyKgOrbgvY+Aa5dO0H1IyrqkFLeE8+idAeiu9Q9lrV0h9dcNW9f7H2 I5uyYd1Y40FPqlQ9RyAgJZ1ZHVX/LNCcokKW/r26zwq8+Pv4+x+caLH+ph5w/NA30pDU lnzA== X-Gm-Message-State: AOJu0Yw9DQI2vGGdX1gdtnCFmu5/sVLxCq5aI6XnF05okCCPwWjARKFO LDgztWS8w9eGBt8hxrWc7Yc= X-Google-Smtp-Source: AGHT+IGW6Mnz5eiZOzB0GyC2Mxdqc4toEBDK29wnYzzWWxk/0O8RTvhbGwKvv+gZCMgY+cnRZ51JeQ== X-Received: by 2002:a05:6a20:9497:b0:141:d640:794a with SMTP id hs23-20020a056a20949700b00141d640794amr889158pzb.39.1693357873498; Tue, 29 Aug 2023 18:11:13 -0700 (PDT) Received: from [192.168.0.105] ([103.124.138.83]) by smtp.gmail.com with ESMTPSA id jc17-20020a17090325d100b001aaf2e8b1eesm10032834plb.248.2023.08.29.18.11.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 29 Aug 2023 18:11:12 -0700 (PDT) Message-ID: <88ffb216-96f9-f232-7fe5-48bf82e6aa70@gmail.com> Date: Wed, 30 Aug 2023 08:11:06 +0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Content-Language: en-US To: Bjorn Helgaas , "Maciej W. Rozycki" , Kuppuswamy Sathyanarayanan , Ajay Agarwal , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vidya Sagar , Michael Bottini , "Rafael J. Wysocki" , Brett Hassall Cc: Linux Kernel Mailing List , Linux PCI , Linux Power Management From: Bagas Sanjaya Subject: Fwd: upstream linux cannot achieve package C8 power saving Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Hi, I notice a bug report on Bugzilla [1]. Quoting from it: > v6.5 (and at least v5.15, v5.19 and v6.4 as well) will not go to a higher power saving level than package C3. > > With the inclusion of a patch that combines 3 Ubuntu commits related to VMD ASPM & LTR, package C8 is used. See Bugzilla for the full thread. FYI, the attached proposed fix is the same as Brett's another BZ report [2]. I include it for upstreaming. To Brett: Would you like to submit the proper, formal patch (see Documentation/process/submitting-patches.rst for details)? Thanks. [1]: https://bugzilla.kernel.org/show_bug.cgi?id=217841 [2]: https://bugzilla.kernel.org/show_bug.cgi?id=217828 commit 9491bfa0b1e5f658e09135759e7ebc373d9a72ce Author: brett.hassall Date: Thu Aug 24 19:26:36 2023 +1000 combined commit of: 71315b8e - UBUNTU: SAUCE: PCI/ASPM: Enable ASPM for links under VMD domain; d83e6f6e - UBUNTU: SAUCE: PCI/ASPM: Enable LTR for endpoints behind VMD; 069d0523 - UBUNTU: SAUCE: vmd: fixup bridge ASPM by driver name instead diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 66d7514ca111..29f2f62aaefa 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -682,7 +682,8 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) aspm_l1ss_init(link); /* Save default state */ - link->aspm_default = link->aspm_enabled; + link->aspm_default = parent->dev_flags & PCI_DEV_FLAGS_ENABLE_ASPM ? + ASPM_STATE_ALL : link->aspm_enabled; /* Setup initial capable state. Will be updated later */ link->aspm_capable = link->aspm_support; diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index c525867760bf..a77e2e38fa6a 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -6041,3 +6041,71 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x9a2d, dpc_log_size); DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x9a2f, dpc_log_size); DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x9a31, dpc_log_size); #endif + +/* + * BIOS may not be able to access config space of devices under VMD domain, so + * it relies on software to enable ASPM for links under VMD. + */ +static bool pci_fixup_is_vmd_bridge(struct pci_dev *pdev) +{ + struct pci_bus *bus = pdev->bus; + struct device *dev; + struct pci_driver *pdrv; + + if (!pci_is_root_bus(bus)) + return false; + + dev = bus->bridge->parent; + if (dev == NULL) + return false; + + pdrv = pci_dev_driver(to_pci_dev(dev)); + if (pdrv == NULL || strcmp("vmd", pdrv->name)) + return false; + + return true; +} + +static void pci_fixup_enable_aspm(struct pci_dev *pdev) +{ + if (!pci_fixup_is_vmd_bridge(pdev)) + return; + + pdev->dev_flags |= PCI_DEV_FLAGS_ENABLE_ASPM; + pci_info(pdev, "enable ASPM for pci bridge behind vmd"); +} +DECLARE_PCI_FIXUP_CLASS_HEADER(PCI_VENDOR_ID_INTEL, PCI_ANY_ID, + PCI_CLASS_BRIDGE_PCI, 8, pci_fixup_enable_aspm); + +static void pci_fixup_enable_vmd_nvme_ltr(struct pci_dev *pdev) +{ + struct pci_dev *parent; + int pos; + u16 val; + + parent = pci_upstream_bridge(pdev); + if (!parent) + return; + + if (!pci_fixup_is_vmd_bridge(parent)) + return; + + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_LTR); + if (!pos) + return; + + pci_read_config_word(pdev, pos + PCI_LTR_MAX_SNOOP_LAT, &val); + if (val) + return; + + pci_read_config_word(pdev, pos + PCI_LTR_MAX_NOSNOOP_LAT, &val); + if (val) + return; + + /* 3145728ns, i.e. 0x300000ns */ + pci_write_config_word(pdev, pos + PCI_LTR_MAX_SNOOP_LAT, 0x1003); + pci_write_config_word(pdev, pos + PCI_LTR_MAX_NOSNOOP_LAT, 0x1003); + pci_info(pdev, "enable LTR for nvme behind vmd"); +} +DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_ANY_ID, PCI_ANY_ID, + PCI_CLASS_STORAGE_EXPRESS, 0, pci_fixup_enable_vmd_nvme_ltr); diff --git a/include/linux/pci.h b/include/linux/pci.h index 60b8772b5bd4..d3f96d111250 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -245,6 +245,8 @@ enum pci_dev_flags { PCI_DEV_FLAGS_NO_RELAXED_ORDERING = (__force pci_dev_flags_t) (1 << 11), /* Device does honor MSI masking despite saying otherwise */ PCI_DEV_FLAGS_HAS_MSI_MASKING = (__force pci_dev_flags_t) (1 << 12), + /* Enable ASPM regardless of how LnkCtl is programmed */ + PCI_DEV_FLAGS_ENABLE_ASPM = (__force pci_dev_flags_t) (1 << 13), }; enum pci_irq_reroute_variant {