From patchwork Tue Sep 24 07:14:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 13810348 Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4952A145B24 for ; Tue, 24 Sep 2024 07:24:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727162649; cv=none; b=H4dpKB79tNbwlud3UAOCOjB+4uQuRyG3KBjh6spQ1UNfwA4ax/wDq0SDCgA6uNjjJW+IiAJOiF1DFy6oBCdGl+RSuRk5j20zDh1qOYqqMmX2Eop1oo/l5qR/Rkw7TCjvpi9+GBOEFDA9c6qrJvK73HjdQ98I7wS0uqgetRBMgIA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727162649; c=relaxed/simple; bh=/kR8Y+aQ/hoCK+GI5dTp0cvDs3+d+f/zp1O4+MFbywU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cTvgCWqj/b8MP9Dkw5IawSMMky9uvrmZvgInQIG0ymy0iNdehpMowAxcTw38nDC4AvnOZbzyc1N6W6sNDc3lkAopyVsqgxb7m7jlWitTehDPOdJnH9SaPMWD5CNIs8NQB215UXZOgQ2uvRajWOMbldPxDfJtdUlwUBsmFjlcYos= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org; spf=pass smtp.mailfrom=endlessos.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b=Z7Am/Vyq; arc=none smtp.client-ip=209.85.167.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=endlessos.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b="Z7Am/Vyq" Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3e03f0564c6so3091394b6e.2 for ; Tue, 24 Sep 2024 00:24:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessos.org; s=google; t=1727162647; x=1727767447; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4LU6Uewc0Mc72sE2KkcnJ2E4+BrFGvMvKo0jKAieZwk=; b=Z7Am/VyqG5eryauQHR/0K3VIVDkloYGRxBs3Sod+jwoKUu0hm1NMKYaJ7QkV+8rt2a J961QCjtB0MVyVNkmMd0enE8W7FSnxVflJKX/L6PfMLetYo101HyxIuUcn9Gk4bqvx7o g+W5z+AbOKRR14oSOztGdWkmnblBLD1yN0tKWUBVHlZPb40MlLAPXh3ehGBfkGQC+sPr FrVbhm4YJxSTGALVklzyHWfbIgxQE9ZgBvm8h/7XG8oNK0SUVG7hgfoEQmly95vW7pQ9 4k7FrfwTBWVgMwmrNn4oIFJibzebhLyayQg/Cn+ktllepvn1LrDIReJloqzrstzxvTan 68EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727162647; x=1727767447; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4LU6Uewc0Mc72sE2KkcnJ2E4+BrFGvMvKo0jKAieZwk=; b=jnUMtjvoI1lwDHl2zk0H0HZ0I20PdiJ7AMYvRAJRgpaPgIacvxgD4t8Rdi6pwW5/vh dphNzGTxfXk29ZuwMVPbsNsPJXz9Pn3degSH+hxBXmBvzwHFz3bn0xxYZMpfYuzjJcU9 ilkSGGJPSWpehmVyQGhq1zwjrO2GkvDONghQO9S53tQajcnvmNZxOS9QGb1pC4ZQ0Wsc 6Aw4VdyZfjPO/C8GFa+7W5Rplmm1Qhy4lCs+Chr8vjipVEhkJZa91SebzAWz4WDqyL0K FZgtbowqclVpeAsxn9Ust4OnZ5L4BVxDHDdjQV3i5BrDukYcWypTxs1eyQRncaUNKyBE 3O5g== X-Forwarded-Encrypted: i=1; AJvYcCXO3ttbJSTNoKYIDsqJc6PQP6ZIkPh27JqF/l4OUioe6hO8X3Duhc4krCF1t4gz+8azh6aZRHJyfok=@vger.kernel.org X-Gm-Message-State: AOJu0Yy3vz/kogeDX5T1+x/3mkI+tFPUyNECTe+9IzVk3/nF1+ejRb3k WNEAvO9aptSpdhjAibHV3iPbMKz+6b3NJAiDK1fa1EyzQ88GI4uz2hlwHkV2tAI= X-Google-Smtp-Source: AGHT+IGadQg5IXmezactvoif7KfUxM+L9h4qHNoX87Y/FzZtP8aQ7na2cHB8Cu0x/pJ33tZPRInQeg== X-Received: by 2002:a05:6808:3389:b0:3e0:3547:b0a with SMTP id 5614622812f47-3e271ce046emr8037918b6e.35.1727162647220; Tue, 24 Sep 2024 00:24:07 -0700 (PDT) Received: from localhost.localdomain ([123.51.167.56]) by smtp.googlemail.com with ESMTPSA id 41be03b00d2f7-7e6b7c8675esm599530a12.92.2024.09.24.00.24.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Sep 2024 00:24:06 -0700 (PDT) From: Jian-Hong Pan To: Bjorn Helgaas Cc: Johan Hovold , David Box , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Kuppuswamy Sathyanarayanan , Nirmal Patel , Jonathan Derrick , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessos.org, Jian-Hong Pan Subject: [PATCH v9 1/3] PCI: vmd: Set PCI devices to D0 before enable PCI PM's L1 substates Date: Tue, 24 Sep 2024 15:14:20 +0800 Message-ID: <20240924071419.15350-2-jhp@endlessos.org> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20240924070551.14976-2-jhp@endlessos.org> References: <20240924070551.14976-2-jhp@endlessos.org> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The remapped PCIe Root Port and the child device have PCI PM L1 substates capability, but they are disabled originally. Here is a failed example on ASUS B1400CEAE: Capabilities: [900 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+ PortCommonModeRestoreTime=32us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=101376ns L1SubCtl2: T_PwrOn=50us Power on all of the VMD remapped PCI devices to D0 before enable PCI-PM L1 PM Substates by following "PCIe r6.0, sec 5.5.4". Link: https://bugzilla.kernel.org/show_bug.cgi?id=218394 Signed-off-by: Jian-Hong Pan --- v2: - Power on the VMD remapped devices with pci_set_power_state_locked() - Prepare the PCIe LTR parameters before enable L1 Substates - Add note into the comments of both pci_enable_link_state() and pci_enable_link_state_locked() for kernel-doc. - The original patch set can be split as individual patches. v3: - Re-send for the missed version information. - Split drivers/pci/pcie/aspm.c modification into following patches. - Fix the comment for enasuring the PCI devices in D0. v4: - The same v5: - Tweak the commit title and message - Change the goto label from out_enable_link_state to out_state_change v6~8: - The same v9: - Update L1 PM Substates information against kernel v6.11 in commit message drivers/pci/controller/vmd.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 264a180403a0..11870d1fc818 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -740,11 +740,9 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) if (!(features & VMD_FEAT_BIOS_PM_QUIRK)) return 0; - pci_enable_link_state_locked(pdev, PCIE_LINK_STATE_ALL); - pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_LTR); if (!pos) - return 0; + goto out_state_change; /* * Skip if the max snoop LTR is non-zero, indicating BIOS has set it @@ -752,7 +750,7 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) */ pci_read_config_dword(pdev, pos + PCI_LTR_MAX_SNOOP_LAT, <r_reg); if (!!(ltr_reg & (PCI_LTR_VALUE_MASK | PCI_LTR_SCALE_MASK))) - return 0; + goto out_state_change; /* * Set the default values to the maximum required by the platform to @@ -764,6 +762,13 @@ static int vmd_pm_enable_quirk(struct pci_dev *pdev, void *userdata) pci_write_config_dword(pdev, pos + PCI_LTR_MAX_SNOOP_LAT, ltr_reg); pci_info(pdev, "VMD: Default LTR value set by driver\n"); +out_state_change: + /* + * Ensure devices are in D0 before enabling PCI-PM L1 PM Substates, per + * PCIe r6.0, sec 5.5.4. + */ + pci_set_power_state_locked(pdev, PCI_D0); + pci_enable_link_state_locked(pdev, PCIE_LINK_STATE_ALL); return 0; } From patchwork Tue Sep 24 07:26:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 13810400 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DD371459F7 for ; Tue, 24 Sep 2024 07:27:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727162872; cv=none; b=n7Ci7JUK4m96Mcwms9NGNBWz7BIriwOCX+2Zy/Zpiz20lt1d83bDBAAYXYNoHqEG7FKJg762gE8EzNoBH2bzD71wqImfGnoKIXddeVuoePaphQvQOsRBAv55FCwAqej2152T8ihiVcF/bZCuDsZwO7kHXF8JXgbQwIfEaJ4LUhU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727162872; c=relaxed/simple; bh=wDk5fvI7DTke43bpHTJyGzKtNvv1AEHrlqAhxcXKJyU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=usZk13IDasYzVpgV7kXgg5g+6a6AljtIWSGzGtP11Q7rVNhTLZ6/Clh2REapBN5jG5GV4UmLQrVElrA3w6E+mlgJ+LReM4pJgrbCNfS65eKmwZe0MsMB3Y4N7hDyPB68TRifB4XQgp2VdI50pUrsSbVdkPrjssnFqiYDoqwRBzA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org; spf=pass smtp.mailfrom=endlessos.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b=vcSLO3u1; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=endlessos.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b="vcSLO3u1" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-206aee40676so41753305ad.0 for ; Tue, 24 Sep 2024 00:27:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessos.org; s=google; t=1727162870; x=1727767670; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4BdXjDYFtB7GyHLjEK5w/+zeuqBSDXib6zpEUFEHzsk=; b=vcSLO3u14gwAzyAJ9QbqfhRAADrAQ/7cKES+KTEINYtZQGFeX2txKIxRGwZhDbmAfv Aoqj0Zhmi3fEe0AmCp6jKpEsShvHgmcmEIwt6/ZjmTucf807KEq0zEL/pBdPV5O3KPUu eSq9SSqARTMCR4vXY3ZsgeyXuiVpfXSMR9Tm6Py25DUhmoP9RbDWfSuiaqNlKYNvpp1k ev9Umm1Y5/ngCDlfOxvNe56eeuMtZwLv2l6oq9OrbhoL3XTIo+3niLgtEmxnbyC0sEON umRCuVDo+//bzFcQc0RPvNHtcBrCrSoBq8PNPA+8IqxqY0Op5I5YLwGFQNUb+v7mHp1N 599A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727162870; x=1727767670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4BdXjDYFtB7GyHLjEK5w/+zeuqBSDXib6zpEUFEHzsk=; b=OifQkyXtc+kSfhJJeWEwB/gwICKKBX1S6sMgsQ+jlBdcQj5qgKpXJO4Wl9nndkpBxt y0uDS9bcV+DCWyDFVPoO3qhVf+j/vWyuweklg379FKDTSXqbT765Mtu2vlfJjHgdktn0 b0HyOiAz7azt1m1m1y5/pkYtI8hdg4u7t5TQaeYqLwFwSBK/kCis0maeD/QHQ8ROQlnl Tu/qpF6yCrtGoKzIRZQYZS6JuzGP3m2lMTNC6KfRKG+h+NN0i6zeKT/sGcakf4tddG0d T4kU/N/D+LHF7n5UC2eT6hBSv/ObAWJJUHKqYRxZwOuujAJohDgDuh80MKFgbV+Ot1l9 KISA== X-Forwarded-Encrypted: i=1; AJvYcCV7RGwAEJH2j9qnbeLDHfuwZKElCJCS78ouTxxgTIM/h5ZnkebqEMFZtdJs7NlNHHAxYVfmgiI3r88=@vger.kernel.org X-Gm-Message-State: AOJu0YyG6flVyH2kVOivmHaOmQ7EKot8oY1gKC0dAXewAEhYWiHGAWQ9 icFLxg/wBHBAdDgpTggAapcU9ilPgPdWPUjcgmKGlODFsoCWke/Dt81LtDSGkas= X-Google-Smtp-Source: AGHT+IE+tL5xfdcYObJI4/sBy4cebqhvd85Sg72wdeYMWr3vRZx/oCsnaZOqqeoRvYg+uyBl8hgjsw== X-Received: by 2002:a17:903:192:b0:207:1825:c65e with SMTP id d9443c01a7336-208d838654amr230680435ad.18.1727162870448; Tue, 24 Sep 2024 00:27:50 -0700 (PDT) Received: from localhost.localdomain ([123.51.167.56]) by smtp.googlemail.com with ESMTPSA id d9443c01a7336-20af16e02c8sm5498105ad.31.2024.09.24.00.27.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Sep 2024 00:27:50 -0700 (PDT) From: Jian-Hong Pan To: Bjorn Helgaas Cc: Johan Hovold , David Box , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Kuppuswamy Sathyanarayanan , Nirmal Patel , Jonathan Derrick , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessos.org, Jian-Hong Pan , Bjorn Helgaas Subject: [PATCH v9 2/3] PCI/ASPM: Add notes about enabling PCI-PM L1SS to pci_enable_link_state(_locked) Date: Tue, 24 Sep 2024 15:26:58 +0800 Message-ID: <20240924072657.15887-2-jhp@endlessos.org> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20240924070551.14976-2-jhp@endlessos.org> References: <20240924070551.14976-2-jhp@endlessos.org> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 According to "PCIe r6.0, sec 5.5.4", add note about D0 requirement in pci_enable_link_state() kernel-doc. Signed-off-by: Jian-Hong Pan Acked-by: Bjorn Helgaas Reviewed-by: Kuppuswamy Sathyanarayanan --- v3: - Fix as readable comments v4: - The same v5: - Tweak and simplify the commit message v6~9: - The same drivers/pci/pcie/aspm.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index cee2365e54b8..bd0a8a05647e 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -1442,6 +1442,9 @@ static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool locked) * touch the LNKCTL register. Also note that this does not enable states * disabled by pci_disable_link_state(). Return 0 or a negative errno. * + * Note: Ensure devices are in D0 before enabling PCI-PM L1 PM Substates, per + * PCIe r6.0, sec 5.5.4. + * * @pdev: PCI device * @state: Mask of ASPM link states to enable */ @@ -1458,6 +1461,9 @@ EXPORT_SYMBOL(pci_enable_link_state); * can't touch the LNKCTL register. Also note that this does not enable states * disabled by pci_disable_link_state(). Return 0 or a negative errno. * + * Note: Ensure devices are in D0 before enabling PCI-PM L1 PM Substates, per + * PCIe r6.0, sec 5.5.4. + * * @pdev: PCI device * @state: Mask of ASPM link states to enable * From patchwork Tue Sep 24 07:29:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 13810401 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 989F2770F5 for ; Tue, 24 Sep 2024 07:30:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727163050; cv=none; b=EBCrt4Mdx5MP59VIMOE+T+K93w9gY/SSQU5L4X/um41jP3lCYrCVhWM8am3j0L/DSAkhwx2OtiOYzRCeCA5Ic4oPp/Wm+2v6rSN2169nO1+ktAmGliS+yHg8NsAvuQ8Tdr9+wtbFC+iVYoSIhibcXHpk/OdJwTfZbQhRc2kAz4c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727163050; c=relaxed/simple; bh=Io2e8sR2b43y3WIcEHaQwF43fQuzPU2cLraygvnVFQ4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=khP3seOFoHeZ35Vcuuie+DhchM4fIMdc+kivSmo87FlI6Ixke0YnTIalm3MzzG4E0cc3nWh0V93tP6iaNhpeI9O/i4V8X3HGAQ+hkNeUJWPdg2Jpqub9Wx01MusVe8B5GHGn1OcBabC5D8bHtq9/wpGjPkUj6Wp3S/40a9P+oX8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org; spf=pass smtp.mailfrom=endlessos.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b=ZjUqXY14; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=endlessos.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=endlessos.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=endlessos.org header.i=@endlessos.org header.b="ZjUqXY14" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3e27a6d0bb5so1351119b6e.2 for ; Tue, 24 Sep 2024 00:30:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessos.org; s=google; t=1727163047; x=1727767847; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5Vs1iiE3td9GEXbCoIhFdbgrLQRtaEywEsGPYjRBlwo=; b=ZjUqXY14qYief/lxiF5oTFAcTyiziy2SKP6qqU1GttFqp7Ko4SqxpbuMf+ficti6dS yUOMDf48B9SurlTekzt7oJy7/DtSxQW8aEqcvKBiYM/tktqbBgXno+AkFIXZeUEy1CfJ 6N8m3sHJoldLMaOxziRMOi0YxdQoLrbCrl0W4vECuJy7sEVlG2T16TNErLWWHT5qBGRT D6cTNTt4omaqNzZWLXVNW8aiSxJkKmmEgJmf3lscjZllkdSim3i4zBwDOdrLgOpennEg ktuwpNngPy/cHMd7pMRyPhHEdDCotFO6JWFUyp5OCHszfE4BAlWL34slydWfY8flFPu6 5U2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727163047; x=1727767847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5Vs1iiE3td9GEXbCoIhFdbgrLQRtaEywEsGPYjRBlwo=; b=uDYRSpliM1whQakqyKhDE3blijuasYEXWJr696GYiiGM1MX0VzyU4OJQ8idcY6jLyV zWigRCJp5qSkOlP4BSkltNmB/HuEbvtgOgA9xTkABW7X0LOqLk6JMA7Un/xNBYSnBv7I Aze/BgHY9pAERJRJ+5T/MYUFIcdie8Qmt8IBkTe11e6zISsGce4ood144YwOecxIUtnk tvuvVw1p5jp2VgyygLa2+KTnD/SXWrXP0VMFdQrDhEreCIF+++kCeo1nhVmFFs/FYgw1 XSjH4HrJa/vOnD7K1C/csV0ZJ7J75fAZyxjZHhPkuPOmaqWGnmr7qzIeN43ePefITHLJ Ln4A== X-Forwarded-Encrypted: i=1; AJvYcCWvoXjovduL/sw2sxmRoR2EKg/4QRfinBNBigtTHEZf7uV+L2gkR4QTLNR0owjzh0x0N9ToTaykAak=@vger.kernel.org X-Gm-Message-State: AOJu0YwsKGZUdSY6b7IF6f5m21nInamLe1DclrqwoGn6wwZgHAztmEU1 3IlskHJKA0nYVQ+p2r7psv41aIj7b5vgZT6uUMaTpGeSOVpW2wUIglMFsnuT1uo= X-Google-Smtp-Source: AGHT+IHezUjchVICcSwZFfFfrE/47nceOgC12FAXIYBCUf1b2OCTgPU2o6ID1sRV3TuR7apW1nA14w== X-Received: by 2002:a05:6870:5b88:b0:277:eea4:a436 with SMTP id 586e51a60fabf-2803a57f035mr8431095fac.7.1727163047552; Tue, 24 Sep 2024 00:30:47 -0700 (PDT) Received: from localhost.localdomain ([123.51.167.56]) by smtp.googlemail.com with ESMTPSA id d2e1a72fcca58-71afc8342dfsm663723b3a.3.2024.09.24.00.30.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Sep 2024 00:30:47 -0700 (PDT) From: Jian-Hong Pan To: Bjorn Helgaas Cc: Johan Hovold , David Box , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Kuppuswamy Sathyanarayanan , Nirmal Patel , Jonathan Derrick , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessos.org, Jian-Hong Pan Subject: [PATCH v9 3/3] PCI: vmd: Save/restore PCIe bridge states before/after pci_reset_bus() Date: Tue, 24 Sep 2024 15:29:00 +0800 Message-ID: <20240924072858.15929-3-jhp@endlessos.org> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20240924070551.14976-2-jhp@endlessos.org> References: <20240924070551.14976-2-jhp@endlessos.org> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 PCI devices' parameters on the VMD bus have been programmed properly originally. But, cleared after pci_reset_bus() and have not been restored correctly. This leads the link's L1.2 between PCIe Root Port and child device gets wrong configs. Here is a failed example on ASUS B1400CEAE with enabled VMD. Both PCIe bridge and NVMe device should have the same LTR1.2_Threshold value. However, they are configured as different values in this case: 10000:e0:06.0 PCI bridge [0604]: Intel Corporation 11th Gen Core Processor PCIe Controller [8086:9a09] (rev 01) (prog-if 00 [Normal decode]) ... Capabilities: [200 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=45us PortTPowerOnTime=50us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=0us 10000:e1:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] (rev 01) (prog-if 02 [NVM Express]) ... Capabilities: [900 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+ PortCommonModeRestoreTime=32us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=101376ns L1SubCtl2: T_PwrOn=50us Here is VMD mapped PCI device tree: -+-[0000:00]-+-00.0 Intel Corporation Device 9a04 | ... \-[10000:e0]-+-06.0-[e1]----00.0 Sandisk Corp WD Blue SN550 NVMe SSD \-17.0 Intel Corporation Tiger Lake-LP SATA Controller When pci_reset_bus() resets the bus [e1] of the NVMe, it only saves and restores NVMe's state before and after reset. Because bus [e1] has only one device: 10000:e1:00.0 NVMe. The PCIe bridge is missed. However, when it restores the NVMe's state, it also restores the ASPM L1SS between the PCIe bridge and the NVMe by pci_restore_aspm_l1ss_state(). The NVMe's L1SS is restored correctly. But, the PCIe bridge's L1SS is restored with the wrong value 0x0. Becuase, the PCIe bridge's state is not saved before reset. That is why pci_restore_aspm_l1ss_state() gets the parent device (PCIe bridge)'s saved state capability data and restores L1SS with value 0. So, save the PCIe bridge's state before pci_reset_bus(). Then, restore the state after pci_reset_bus(). The restoring state also consumes the saving state. Link: https://www.spinics.net/lists/linux-pci/msg159267.html Signed-off-by: Jian-Hong Pan --- v9: - Drop the v8 fix about drivers/pci/pcie/aspm.c. Use this in VMD instead. drivers/pci/controller/vmd.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 11870d1fc818..2820005165b4 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -938,9 +938,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) if (!list_empty(&child->devices)) { dev = list_first_entry(&child->devices, struct pci_dev, bus_list); + /* To avoid pci_reset_bus restore wrong ASPM L1SS + * configuration due to missed saving parent device's + * states, save & restore the parent device's states + * as well. + */ + pci_save_state(dev->bus->self); ret = pci_reset_bus(dev); if (ret) pci_warn(dev, "can't reset device: %d\n", ret); + pci_restore_state(dev->bus->self); break; }