From patchwork Wed Dec 27 09:44:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505252 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FD5724B2F for ; Wed, 27 Dec 2023 09:46:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="QBL/iHH0"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="Vj9LGVKZ" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com 32DF9C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670361; bh=SBhBMSD/b041jxXSGOAXTq5IBUgFoQ2AEafdsjO0oPw=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=QBL/iHH08aFeUTb13vnjMrlymbnGA0PInTKP1c8DV9fH82J3pmemwcFGnw+G2NAm9 3bsQgIA5uF8Ccse8U01vLlD0qpeyrJPWOVFLADMCuMPgmsUI3LtstnDc00tSCXOeO0 X+MIksIFI+ayEExRpRQRsFiSnilEwVPpthedT+l1Q26w7iMryqTj4F7NWRBFLay6uY Xag6480UhGnwCSIGaJ3lPrRzh7/fPmcQ/xSdwqzyBs6LJ2A4i4Sstk8M+7X5H5JzGc Sgf13yK5ccnkKn4wce3K10KxpT4PJjETe9rkQtA60//h6LaBHdUzJjKw6lQYYxeZ8f Ux0igA/dbFT0Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670361; bh=SBhBMSD/b041jxXSGOAXTq5IBUgFoQ2AEafdsjO0oPw=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=Vj9LGVKZ0ypC8e6QD4u6K9kOyByCNNjwolgEVntqVDOYyGJcUvbnuFDu9k3VCeNxU Did5SBr17URGWTjqBi+0P9PtJtE7Tr26kZ6v3HdGkONVO8VfzLwX3fv2nzmX0MJG+M bweyhHJMYWYESJNiF3nfaPkXSfn3UU3wUf6YGgvxxWvGW7etmNcQ+PKl4BCgzYkNd5 zPA7htUQa+rnm7+ywcGegrM87VeXPYBZV09WLskhXwZjQaa7hGfURDbHq9wgjUMrVp CCP4B4HyiLvRiWYIJQFW7PMOxCxzdl4reoPIKEJ93062mdYb8LXqV4a8JghTAoJ8zU BSwZuhO53DeMA== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 01/15] pciutils-lspci: Fix unsynchronized caches in lspci struct device and pci struct pci_dev Date: Wed, 27 Dec 2023 14:44:50 +0500 Message-ID: <20231227094504.32257-2-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) lspci initializes both caches for the device to the same memory block in its scan_device function. Latter calls to config_fetch function will realloc cache in struct device, but not in struct pci_dev leading to the invalid pointer in the latter. pci_dev cache is used by pci_read_* functions, what will lead to a possible use-after-free situations. Example: With patch: diff --git a/ls-caps.c b/ls-caps.c index a481b16..b454843 100644 --- a/ls-caps.c +++ b/ls-caps.c @@ -1802,6 +1802,7 @@ show_caps(struct device *d, int where) break; case PCI_CAP_ID_EXP: type = cap_express(d, where, cap); + struct pci_cap* test = pci_find_cap(d->dev, PCI_CAP_ID_EXP, PCI_CAP_NORMAL); can_have_ext_caps = 1; break; case PCI_CAP_ID_MSIX: valgrind run: valgrind ./lspci -vvvs 7:0.0 ... ==22835== Invalid read of size 2 ==22835== at 0x11A90A: pci_read_word (in /home/merlin/git/pciutils/lspci) ==22835== by 0x11EBEC: pci_scan_caps (in /home/merlin/git/pciutils/lspci) ==22835== by 0x11AC00: pci_fill_info_v38 (in /home/merlin/git/pciutils/lspci) ==22835== by 0x11ED73: pci_find_cap (in /home/merlin/git/pciutils/lspci) ==22835== by 0x1126FA: show_caps (in /home/merlin/git/pciutils/lspci) ==22835== by 0x10E860: show_device (in /home/merlin/git/pciutils/lspci) ==22835== by 0x10BFA3: main (in /home/merlin/git/pciutils/lspci) ==22835== Address 0x5249276 is 6 bytes inside a block of size 64 free'd ==22835== at 0x4E0A13B: realloc (vg_replace_malloc.c:1649) ==22835== by 0x119BCC: xrealloc (in /home/merlin/git/pciutils/lspci) ==22835== by 0x10CD2C: config_fetch (in /home/merlin/git/pciutils/lspci) ==22835== by 0x110DAA: show_caps (in /home/merlin/git/pciutils/lspci) ==22835== by 0x10E860: show_device (in /home/merlin/git/pciutils/lspci) ==22835== by 0x10BFA3: main (in /home/merlin/git/pciutils/lspci) ==22835== Block was alloc'd at ==22835== at 0x4E050B5: malloc (vg_replace_malloc.c:431) ==22835== by 0x119B9C: xmalloc (in /home/merlin/git/pciutils/lspci) ==22835== by 0x10CE80: scan_device (in /home/merlin/git/pciutils/lspci) ==22835== by 0x10BF0F: main (in /home/merlin/git/pciutils/lspci) ... Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lspci.c | 1 + 1 file changed, 1 insertion(+) diff --git a/lspci.c b/lspci.c index 9452cd3..071cc11 100644 --- a/lspci.c +++ b/lspci.c @@ -107,6 +107,7 @@ config_fetch(struct device *d, unsigned int pos, unsigned int len) d->config = xrealloc(d->config, d->config_bufsize); d->present = xrealloc(d->present, d->config_bufsize); memset(d->present + orig_size, 0, d->config_bufsize - orig_size); + pci_setup_cache(d->dev, d->config, d->dev->cache_len); } result = pci_read_block(d->dev, pos, d->config + pos, len); if (result) From patchwork Wed Dec 27 09:44:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505253 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C00F1095A for ; Wed, 27 Dec 2023 09:46:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="SUuvw1G2"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="Zw5EYan4" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com AF1F4C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670363; bh=xUVa7prqwAVqNbBX5/MWuagYaqe7bVGwSoh/co5ZIog=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=SUuvw1G24IP9ZKBS9YpGbwTCmp/IV3SEru1T+ZrQeE0wt30kpXsQWiokN4T5OIF98 Lbqix4FjFyc6R/HSeTnJfm6938lCKftQH41CN47MF6iYpB0Hv+wUFmv29rxh1yxBZF t6b64Kby6JcyofhOuY46DMiLAB+IhUce8ORAUaV4yhbune743pxsSWq7giAKOpmoi0 2VNVEqhfm3YyKYZFLt5Qbyxqy2d11vLkmxhZoIpJdSzerEZOblogssnl7JxqjTtpSG Va3jANqxLU8n8d/fWFa3GiPwOPyYdOdxVJ0bKX3k2/U41friPVkXVthdA+ccc43Q0G iIU+fxsXJIoKA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670363; bh=xUVa7prqwAVqNbBX5/MWuagYaqe7bVGwSoh/co5ZIog=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=Zw5EYan49WgzUgyjaD/055f9yEy29CQTNLz1nS10llBQvNh2oVWa31bpmonVekWdY dfhNt/0BYiCM1l2UzQl1RXWZCZ5lJ+SW4A2BHqZ9G2VV2b9Wc8fHSVVVd1ylwQvB3/ mCT8YRNiW2OqY2jHCuS1LU2j0NNaV9DsCce4pIWh8G/EBaf0r6+VdaTPkKJ1whKFN6 ExYTgoTpblLxeg02vgK7N5Qz/CRB7+7krUi16KytiToyJa0XfQWeWMq4gJy7Jca1HV HPuM7gWrHRT0ScWkoQ0VfbSLeKElYp+ViJN5+7whC0mZxDUarakId6WPPeTiSB4blb 2tHb9zPED3aXw== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 02/15] pciutils: Add constants for Lane Margining at the Receiver Extended Capability Date: Wed, 27 Dec 2023 14:44:51 +0500 Message-ID: <20231227094504.32257-3-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lib/header.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/header.h b/lib/header.h index 2bace93..2141013 100644 --- a/lib/header.h +++ b/lib/header.h @@ -1414,6 +1414,13 @@ #define PCI_DOE_STS_ERROR 0x4 /* DOE Error */ #define PCI_DOE_STS_OBJECT_READY 0x80000000 /* Data Object Ready */ +/* Lane Margining at the Receiver Extended Capability */ +#define PCI_LMR_CAPS 0x4 /* Margining Port Capabilities Register */ +#define PCI_LMR_CAPS_DRVR 0x1 /* Margining uses Driver Software */ +#define PCI_LMR_PORT_STS 0x6 /* Margining Port Status Register */ +#define PCI_LMR_PORT_STS_READY 0x1 /* Margining Ready */ +#define PCI_LMR_PORT_STS_SOFT_READY 0x2 /* Margining Software Ready */ + /* * The PCI interface treats multi-function devices as independent * devices. The slot/function address of each device is encoded From patchwork Wed Dec 27 09:44:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505254 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FEBC286B5 for ; Wed, 27 Dec 2023 09:46:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="IT9v5NBB"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="JcwKLlh3" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com CF1A1C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670365; bh=T7/F0FWKC0Ow4uaEbZVvDFdSptX9rapt1qbYdNZ95tg=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=IT9v5NBBcpQupkG58WQRm2EVoSl5TKp9wRya8cJnbb21hEcmy7zxwqvOlUhxfrMyV VhE+jD/DZJkYIgbApNprwZZn5L4kcwhXG8TjYzInx3ehwyOWYKCPSXxn7K4UFDalPj GBIA+eIB0cG6az7gfWnV/kdV8AM8/rAvS8phI1YnbuMf58jSpAd9H9NT1+CGcEapUr b0Wr34HRpyQ4K0QpgcW9KTFVeAfiBeSZoK1GQzdg+bESc1PC2dadLFFkh4PISKjOsb SucGa5EFD41+9mSWGx3Q1YiyFFhltH5JJzp6g2hbkrKNqaKjz/7wZKc9dOv4l1Khwd DHvhwDeg/rX4Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670365; bh=T7/F0FWKC0Ow4uaEbZVvDFdSptX9rapt1qbYdNZ95tg=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=JcwKLlh3YoMDztxRqK4lI2pN0A4S4VLfV2QK0ZAA3dxFTVPf04PajHfneoYIXxbUm kftR1fMduaqsfRjnEfQdscCMOlUyvVfLmGHD3pRtTSwfOPGBH5XCPB9/VMDQk54jwc feUgDCs9jpxqb1FmodfPgWEnJMyAYRh9lYg5XsatdFRw/E7AfVN4/I3nhvzsDGMxuP 5RD7EkdFjgX8fPpFhnyC5ffjwaxqEJi1S2VTKAXCBo776KBoPN4NIIEhvhXoA/w6K0 prAi37h3hjsrCwdTuikUpAvx+qNrYwRtdnX1ZxVRpSFbFBGMHPdz2h4gQrlSAnwmBp 7i4z1U55VIp7w== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 03/15] pciutils-lspci: Add Lane Margining support to the lspci Date: Wed, 27 Dec 2023 14:44:52 +0500 Message-ID: <20231227094504.32257-4-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Gather all the info available without writing to the config space. Without any commands margining capability exposes only 3 status bits to read through Margining Port Capabilities and Margining Port Status registers. It makes sense to show them anyway. For example, Margining Ready bit indicates whether the device is actually ready for the margining process. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- ls-ecaps.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/ls-ecaps.c b/ls-ecaps.c index 6d2e7b0..9810044 100644 --- a/ls-ecaps.c +++ b/ls-ecaps.c @@ -691,6 +691,26 @@ cap_rcec(struct device *d, int where) printf("\t\tAssociatedBusNumbers: %02x-%02x\n", nextbusn, lastbusn ); } +static void +cap_lmr(struct device *d, int where) +{ + printf("Lane Margining at the Receiver\n"); + + if (verbose < 2) + return; + + if (!config_fetch(d, where, 8)) + return; + + u16 port_caps = get_conf_word(d, where + PCI_LMR_CAPS); + u16 port_status = get_conf_word(d, where + PCI_LMR_PORT_STS); + + printf("\t\tPortCap: Uses Driver%c\n", FLAG(port_caps, PCI_LMR_CAPS_DRVR)); + printf("\t\tPortSta: MargReady%c MargSoftReady%c\n", + FLAG(port_status, PCI_LMR_PORT_STS_READY), + FLAG(port_status, PCI_LMR_PORT_STS_SOFT_READY)); +} + static void cxl_range(u64 base, u64 size, int n) { @@ -1610,7 +1630,7 @@ show_ext_caps(struct device *d, int type) printf("Physical Layer 16.0 GT/s \n"); break; case PCI_EXT_CAP_ID_LMR: - printf("Lane Margining at the Receiver \n"); + cap_lmr(d, where); break; case PCI_EXT_CAP_ID_HIER_ID: printf("Hierarchy ID \n"); From patchwork Wed Dec 27 09:44:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505255 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DAB828E2D for ; Wed, 27 Dec 2023 09:46:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="veDZcvx6"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="Ut4CAezZ" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com 3940BC000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670367; bh=CTJjAjTpq/uv59gXUUxIWe/fKK/pQbwK97FgZzzA+Mw=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=veDZcvx67IRqjiGn+RPkuFqlgaV1sEc4jFK8rHJPRK7dldGTxDve1/TGtSezL7lZ+ eICatC7++RtGrWVkEt22VgODTELpHxXM4We+LTNGlxWGDaaaiRx6UnuSQl8bCggppl QJycu6BgtXEnhlyyyFyvx1IKEbyMWBpSDlv3BUAgxkoQ/UlMVcyNpOOI0zm1C6kGKZ 4cWj0VoltHmHvtFFQ6wQP3vay5FD6O7CmvAl2sN4Eocopj8HdKg7rrvzR6BVtGsEX4 n9pQfxT1Q6wA5x1dAknni1kv3ZKRV6wDgHTsJVA/KAnXphCrVvFDqYk7YFd/CcUKi0 Hi5ugBCuemyWg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670367; bh=CTJjAjTpq/uv59gXUUxIWe/fKK/pQbwK97FgZzzA+Mw=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=Ut4CAezZjNbrwp7FyO4u7+m+N3epOnry5KtfQt1txf6uH+SzmJb0RTTZAoY4edTpj pM5Cfmy2S+Jwj5ZVBcnkyW59DdvWO6YhpH2ZGmyjuroeHtRZQEFm/JG8o7aaQKYzJs JkqFW3w7KMwEud77qDdZBVpLZNaRr5ajP4wW1M7RFt5E0ftQo4kdTzb2dnB4i1Uydh Jr0y89RkIZboej0cuMp+0Btv3IZ558jfqhFY2BRVtKq6DJxuweMamovFMdWSMXtOU1 ajgAfi9Ew7gMeMhfTAr+KFN9Mv0vSZ22Rr1NhAwJ2Ff8CP1/KOtTYj9Vu8+HHayqNW LJzUA8cn61tUw== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 04/15] pciutils-pcilib: Add separate file for bit manipulation functions Date: Wed, 27 Dec 2023 14:44:53 +0500 Message-ID: <20231227094504.32257-5-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Move several macros from lspci and add some more for operations with bit masks. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- Makefile | 2 +- lib/bitops.h | 39 +++++++++++++++++++++++++++++++++++++++ lib/pci.h | 1 + lspci.h | 6 ------ 4 files changed, 41 insertions(+), 7 deletions(-) create mode 100644 lib/bitops.h diff --git a/Makefile b/Makefile index 228cb56..52538e8 100644 --- a/Makefile +++ b/Makefile @@ -62,7 +62,7 @@ LIBNAME=libpci -include lib/config.mk -PCIINC=lib/config.h lib/header.h lib/pci.h lib/types.h lib/sysdep.h +PCIINC=lib/config.h lib/header.h lib/pci.h lib/types.h lib/sysdep.h lib/bitops.h PCIINC_INS=lib/config.h lib/header.h lib/pci.h lib/types.h export diff --git a/lib/bitops.h b/lib/bitops.h new file mode 100644 index 0000000..029741e --- /dev/null +++ b/lib/bitops.h @@ -0,0 +1,39 @@ +/* + * The PCI Utilities -- Decode bits and bit fields + * + * Copyright (c) 2023 Martin Mares + * Copyright (c) 2023 KNS Group LLC (YADRO) + * + * Can be freely distributed and used under the terms of the GNU GPL v2+. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef _BITOPS_H +#define _BITOPS_H + +#ifndef _PCI_LIB_H +#error Import only from pci.h +#endif + +/* Useful macros for decoding of bits and bit fields */ + +#define FLAG(x, y) ((x & y) ? '+' : '-') + +// Generate mask + +#define BIT(at) ((u64)1 << (at)) +// Boundaries inclusive +#define MASK(h, l) ((((u64)1 << ((h) + 1)) - 1) & ~(((u64)1 << (l)) - 1)) + +// Get/set from register + +#define BITS(x, at, width) (((x) >> (at)) & ((1 << (width)) - 1)) +#define GET_REG_MASK(reg, mask) (((reg) & (mask)) / ((mask) & ~((mask) << 1))) +#define SET_REG_MASK(reg, mask, val) \ + (((reg) & ~(mask)) | (((val) * ((mask) & ~((mask) << 1))) & (mask))) + +#define TABLE(tab, x, buf) \ + ((x) < sizeof(tab) / sizeof((tab)[0]) ? (tab)[x] : (sprintf((buf), "??%d", (x)), (buf))) + +#endif diff --git a/lib/pci.h b/lib/pci.h index 2322bf7..6cc46b3 100644 --- a/lib/pci.h +++ b/lib/pci.h @@ -17,6 +17,7 @@ #include "header.h" #include "types.h" +#include "bitops.h" #define PCI_LIB_VERSION 0x030a00 diff --git a/lspci.h b/lspci.h index c5a9ec7..4d711a5 100644 --- a/lspci.h +++ b/lspci.h @@ -58,12 +58,6 @@ u32 get_conf_long(struct device *d, unsigned int pos); word get_conf_word(struct device *d, unsigned int pos); byte get_conf_byte(struct device *d, unsigned int pos); -/* Useful macros for decoding of bits and bit fields */ - -#define FLAG(x,y) ((x & y) ? '+' : '-') -#define BITS(x,at,width) (((x) >> (at)) & ((1 << (width)) - 1)) -#define TABLE(tab,x,buf) ((x) < sizeof(tab)/sizeof((tab)[0]) ? (tab)[x] : (sprintf((buf), "??%d", (x)), (buf))) - /* ls-vpd.c */ void cap_vpd(struct device *d); From patchwork Wed Dec 27 09:44:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505256 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E74922C680 for ; Wed, 27 Dec 2023 09:46:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="rS2gEy54"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="QuyBHjR5" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com 85CF3C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670368; bh=K1q0r4daYcjwSfUXX+zXBzWyzsX+BJADPAIW1bb5nK4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=rS2gEy54njf5px1RGyznH59Oqa4Ba8Sz/1Y/sn3n4CwCl/zctF8oU0eMEK5fxX+6A NXbf5XPCp84z0wVBHV0aBe3Hm+k7RF+y14tn9Ah0C6s8Aeaj/YL5SztdLNVKLvEmki 9lzuhN9FGhydeLXtACEyJPS5JBD2XNVXkAa2cXaXsSJOHAj3noWepdTXtlmecPrRJj TldXXSeWEK888osKwCCvcKIsatdF1NG8DoU+LGelIuk9iONbjPRyDhnUhzpLHJyny6 p1VwaRFuiWb+7/sNbBPX9XHjv1YFGylEl0PdrryIaaK1oAJJNsbp1ST6WVBNwifUPh K4vA+6nLnQJow== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670368; bh=K1q0r4daYcjwSfUXX+zXBzWyzsX+BJADPAIW1bb5nK4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=QuyBHjR5qhwsbVnfWdgXo+BXZf3LKQyZK1V1rnc6I6wZYEQUrnYkA5QZLFZ8bGrqP eHfhuZmzjpVnZ/rMGhBzJ6mSXJSRaq9zhHi1QtqaYWROmM68gu3dGtmN90KzmW3Axw JKA2sCq/txDmVaruczJp+9pJ5pC4bAgDgbxgXvEd7eG20jwV/1bGakCfrO+BvifEui C/ZkYc/Lc0htkQh6+sjs3oqUbVWd7Fco0N43Kb49vSRWzQcntk/ZE3RE9RLif8+sTy 3zBpYqFoIuxMLeIewPw84QREEvywh4ypvQvTEZy2wPf4HiyDZ5KdRbFABqEo3f7JnH arPjq3EEBPptQ== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 05/15] pciutils-pcilmr: Add functions for device checking and preparations before main margining processes Date: Wed, 27 Dec 2023 14:44:54 +0500 Message-ID: <20231227094504.32257-6-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Follow the checklist from PCIe Base Spec Rev 5.0 section 4.2.13.3 "Receiver Margin Testing Requirements": * Verify the Link is at 16 GT/s or higher data rate, in DO PM state; * Verify that Margining Ready bit of the device is set; * Disable the ASPM and Autonomous Speed/Width features for the duration of the test. Also verify that Upstream Port of the Link is Function 0 of a Device, according to spec, only it must implement margining registers. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lmr/lmr.h | 55 +++++++++++++++++++ lmr/margin_hw.c | 138 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 193 insertions(+) create mode 100644 lmr/lmr.h create mode 100644 lmr/margin_hw.c diff --git a/lmr/lmr.h b/lmr/lmr.h new file mode 100644 index 0000000..67fe0b0 --- /dev/null +++ b/lmr/lmr.h @@ -0,0 +1,55 @@ +/* + * The PCI Utilities -- Margining utility main header + * + * Copyright (c) 2023 KNS Group LLC (YADRO) + * + * Can be freely distributed and used under the terms of the GNU GPL v2+. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef _LMR_H +#define _LMR_H + +#include + +#include "pciutils.h" + +/* PCI Device wrapper for margining functions */ +struct margin_dev { + struct pci_dev *dev; + int lmr_cap_addr; + u8 width; + u8 retimers_n; + u8 link_speed; + + /* Saved Device settings to restore after margining */ + u8 aspm; + bool hasd; // Hardware Autonomous Speed Disable + bool hawd; // Hardware Autonomous Width Disable +}; + +struct margin_link { + struct margin_dev down_port; + struct margin_dev up_port; +}; + +/* margin_hw */ + +/* Verify that devices form the link with 16 GT/s or 32 GT/s data rate */ +bool margin_verify_link(struct pci_dev *down_port, struct pci_dev *up_port); + +/* Check Margining Ready bit from Margining Port Status Register */ +bool margin_check_ready_bit(struct pci_dev *dev); + +/* Verify link and fill wrappers */ +bool margin_fill_link(struct pci_dev *down_port, struct pci_dev *up_port, + struct margin_link *wrappers); + +/* Disable ASPM, set Hardware Autonomous Speed/Width Disable bits */ +bool margin_prep_link(struct margin_link *link); + +/* Restore ASPM, Hardware Autonomous Speed/Width settings */ +void margin_restore_link(struct margin_link *link); + +#endif diff --git a/lmr/margin_hw.c b/lmr/margin_hw.c new file mode 100644 index 0000000..c000132 --- /dev/null +++ b/lmr/margin_hw.c @@ -0,0 +1,138 @@ +/* + * The PCI Utilities -- Verify and prepare devices before margining + * + * Copyright (c) 2023 KNS Group LLC (YADRO) + * + * Can be freely distributed and used under the terms of the GNU GPL v2+. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "lmr.h" + +bool +margin_verify_link(struct pci_dev *down_port, struct pci_dev *up_port) +{ + struct pci_cap *cap = pci_find_cap(down_port, PCI_CAP_ID_EXP, PCI_CAP_NORMAL); + if (!cap) + return false; + if ((pci_read_word(down_port, cap->addr + PCI_EXP_LNKSTA) & PCI_EXP_LNKSTA_SPEED) < 4) + return false; + if ((pci_read_word(down_port, cap->addr + PCI_EXP_LNKSTA) & PCI_EXP_LNKSTA_SPEED) > 5) + return false; + + u8 down_type = pci_read_byte(down_port, PCI_HEADER_TYPE) & 0x7F; + u8 down_sec = pci_read_byte(down_port, PCI_SECONDARY_BUS); + u8 down_dir + = GET_REG_MASK(pci_read_word(down_port, cap->addr + PCI_EXP_FLAGS), PCI_EXP_FLAGS_TYPE); + + // Verify that devices are linked, down_port is Root Port or Downstream Port of Switch, + // up_port is Function 0 of a Device + if (!(down_sec == up_port->bus && down_type == PCI_HEADER_TYPE_BRIDGE + && (down_dir == PCI_EXP_TYPE_ROOT_PORT || down_dir == PCI_EXP_TYPE_DOWNSTREAM) + && up_port->func == 0)) + return false; + + struct pci_cap *pm = pci_find_cap(up_port, PCI_CAP_ID_PM, PCI_CAP_NORMAL); + return pm && !(pci_read_word(up_port, pm->addr + PCI_PM_CTRL) & PCI_PM_CTRL_STATE_MASK); // D0 +} + +bool +margin_check_ready_bit(struct pci_dev *dev) +{ + struct pci_cap *lmr = pci_find_cap(dev, PCI_EXT_CAP_ID_LMR, PCI_CAP_EXTENDED); + return lmr && (pci_read_word(dev, lmr->addr + PCI_LMR_PORT_STS) & PCI_LMR_PORT_STS_READY); +} + +/* Awaits device at 16 GT/s or higher */ +static struct margin_dev +fill_dev_wrapper(struct pci_dev *dev) +{ + struct pci_cap *cap = pci_find_cap(dev, PCI_CAP_ID_EXP, PCI_CAP_NORMAL); + struct margin_dev res + = { .dev = dev, + .lmr_cap_addr = pci_find_cap(dev, PCI_EXT_CAP_ID_LMR, PCI_CAP_EXTENDED)->addr, + .width = GET_REG_MASK(pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA), PCI_EXP_LNKSTA_WIDTH), + .retimers_n + = (!!(pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA2) & PCI_EXP_LINKSTA2_RETIMER)) + + (!!(pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA2) & PCI_EXP_LINKSTA2_2RETIMERS)), + .link_speed = (pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA) & PCI_EXP_LNKSTA_SPEED) }; + return res; +} + +bool +margin_fill_link(struct pci_dev *down_port, struct pci_dev *up_port, struct margin_link *wrappers) +{ + if (!margin_verify_link(down_port, up_port)) + return false; + wrappers->down_port = fill_dev_wrapper(down_port); + wrappers->up_port = fill_dev_wrapper(up_port); + return true; +} + +/* Disable ASPM, set Hardware Autonomous Speed/Width Disable bits */ +static bool +margin_prep_dev(struct margin_dev *dev) +{ + struct pci_cap *pcie = pci_find_cap(dev->dev, PCI_CAP_ID_EXP, PCI_CAP_NORMAL); + if (!pcie) + return false; + + u16 lnk_ctl = pci_read_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL); + dev->aspm = lnk_ctl & PCI_EXP_LNKCTL_ASPM; + dev->hawd = !!(lnk_ctl & PCI_EXP_LNKCTL_HWAUTWD); + lnk_ctl &= ~PCI_EXP_LNKCTL_ASPM; + pci_write_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL, lnk_ctl); + if (pci_read_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL) & PCI_EXP_LNKCTL_ASPM) + return false; + + lnk_ctl |= PCI_EXP_LNKCTL_HWAUTWD; + pci_write_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL, lnk_ctl); + + u16 lnk_ctl2 = pci_read_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL2); + dev->hasd = !!(lnk_ctl2 & PCI_EXP_LNKCTL2_SPEED_DIS); + lnk_ctl2 |= PCI_EXP_LNKCTL2_SPEED_DIS; + pci_write_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL2, lnk_ctl2); + + return true; +} + +/* Restore Device ASPM, Hardware Autonomous Speed/Width settings */ +static void +margin_restore_dev(struct margin_dev *dev) +{ + struct pci_cap *pcie = pci_find_cap(dev->dev, PCI_CAP_ID_EXP, PCI_CAP_NORMAL); + if (!pcie) + return; + + u16 lnk_ctl = pci_read_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL); + lnk_ctl = SET_REG_MASK(lnk_ctl, PCI_EXP_LNKCAP_ASPM, dev->aspm); + lnk_ctl = SET_REG_MASK(lnk_ctl, PCI_EXP_LNKCTL_HWAUTWD, dev->hawd); + pci_write_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL, lnk_ctl); + + u16 lnk_ctl2 = pci_read_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL2); + lnk_ctl2 = SET_REG_MASK(lnk_ctl2, PCI_EXP_LNKCTL2_SPEED_DIS, dev->hasd); + pci_write_word(dev->dev, pcie->addr + PCI_EXP_LNKCTL2, lnk_ctl2); +} + +bool +margin_prep_link(struct margin_link *link) +{ + if (!link) + return false; + if (!margin_prep_dev(&link->down_port)) + return false; + if (!margin_prep_dev(&link->up_port)) + { + margin_restore_dev(&link->down_port); + return false; + } + return true; +} + +void +margin_restore_link(struct margin_link *link) +{ + margin_restore_dev(&link->down_port); + margin_restore_dev(&link->up_port); +} From patchwork Wed Dec 27 09:44:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505257 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC7A428E2D for ; Wed, 27 Dec 2023 09:46:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="uuTfaDlY"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="INQxQeuE" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com 3C782C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670370; bh=944V6eKFcFh6oSxefV06MWTrb4ZVIRNgXCaMPIHHVPU=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=uuTfaDlY8Ink4o6KSTi08+1CqgAtp1PWNcznxObBq5Sp9ETvRXHgVClGqmLrkT+y4 0Kz1b5trtYjevy8sNzCpie8DYt5rLusNd27ho7f1ctLRsmIeBGvMzR5BMH6G49q8mg FK4Tc6+xzkkBANCanMV1pMKC4zMOgWKaCMEreQuUEcfiscg1un5C8Ea8EopG3AptbL sS4F3C4hXrG+X8EV5dm4zq5YzBlRm6VVmK7Acm6gBFkA0yWroB4oaP3884hOnTCG7L eux32qdCqGnH6J8QTSlKQFYdf9qt875RDbKKwAN1ELh/EVFgO3hIvavGq22lqtH6Rw pO+7k9Wlv2Duw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670370; bh=944V6eKFcFh6oSxefV06MWTrb4ZVIRNgXCaMPIHHVPU=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=INQxQeuElbJSVeMhlFkExCzHxgzmVSjB7mbZixTZWzlxQQPSXbt/UWOUhlvfEEwqU BMQPF446rXbNKsnSczIqJZoRGNUaTaSEcDsVV4rin5dsLCk9oTOv7ZQjRRl6fYRI3Q l3RhD3mJJhvm3c95bSKEsrICGPJso5LI7Ppib8zepgIIzhg+8j3VBD0PKQemMYyvZR rw2LZJUuvCl3p5C2IGTqQeFOgpckyLbtYdGv1JhW0/Yaa6cPX4tgwHY8JOKp6ExS6a odV2enExYzeTL5iLC/IhF6rcsyKfKyODCFpWo7RslvYr35hSgCaYgxHvJcV5koPe6R SMiAxVhByMedA== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 06/15] pciutils-pcilmr: Add margining process functions Date: Wed, 27 Dec 2023 14:44:55 +0500 Message-ID: <20231227094504.32257-7-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) * Implement the margining flow as described in the section "Example Software Flow for Lane Margining at Receiver" of the PCIe Base Spec Rev 5.0; * Implement margining commands formation and response parsing according to the PCIe Base Spec Rev 5.0 table 4-26; * Use Receiver margining parameters as described in the PCIe Base Spec Rev 5.0 table 8-11; * Support lane reversal and simultaneous margining of several link lanes. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lmr/lmr.h | 129 ++++++++++++ lmr/margin.c | 539 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 668 insertions(+) create mode 100644 lmr/margin.c diff --git a/lmr/lmr.h b/lmr/lmr.h index 67fe0b0..fa627a3 100644 --- a/lmr/lmr.h +++ b/lmr/lmr.h @@ -15,6 +15,8 @@ #include "pciutils.h" +#define MARGIN_STEP_MS 1000 + /* PCI Device wrapper for margining functions */ struct margin_dev { struct pci_dev *dev; @@ -34,6 +36,118 @@ struct margin_link { struct margin_dev up_port; }; +/* Specification Revision 5.0 Table 8-11 */ +struct margin_params { + bool ind_error_sampler; + bool sample_report_method; + bool ind_left_right_tim; + bool ind_up_down_volt; + bool volt_support; + + u8 max_lanes; + + u8 timing_steps; + u8 timing_offset; + + u8 volt_steps; + u8 volt_offset; + + u8 sample_rate_v; + u8 sample_rate_t; +}; + +/* Step Margin Execution Status - Step command response */ +enum margin_step_exec_sts { + MARGIN_NAK = 0, // NAK/Set up for margin + MARGIN_LIM, // Too many errors (device limit) + MARGIN_THR // Test threshold has been reached +}; + +enum margin_dir { VOLT_UP = 0, VOLT_DOWN, TIM_LEFT, TIM_RIGHT }; + +/* Margining results of one lane of the receiver */ +struct margin_res_lane { + u8 lane; + u8 steps[4]; + enum margin_step_exec_sts statuses[4]; +}; + +/* Reason not to run margining test on the Link/Receiver */ +enum margin_test_status { + MARGIN_TEST_OK = 0, + MARGIN_TEST_READY_BIT, + MARGIN_TEST_CAPS, + + // Couldn't run test + MARGIN_TEST_PREREQS, + MARGIN_TEST_ARGS_LANES, + MARGIN_TEST_ARGS_RECVS, + MARGIN_TEST_ASPM +}; + +/* All lanes Receiver results */ +struct margin_results { + u8 recvn; // Receiver Number + struct margin_params params; + bool lane_reversal; + u8 link_speed; + + enum margin_test_status test_status; + + /* Used to convert steps to physical quantity. + Calculated from MaxOffset and NumSteps */ + double tim_coef; + double volt_coef; + + u8 lanes_n; + struct margin_res_lane *lanes; +}; + +/* pcilmr arguments */ +struct margin_args { + u8 steps_t; // 0 == use NumTimingSteps + u8 steps_v; // 0 == use NumVoltageSteps + u8 parallel_lanes; // [1; MaxLanes + 1] + u8 error_limit; // [0; 63] + u8 recvs[6]; // Receivers Numbers + u8 recvs_n; // 0 == margin all available receivers + u8 lanes[32]; // Lanes to Margin + u8 lanes_n; // 0 == margin all available lanes + bool run_margin; // Or print params only + u8 verbosity; // 0 - basic; + // 1 - add info about remaining time and lanes in progress during margining + + u64 *steps_utility; // For ETA logging +}; + +/* Receiver structure */ +struct margin_recv { + struct margin_dev *dev; + u8 recvn; // Receiver Number + bool lane_reversal; + struct margin_params *params; + + u8 parallel_lanes; + u8 error_limit; +}; + +struct margin_lanes_data { + struct margin_recv *recv; + + struct margin_res_lane *results; + u8 *lanes_numbers; + u8 lanes_n; + + bool ind; + enum margin_dir dir; + + u8 steps_lane_done; + u8 steps_lane_total; + u64 *steps_utility; + + u8 verbosity; +}; + /* margin_hw */ /* Verify that devices form the link with 16 GT/s or 32 GT/s data rate */ @@ -52,4 +166,19 @@ bool margin_prep_link(struct margin_link *link); /* Restore ASPM, Hardware Autonomous Speed/Width settings */ void margin_restore_link(struct margin_link *link); +/* margin */ + +/* Fill margin_params without calling other functions */ +bool margin_read_params(struct pci_access *pacc, struct pci_dev *dev, u8 recvn, + struct margin_params *params); + +enum margin_test_status margin_process_args(struct margin_dev *dev, struct margin_args *args); + +/* Awaits that args are prepared through process_args. + Returns number of margined Receivers through recvs_n */ +struct margin_results *margin_test_link(struct margin_link *link, struct margin_args *args, + u8 *recvs_n); + +void margin_free_results(struct margin_results *results, u8 results_n); + #endif diff --git a/lmr/margin.c b/lmr/margin.c new file mode 100644 index 0000000..57bef88 --- /dev/null +++ b/lmr/margin.c @@ -0,0 +1,539 @@ +/* + * The PCI Utilities -- Obtain the margin information of the Link + * + * Copyright (c) 2023 KNS Group LLC (YADRO) + * + * Can be freely distributed and used under the terms of the GNU GPL v2+. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include +#include +#include + +#include "lmr.h" + +/* Macro helpers for Margining command parsing */ + +typedef u16 margin_cmd; + +/* Margining command parsing */ + +#define LMR_CMD_RECVN MASK(2, 0) +#define LMR_CMD_TYPE MASK(5, 3) +#define LMR_CMD_PAYLOAD MASK(15, 8) + +// Payload parsing + +// Report Capabilities +#define LMR_PLD_VOLT_SUPPORT BIT(8) +#define LMR_PLD_IND_U_D_VOLT BIT(9) +#define LMR_PLD_IND_L_R_TIM BIT(10) +#define LMR_PLD_SAMPLE_REPORT_METHOD BIT(11) +#define LMR_PLD_IND_ERR_SAMPLER BIT(12) + +#define LMR_PLD_MAX_T_STEPS MASK(13, 8) +#define LMR_PLD_MAX_V_STEPS MASK(14, 8) +#define LMR_PLD_MAX_OFFSET MASK(14, 8) +#define LMR_PLD_MAX_LANES MASK(12, 8) +#define LMR_PLD_SAMPLE_RATE MASK(13, 8) + +// Timing Step +#define LMR_PLD_MARGIN_T_STEPS MASK(13, 8) +#define LMR_PLD_T_GO_LEFT BIT(14) + +// Voltage Timing +#define LMR_PLD_MARGIN_V_STEPS MASK(14, 8) +#define LMR_PLD_V_GO_DOWN BIT(15) + +// Step Response +#define LMR_PLD_ERR_CNT MASK(13, 8) +#define LMR_PLD_MARGIN_STS MASK(15, 14) + +/* Address calc macro for Lanes Margining registers */ + +#define LMR_LANE_CTRL(lmr_cap_addr, lane) ((lmr_cap_addr) + 8 + 4 * (lane)) +#define LMR_LANE_STATUS(lmr_cap_addr, lane) ((lmr_cap_addr) + 10 + 4 * (lane)) + +/* Margining Commands */ + +#define MARG_TIM(go_left, step, recvn) margin_make_cmd(((go_left) << 6) | (step), 3, recvn) +#define MARG_VOLT(go_down, step, recvn) margin_make_cmd(((go_down) << 7) | (step), 4, recvn) + +// Report commands +#define REPORT_CAPS(recvn) margin_make_cmd(0x88, 1, recvn) +#define REPORT_VOL_STEPS(recvn) margin_make_cmd(0x89, 1, recvn) +#define REPORT_TIM_STEPS(recvn) margin_make_cmd(0x8A, 1, recvn) +#define REPORT_TIM_OFFSET(recvn) margin_make_cmd(0x8B, 1, recvn) +#define REPORT_VOL_OFFSET(recvn) margin_make_cmd(0x8C, 1, recvn) +#define REPORT_SAMPL_RATE_V(recvn) margin_make_cmd(0x8D, 1, recvn) +#define REPORT_SAMPL_RATE_T(recvn) margin_make_cmd(0x8E, 1, recvn) +#define REPORT_SAMPLE_CNT(recvn) margin_make_cmd(0x8F, 1, recvn) +#define REPORT_MAX_LANES(recvn) margin_make_cmd(0x90, 1, recvn) + +// Set commands +#define NO_COMMAND margin_make_cmd(0x9C, 7, 0) +#define CLEAR_ERROR_LOG(recvn) margin_make_cmd(0x55, 2, recvn) +#define GO_TO_NORMAL_SETTINGS(recvn) margin_make_cmd(0xF, 2, recvn) +#define SET_ERROR_LIMIT(error_limit, recvn) margin_make_cmd(0xC0 | (error_limit), 2, recvn) + +static int +msleep(long msec) +{ + struct timespec ts; + int res; + + if (msec < 0) + { + errno = EINVAL; + return -1; + } + + ts.tv_sec = msec / 1000; + ts.tv_nsec = (msec % 1000) * 1000000; + + do + { + res = nanosleep(&ts, &ts); + } while (res && errno == EINTR); + + return res; +} + +static margin_cmd +margin_make_cmd(u8 payload, u8 type, u8 recvn) +{ + return SET_REG_MASK(0, LMR_CMD_PAYLOAD, payload) | SET_REG_MASK(0, LMR_CMD_TYPE, type) + | SET_REG_MASK(0, LMR_CMD_RECVN, recvn); +} + +static bool +margin_set_cmd(struct margin_dev *dev, u8 lane, margin_cmd cmd) +{ + pci_write_word(dev->dev, LMR_LANE_CTRL(dev->lmr_cap_addr, lane), cmd); + msleep(10); + return pci_read_word(dev->dev, LMR_LANE_STATUS(dev->lmr_cap_addr, lane)) == cmd; +} + +static bool +margin_report_cmd(struct margin_dev *dev, u8 lane, margin_cmd cmd, margin_cmd *result) +{ + pci_write_word(dev->dev, LMR_LANE_CTRL(dev->lmr_cap_addr, lane), cmd); + msleep(10); + *result = pci_read_word(dev->dev, LMR_LANE_STATUS(dev->lmr_cap_addr, lane)); + return GET_REG_MASK(*result, LMR_CMD_TYPE) == GET_REG_MASK(cmd, LMR_CMD_TYPE) + && GET_REG_MASK(*result, LMR_CMD_RECVN) == GET_REG_MASK(cmd, LMR_CMD_RECVN) + && margin_set_cmd(dev, lane, NO_COMMAND); +} + +static bool +read_params_internal(struct margin_dev *dev, u8 recvn, bool lane_reversal, + struct margin_params *params) +{ + margin_cmd resp; + u8 lane = lane_reversal ? dev->width - 1 : 0; + margin_set_cmd(dev, lane, NO_COMMAND); + bool status = margin_report_cmd(dev, lane, REPORT_CAPS(recvn), &resp); + if (status) + { + params->volt_support = GET_REG_MASK(resp, LMR_PLD_VOLT_SUPPORT); + params->ind_up_down_volt = GET_REG_MASK(resp, LMR_PLD_IND_U_D_VOLT); + params->ind_left_right_tim = GET_REG_MASK(resp, LMR_PLD_IND_L_R_TIM); + params->sample_report_method = GET_REG_MASK(resp, LMR_PLD_SAMPLE_REPORT_METHOD); + params->ind_error_sampler = GET_REG_MASK(resp, LMR_PLD_IND_ERR_SAMPLER); + status = margin_report_cmd(dev, lane, REPORT_VOL_STEPS(recvn), &resp); + } + if (status) + { + params->volt_steps = GET_REG_MASK(resp, LMR_PLD_MAX_V_STEPS); + status = margin_report_cmd(dev, lane, REPORT_TIM_STEPS(recvn), &resp); + } + if (status) + { + params->timing_steps = GET_REG_MASK(resp, LMR_PLD_MAX_T_STEPS); + status = margin_report_cmd(dev, lane, REPORT_TIM_OFFSET(recvn), &resp); + } + if (status) + { + params->timing_offset = GET_REG_MASK(resp, LMR_PLD_MAX_OFFSET); + status = margin_report_cmd(dev, lane, REPORT_VOL_OFFSET(recvn), &resp); + } + if (status) + { + params->volt_offset = GET_REG_MASK(resp, LMR_PLD_MAX_OFFSET); + status = margin_report_cmd(dev, lane, REPORT_SAMPL_RATE_V(recvn), &resp); + } + if (status) + { + params->sample_rate_v = GET_REG_MASK(resp, LMR_PLD_SAMPLE_RATE); + status = margin_report_cmd(dev, lane, REPORT_SAMPL_RATE_T(recvn), &resp); + } + if (status) + { + params->sample_rate_t = GET_REG_MASK(resp, LMR_PLD_SAMPLE_RATE); + status = margin_report_cmd(dev, lane, REPORT_MAX_LANES(recvn), &resp); + } + if (status) + params->max_lanes = GET_REG_MASK(resp, LMR_PLD_MAX_LANES); + return status; +} + +/* Margin all lanes_n lanes simultaneously */ +static void +margin_test_lanes(struct margin_lanes_data arg) +{ + u8 steps_done = 0; + margin_cmd lane_status; + u8 marg_type; + margin_cmd step_cmd; + bool timing = (arg.dir == TIM_LEFT || arg.dir == TIM_RIGHT); + + if (timing) + { + marg_type = 3; + step_cmd = MARG_TIM(arg.dir == TIM_LEFT, steps_done, arg.recv->recvn); + } + else + { + marg_type = 4; + step_cmd = MARG_VOLT(arg.dir == VOLT_DOWN, steps_done, arg.recv->recvn); + } + + bool failed_lanes[32] = { 0 }; + u8 alive_lanes = arg.lanes_n; + + for (int i = 0; i < arg.lanes_n; i++) + { + margin_set_cmd(arg.recv->dev, arg.results[i].lane, NO_COMMAND); + margin_set_cmd(arg.recv->dev, arg.results[i].lane, + SET_ERROR_LIMIT(arg.recv->error_limit, arg.recv->recvn)); + margin_set_cmd(arg.recv->dev, arg.results[i].lane, NO_COMMAND); + arg.results[i].steps[arg.dir] = arg.steps_lane_total; + arg.results[i].statuses[arg.dir] = MARGIN_THR; + } + + while (alive_lanes > 0 && steps_done < arg.steps_lane_total) + { + alive_lanes = 0; + steps_done++; + if (timing) + step_cmd = SET_REG_MASK(step_cmd, LMR_PLD_MARGIN_T_STEPS, steps_done); + else + step_cmd = SET_REG_MASK(step_cmd, LMR_PLD_MARGIN_V_STEPS, steps_done); + + for (int i = 0; i < arg.lanes_n; i++) + { + if (!failed_lanes[i]) + { + alive_lanes++; + int ctrl_addr = LMR_LANE_CTRL(arg.recv->dev->lmr_cap_addr, arg.results[i].lane); + pci_write_word(arg.recv->dev->dev, ctrl_addr, step_cmd); + } + } + msleep(MARGIN_STEP_MS); + + for (int i = 0; i < arg.lanes_n; i++) + { + if (!failed_lanes[i]) + { + int status_addr = LMR_LANE_STATUS(arg.recv->dev->lmr_cap_addr, arg.results[i].lane); + lane_status = pci_read_word(arg.recv->dev->dev, status_addr); + u8 step_status = GET_REG_MASK(lane_status, LMR_PLD_MARGIN_STS); + if (!(GET_REG_MASK(lane_status, LMR_CMD_TYPE) == marg_type + && GET_REG_MASK(lane_status, LMR_CMD_RECVN) == arg.recv->recvn + && step_status == 2 + && GET_REG_MASK(lane_status, LMR_PLD_ERR_CNT) <= arg.recv->error_limit + && margin_set_cmd(arg.recv->dev, arg.results[i].lane, NO_COMMAND))) + { + alive_lanes--; + failed_lanes[i] = true; + arg.results[i].steps[arg.dir] = steps_done - 1; + arg.results[i].statuses[arg.dir] + = (step_status == 3 || step_status == 1 ? MARGIN_NAK : MARGIN_LIM); + } + } + } + + arg.steps_lane_done = steps_done; + } + + for (int i = 0; i < arg.lanes_n; i++) + { + margin_set_cmd(arg.recv->dev, arg.results[i].lane, NO_COMMAND); + margin_set_cmd(arg.recv->dev, arg.results[i].lane, CLEAR_ERROR_LOG(arg.recv->recvn)); + margin_set_cmd(arg.recv->dev, arg.results[i].lane, NO_COMMAND); + margin_set_cmd(arg.recv->dev, arg.results[i].lane, GO_TO_NORMAL_SETTINGS(arg.recv->recvn)); + margin_set_cmd(arg.recv->dev, arg.results[i].lane, NO_COMMAND); + } +} + +/* Awaits that Receiver is prepared through prep_dev function */ +static bool +margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, + struct margin_results *results) +{ + u8 *lanes_to_margin = args->lanes; + u8 lanes_n = args->lanes_n; + + struct margin_params params; + struct margin_recv recv = { .dev = dev, + .recvn = recvn, + .lane_reversal = false, + .params = ¶ms, + .parallel_lanes = args->parallel_lanes ? args->parallel_lanes : 1, + .error_limit = args->error_limit }; + + results->recvn = recvn; + results->lanes_n = lanes_n; + + if (!margin_check_ready_bit(dev->dev)) + { + results->test_status = MARGIN_TEST_READY_BIT; + return false; + } + + if (!read_params_internal(dev, recvn, recv.lane_reversal, ¶ms)) + { + recv.lane_reversal = true; + if (!read_params_internal(dev, recvn, recv.lane_reversal, ¶ms)) + { + results->test_status = MARGIN_TEST_CAPS; + return false; + } + } + + results->params = params; + + if (recv.parallel_lanes > params.max_lanes + 1) + recv.parallel_lanes = params.max_lanes + 1; + + results->tim_coef = (double)params.timing_offset / (double)params.timing_steps; + results->volt_coef = (double)params.volt_offset / (double)params.volt_steps * 10.0; + + results->lane_reversal = recv.lane_reversal; + results->link_speed = dev->link_speed; + results->test_status = MARGIN_TEST_OK; + + results->lanes = xmalloc(sizeof(struct margin_res_lane) * lanes_n); + for (int i = 0; i < lanes_n; i++) + { + results->lanes[i].lane + = recv.lane_reversal ? dev->width - lanes_to_margin[i] - 1 : lanes_to_margin[i]; + } + + if (args->run_margin) + { + struct margin_lanes_data lanes_data + = { .recv = &recv, .verbosity = args->verbosity, .steps_utility = args->steps_utility }; + + enum margin_dir dir[] = { TIM_LEFT, TIM_RIGHT, VOLT_UP, VOLT_DOWN }; + + u8 lanes_done = 0; + u8 use_lanes = 0; + u8 steps_t = args->steps_t ? args->steps_t : params.timing_steps; + u8 steps_v = args->steps_v ? args->steps_v : params.volt_steps; + + while (lanes_done != lanes_n) + { + use_lanes = (lanes_done + recv.parallel_lanes > lanes_n) ? lanes_n - lanes_done : + recv.parallel_lanes; + lanes_data.lanes_numbers = lanes_to_margin + lanes_done; + lanes_data.lanes_n = use_lanes; + lanes_data.results = results->lanes + lanes_done; + + for (int i = 0; i < 4; i++) + { + bool timing = dir[i] == TIM_LEFT || dir[i] == TIM_RIGHT; + if (!timing && !params.volt_support) + continue; + if (dir[i] == TIM_RIGHT && !params.ind_left_right_tim) + continue; + if (dir[i] == VOLT_DOWN && !params.ind_up_down_volt) + continue; + + lanes_data.ind = timing ? params.ind_left_right_tim : params.ind_up_down_volt; + lanes_data.dir = dir[i]; + lanes_data.steps_lane_total = timing ? steps_t : steps_v; + if (*args->steps_utility >= lanes_data.steps_lane_total) + *args->steps_utility -= lanes_data.steps_lane_total; + else + *args->steps_utility = 0; + margin_test_lanes(lanes_data); + } + lanes_done += use_lanes; + } + if (recv.lane_reversal) + { + for (int i = 0; i < lanes_n; i++) + results->lanes[i].lane = lanes_to_margin[i]; + } + } + + return true; +} + +bool +margin_read_params(struct pci_access *pacc, struct pci_dev *dev, u8 recvn, + struct margin_params *params) +{ + struct pci_cap *cap = pci_find_cap(dev, PCI_CAP_ID_EXP, PCI_CAP_NORMAL); + if (!cap) + return false; + u8 dev_dir = GET_REG_MASK(pci_read_word(dev, cap->addr + PCI_EXP_FLAGS), PCI_EXP_FLAGS_TYPE); + + bool dev_down; + if (dev_dir == PCI_EXP_TYPE_ROOT_PORT || dev_dir == PCI_EXP_TYPE_DOWNSTREAM) + dev_down = true; + else + dev_down = false; + + if (recvn == 0) + { + if (dev_down) + recvn = 1; + else + recvn = 6; + } + + if (recvn > 6) + return false; + if (dev_down && recvn == 6) + return false; + if (!dev_down && recvn != 6) + return false; + + struct pci_dev *down = NULL; + struct pci_dev *up = NULL; + struct margin_link link; + + for (struct pci_dev *p = pacc->devices; p; p = p->next) + { + if (dev_down && pci_read_byte(dev, PCI_SECONDARY_BUS) == p->bus && dev->domain == p->domain + && p->func == 0) + { + down = dev; + up = p; + break; + } + else if (!dev_down && pci_read_byte(p, PCI_SECONDARY_BUS) == dev->bus + && dev->domain == p->domain) + { + down = p; + up = dev; + break; + } + } + + if (!down) + return false; + + if (!margin_fill_link(down, up, &link)) + return false; + + struct margin_dev *dut = (dev_down ? &link.down_port : &link.up_port); + if (!margin_check_ready_bit(dut->dev)) + return false; + + if (!margin_prep_link(&link)) + return false; + + bool status; + bool lane_reversal = false; + status = read_params_internal(dut, recvn, lane_reversal, params); + if (!status) + { + lane_reversal = true; + status = read_params_internal(dut, recvn, lane_reversal, params); + } + + margin_restore_link(&link); + + return status; +} + +enum margin_test_status +margin_process_args(struct margin_dev *dev, struct margin_args *args) +{ + u8 receivers_n = 2 + 2 * dev->retimers_n; + + if (!args->recvs_n) + { + for (int i = 1; i < receivers_n; i++) + args->recvs[i - 1] = i; + args->recvs[receivers_n - 1] = 6; + args->recvs_n = receivers_n; + } + else + { + for (int i = 0; i < args->recvs_n; i++) + { + u8 recvn = args->recvs[i]; + if (recvn < 1 || recvn > 6 || (recvn != 6 && recvn > receivers_n - 1)) + { + return MARGIN_TEST_ARGS_RECVS; + } + } + } + + if (!args->lanes_n) + { + args->lanes_n = dev->width; + for (int i = 0; i < args->lanes_n; i++) + args->lanes[i] = i; + } + else + { + for (int i = 0; i < args->lanes_n; i++) + { + if (args->lanes[i] >= dev->width) + { + return MARGIN_TEST_ARGS_LANES; + } + } + } + + return MARGIN_TEST_OK; +} + +struct margin_results * +margin_test_link(struct margin_link *link, struct margin_args *args, u8 *recvs_n) +{ + bool status = margin_prep_link(link); + + u8 receivers_n = status ? args->recvs_n : 1; + u8 *receivers = args->recvs; + + struct margin_results *results = xmalloc(sizeof(*results) * receivers_n); + + if (!status) + { + results[0].test_status = MARGIN_TEST_ASPM; + } + + if (status) + { + struct margin_dev *dut; + for (int i = 0; i < receivers_n; i++) + { + dut = receivers[i] == 6 ? &link->up_port : &link->down_port; + margin_test_receiver(dut, receivers[i], args, &results[i]); + } + + margin_restore_link(link); + } + + *recvs_n = receivers_n; + return results; +} + +void +margin_free_results(struct margin_results *results, u8 results_n) +{ + for (int i = 0; i < results_n; i++) + { + if (results[i].test_status == MARGIN_TEST_OK) + free(results[i].lanes); + } + free(results); +} From patchwork Wed Dec 27 09:44:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505258 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1444A2869B for ; Wed, 27 Dec 2023 09:46:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="ykcyHotF"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="jfJC+cj3" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com A6A48C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670371; bh=Ax3NW0xu1vCClYRutt/b2zIyVE//qDwQnJ9GXTEPJXk=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=ykcyHotFy/RtHGZjBV3MzQmfVzXW3dN0PIb9cAcP47Noe430zood+NPKf5/rU26qy NT37/fyGrwjW+SUWUpvFcIXx9pr9M+x+svJHl6rI86+T6r3mE5wlrA+eoMjAF56OrT YQ1c/uxN9cJbKIkQR87rga9df85RRmydrkV87np8pcWE1nheUr8GggXoMROOAPD1lN 05p9qeOvANEVtsqAfOXOcSuvtaph6/0vhatqSTeuSFy6IVgeIrXut+VEudOSLfDBR8 GYo0LDCjRkqgaaXqfAoZT67gj/lgQQXq+KpYQPU7eYGPoIeotYyk6wD5VLowpBMhx9 KX1mcofZaLTDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670371; bh=Ax3NW0xu1vCClYRutt/b2zIyVE//qDwQnJ9GXTEPJXk=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=jfJC+cj3UWc5XuRvqHh20ekCV5Sxbf8ZSTUF3enZjLnflwp0GECnvL2BQZNE8HBiJ VMSZrJojgSHYgUfrhYNah8nfMeoyXI1bB5IHY1YpqXCyekuweundfEMrd4LlaCTZsB 171PiV6z4bHUhXsKAcvGbnY2W/mDcu7XhfpGmf4NCxVOBL10f7NWZ3gMBvpVJXKv9/ vxfqAcNtkXCgE4gt35kaosB6u0xpZY+fs29nOQaJaj1bHsyLUEzAyLeJHfAYMaDXeS NHVjdJ1tsjKL3bwIha+Ttns0cuRG3fyIx5NWPaXjMI+zhKkYLInTg4HAHV8SFSU8l9 xcNSO0d3h3vAg== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 07/15] pciutils-pcilmr: Add logging functions for margining Date: Wed, 27 Dec 2023 14:44:56 +0500 Message-ID: <20231227094504.32257-8-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) * Implement option to turn on/off logging for margining; * Support systems with several PCI domains; * margin_log_margining function prints margining in progress log using one line messages for each Receiver in the form: "Margining - - Lanes [] - ETA: Steps: Total ETA: ". Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lmr/lmr.h | 24 +++++++++ lmr/margin.c | 13 +++++ lmr/margin_log.c | 135 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 172 insertions(+) create mode 100644 lmr/margin_log.c diff --git a/lmr/lmr.h b/lmr/lmr.h index fa627a3..c3a5039 100644 --- a/lmr/lmr.h +++ b/lmr/lmr.h @@ -181,4 +181,28 @@ struct margin_results *margin_test_link(struct margin_link *link, struct margin_ void margin_free_results(struct margin_results *results, u8 results_n); +/* margin_log */ + +extern bool margin_global_logging; +extern bool margin_print_domain; + +void margin_log(char *format, ...); + +/* b:d.f -> b:d.f */ +void margin_log_bdfs(struct pci_dev *down_port, struct pci_dev *up_port); + +/* Print Link header (bdfs, width, speed) */ +void margin_log_link(struct margin_link *link); + +void margin_log_params(struct margin_params *params); + +/* Print receiver number */ +void margin_log_recvn(struct margin_recv *recv); + +/* Print full info from Receiver struct */ +void margin_log_receiver(struct margin_recv *recv); + +/* Margining in progress log */ +void margin_log_margining(struct margin_lanes_data arg); + #endif diff --git a/lmr/margin.c b/lmr/margin.c index 57bef88..1f1fa2f 100644 --- a/lmr/margin.c +++ b/lmr/margin.c @@ -256,6 +256,7 @@ margin_test_lanes(struct margin_lanes_data arg) } arg.steps_lane_done = steps_done; + margin_log_margining(arg); } for (int i = 0; i < arg.lanes_n; i++) @@ -286,9 +287,11 @@ margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, results->recvn = recvn; results->lanes_n = lanes_n; + margin_log_recvn(&recv); if (!margin_check_ready_bit(dev->dev)) { + margin_log("\nMargining Ready bit is Clear.\n"); results->test_status = MARGIN_TEST_READY_BIT; return false; } @@ -298,6 +301,7 @@ margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, recv.lane_reversal = true; if (!read_params_internal(dev, recvn, recv.lane_reversal, ¶ms)) { + margin_log("\nError during caps reading.\n"); results->test_status = MARGIN_TEST_CAPS; return false; } @@ -315,6 +319,8 @@ margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, results->link_speed = dev->link_speed; results->test_status = MARGIN_TEST_OK; + margin_log_receiver(&recv); + results->lanes = xmalloc(sizeof(struct margin_res_lane) * lanes_n); for (int i = 0; i < lanes_n; i++) { @@ -324,6 +330,8 @@ margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, if (args->run_margin) { + if (args->verbosity > 0) + margin_log("\n"); struct margin_lanes_data lanes_data = { .recv = &recv, .verbosity = args->verbosity, .steps_utility = args->steps_utility }; @@ -363,6 +371,8 @@ margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, } lanes_done += use_lanes; } + if (args->verbosity > 0) + margin_log("\n"); if (recv.lane_reversal) { for (int i = 0; i < lanes_n; i++) @@ -504,11 +514,14 @@ margin_test_link(struct margin_link *link, struct margin_args *args, u8 *recvs_n u8 receivers_n = status ? args->recvs_n : 1; u8 *receivers = args->recvs; + margin_log_link(link); + struct margin_results *results = xmalloc(sizeof(*results) * receivers_n); if (!status) { results[0].test_status = MARGIN_TEST_ASPM; + margin_log("\nCouldn't disable ASPM on the given Link.\n"); } if (status) diff --git a/lmr/margin_log.c b/lmr/margin_log.c new file mode 100644 index 0000000..6833d1a --- /dev/null +++ b/lmr/margin_log.c @@ -0,0 +1,135 @@ +/* + * The PCI Utilities -- Log margining process + * + * Copyright (c) 2023 KNS Group LLC (YADRO) + * + * Can be freely distributed and used under the terms of the GNU GPL v2+. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include +#include + +#include "lmr.h" + +bool margin_global_logging = false; +bool margin_print_domain = true; + +void +margin_log(char *format, ...) +{ + va_list arg; + va_start(arg, format); + if (margin_global_logging) + vprintf(format, arg); + va_end(arg); +} + +void +margin_log_bdfs(struct pci_dev *down, struct pci_dev *up) +{ + if (margin_print_domain) + margin_log("%x:%x:%x.%x -> %x:%x:%x.%x", down->domain, down->bus, down->dev, down->func, + up->domain, up->bus, up->dev, up->func); + else + margin_log("%x:%x.%x -> %x:%x.%x", down->bus, down->dev, down->func, up->bus, up->dev, + up->func); +} + +void +margin_log_link(struct margin_link *link) +{ + margin_log("Link "); + margin_log_bdfs(link->down_port.dev, link->up_port.dev); + margin_log("\nNegotiated Link Width: %d\n", link->down_port.width); + margin_log("Link Speed: %d.0 GT/s = Gen %d\n", (link->down_port.link_speed - 3) * 16, + link->down_port.link_speed); + margin_log("Available receivers: "); + int receivers_n = 2 + 2 * link->down_port.retimers_n; + for (int i = 1; i < receivers_n; i++) + margin_log("Rx(%X) - %d, ", 10 + i - 1, i); + margin_log("Rx(F) - 6\n"); +} + +void +margin_log_params(struct margin_params *params) +{ + margin_log("Independent Error Sampler: %d\n", params->ind_error_sampler); + margin_log("Sample Reporting Method: %d\n", params->sample_report_method); + margin_log("Independent Left and Right Timing Margining: %d\n", params->ind_left_right_tim); + margin_log("Voltage Margining Supported: %d\n", params->volt_support); + margin_log("Independent Up and Down Voltage Margining: %d\n", params->ind_up_down_volt); + margin_log("Number of Timing Steps: %d\n", params->timing_steps); + margin_log("Number of Voltage Steps: %d\n", params->volt_steps); + margin_log("Max Timing Offset: %d\n", params->timing_offset); + margin_log("Max Voltage Offset: %d\n", params->volt_offset); + margin_log("Max Lanes: %d\n", params->max_lanes); +} + +void +margin_log_recvn(struct margin_recv *recv) +{ + margin_log("\nReceiver = Rx(%X)\n", 10 + recv->recvn - 1); +} + +void +margin_log_receiver(struct margin_recv *recv) +{ + margin_log("\nError Count Limit = %d\n", recv->error_limit); + margin_log("Parallel Lanes: %d\n\n", recv->parallel_lanes); + + margin_log_params(recv->params); + + if (recv->lane_reversal) + { + margin_log("\nWarning: device uses Lane Reversal.\n"); + margin_log("However, utility uses logical lane numbers in arguments and for logging.\n"); + } +} + +void +margin_log_margining(struct margin_lanes_data arg) +{ + char *ind_dirs[] = { "Up", "Down", "Left", "Right" }; + char *non_ind_dirs[] = { "Voltage", "", "Timing" }; + + if (arg.verbosity > 0) + { + margin_log("\033[2K\rMargining - "); + if (arg.ind) + margin_log("%s", ind_dirs[arg.dir]); + else + margin_log("%s", non_ind_dirs[arg.dir]); + + u8 lanes_counter = 0; + margin_log(" - Lanes "); + margin_log("[%d", arg.lanes_numbers[0]); + for (int i = 1; i < arg.lanes_n; i++) + { + if (arg.lanes_numbers[i] - 1 == arg.lanes_numbers[i - 1]) + { + lanes_counter++; + if (lanes_counter == 1) + margin_log("-"); + if (i + 1 == arg.lanes_n) + margin_log("%d", arg.lanes_numbers[i]); + } + else + { + if (lanes_counter > 0) + margin_log("%d", arg.lanes_numbers[i - 1]); + margin_log(",%d", arg.lanes_numbers[i]); + lanes_counter = 0; + } + } + margin_log("]"); + + u64 lane_eta_s = (arg.steps_lane_total - arg.steps_lane_done) * MARGIN_STEP_MS / 1000; + u64 total_eta_s = *arg.steps_utility * MARGIN_STEP_MS / 1000 + lane_eta_s; + margin_log(" - ETA: %3ds Steps: %3d Total ETA: %3dm %2ds", lane_eta_s, arg.steps_lane_done, + total_eta_s / 60, total_eta_s % 60); + + fflush(stdout); + } +} From patchwork Wed Dec 27 09:44:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505259 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EB6C24B2F for ; Wed, 27 Dec 2023 09:46:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="FaBpZNlf"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="ZHuvVNFI" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com DE4F7C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670373; bh=CcF98B8RDvfXdgFqkGZKx6NB2ZRea34z09tQdPsmn/I=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=FaBpZNlfj4qxqu65a6o3degW4NBCd1c2gAM4DydYY3pkRFeM26l+0dwTzcgDMt4vO iib0K5AJtLtc5cpGWKDHW2OQKy9wh+JvLHJbx9nde5fFn7zNXDk2U6AfJoATD9mYqc OQpRWLdcbo8rirAA08xxyUXVRAtxso7ndm1alRKiA/au9x70/vD8kS3WJplxAu/sfT NsKexJPLrHy9Aa74Hs7KtNhIMVmVlRZva+ORpiy1r03Fv0Ndd7sMIxiRlt/63TuG62 HVvssGzXtKcNCcAbeJRtEQreZWGtohMtsCvWCuGsqRZUw3JjKOn38RpohHt1tShJnV TmFudeJ4Od+og== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670373; bh=CcF98B8RDvfXdgFqkGZKx6NB2ZRea34z09tQdPsmn/I=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=ZHuvVNFI7qiB+n3+zhWcgko7+H14kf6/7RyOcxDCVIkrNzCjbhIRPpAIkwXb6RSKg +5cxO1dXTSPyOR1j3FI01cxMJyANnT2R1aMVbOMR3C8ePQUU20eFM2WGPhU1xyv94P g9ASzdxfbNIWP9ck4VW/7eJBxgQryxLwVfp7RIkLABRuLzQTP0zcTsB/Dv4tWK3070 oncNsaLsG22ACky91NGwSyOZKFT8tirvu0sW2k9aG1uOTTs02I0wf+tdra7LWUSw0d q5cSgl3QloaWCc6HtIJ8izLBWXXv8BjsxQ9Z08AX8RjKxxPTOr59QDMXVQXiPuq3/G k7+Vc7Yn0Pdiw== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 08/15] pciutils-pcilmr: Add function for default margining results log Date: Wed, 27 Dec 2023 14:44:57 +0500 Message-ID: <20231227094504.32257-9-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Lanes are rated according to the minimum/recommended values. The minimum values are taken from PCIe Base Spec Rev 5.0 section 8.4.4. 30% UI recommended value for timing is taken from NVIDIA presentation "PCIe 4.0 Mass Electrical Margins Data Collection". Receiver lanes are called 'Weird' if all results of all receiver lanes are equal to the spec minimum value. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lmr/lmr.h | 8 +++ lmr/margin_results.c | 159 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 lmr/margin_results.c diff --git a/lmr/lmr.h b/lmr/lmr.h index c3a5039..d35b8ae 100644 --- a/lmr/lmr.h +++ b/lmr/lmr.h @@ -17,6 +17,10 @@ #define MARGIN_STEP_MS 1000 +#define MARGIN_TIM_MIN 20 +#define MARGIN_TIM_RECOMMEND 30 +#define MARGIN_VOLT_MIN 50 + /* PCI Device wrapper for margining functions */ struct margin_dev { struct pci_dev *dev; @@ -205,4 +209,8 @@ void margin_log_receiver(struct margin_recv *recv); /* Margining in progress log */ void margin_log_margining(struct margin_lanes_data arg); +/* margin_results */ + +void margin_results_print_brief(struct margin_results *results, u8 recvs_n); + #endif diff --git a/lmr/margin_results.c b/lmr/margin_results.c new file mode 100644 index 0000000..5ee065d --- /dev/null +++ b/lmr/margin_results.c @@ -0,0 +1,159 @@ +/* + * The PCI Utilities -- Display/save margining results + * + * Copyright (c) 2023 KNS Group LLC (YADRO) + * + * Can be freely distributed and used under the terms of the GNU GPL v2+. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include +#include +#include +#include + +#include "lmr.h" + +enum lane_rating { + BAD = 0, + OKAY, + PERFECT, + WEIRD, + INIT, +}; + +static char *const grades[] = { "Bad", "Okay", "Perfect", "Weird" }; +static char *const sts_strings[] = { "NAK", "LIM", "THR" }; +static const double ui[] = { 62.5 / 100, 31.25 / 100 }; + +static enum lane_rating +rate_lane(double value, double min, double recommended, enum lane_rating cur_rate) +{ + enum lane_rating res = PERFECT; + if (value < recommended) + res = OKAY; + if (value < min) + res = BAD; + if (cur_rate == INIT) + return res; + if (res < cur_rate) + return res; + else + return cur_rate; +} + +static bool +check_recv_weird(struct margin_results *results, double tim_min, double volt_min) +{ + bool result = true; + + struct margin_res_lane *lane; + for (int i = 0; i < results->lanes_n && result; i++) + { + lane = &(results->lanes[i]); + if (lane->steps[TIM_LEFT] * results->tim_coef != tim_min) + result = false; + if (results->params.ind_left_right_tim + && lane->steps[TIM_RIGHT] * results->tim_coef != tim_min) + result = false; + if (results->params.volt_support) + { + if (lane->steps[VOLT_UP] * results->volt_coef != volt_min) + result = false; + if (results->params.ind_up_down_volt + && lane->steps[VOLT_DOWN] * results->volt_coef != volt_min) + result = false; + } + } + return result; +} + +void +margin_results_print_brief(struct margin_results *results, u8 recvs_n) +{ + struct margin_res_lane *lane; + struct margin_results *res; + struct margin_params params; + + enum lane_rating lane_rating; + + u8 link_speed; + + char *no_test_msgs[] = { "", + "Margining Ready bit is Clear", + "Error during caps reading", + "Margining prerequisites are not satisfied (16/32 GT/s, D0)", + "Invalid lanes specified with arguments", + "Invalid receivers specified with arguments", + "Couldn't disable ASPM" }; + + for (int i = 0; i < recvs_n; i++) + { + res = &(results[i]); + params = res->params; + link_speed = res->link_speed - 4; + + if (res->test_status != MARGIN_TEST_OK) + { + if (res->test_status < MARGIN_TEST_PREREQS) + printf("Rx(%X) -", 10 + res->recvn - 1); + printf(" Couldn't run test (%s)\n\n", no_test_msgs[res->test_status]); + continue; + } + + if (res->lane_reversal) + printf("Rx(%X) - Lane Reversal\n", 10 + res->recvn - 1); + + if (check_recv_weird(res, MARGIN_TIM_MIN, MARGIN_VOLT_MIN)) + lane_rating = WEIRD; + else + lane_rating = INIT; + + for (u8 j = 0; j < res->lanes_n; j++) + { + lane = &(res->lanes[j]); + double left_ui = lane->steps[TIM_LEFT] * res->tim_coef; + double right_ui = lane->steps[TIM_RIGHT] * res->tim_coef; + double up_volt = lane->steps[VOLT_UP] * res->volt_coef; + double down_volt = lane->steps[VOLT_DOWN] * res->volt_coef; + + if (lane_rating != WEIRD) + { + lane_rating = rate_lane(left_ui, MARGIN_TIM_MIN, MARGIN_TIM_RECOMMEND, INIT); + if (params.ind_left_right_tim) + lane_rating + = rate_lane(right_ui, MARGIN_TIM_MIN, MARGIN_TIM_RECOMMEND, lane_rating); + if (params.volt_support) + { + lane_rating = rate_lane(up_volt, MARGIN_VOLT_MIN, MARGIN_VOLT_MIN, lane_rating); + if (params.ind_up_down_volt) + lane_rating + = rate_lane(down_volt, MARGIN_VOLT_MIN, MARGIN_VOLT_MIN, lane_rating); + } + } + + printf("Rx(%X) Lane %2d - %s\t", 10 + res->recvn - 1, lane->lane, grades[lane_rating]); + if (params.ind_left_right_tim) + printf("L %4.1f%% UI - %5.2fps - %2dst %s, R %4.1f%% UI - %5.2fps - %2dst %s", left_ui, + left_ui * ui[link_speed], lane->steps[TIM_LEFT], + sts_strings[lane->statuses[TIM_LEFT]], right_ui, right_ui * ui[link_speed], + lane->steps[TIM_RIGHT], sts_strings[lane->statuses[TIM_RIGHT]]); + else + printf("T %4.1f%% UI - %5.2fps - %2dst %s", left_ui, left_ui * ui[link_speed], + lane->steps[TIM_LEFT], sts_strings[lane->statuses[TIM_LEFT]]); + if (params.volt_support) + { + if (params.ind_up_down_volt) + printf(", U %5.1f mV - %3dst %s, D %5.1f mV - %3dst %s", up_volt, + lane->steps[VOLT_UP], sts_strings[lane->statuses[VOLT_UP]], down_volt, + lane->steps[VOLT_DOWN], sts_strings[lane->statuses[VOLT_DOWN]]); + else + printf(", V %5.1f mV - %3dst %s", up_volt, lane->steps[VOLT_UP], + sts_strings[lane->statuses[VOLT_UP]]); + } + printf("\n"); + } + printf("\n"); + } +} From patchwork Wed Dec 27 09:44:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505260 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B2D91095A for ; Wed, 27 Dec 2023 09:46:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="HOVRHUEW"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="Yw0gVHd6" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com 8B39DC000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670375; bh=CHiHv/s9vtWrbYFqTgEJBRebx92tNk5nrBIj8xENWms=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=HOVRHUEW532lBgcoaf+NBCOxLqvT4x+j3K2AiojLnoCNNMvEQlI3H5wui+hgQBDJX oclQ44AtIRwHWhF3NgV4bqofuJ6srKR+qEGU2OQhFkJmSwGx8VQjAN2j5PMjTHD0fj UIs1lufO0pxlWQlOTaLVaaOSLMjQwUVAO/KPFS4YrY+wu9uPDHsCVHhrnSFqULh6lJ Qf3935bNO+Bx0McJ9JzYxIOSBmBdBi2bXnzleWbBjtXmEEJ8EDcbvHt4t7V+ictHwL 6TpmVf4hjOc9vU5Yyax4UNa2VOpf2AwZZx7d0o8JRbUFXKFNrvAdLKP6kjlSvdlxDs EWJpkhGGIo77w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670375; bh=CHiHv/s9vtWrbYFqTgEJBRebx92tNk5nrBIj8xENWms=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=Yw0gVHd6ZppwU7CNG0c1eUc8Oge7AHJeoQ6i1yFRd1xc35LLG/9Meyro8XQqWOBGs +GiPZap3/4moVcd0gqQR9dGcNbEGcMSsEhPXwl64TegMbTJ2IsUi0xx6Dpa83zbfS7 axsvgQleL/SNCqxtBKNy86PefXfoEi7EGVQqqREsKnt/ykIrgoXHxeJkGcMV9on1dM 7dRa2ghp/N6JTo1qTHTub53P3sIMpFVo1AvmbLof1qMTGDeLwoCctPgkEEqulYXLWC TzqKFpS9kcPDE4R4SlTKFLjx+6ixE7MrF+nf26PsAlX55i0DntqXV35fyQj3FWull1 bBXTMX3Y8XXNQ== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 09/15] pciutils-pcilmr: Add utility main function Date: Wed, 27 Dec 2023 14:44:58 +0500 Message-ID: <20231227094504.32257-10-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- Makefile | 14 ++- pcilmr.c | 294 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 306 insertions(+), 2 deletions(-) create mode 100644 pcilmr.c diff --git a/Makefile b/Makefile index 52538e8..ee8e8ee 100644 --- a/Makefile +++ b/Makefile @@ -65,9 +65,13 @@ LIBNAME=libpci PCIINC=lib/config.h lib/header.h lib/pci.h lib/types.h lib/sysdep.h lib/bitops.h PCIINC_INS=lib/config.h lib/header.h lib/pci.h lib/types.h +LMR=margin_hw.o margin.o margin_log.o margin_results.o +LMROBJS=$(addprefix lmr/,$(LMR)) +LMRINC=lmr/lmr.h + export -all: lib/$(PCIIMPLIB) lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lspci.8 setpci.8 pcilib.7 pci.ids.5 update-pciids update-pciids.8 $(PCI_IDS) +all: lib/$(PCIIMPLIB) lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lspci.8 setpci.8 pcilib.7 pci.ids.5 update-pciids update-pciids.8 $(PCI_IDS) pcilmr lib/$(PCIIMPLIB): $(PCIINC) force $(MAKE) -C lib all @@ -110,6 +114,12 @@ update-pciids: update-pciids.sh example$(EXEEXT): example.o lib/$(PCIIMPLIB) example.o: example.c $(PCIINC) +$(LMROBJS) pcilmr.o: CFLAGS+=-I . +$(LMROBJS): %.o: %.c $(LMRINC) $(PCIINC) pciutils.h + +pcilmr: pcilmr.o lib/$(PCIIMPLIB) $(LMROBJS) $(COMMON) +pcilmr.o: pcilmr.c $(LMRINC) $(PCIINC) pciutils.h + %$(EXEEXT): %.o $(CC) $(LDFLAGS) $(TARGET_ARCH) $^ $(LDLIBS) -o $@ @@ -141,7 +151,7 @@ TAGS: clean: rm -f `find . -name "*~" -o -name "*.[oa]" -o -name "\#*\#" -o -name TAGS -o -name core -o -name "*.orig"` - rm -f update-pciids lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lib/config.* *.[578] pci.ids.gz lib/*.pc lib/*.so lib/*.so.* lib/*.dll lib/*.def lib/dllrsrc.rc *-rsrc.rc tags + rm -f update-pciids lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lib/config.* *.[578] pci.ids.gz lib/*.pc lib/*.so lib/*.so.* lib/*.dll lib/*.def lib/dllrsrc.rc *-rsrc.rc tags pcilmr rm -rf maint/dist distclean: clean diff --git a/pcilmr.c b/pcilmr.c new file mode 100644 index 0000000..3634c97 --- /dev/null +++ b/pcilmr.c @@ -0,0 +1,294 @@ +/* + * The PCI Utilities -- Margining utility main function + * + * Copyright (c) 2023 KNS Group LLC (YADRO) + * + * Can be freely distributed and used under the terms of the GNU GPL v2+. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include +#include +#include +#include + +#include "lmr/lmr.h" + +const char program_name[] = "pcilmr"; + +static const char usage_msg[] + = "Usage:\n" + "pcilmr [] \n\n" + "Device Specifier:\n" + ":\t[:]:.\n\n" + "Margining options:\n\n" + "Margining Test settings:\n" + "-c\t\t\tPrint Device Lane Margining Capabilities only. Do not run margining.\n" + "-l [,...]\tSpecify lanes for margining. Default: all link lanes.\n" + "\t\t\tRemember that Device may use Lane Reversal for Lane numbering.\n" + "\t\t\tHowever, utility uses logical lane numbers in arguments and for logging.\n" + "\t\t\tUtility will automatically determine Lane Reversal and tune its calls.\n" + "-e \t\tSpecify Error Count Limit for margining. Default: 4.\n" + "-r [,...]\tSpecify Receivers to select margining targets.\n" + "\t\t\tDefault: all available Receivers (including Retimers).\n" + "-p \tSpecify number of lanes to margin simultaneously.\n" + "\t\t\tDefault: 1.\n" + "\t\t\tAccording to spec it's possible for Receiver to margin up\n" + "\t\t\tto MaxLanes + 1 lanes simultaneously, but usually this works\n" + "\t\t\tbad, so this option is for experiments mostly.\n" + "-T\t\t\tTime Margining will continue until the Error Count is no more\n" + "\t\t\tthan an Error Count Limit. Use this option to find Link limit.\n" + "-V\t\t\tSame as -T option, but for Voltage.\n" + "-t \t\tSpecify maximum number of steps for Time Margining.\n" + "-v \t\tSpecify maximum number of steps for Voltage Margining.\n" + "Use only one of -T/-t options at the same time (same for -V/-v).\n" + "Without these options utility will use MaxSteps from Device\n" + "capabilities as test limit.\n\n"; + +static struct pci_dev * +dev_for_filter(struct pci_access *pacc, char *filter) +{ + struct pci_filter pci_filter; + char dev[17] = { 0 }; + strncpy(dev, filter, sizeof(dev) - 1); + pci_filter_init(pacc, &pci_filter); + if (pci_filter_parse_slot(&pci_filter, dev)) + die("Invalid device ID: %s\n", filter); + + if (pci_filter.bus == -1 || pci_filter.slot == -1 || pci_filter.func == -1) + die("Invalid device ID: %s\n", filter); + + if (pci_filter.domain == -1) + pci_filter.domain = 0; + + for (struct pci_dev *p = pacc->devices; p; p = p->next) + { + if (pci_filter_match(&pci_filter, p)) + return p; + } + + die("No such PCI device: %s or you don't have enough privileges.\n", filter); +} + +static struct pci_dev * +find_down_port_for_up(struct pci_access *pacc, struct pci_dev *up) +{ + struct pci_dev *down = NULL; + for (struct pci_dev *p = pacc->devices; p; p = p->next) + { + if (pci_read_byte(p, PCI_SECONDARY_BUS) == up->bus && up->domain == p->domain) + { + down = p; + break; + } + } + return down; +} + +static u8 +parse_csv_arg(char *arg, u8 *vals) +{ + u8 cnt = 0; + char *token = strtok(arg, ","); + while (token) + { + vals[cnt] = atoi(token); + cnt++; + token = strtok(NULL, ","); + } + return cnt; +} + +int +main(int argc, char **argv) +{ + struct pci_access *pacc; + + struct pci_dev *up_port; + struct pci_dev *down_port; + + struct margin_link link; + + bool status = true; + + struct margin_results *results; + u8 results_n; + + struct margin_args args; + + u8 steps_t_arg = 0; + u8 steps_v_arg = 0; + u8 parallel_lanes_arg = 1; + u8 error_limit = 4; + + u8 lanes_n = 0; + u8 recvs_n = 0; + + bool run_margin = true; + + u64 total_steps = 0; + + pacc = pci_alloc(); + pci_init(pacc); + pci_scan_bus(pacc); + + margin_print_domain = false; + for (struct pci_dev *dev = pacc->devices; dev; dev = dev->next) + { + if (dev->domain != 0) + { + margin_print_domain = true; + break; + } + } + + margin_global_logging = true; + + int c; + + while ((c = getopt(argc, argv, ":r:e:l:cp:t:v:VT")) != -1) + { + switch (c) + { + case 't': + steps_t_arg = atoi(optarg); + break; + case 'T': + steps_t_arg = 63; + break; + case 'v': + steps_v_arg = atoi(optarg); + break; + case 'V': + steps_v_arg = 127; + break; + case 'p': + parallel_lanes_arg = atoi(optarg); + break; + case 'c': + run_margin = false; + break; + case 'l': + lanes_n = parse_csv_arg(optarg, args.lanes); + break; + case 'e': + error_limit = atoi(optarg); + break; + case 'r': + recvs_n = parse_csv_arg(optarg, args.recvs); + break; + default: + die("Invalid arguments\n\n%s", usage_msg); + } + } + + if (optind != argc - 1) + status = false; + if (!status && argc > 1) + die("Invalid arguments\n\n%s", usage_msg); + if (!status) + { + printf("%s", usage_msg); + exit(0); + } + + up_port = dev_for_filter(pacc, argv[argc - 1]); + + down_port = find_down_port_for_up(pacc, up_port); + if (!down_port) + die("Cannot find Upstream Component for the specified device: %s\n", argv[argc - 1]); + + if (!pci_find_cap(up_port, PCI_CAP_ID_EXP, PCI_CAP_NORMAL)) + die("Looks like you don't have enough privileges to access " + "Device Configuration Space.\nTry to run utility as root.\n"); + + if (!margin_fill_link(down_port, up_port, &link)) + { + printf("Link "); + margin_log_bdfs(down_port, up_port); + printf(" is not ready for margining.\n" + "Link data rate must be 16 GT/s or 32 GT/s.\n" + "Downstream Component must be at D0 PM state.\n"); + status = false; + } + + if (status) + { + args.error_limit = error_limit; + args.lanes_n = lanes_n; + args.recvs_n = recvs_n; + args.steps_t = steps_t_arg; + args.steps_v = steps_v_arg; + args.parallel_lanes = parallel_lanes_arg; + args.run_margin = run_margin; + args.verbosity = 1; + args.steps_utility = &total_steps; + + enum margin_test_status args_status; + + if ((args_status = margin_process_args(&link.down_port, &args)) != MARGIN_TEST_OK) + { + status = false; + margin_log_link(&link); + if (args_status == MARGIN_TEST_ARGS_RECVS) + margin_log("\nInvalid RecNums specified.\n"); + else if (args_status == MARGIN_TEST_ARGS_LANES) + margin_log("\nInvalid lanes specified.\n"); + } + } + + if (status) + { + struct margin_params params; + + for (int i = 0; i < args.recvs_n; i++) + { + if (margin_read_params(pacc, args.recvs[i] == 6 ? up_port : down_port, args.recvs[i], + ¶ms)) + { + u8 steps_t = steps_t_arg ? steps_t_arg : params.timing_steps; + u8 steps_v = steps_v_arg ? steps_v_arg : params.volt_steps; + u8 parallel_recv = parallel_lanes_arg > params.max_lanes + 1 ? params.max_lanes + 1 : + parallel_lanes_arg; + + u8 step_multiplier + = args.lanes_n / parallel_recv + ((args.lanes_n % parallel_recv) > 0); + + total_steps += steps_t * step_multiplier; + if (params.ind_left_right_tim) + total_steps += steps_t * step_multiplier; + if (params.volt_support) + { + total_steps += steps_v * step_multiplier; + if (params.ind_up_down_volt) + total_steps += steps_v * step_multiplier; + } + } + } + + results = margin_test_link(&link, &args, &results_n); + } + + if (status && run_margin) + { + printf("\nResults:\n"); + printf("\nPass/fail criteria:\nTiming:\n"); + printf("Minimum Offset (spec): %d %% UI\nRecommended Offset: %d %% UI\n", MARGIN_TIM_MIN, + MARGIN_TIM_RECOMMEND); + printf("\nVoltage:\nMinimum Offset (spec): %d mV\n\n", MARGIN_VOLT_MIN); + printf( + "Margining statuses:\nLIM -\tErrorCount exceeded Error Count Limit (found device limit)\n"); + printf("NAK -\tDevice didn't execute last command, \n\tso result may be less reliable\n"); + printf("THR -\tThe set (using the utility options) \n\tstep threshold has been reached\n\n"); + printf("Notations:\nst - steps\n\n"); + + margin_results_print_brief(results, results_n); + } + + if (status) + margin_free_results(results, results_n); + + pci_cleanup(pacc); + return 0; +} From patchwork Wed Dec 27 09:44:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505261 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1A052940A for ; Wed, 27 Dec 2023 09:46:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="MOd4VX03"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="nkoVJRgO" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com 3A0D9C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670377; bh=Qgf3HY9Q/npWtHJ8zz2bI8mQD5n++N9iMXdOnjtXbJA=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=MOd4VX03Vt1OzP8VhUYhClq5O6avM53NESMS2VDfjNQIxnWxdf4R9f0fgqyB56lOe gh2byAM7jDMlLDq/Md39EULYj371I2nBfuV0O+tDjJT2PN6+COQMDpEtC6rpHfyopG B4XhfSdnCiXeZLczWA+a6lLGyS7M3DPtxpBo3MMSXjOLhse5Z8nWmXnR+hBD2J7x/i pG+WRs7eOURqpnxRgIU54zDIeucvdKMpsWxAML2MjR7b33c8ZyuNdRlLWaX1hhOaMC bo6CzsoPGphUZ73ZkPSzGn+8pXX9LaKBtsbDSKny/8XYtC9U8wylyHNb8hIZ4VGv6t RJ4rYs8nyeYwQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670377; bh=Qgf3HY9Q/npWtHJ8zz2bI8mQD5n++N9iMXdOnjtXbJA=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=nkoVJRgOIMKXnGiQZ9dgMtgJYG1POI+78z1rd53HQ2IICe9vXxUMDZX1PcG0jWzOX qQa1Tvna+y8hCm4edOFmHUrw3arDZNglEnj+xHeE09NNw9TgTE0Mgt1j2Y48Ku5fgP Gz9kKVpdqxD3IGvIvIEpyxpKmnaIFp3vP/shH0Z4k7jsezvgWFfu38P4u/h7igs4bX yCR/1Av7/L6WQ0AUtR0aH/8kctTHKeXd7AGPGaNGTnySLkaYVargxGTi0lHwbh4NV3 rxXja24wslK0fsvUOweJ8Kn8mB0XXxGKhvQqMr+edLNcGgg6T/r+CkMXR6M6Tu3sRo +WwYVrv7xIleQ== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 10/15] pciutils-pcilmr: Add support for unique hardware quirks Date: Wed, 27 Dec 2023 14:44:59 +0500 Message-ID: <20231227094504.32257-11-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Make it possible to change receiver margining parameters depending on current hardware specificity. In our tests Intel Ice Lake CPUs RC ports reported MaxVoltageOffset = 50 (RxA), which led to results several times bigger than the results of the hardware debugger. Looks like in Intel Sapphire Rapids this was fixed, these CPU RC ports report MaxVoltageOffset = 12 (RxA). To solve the problem it was decided to hardcode Volt Offset to 12 (120 mV) for Ice Lake RC ports. In the case of margining a specific link, only information about Downstream and Upstream ports should be sufficient to decide whether to use quirks, so the feature was implemented based on a list of devices (vendor - device - revision triples), whose problems are known. Back to Ice Lake ports, according to Integrators List on the pci-sig site, the list of possible RC ports of Ice Lake Xeon's includes at least three more options (with ids 347B/C/D) besides the one used in this commit, but we don't have such processors to check the relevance of the MaxVoltageOffset problem for these ports. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lmr/lmr.h | 6 ++++++ lmr/margin.c | 16 ++++++++++++++++ lmr/margin_hw.c | 24 +++++++++++++++++++++++- lmr/margin_log.c | 16 ++++++++++++++++ 4 files changed, 61 insertions(+), 1 deletion(-) diff --git a/lmr/lmr.h b/lmr/lmr.h index d35b8ae..bb188fc 100644 --- a/lmr/lmr.h +++ b/lmr/lmr.h @@ -21,6 +21,8 @@ #define MARGIN_TIM_RECOMMEND 30 #define MARGIN_VOLT_MIN 50 +enum margin_hw { MARGIN_HW_DEFAULT, MARGIN_ICE_LAKE_RC }; + /* PCI Device wrapper for margining functions */ struct margin_dev { struct pci_dev *dev; @@ -29,6 +31,8 @@ struct margin_dev { u8 retimers_n; u8 link_speed; + enum margin_hw hw; + /* Saved Device settings to restore after margining */ u8 aspm; bool hasd; // Hardware Autonomous Speed Disable @@ -209,6 +213,8 @@ void margin_log_receiver(struct margin_recv *recv); /* Margining in progress log */ void margin_log_margining(struct margin_lanes_data arg); +void margin_log_hw_quirks(struct margin_recv *recv); + /* margin_results */ void margin_results_print_brief(struct margin_results *results, u8 recvs_n); diff --git a/lmr/margin.c b/lmr/margin.c index 1f1fa2f..cc142fa 100644 --- a/lmr/margin.c +++ b/lmr/margin.c @@ -127,6 +127,20 @@ margin_report_cmd(struct margin_dev *dev, u8 lane, margin_cmd cmd, margin_cmd *r && margin_set_cmd(dev, lane, NO_COMMAND); } +static void +margin_apply_hw_quirks(struct margin_recv *recv) +{ + switch (recv->dev->hw) + { + case MARGIN_ICE_LAKE_RC: + if (recv->recvn == 1) + recv->params->volt_offset = 12; + break; + default: + break; + } +} + static bool read_params_internal(struct margin_dev *dev, u8 recvn, bool lane_reversal, struct margin_params *params) @@ -311,6 +325,8 @@ margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, if (recv.parallel_lanes > params.max_lanes + 1) recv.parallel_lanes = params.max_lanes + 1; + margin_apply_hw_quirks(&recv); + margin_log_hw_quirks(&recv); results->tim_coef = (double)params.timing_offset / (double)params.timing_steps; results->volt_coef = (double)params.volt_offset / (double)params.volt_steps * 10.0; diff --git a/lmr/margin_hw.c b/lmr/margin_hw.c index c000132..fc427c8 100644 --- a/lmr/margin_hw.c +++ b/lmr/margin_hw.c @@ -10,6 +10,27 @@ #include "lmr.h" +static u16 special_hw[][4] = + // Vendor ID, Device ID, Revision ID, margin_hw + { { 0x8086, 0x347A, 0x4, MARGIN_ICE_LAKE_RC }, + { 0xFFFF, 0, 0, MARGIN_HW_DEFAULT } + }; + +static enum margin_hw +detect_unique_hw(struct pci_dev *dev) +{ + u16 vendor = pci_read_word(dev, PCI_VENDOR_ID); + u16 device = pci_read_word(dev, PCI_DEVICE_ID); + u8 revision = pci_read_byte(dev, PCI_REVISION_ID); + + for (int i = 0; special_hw[i][0] != 0xFFFF; i++) + { + if (vendor == special_hw[i][0] && device == special_hw[i][1] && revision == special_hw[i][2]) + return special_hw[i][3]; + } + return MARGIN_HW_DEFAULT; +} + bool margin_verify_link(struct pci_dev *down_port, struct pci_dev *up_port) { @@ -56,7 +77,8 @@ fill_dev_wrapper(struct pci_dev *dev) .retimers_n = (!!(pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA2) & PCI_EXP_LINKSTA2_RETIMER)) + (!!(pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA2) & PCI_EXP_LINKSTA2_2RETIMERS)), - .link_speed = (pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA) & PCI_EXP_LNKSTA_SPEED) }; + .link_speed = (pci_read_word(dev, cap->addr + PCI_EXP_LNKSTA) & PCI_EXP_LNKSTA_SPEED), + .hw = detect_unique_hw(dev) }; return res; } diff --git a/lmr/margin_log.c b/lmr/margin_log.c index 6833d1a..a93e07b 100644 --- a/lmr/margin_log.c +++ b/lmr/margin_log.c @@ -133,3 +133,19 @@ margin_log_margining(struct margin_lanes_data arg) fflush(stdout); } } + +void +margin_log_hw_quirks(struct margin_recv *recv) +{ + switch (recv->dev->hw) + { + case MARGIN_ICE_LAKE_RC: + if (recv->recvn == 1) + margin_log("\nRx(A) is Intel Ice Lake RC port.\n" + "Applying next quirks for margining process:\n" + " - Set MaxVoltageOffset to 12 (120 mV).\n"); + break; + default: + break; + } +} From patchwork Wed Dec 27 09:45:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505262 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 177AA2C6A2 for ; Wed, 27 Dec 2023 09:46:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="FFYNcvll"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="MF2hTNGI" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com 80AACC000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670378; bh=f9HOiVP1j/am+NoHSSBRYmNnX6zSQKYjaB2+zJLZ7QE=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=FFYNcvllV1ZIOasw6TwIGyL+SH/phvu612TqckWyc89nMgD8DzmK5NmEg+gTwFInt xhXoHCJUc9HwNjo3VKGNSFf7HEX1rDmqHCBO3cHSPI4nGLEDBRBsc2aPedjW+E6ujW eqNpZRpZhLWl6JDgt13koslDAKpTHKFQB9SIpBVPPklN/urIyFS7wDNv2vWEc5at/e ibq9+Kzl0cO1MxdnCrAJMFSEQoTa6AUcUrhhWJTvLS+CzT4KNhz3JMpXAOoXM/jiTy RHygSroHFvQ1rHmoHvT0mIgd7SadlZGLvi2EhuHpcPTtDXAKkDcMDC//yhiFBfjprf s+A/IxOtSCT6Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670378; bh=f9HOiVP1j/am+NoHSSBRYmNnX6zSQKYjaB2+zJLZ7QE=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=MF2hTNGITSKCTwWZs7w1oH3p2nQZcaeEqHOcjSuO5MaBGCny7iXx/54kGTT0jA683 bsISOci8eiwvuI5YXdVK+Ui2CaRTxZdzUZFP9Yy7ecQgH4fLwNANkGU3fUoWUjoiTX 2K1zyR+yubq+b6kR5rsPFdKoJbp6+VkjHM6lNolCLqypN4wKquZfPR0d2dmPQv0DjG 8R3uSilIsFyDdWLt4F/IByGtU9N429Tg6RrEAV5Ekj6lPc7Rn2UiWFQenwUrl3omnt eL9k3P9eVZ5RM1WTWGFNR2TN6Wt6rh8zMluIbEKjS+tJX0AROSyAW19gZI9esMRWSP WoRVPWv/XJNLQ== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 11/15] pciutils-pcilmr: Add the ability to pass multiple links to the utility Date: Wed, 27 Dec 2023 14:45:00 +0500 Message-ID: <20231227094504.32257-12-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) * Add support for different utility modes; * Make the default (now --margin) mode capable to accept several components and run test for all of them; * Add --full mode for sequential start of the test on all ready links in the system; * The complication of the main function is due to the need to pre-read the parameters of the devices before starting the tests in order to calculate Total ETA of the utility. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- pcilmr.c | 243 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 189 insertions(+), 54 deletions(-) diff --git a/pcilmr.c b/pcilmr.c index 3634c97..3c2f250 100644 --- a/pcilmr.c +++ b/pcilmr.c @@ -17,11 +17,17 @@ const char program_name[] = "pcilmr"; +enum mode { MARGIN, FULL }; + static const char usage_msg[] = "Usage:\n" - "pcilmr [] \n\n" + "pcilmr [--margin] [] ...\n" + "pcilmr --full []\n" "Device Specifier:\n" ":\t[:]:.\n\n" + "Modes:\n" + "--margin\t\tMargin selected Links\n" + "--full\t\t\tMargin all ready for testing Links in the system (one by one)\n" "Margining options:\n\n" "Margining Test settings:\n" "-c\t\t\tPrint Device Lane Margining Capabilities only. Do not run margining.\n" @@ -100,27 +106,59 @@ parse_csv_arg(char *arg, u8 *vals) return cnt; } +static u8 +find_ready_links(struct pci_access *pacc, struct pci_dev **down_ports, struct pci_dev **up_ports, + bool cnt_only) +{ + u8 cnt = 0; + for (struct pci_dev *up = pacc->devices; up; up = up->next) + { + if (pci_find_cap(up, PCI_EXT_CAP_ID_LMR, PCI_CAP_EXTENDED)) + { + struct pci_dev *down = find_down_port_for_up(pacc, up); + + if (down && margin_verify_link(down, up) + && (margin_check_ready_bit(down) || margin_check_ready_bit(up))) + { + if (!cnt_only) + { + up_ports[cnt] = up; + down_ports[cnt] = down; + } + cnt++; + } + } + } + return cnt; +} + int main(int argc, char **argv) { struct pci_access *pacc; - struct pci_dev *up_port; - struct pci_dev *down_port; + struct pci_dev **up_ports; + struct pci_dev **down_ports; + u8 ports_n = 0; - struct margin_link link; + struct margin_link *links; + bool *checks_status_ports; bool status = true; + enum mode mode; - struct margin_results *results; - u8 results_n; + /* each link has several receivers -> several results */ + struct margin_results **results; + u8 *results_n; - struct margin_args args; + struct margin_args *args; u8 steps_t_arg = 0; u8 steps_v_arg = 0; u8 parallel_lanes_arg = 1; u8 error_limit = 4; + u8 lanes_arg[32]; + u8 recvs_arg[6]; u8 lanes_n = 0; u8 recvs_n = 0; @@ -145,7 +183,29 @@ main(int argc, char **argv) margin_global_logging = true; + struct option long_options[] + = { { .name = "margin", .has_arg = no_argument, .flag = NULL, .val = 0 }, + { .name = "full", .has_arg = no_argument, .flag = NULL, .val = 1 }, + { 0, 0, 0, 0 } }; + int c; + c = getopt_long(argc, argv, ":", long_options, NULL); + + switch (c) + { + case -1: /* no options (strings like component are possible) */ + /* FALLTHROUGH */ + case 0: + mode = MARGIN; + break; + case 1: + mode = FULL; + break; + default: /* unknown option symbol */ + mode = MARGIN; + optind--; + break; + } while ((c = getopt(argc, argv, ":r:e:l:cp:t:v:VT")) != -1) { @@ -170,20 +230,22 @@ main(int argc, char **argv) run_margin = false; break; case 'l': - lanes_n = parse_csv_arg(optarg, args.lanes); + lanes_n = parse_csv_arg(optarg, lanes_arg); break; case 'e': error_limit = atoi(optarg); break; case 'r': - recvs_n = parse_csv_arg(optarg, args.recvs); + recvs_n = parse_csv_arg(optarg, recvs_arg); break; default: die("Invalid arguments\n\n%s", usage_msg); } } - if (optind != argc - 1) + if (mode == FULL && optind != argc) + status = false; + if (mode == MARGIN && optind == argc) status = false; if (!status && argc > 1) die("Invalid arguments\n\n%s", usage_msg); @@ -193,59 +255,91 @@ main(int argc, char **argv) exit(0); } - up_port = dev_for_filter(pacc, argv[argc - 1]); + if (mode == FULL) + { + ports_n = find_ready_links(pacc, NULL, NULL, true); + if (ports_n == 0) + { + die("Links not found or you don't have enough privileges.\n"); + } + else + { + up_ports = xmalloc(ports_n * sizeof(*up_ports)); + down_ports = xmalloc(ports_n * sizeof(*down_ports)); + find_ready_links(pacc, down_ports, up_ports, false); + } + } + else if (mode == MARGIN) + { + ports_n = argc - optind; + up_ports = xmalloc(ports_n * sizeof(*up_ports)); + down_ports = xmalloc(ports_n * sizeof(*down_ports)); - down_port = find_down_port_for_up(pacc, up_port); - if (!down_port) - die("Cannot find Upstream Component for the specified device: %s\n", argv[argc - 1]); + u8 cnt = 0; + while (optind != argc) + { + up_ports[cnt] = dev_for_filter(pacc, argv[optind]); + down_ports[cnt] = find_down_port_for_up(pacc, up_ports[cnt]); + if (!down_ports[cnt]) + die("Cannot find Upstream Component for the specified device: %s\n", argv[optind]); + cnt++; + optind++; + } + } + else + die("Bug in the args parsing!\n"); - if (!pci_find_cap(up_port, PCI_CAP_ID_EXP, PCI_CAP_NORMAL)) + if (!pci_find_cap(up_ports[0], PCI_CAP_ID_EXP, PCI_CAP_NORMAL)) die("Looks like you don't have enough privileges to access " "Device Configuration Space.\nTry to run utility as root.\n"); - if (!margin_fill_link(down_port, up_port, &link)) - { - printf("Link "); - margin_log_bdfs(down_port, up_port); - printf(" is not ready for margining.\n" - "Link data rate must be 16 GT/s or 32 GT/s.\n" - "Downstream Component must be at D0 PM state.\n"); - status = false; - } + results = xmalloc(ports_n * sizeof(*results)); + results_n = xmalloc(ports_n * sizeof(*results_n)); + links = xmalloc(ports_n * sizeof(*links)); + checks_status_ports = xmalloc(ports_n * sizeof(*checks_status_ports)); + args = xmalloc(ports_n * sizeof(*args)); - if (status) + for (int i = 0; i < ports_n; i++) { - args.error_limit = error_limit; - args.lanes_n = lanes_n; - args.recvs_n = recvs_n; - args.steps_t = steps_t_arg; - args.steps_v = steps_v_arg; - args.parallel_lanes = parallel_lanes_arg; - args.run_margin = run_margin; - args.verbosity = 1; - args.steps_utility = &total_steps; + args[i].error_limit = error_limit; + args[i].parallel_lanes = parallel_lanes_arg; + args[i].run_margin = run_margin; + args[i].verbosity = 1; + args[i].steps_t = steps_t_arg; + args[i].steps_v = steps_v_arg; + for (int j = 0; j < recvs_n; j++) + args[i].recvs[j] = recvs_arg[j]; + args[i].recvs_n = recvs_n; + for (int j = 0; j < lanes_n; j++) + args[i].lanes[j] = lanes_arg[j]; + args[i].lanes_n = lanes_n; + args[i].steps_utility = &total_steps; enum margin_test_status args_status; - if ((args_status = margin_process_args(&link.down_port, &args)) != MARGIN_TEST_OK) + if (!margin_fill_link(down_ports[i], up_ports[i], &links[i])) { - status = false; - margin_log_link(&link); - if (args_status == MARGIN_TEST_ARGS_RECVS) - margin_log("\nInvalid RecNums specified.\n"); - else if (args_status == MARGIN_TEST_ARGS_LANES) - margin_log("\nInvalid lanes specified.\n"); + checks_status_ports[i] = false; + results[i] = xmalloc(sizeof(*results[i])); + results[i]->test_status = MARGIN_TEST_PREREQS; + continue; } - } - if (status) - { + if ((args_status = margin_process_args(&links[i].down_port, &args[i])) != MARGIN_TEST_OK) + { + checks_status_ports[i] = false; + results[i] = xmalloc(sizeof(*results[i])); + results[i]->test_status = args_status; + continue; + } + + checks_status_ports[i] = true; struct margin_params params; - for (int i = 0; i < args.recvs_n; i++) + for (int j = 0; j < args[i].recvs_n; j++) { - if (margin_read_params(pacc, args.recvs[i] == 6 ? up_port : down_port, args.recvs[i], - ¶ms)) + if (margin_read_params(pacc, args[i].recvs[j] == 6 ? up_ports[i] : down_ports[i], + args[i].recvs[j], ¶ms)) { u8 steps_t = steps_t_arg ? steps_t_arg : params.timing_steps; u8 steps_v = steps_v_arg ? steps_v_arg : params.volt_steps; @@ -253,7 +347,7 @@ main(int argc, char **argv) parallel_lanes_arg; u8 step_multiplier - = args.lanes_n / parallel_recv + ((args.lanes_n % parallel_recv) > 0); + = args[i].lanes_n / parallel_recv + ((args[i].lanes_n % parallel_recv) > 0); total_steps += steps_t * step_multiplier; if (params.ind_left_right_tim) @@ -266,13 +360,40 @@ main(int argc, char **argv) } } } + } - results = margin_test_link(&link, &args, &results_n); + for (int i = 0; i < ports_n; i++) + { + if (checks_status_ports[i]) + results[i] = margin_test_link(&links[i], &args[i], &results_n[i]); + else + { + results_n[i] = 1; + if (results[i]->test_status == MARGIN_TEST_PREREQS) + { + printf("Link "); + margin_log_bdfs(down_ports[i], up_ports[i]); + printf(" is not ready for margining.\n" + "Link data rate must be 16 GT/s or 32 GT/s.\n" + "Downstream Component must be at D0 PM state.\n"); + } + else if (results[i]->test_status == MARGIN_TEST_ARGS_RECVS) + { + margin_log_link(&links[i]); + printf("\nInvalid RecNums specified.\n"); + } + else if (results[i]->test_status == MARGIN_TEST_ARGS_LANES) + { + margin_log_link(&links[i]); + printf("\nInvalid lanes specified.\n"); + } + } + printf("\n----\n\n"); } - if (status && run_margin) + if (run_margin) { - printf("\nResults:\n"); + printf("Results:\n"); printf("\nPass/fail criteria:\nTiming:\n"); printf("Minimum Offset (spec): %d %% UI\nRecommended Offset: %d %% UI\n", MARGIN_TIM_MIN, MARGIN_TIM_RECOMMEND); @@ -283,11 +404,25 @@ main(int argc, char **argv) printf("THR -\tThe set (using the utility options) \n\tstep threshold has been reached\n\n"); printf("Notations:\nst - steps\n\n"); - margin_results_print_brief(results, results_n); + for (int i = 0; i < ports_n; i++) + { + printf("Link "); + margin_log_bdfs(down_ports[i], up_ports[i]); + printf(":\n\n"); + margin_results_print_brief(results[i], results_n[i]); + printf("\n"); + } } - if (status) - margin_free_results(results, results_n); + for (int i = 0; i < ports_n; i++) + margin_free_results(results[i], results_n[i]); + free(results_n); + free(results); + free(up_ports); + free(down_ports); + free(links); + free(checks_status_ports); + free(args); pci_cleanup(pacc); return 0; From patchwork Wed Dec 27 09:45:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505263 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 281D7288B6 for ; Wed, 27 Dec 2023 09:46:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="NwiDeCDI"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="Uzpn3vrU" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com AE0F7C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670379; bh=d7HCpL3hF76xZ1LCs3g4aQfkoFICVTAarCn9H2xnV7E=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=NwiDeCDId1jSv0+EN06nBhgalfI5H10CenpZmBFHI1i4DBQzvyq4PivrVo2om7LsU hh0tV2cDOWilotBCKp96GN1zwcXujpoz+YuGTTXvgdH2MrikWLz5X0larmfG2gsm27 oxkThqS4VMSFXzsO4x9We5rNM1lrwr63Xjl76B1Y+vl7v77EMLDQnxgWfdIDnNSJ07 1RX/qUm6Sjl4OzGhPly4b90uEi0Ma3o2M1Klnqhvv+0depNkDNR4sBtpWnNxWgmIZK ct2WXpPGt6NWhPsjw9e8lrv7kGbRrR7Yg2lnS6HKhpYuv1cZAW3dM3MhMxk3+gulTb inkb4Lz9DcEhA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670379; bh=d7HCpL3hF76xZ1LCs3g4aQfkoFICVTAarCn9H2xnV7E=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=Uzpn3vrUEGJhZsgIHHGDb0JjTPhnT4s6wLNuH/Bc+/HmSDZiFBintACc5IcFRNxJT M2e645S9BdMCuImVkgCtaT2zopOP5soGazCUprzMtyRb273spwzrYjNNunBrzTTpVm 0mutnWkDK8Mxq3LdQW2lwuSoKyx07MQyCiHl683exZsLmA7s6yrhRUIeMnw1knC73L LgvkskDYj32VFLwz1OzX79fwIXosD6AZirKn9q4JT6lEgS1WyU4HlyIdsrKWi/zr09 hHRaKflLeqVI8LZdv55jywkGNcwpOBjYw7X4swFGgcOsyeQg/QaLbsJhyiT2mPDWJ6 d4MZEzetU5uZg== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 12/15] pciutils-pcilmr: Add --scan mode to search for all LMR-capable Links Date: Wed, 27 Dec 2023 14:45:01 +0500 Message-ID: <20231227094504.32257-13-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- pcilmr.c | 43 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/pcilmr.c b/pcilmr.c index 3c2f250..43e791d 100644 --- a/pcilmr.c +++ b/pcilmr.c @@ -17,17 +17,19 @@ const char program_name[] = "pcilmr"; -enum mode { MARGIN, FULL }; +enum mode { MARGIN, FULL, SCAN }; static const char usage_msg[] = "Usage:\n" "pcilmr [--margin] [] ...\n" "pcilmr --full []\n" + "pcilmr --scan\n\n" "Device Specifier:\n" ":\t[:]:.\n\n" "Modes:\n" "--margin\t\tMargin selected Links\n" "--full\t\t\tMargin all ready for testing Links in the system (one by one)\n" + "--scan\t\t\tScan for Links available for margining\n\n" "Margining options:\n\n" "Margining Test settings:\n" "-c\t\t\tPrint Device Lane Margining Capabilities only. Do not run margining.\n" @@ -106,6 +108,36 @@ parse_csv_arg(char *arg, u8 *vals) return cnt; } +static void +scan_links(struct pci_access *pacc, bool only_ready) +{ + if (only_ready) + printf("Links ready for margining:\n"); + else + printf("Links with Lane Margining at the Receiver capabilities:\n"); + bool flag = true; + for (struct pci_dev *up = pacc->devices; up; up = up->next) + { + if (pci_find_cap(up, PCI_EXT_CAP_ID_LMR, PCI_CAP_EXTENDED)) + { + struct pci_dev *down = find_down_port_for_up(pacc, up); + + if (down && margin_verify_link(down, up)) + { + margin_log_bdfs(down, up); + if (!only_ready && (margin_check_ready_bit(down) || margin_check_ready_bit(up))) + printf(" - Ready"); + printf("\n"); + flag = false; + } + } + } + if (flag) + printf("Links not found or you don't have enough privileges.\n"); + pci_cleanup(pacc); + exit(0); +} + static u8 find_ready_links(struct pci_access *pacc, struct pci_dev **down_ports, struct pci_dev **up_ports, bool cnt_only) @@ -185,7 +217,8 @@ main(int argc, char **argv) struct option long_options[] = { { .name = "margin", .has_arg = no_argument, .flag = NULL, .val = 0 }, - { .name = "full", .has_arg = no_argument, .flag = NULL, .val = 1 }, + { .name = "scan", .has_arg = no_argument, .flag = NULL, .val = 1 }, + { .name = "full", .has_arg = no_argument, .flag = NULL, .val = 2 }, { 0, 0, 0, 0 } }; int c; @@ -199,6 +232,12 @@ main(int argc, char **argv) mode = MARGIN; break; case 1: + mode = SCAN; + if (optind == argc) + scan_links(pacc, false); + optind--; + break; + case 2: mode = FULL; break; default: /* unknown option symbol */ From patchwork Wed Dec 27 09:45:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505264 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76F1624B3C for ; Wed, 27 Dec 2023 09:46:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="lQfoeny2"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="TTf9/Rdz" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com D0FD0C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670380; bh=7q54cSreJ48wFoWwiSu6CwGJIg4nfa1QOd46uDlpZq4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=lQfoeny2OhiTrSwaMTWdUcNAi5ROkXKgpbY8nc/RhY1AwXGRzM/PgeSf3qxaTNPxv 1pX104zMpcGnfqqjKdMMj/a+r5EP6Dskt9boj6B5+rvKYzlzgvr/H33kJaXRUM0qXs WFip1OViTr1zGvIWiPLveKNCgU2sBveUJ9jatCJpHsx/j4wMwbNFBMTScd5wrQo+xJ CtLXBQZP0GrJQFzosyvZU849HUiIsVaq9Foj8+hhiBVxEUTTGnIBpdAoGRqOjc/hPV TPdyr4wzVU/t/dQqGVW3E1KuBnfzH0yuBmrSW1gH+AsF99xYPdmTp1EhwhXAhVN9nP qXhUKjREoKFxA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670380; bh=7q54cSreJ48wFoWwiSu6CwGJIg4nfa1QOd46uDlpZq4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=TTf9/RdzWav+BEh3FvIW3kKT1EAKOaYCoSKbRXiP6yPUA8OerCewaw+FyGpVpQsbQ a+mehZoLvJGTy4INOqCkyVt4CCs+2Z0aOG8icfwIQoLS8DsQXdf4BbCzE45Qs3Awcw wY3kPOVSqSFMn5qzWAGrZredyj0oz6e8gw3OBXbUHBZ5+O3EX94TDaYulPIIu+//Ia K3zOyC9CtcVCuDBVqI0WkE4hZ5BY9Fsdpah1ly+c/MPZ3n6d/Fqj/wVKNQr4Plo7th ASHSRdjANWOUINvjbOjS3Y1EdLXROx/XbYBPvCn7iRrtOOZ+WOOJ2DH+8gEWsV4pr2 IVWDmkkfQ8tWA== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 13/15] pciutils-pcilmr: Add option to save margining results in csv form Date: Wed, 27 Dec 2023 14:45:02 +0500 Message-ID: <20231227094504.32257-14-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lmr/lmr.h | 3 ++ lmr/margin_results.c | 111 +++++++++++++++++++++++++++++++++++++++++++ pcilmr.c | 18 ++++++- 3 files changed, 130 insertions(+), 2 deletions(-) diff --git a/lmr/lmr.h b/lmr/lmr.h index bb188fc..c9dcd69 100644 --- a/lmr/lmr.h +++ b/lmr/lmr.h @@ -219,4 +219,7 @@ void margin_log_hw_quirks(struct margin_recv *recv); void margin_results_print_brief(struct margin_results *results, u8 recvs_n); +void margin_results_save_csv(struct margin_results *results, u8 recvs_n, char *dir, + struct pci_dev *up_port); + #endif diff --git a/lmr/margin_results.c b/lmr/margin_results.c index 5ee065d..aca3ab7 100644 --- a/lmr/margin_results.c +++ b/lmr/margin_results.c @@ -157,3 +157,114 @@ margin_results_print_brief(struct margin_results *results, u8 recvs_n) printf("\n"); } } + +void +margin_results_save_csv(struct margin_results *results, u8 recvs_n, char *dir, + struct pci_dev *up_port) +{ + char timestamp[64]; + time_t tim = time(NULL); + strftime(timestamp, sizeof(timestamp), "%FT%H.%M.%S", gmtime(&tim)); + + size_t pathlen = strlen(dir) + 128; + char *path = xmalloc(pathlen); + FILE *csv; + + struct margin_res_lane *lane; + struct margin_results *res; + struct margin_params params; + + enum lane_rating lane_rating; + u8 link_speed; + + for (int i = 0; i < recvs_n; i++) + { + res = &(results[i]); + params = res->params; + link_speed = res->link_speed - 4; + + if (res->test_status != MARGIN_TEST_OK) + continue; + snprintf(path, pathlen, "%s/lmr_%0*x.%02x.%02x.%x_Rx%X_%s.csv", dir, + up_port->domain_16 == 0xffff ? 8 : 4, up_port->domain, up_port->bus, up_port->dev, + up_port->func, 10 + res->recvn - 1, timestamp); + csv = fopen(path, "w"); + if (!csv) + die("Error while saving %s\n", path); + + fprintf(csv, "Lane,Lane Status,Left %% UI,Left ps,Left Steps,Left Status," + "Right %% UI,Right ps,Right Steps,Right Status," + "Time %% UI,Time ps,Time Steps,Time Status," + "Up mV,Up Steps,Up Status,Down mV,Down Steps,Down Status," + "Voltage mV,Voltage Steps,Voltage Status\n"); + + if (check_recv_weird(res, MARGIN_TIM_MIN, MARGIN_VOLT_MIN)) + lane_rating = WEIRD; + else + lane_rating = INIT; + + for (int j = 0; j < res->lanes_n; j++) + { + lane = &(res->lanes[j]); + double left_ui = lane->steps[TIM_LEFT] * res->tim_coef; + double right_ui = lane->steps[TIM_RIGHT] * res->tim_coef; + double up_volt = lane->steps[VOLT_UP] * res->volt_coef; + double down_volt = lane->steps[VOLT_DOWN] * res->volt_coef; + + if (lane_rating != WEIRD) + { + lane_rating = rate_lane(left_ui, MARGIN_TIM_MIN, MARGIN_TIM_RECOMMEND, INIT); + if (params.ind_left_right_tim) + lane_rating + = rate_lane(right_ui, MARGIN_TIM_MIN, MARGIN_TIM_RECOMMEND, lane_rating); + if (params.volt_support) + { + lane_rating = rate_lane(up_volt, MARGIN_VOLT_MIN, MARGIN_VOLT_MIN, lane_rating); + if (params.ind_up_down_volt) + lane_rating + = rate_lane(down_volt, MARGIN_VOLT_MIN, MARGIN_VOLT_MIN, lane_rating); + } + } + + fprintf(csv, "%d,%s,", lane->lane, grades[lane_rating]); + if (params.ind_left_right_tim) + { + fprintf(csv, "%f,%f,%d,%s,%f,%f,%d,%s,NA,NA,NA,NA,", left_ui, + left_ui * ui[link_speed], lane->steps[TIM_LEFT], + sts_strings[lane->statuses[TIM_LEFT]], right_ui, right_ui * ui[link_speed], + lane->steps[TIM_RIGHT], sts_strings[lane->statuses[TIM_RIGHT]]); + } + else + { + for (int k = 0; k < 8; k++) + fprintf(csv, "NA,"); + fprintf(csv, "%f,%f,%d,%s,", left_ui, left_ui * ui[link_speed], lane->steps[TIM_LEFT], + sts_strings[lane->statuses[TIM_LEFT]]); + } + if (params.volt_support) + { + if (params.ind_up_down_volt) + { + fprintf(csv, "%f,%d,%s,%f,%d,%s,NA,NA,NA\n", up_volt, lane->steps[VOLT_UP], + sts_strings[lane->statuses[VOLT_UP]], down_volt, lane->steps[VOLT_DOWN], + sts_strings[lane->statuses[VOLT_DOWN]]); + } + else + { + for (int k = 0; k < 6; k++) + fprintf(csv, "NA,"); + fprintf(csv, "%f,%d,%s\n", up_volt, lane->steps[VOLT_UP], + sts_strings[lane->statuses[VOLT_UP]]); + } + } + else + { + for (int k = 0; k < 8; k++) + fprintf(csv, "NA,"); + fprintf(csv, "NA\n"); + } + } + fclose(csv); + } + free(path); +} diff --git a/pcilmr.c b/pcilmr.c index 43e791d..eb5d947 100644 --- a/pcilmr.c +++ b/pcilmr.c @@ -52,7 +52,12 @@ static const char usage_msg[] "-v \t\tSpecify maximum number of steps for Voltage Margining.\n" "Use only one of -T/-t options at the same time (same for -V/-v).\n" "Without these options utility will use MaxSteps from Device\n" - "capabilities as test limit.\n\n"; + "capabilities as test limit.\n\n" + "Margining Log settings:\n" + "-o \t\tSave margining results in csv form into the\n" + "\t\t\tspecified directory. Utility will generate file with the\n" + "\t\t\tname in form of 'lmr__Rx#_.csv'\n" + "\t\t\tfor each successfully tested receiver.\n"; static struct pci_dev * dev_for_filter(struct pci_access *pacc, char *filter) @@ -197,6 +202,9 @@ main(int argc, char **argv) bool run_margin = true; + char *dir_for_csv = NULL; + bool save_csv = false; + u64 total_steps = 0; pacc = pci_alloc(); @@ -246,7 +254,7 @@ main(int argc, char **argv) break; } - while ((c = getopt(argc, argv, ":r:e:l:cp:t:v:VT")) != -1) + while ((c = getopt(argc, argv, ":r:e:l:cp:t:v:VTo:")) != -1) { switch (c) { @@ -277,6 +285,10 @@ main(int argc, char **argv) case 'r': recvs_n = parse_csv_arg(optarg, recvs_arg); break; + case 'o': + dir_for_csv = optarg; + save_csv = true; + break; default: die("Invalid arguments\n\n%s", usage_msg); } @@ -449,6 +461,8 @@ main(int argc, char **argv) margin_log_bdfs(down_ports[i], up_ports[i]); printf(":\n\n"); margin_results_print_brief(results[i], results_n[i]); + if (save_csv) + margin_results_save_csv(results[i], results_n[i], dir_for_csv, up_ports[i]); printf("\n"); } } From patchwork Wed Dec 27 09:45:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505265 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DF142869B for ; Wed, 27 Dec 2023 09:46:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="wPU8GMx1"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="uGLdwg8X" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com ECFAAC000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670383; bh=fawPkXjGzoh/XKSOhgqVt8rUb5W7gXYgFRqlnvPfLL8=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=wPU8GMx1ShhCiOKpKXoj3GAPIMXgXBkJafBxcKfIeCPPGdGf5pRtqlBaJp/PrRHw3 sCuL8SbNnncB3FxaJeWHR/PLaUuaJh74kPNv8UNJS+4Nz1gXZ7LApHEH27aRMsl6NB h+QoVBgwTNptiVJYkLxSiUrJMGkZMqScCNovjw+1q6NdkyuF+CV0+b0XdVeTKfTXAM eBRQUCiz9jT4hGK+fvAHprj6t6X4mugzF0fTHrg07TNnknBy7DtpLiTae8fpAjTfFH pcf6OTY01TudqKRkbaYPz3MAdHiK5PFoZ4b1ga0FOV0iL/dzWfqU4ywCoUBIMLlYOt Z2rbIWmR3m4Yw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670383; bh=fawPkXjGzoh/XKSOhgqVt8rUb5W7gXYgFRqlnvPfLL8=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=uGLdwg8X4KP+uwAOuAQkAIkisaCWXOZ8sMJ+yw32+cTvInEtuIB6h7TKJyVbwFK0i d2VkXCpQDXrQqX12inh2fsyDVtnfHoZh8K1khUTAJgw+9Ivlm7aP10+mtN8MJXjPoK S6M+or3ZjnGQEeQCWCkwocwvgq7LdUjNqQb2iU90zHey/ZcWUI0GsYAKTc+GJrmasU EsEDNlDBltkXDVZeREnu9QocHRN9rMJiRps+4ZVGijPCD1vc8YEKp8sxONPKThS1GP YRvoEAbpU+HuwFphQo/rhnxdanIUayNUJJfkREgisJE9f+2y0gF/379BlRsuFofw5u pVDLY0Ryug+Jg== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 14/15] pciutils-pcilmr: Add handling of situations when device reports its MaxOffset values equal to 0 Date: Wed, 27 Dec 2023 14:45:03 +0500 Message-ID: <20231227094504.32257-15-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) According to spec, for the MaxTimingOffset and MaxVoltageOffset parameters 'A 0 value may be reported if the vendor chooses not to report the offset'. Use max possible Offset value in such situations and report to the user. Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- lmr/lmr.h | 3 +++ lmr/margin.c | 9 +++++++-- lmr/margin_log.c | 7 +++++++ lmr/margin_results.c | 13 +++++++++++++ 4 files changed, 30 insertions(+), 2 deletions(-) diff --git a/lmr/lmr.h b/lmr/lmr.h index c9dcd69..7375c33 100644 --- a/lmr/lmr.h +++ b/lmr/lmr.h @@ -107,6 +107,9 @@ struct margin_results { double tim_coef; double volt_coef; + bool tim_off_reported; + bool volt_off_reported; + u8 lanes_n; struct margin_res_lane *lanes; }; diff --git a/lmr/margin.c b/lmr/margin.c index cc142fa..0b14747 100644 --- a/lmr/margin.c +++ b/lmr/margin.c @@ -328,8 +328,13 @@ margin_test_receiver(struct margin_dev *dev, u8 recvn, struct margin_args *args, margin_apply_hw_quirks(&recv); margin_log_hw_quirks(&recv); - results->tim_coef = (double)params.timing_offset / (double)params.timing_steps; - results->volt_coef = (double)params.volt_offset / (double)params.volt_steps * 10.0; + results->tim_off_reported = params.timing_offset != 0; + results->volt_off_reported = params.volt_offset != 0; + double tim_offset = results->tim_off_reported ? (double)params.timing_offset : 50.0; + double volt_offset = results->volt_off_reported ? (double)params.volt_offset : 50.0; + + results->tim_coef = tim_offset / (double)params.timing_steps; + results->volt_coef = volt_offset / (double)params.volt_steps * 10.0; results->lane_reversal = recv.lane_reversal; results->link_speed = dev->link_speed; diff --git a/lmr/margin_log.c b/lmr/margin_log.c index a93e07b..b3c4bd5 100644 --- a/lmr/margin_log.c +++ b/lmr/margin_log.c @@ -86,6 +86,13 @@ margin_log_receiver(struct margin_recv *recv) margin_log("\nWarning: device uses Lane Reversal.\n"); margin_log("However, utility uses logical lane numbers in arguments and for logging.\n"); } + + if (recv->params->timing_offset == 0) + margin_log("\nWarning: Vendor chose not to report the Max Timing Offset.\n" + "Utility will use its max possible value - 50 (50%% UI).\n"); + if (recv->params->volt_support && recv->params->volt_offset == 0) + margin_log("\nWarning: Vendor chose not to report the Max Voltage Offset.\n" + "Utility will use its max possible value - 50 (500 mV).\n"); } void diff --git a/lmr/margin_results.c b/lmr/margin_results.c index aca3ab7..6d6ed29 100644 --- a/lmr/margin_results.c +++ b/lmr/margin_results.c @@ -105,6 +105,19 @@ margin_results_print_brief(struct margin_results *results, u8 recvs_n) if (res->lane_reversal) printf("Rx(%X) - Lane Reversal\n", 10 + res->recvn - 1); + if (!res->tim_off_reported) + printf("Rx(%X) - Attention: Vendor chose not to report the Max Timing Offset.\n" + "Utility used its max possible value (50%% UI) for calculations of %% UI and ps.\n" + "Keep in mind that for timing results of this receiver only steps values are " + "reliable.\n\n", + 10 + res->recvn - 1); + if (params.volt_support && !res->volt_off_reported) + printf("Rx(%X) - Attention: Vendor chose not to report the Max Voltage Offset.\n" + "Utility used its max possible value (500 mV) for calculations of mV.\n" + "Keep in mind that for voltage results of this receiver only steps values are " + "reliable.\n\n", + 10 + res->recvn - 1); + if (check_recv_weird(res, MARGIN_TIM_MIN, MARGIN_VOLT_MIN)) lane_rating = WEIRD; else From patchwork Wed Dec 27 09:45:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Proshkin X-Patchwork-Id: 13505266 Received: from mta-04.yadro.com (mta-04.yadro.com [89.207.88.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49F87286B5 for ; Wed, 27 Dec 2023 09:46:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=yadro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=yadro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="VTcXYydI"; dkim=pass (2048-bit key) header.d=yadro.com header.i=@yadro.com header.b="wLFHtH0Y" DKIM-Filter: OpenDKIM Filter v2.11.0 mta-04.yadro.com AC3C1C000A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-04; t=1703670385; bh=6SEouK9QkJobnpLhSQ6RU21HAzZB984JfrkiMala3A4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=VTcXYydIYVltk8gLGny0sYeAdIRSni3h+YNxEpzur4oMBMS9UAPJLgWWs0vCxb12g BuX4PssUusUvBjwuRTMr4Pn4HqLC8fMZ1uaIJgEHtrS9SjnXfKxJE8D0qjDsxsxrO7 NlfQn86dg/X5eqV+cu/Wo9ja068yIfMswgQ+PeFOFzpr7Qj1wvDn6F7dFovaYvfz2n WtwGMn4ZzBrjiQDSROixVO8vUltL1NBaRAEPc3+VBeDye0TA3dw3ehslUDIBijYvuH U6Vyl9X1FqkR5Jn2i+1ZGbGTbd3UiQnRA960GbLDwDy3gIMga2+dE/cArsm046qwRT UA2om72+wc+dA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yadro.com; s=mta-03; t=1703670385; bh=6SEouK9QkJobnpLhSQ6RU21HAzZB984JfrkiMala3A4=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=wLFHtH0YdwyuV1HGjagJfOOixHRxNvpxwAjo+JOX/tUYjrZN7TTY4B7rTAsZIPNRG CFukz4qaDOHfc19JaGk7EJ6LgwSCNaJNyfY9BeRV4lFNt2rwNcLdd+jFmkqNol7Hgx 5SkJy0RpK1sP8Ufb0nvEN8UqTkqCuX02pYU7ClrBBkrtmIqcKPzAZ7QU7kNgChu/7O IFbmb/E5oFharHJGdYcqNqFVK9o/7ErA4KVA00zSrQBf9X6AZoO+Bp5Hgt/0OBUoI1 KlalAgybbzSfl4aGwxeb7GgpvyIdYcQiSmypcyO5ZtYwxCN4JZUGZMqgBKU0zDUi30 XNZqJMvrdGjMQ== From: Nikita Proshkin To: , Martin Mares CC: , Bjorn Helgaas , Sergei Miroshnichenko , Nikita Proshkin Subject: [PATCH v2 15/15] pciutils-pcilmr: Add pcilmr man page Date: Wed, 27 Dec 2023 14:45:04 +0500 Message-ID: <20231227094504.32257-16-n.proshkin@yadro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231227094504.32257-1-n.proshkin@yadro.com> References: <20231227094504.32257-1-n.proshkin@yadro.com> Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: T-EXCH-09.corp.yadro.com (172.17.11.59) To T-EXCH-08.corp.yadro.com (172.17.11.58) Reviewed-by: Sergei Miroshnichenko Signed-off-by: Nikita Proshkin --- Makefile | 2 +- pcilmr.c | 3 +- pcilmr.man | 182 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 185 insertions(+), 2 deletions(-) create mode 100644 pcilmr.man diff --git a/Makefile b/Makefile index ee8e8ee..cb002cb 100644 --- a/Makefile +++ b/Makefile @@ -71,7 +71,7 @@ LMRINC=lmr/lmr.h export -all: lib/$(PCIIMPLIB) lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lspci.8 setpci.8 pcilib.7 pci.ids.5 update-pciids update-pciids.8 $(PCI_IDS) pcilmr +all: lib/$(PCIIMPLIB) lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lspci.8 setpci.8 pcilib.7 pci.ids.5 update-pciids update-pciids.8 $(PCI_IDS) pcilmr pcilmr.8 lib/$(PCIIMPLIB): $(PCIINC) force $(MAKE) -C lib all diff --git a/pcilmr.c b/pcilmr.c index eb5d947..bab2a07 100644 --- a/pcilmr.c +++ b/pcilmr.c @@ -20,7 +20,8 @@ const char program_name[] = "pcilmr"; enum mode { MARGIN, FULL, SCAN }; static const char usage_msg[] - = "Usage:\n" + = "! Utility requires preliminary preparation of the system. Refer to the pcilmr man page !\n\n" + "Usage:\n" "pcilmr [--margin] [] ...\n" "pcilmr --full []\n" "pcilmr --scan\n\n" diff --git a/pcilmr.man b/pcilmr.man new file mode 100644 index 0000000..673262f --- /dev/null +++ b/pcilmr.man @@ -0,0 +1,182 @@ +.TH PCILMR 8 "@TODAY@" "@VERSION@" "The PCI Utilities" +.SH NAME +pcilmr \- margin PCIe Links +.SH SYNOPSIS +.B pcilmr +.RB [ "--margin" ] +.RI [ "" ] " ..." +.br +.B pcilmr --full +.RI [ "" ] +.br +.B pcilmr --scan +.SH CONFIGURATION +List of the requirements for links and system settings +to run the margining test. + +.B BIOS settings +(depends on the system, relevant for server baseboards +with Xeon CPUs): +.IP \[bu] 3 +Turn off PCIe Leaky Bucket Feature, Re-Equalization and Link Degradation; +.IP \[bu] +Set Error Thresholds to 0; +.IP \[bu] +Intel VMD for NVMe SSDs - in case of strange behavior of the +.BR pcilmr, +try to run it with the VMD turned off. +.PP +.B Device (link) requirements: +.IP +.I "Configured by the user before running the utility, the utility does not change them:" +.RS +.IP \[bu] 3 +The current Link data rate must be 16.0 GT/s or higher (right now +utility supports 16 GT/s and 32 GT/s Links); +.IP \[bu] +Link Downstream Component must be at D0 Power Management State. +.RE +.IP +.I "Configured by the utility during operation, utility set them to their original " +.I "state after receiving the results:" +.RS +.IP \[bu] 3 +The ASPM must be disabled in both the Downstream Port and Upstream Port; +.IP \[bu] +The Hardware Autonomous Speed Disable bit of the Link Control 2 register must be Set in both the +Downstream Port and Upstream Port; +.IP \[bu] +The Hardware Autonomous Width Disable bit of the Link Control register must be Set in both the +Downstream Port and Upstream Port. +.SH DESCRIPTION +.B pcilmr +utility allows you to take advantage of the PCIe Lane Margining at the Receiver +capability which is mandatory for all Ports supporting a data rate of 16.0 GT/s or +higher, including Pseudo Ports (Retimers). Lane Margining at Receiver enables system +software to obtain the margin information of a given Receiver while the Link is in the +L0 state. The margin information includes both voltage and time, in either direction from +the current Receiver position. Margining support for timing is required, while support +for voltage is optional at 16.0 GT/s and required at 32.0 GT/s and higher data rates. Also, +independent time margining and independent voltage margining is optional. + +Utility allows to get an approximation of the eye margin diagram in the form of a rhombus +(by four points). Lane Margining at the Receiver capability enables users to margin PCIe +links without a hardware debugger and without the need to stop the target system. Utility +can be useful to debug link issues due to receiver margins. + +However, the utility results may be not particularly accurate and, as it was found out during +testing, specific devices provide rather dubious capability support and the reliability of +the information they provide is questionable. The PCIe specification provides reference values +for the eye diagram, which are also used by the +.B pcilmr +to evaluate the results, but it seems that it makes sense to contact the +manufacturer of a particular device for references. + +The PCIe Base Specification Revision 5.0 sets allowed range for Timing Margin from 20%\~UI to 50%\~UI and +for Voltage Margin from 50\~mV to 500\~mV. Utility uses 30%\~UI as the recommended +value for Timing - taken from NVIDIA presentation ("PCIe 4.0 Mass Electrical Margins Data +Collection"). + +.B pcilmr +requires root privileges (to access Extended Configuration Space), but during our testing +there were no problems with the devices and they successfully returned to their normal initial +state after the end of testing. + +.SH OPTIONS +.SS Device Specifier +.B "" \t +.RI [ "" :] : . +(see +.BR lspci (8)) +.SS Utility Modes +.TP +.BI --margin " ..." +Margin selected Links. +.TP +.B --full +Margin all ready for testing (in a meaning similar to the +.B --scan +option) Links in the system (one by one). +.TP +.B --scan +Scan for Links with negotiated speed 16 GT/s or higher. Mark "Ready" those of them +in which at least one of the Link sides have Margining Ready bit set meaning that +these Links are ready for testing and you can run utility on them. +.SS Margining Test options +.TP +.B -c +Print Device Lane Margining Capabilities only. Do not run margining. +.TP +\fB\-l\fI \fP[\fI,...\fP] +Specify lanes for margining. +.br +Remember that Device may use Lane Reversal for Lane numbering. However, utility +uses logical lane numbers in arguments and for logging. Utility will automatically +determine Lane Reversal and tune its calls. +.br +Default: all link lanes. +.TP +.BI -e " " +Specify Error Count Limit for margining. +.br +Default: 4. +.TP +\fB-r\fI \fP[\fI,...\fP] +Specify Receivers to select margining targets. +.br +Default: all available Receivers (including Retimers). +.TP +.BI -p " " +Specify number of lanes to margin simultaneously. +.br +According to spec it's possible for Receiver to margin up to MaxLanes + 1 +lanes simultaneously, but during testing, performing margining on several +lanes simultaneously led to results that were different from sequential +margining, so this feature requires additional verification and +.I -p +option right now is for experiments mostly. +.br +Default: 1. +.PP +.B "Use only one of -T/-t options at the same time (same for -V/-v)." +.br +.B "Without these options utility will use MaxSteps from Device" +.B "capabilities as test limit." +.TP +.B -T +Time Margining will continue until the Error Count is no more +than an Error Count Limit. Use this option to find Link limit. +.TP +.BI -t " " +Specify maximum number of steps for Time Margining. +.TP +.B -V +Same as +.I -T +option, but for Voltage. +.TP +.BI -v " " +Specify maximum number of steps for Voltage Margining. +.SS Margining Log options +.TP +.BI -o " " +Save margining results in csv form into the specified directory. Utility +will generate file with the name in form of +.RI "\[dq]lmr_" "" "_Rx" # _ ".csv\[dq]" +for each successfully tested receiver. + +.SH EXAMPLES +Utility syntax example: +.RS +.BI "pcilmr -l" " 0,1 " "-r" " 1,6 " "-TV" " ab:0.0 52:0.0" +.RE + +.UR https://gist.github.com/bombanya/f2b15263712757ffba1a11eea011c419 +Examples of collected results on different systems. +.UE + +.SH SEE ALSO +.nh +.BR lspci (8), +.B PCI Express Base Specification (Lane Margining at Receiver) +.hy