@@ -71,7 +71,7 @@ LMRINC=lmr/lmr.h
export
-all: lib/$(PCIIMPLIB) lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lspci.8 setpci.8 pcilib.7 pci.ids.5 update-pciids update-pciids.8 $(PCI_IDS) pcilmr
+all: lib/$(PCIIMPLIB) lspci$(EXEEXT) setpci$(EXEEXT) example$(EXEEXT) lspci.8 setpci.8 pcilib.7 pci.ids.5 update-pciids update-pciids.8 $(PCI_IDS) pcilmr pcilmr.8
lib/$(PCIIMPLIB): $(PCIINC) force
$(MAKE) -C lib all
@@ -20,7 +20,8 @@ const char program_name[] = "pcilmr";
enum mode { MARGIN, FULL, SCAN };
static const char usage_msg[]
- = "Usage:\n"
+ = "! Utility requires preliminary preparation of the system. Refer to the pcilmr man page !\n\n"
+ "Usage:\n"
"pcilmr [--margin] [<margining options>] <downstream component> ...\n"
"pcilmr --full [<margining options>]\n"
"pcilmr --scan\n\n"
new file mode 100644
@@ -0,0 +1,182 @@
+.TH PCILMR 8 "@TODAY@" "@VERSION@" "The PCI Utilities"
+.SH NAME
+pcilmr \- margin PCIe Links
+.SH SYNOPSIS
+.B pcilmr
+.RB [ "--margin" ]
+.RI [ "<margining options>" ] " <downstream component> ..."
+.br
+.B pcilmr --full
+.RI [ "<margining options>" ]
+.br
+.B pcilmr --scan
+.SH CONFIGURATION
+List of the requirements for links and system settings
+to run the margining test.
+
+.B BIOS settings
+(depends on the system, relevant for server baseboards
+with Xeon CPUs):
+.IP \[bu] 3
+Turn off PCIe Leaky Bucket Feature, Re-Equalization and Link Degradation;
+.IP \[bu]
+Set Error Thresholds to 0;
+.IP \[bu]
+Intel VMD for NVMe SSDs - in case of strange behavior of the
+.BR pcilmr,
+try to run it with the VMD turned off.
+.PP
+.B Device (link) requirements:
+.IP
+.I "Configured by the user before running the utility, the utility does not change them:"
+.RS
+.IP \[bu] 3
+The current Link data rate must be 16.0 GT/s or higher (right now
+utility supports 16 GT/s and 32 GT/s Links);
+.IP \[bu]
+Link Downstream Component must be at D0 Power Management State.
+.RE
+.IP
+.I "Configured by the utility during operation, utility set them to their original "
+.I "state after receiving the results:"
+.RS
+.IP \[bu] 3
+The ASPM must be disabled in both the Downstream Port and Upstream Port;
+.IP \[bu]
+The Hardware Autonomous Speed Disable bit of the Link Control 2 register must be Set in both the
+Downstream Port and Upstream Port;
+.IP \[bu]
+The Hardware Autonomous Width Disable bit of the Link Control register must be Set in both the
+Downstream Port and Upstream Port.
+.SH DESCRIPTION
+.B pcilmr
+utility allows you to take advantage of the PCIe Lane Margining at the Receiver
+capability which is mandatory for all Ports supporting a data rate of 16.0 GT/s or
+higher, including Pseudo Ports (Retimers). Lane Margining at Receiver enables system
+software to obtain the margin information of a given Receiver while the Link is in the
+L0 state. The margin information includes both voltage and time, in either direction from
+the current Receiver position. Margining support for timing is required, while support
+for voltage is optional at 16.0 GT/s and required at 32.0 GT/s and higher data rates. Also,
+independent time margining and independent voltage margining is optional.
+
+Utility allows to get an approximation of the eye margin diagram in the form of a rhombus
+(by four points). Lane Margining at the Receiver capability enables users to margin PCIe
+links without a hardware debugger and without the need to stop the target system. Utility
+can be useful to debug link issues due to receiver margins.
+
+However, the utility results may be not particularly accurate and, as it was found out during
+testing, specific devices provide rather dubious capability support and the reliability of
+the information they provide is questionable. The PCIe specification provides reference values
+for the eye diagram, which are also used by the
+.B pcilmr
+to evaluate the results, but it seems that it makes sense to contact the
+manufacturer of a particular device for references.
+
+The PCIe Base Specification Revision 5.0 sets allowed range for Timing Margin from 20%\~UI to 50%\~UI and
+for Voltage Margin from 50\~mV to 500\~mV. Utility uses 30%\~UI as the recommended
+value for Timing - taken from NVIDIA presentation ("PCIe 4.0 Mass Electrical Margins Data
+Collection").
+
+.B pcilmr
+requires root privileges (to access Extended Configuration Space), but during our testing
+there were no problems with the devices and they successfully returned to their normal initial
+state after the end of testing.
+
+.SH OPTIONS
+.SS Device Specifier
+.B "<device/component>" \t
+.RI [ "<domain>" :] <bus> : <dev> . <func>
+(see
+.BR lspci (8))
+.SS Utility Modes
+.TP
+.BI --margin " <downstream component> ..."
+Margin selected Links.
+.TP
+.B --full
+Margin all ready for testing (in a meaning similar to the
+.B --scan
+option) Links in the system (one by one).
+.TP
+.B --scan
+Scan for Links with negotiated speed 16 GT/s or higher. Mark "Ready" those of them
+in which at least one of the Link sides have Margining Ready bit set meaning that
+these Links are ready for testing and you can run utility on them.
+.SS Margining Test options
+.TP
+.B -c
+Print Device Lane Margining Capabilities only. Do not run margining.
+.TP
+\fB\-l\fI <lane>\fP[\fI,<lane>...\fP]
+Specify lanes for margining.
+.br
+Remember that Device may use Lane Reversal for Lane numbering. However, utility
+uses logical lane numbers in arguments and for logging. Utility will automatically
+determine Lane Reversal and tune its calls.
+.br
+Default: all link lanes.
+.TP
+.BI -e " <errors>"
+Specify Error Count Limit for margining.
+.br
+Default: 4.
+.TP
+\fB-r\fI <recvn>\fP[\fI,<recvn>...\fP]
+Specify Receivers to select margining targets.
+.br
+Default: all available Receivers (including Retimers).
+.TP
+.BI -p " <parallel_lanes>"
+Specify number of lanes to margin simultaneously.
+.br
+According to spec it's possible for Receiver to margin up to MaxLanes + 1
+lanes simultaneously, but during testing, performing margining on several
+lanes simultaneously led to results that were different from sequential
+margining, so this feature requires additional verification and
+.I -p
+option right now is for experiments mostly.
+.br
+Default: 1.
+.PP
+.B "Use only one of -T/-t options at the same time (same for -V/-v)."
+.br
+.B "Without these options utility will use MaxSteps from Device"
+.B "capabilities as test limit."
+.TP
+.B -T
+Time Margining will continue until the Error Count is no more
+than an Error Count Limit. Use this option to find Link limit.
+.TP
+.BI -t " <steps>"
+Specify maximum number of steps for Time Margining.
+.TP
+.B -V
+Same as
+.I -T
+option, but for Voltage.
+.TP
+.BI -v " <steps>"
+Specify maximum number of steps for Voltage Margining.
+.SS Margining Log options
+.TP
+.BI -o " <directory>"
+Save margining results in csv form into the specified directory. Utility
+will generate file with the name in form of
+.RI "\[dq]lmr_" "<downstream component>" "_Rx" # _ <timestamp> ".csv\[dq]"
+for each successfully tested receiver.
+
+.SH EXAMPLES
+Utility syntax example:
+.RS
+.BI "pcilmr -l" " 0,1 " "-r" " 1,6 " "-TV" " ab:0.0 52:0.0"
+.RE
+
+.UR https://gist.github.com/bombanya/f2b15263712757ffba1a11eea011c419
+Examples of collected results on different systems.
+.UE
+
+.SH SEE ALSO
+.nh
+.BR lspci (8),
+.B PCI Express Base Specification (Lane Margining at Receiver)
+.hy