| Message ID | cover.1631957565.git.mchehab+huawei@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | get_abi.pl: Check for missing symbols at the ABI specs | expand |
On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote: > Hi Greg, > > Add a new feature at get_abi.pl to optionally check for existing symbols > under /sys that won't match a "What:" inside Documentation/ABI. > > Such feature is very useful to detect missing documentation for ABI. > > This series brings a major speedup, plus it fixes a few border cases when > matching regexes that end with a ".*" or \d+. > > patch 1 changes get_abi.pl logic to handle multiple What: lines, in > order to make the script more robust; > > patch 2 adds the basic logic. It runs really quicky (up to 2 > seconds), but it doesn't use sysfs softlinks. > > Patch 3 adds support for parsing softlinks. It makes the script a > lot slower, making it take a couple of minutes to process the entire > sysfs files. It could be optimized in the future by using a graph, > but, for now, let's keep it simple. > > Patch 4 adds an optional parameter to allow filtering the results > using a regex given by the user. When this parameter is used > (which should be the normal usecase), it will only try to find softlinks > if the sysfs node matches a regex. > > Patch 5 improves the report by avoiding it to ignore What: that > ends with a wildcard. > > Patch 6 is a minor speedup. On a Dell Precision 5820, after patch 6, > results are: > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > real 2m35.563s > user 2m34.346s > sys 0m1.220s > 7595 undefined > 896 undefined_symbols > > Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3) > search for links by a logic which handle symlinks using BFS. It > also addresses a border case that was making 'msi-irqs/\d+' regex to > be misparsed. > > After patch 7, it is 11 times faster: > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > real 0m14.137s > user 0m12.795s > sys 0m1.348s > 7030 undefined > 794 undefined_symbols > > (the difference on the number of undefined symbols are due to the fix for > it to properly handle 'msi-irqs/\d+' regex) > > - > > While this series is independent from Documentation/ABI changes, it > works best when applied from this tree, which also contain ABI fixes > and a couple of additions of frequent missed symbols on my machine: > > https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3 I've taken all of these, but get_abi.pl seems to be stuck in an endless loop or something. I gave up and stopped it after 14 minutes. It had stopped printing out anything after finding all of the pci attributes that are not documented :) Anything I can do to help debug this? thanks, greg k-h
Em Tue, 21 Sep 2021 18:52:42 +0200 Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu: > On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote: > > Hi Greg, > > > > Add a new feature at get_abi.pl to optionally check for existing symbols > > under /sys that won't match a "What:" inside Documentation/ABI. > > > > Such feature is very useful to detect missing documentation for ABI. > > > > This series brings a major speedup, plus it fixes a few border cases when > > matching regexes that end with a ".*" or \d+. > > > > patch 1 changes get_abi.pl logic to handle multiple What: lines, in > > order to make the script more robust; > > > > patch 2 adds the basic logic. It runs really quicky (up to 2 > > seconds), but it doesn't use sysfs softlinks. > > > > Patch 3 adds support for parsing softlinks. It makes the script a > > lot slower, making it take a couple of minutes to process the entire > > sysfs files. It could be optimized in the future by using a graph, > > but, for now, let's keep it simple. > > > > Patch 4 adds an optional parameter to allow filtering the results > > using a regex given by the user. When this parameter is used > > (which should be the normal usecase), it will only try to find softlinks > > if the sysfs node matches a regex. > > > > Patch 5 improves the report by avoiding it to ignore What: that > > ends with a wildcard. > > > > Patch 6 is a minor speedup. On a Dell Precision 5820, after patch 6, > > results are: > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > real 2m35.563s > > user 2m34.346s > > sys 0m1.220s > > 7595 undefined > > 896 undefined_symbols > > > > Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3) > > search for links by a logic which handle symlinks using BFS. It > > also addresses a border case that was making 'msi-irqs/\d+' regex to > > be misparsed. > > > > After patch 7, it is 11 times faster: > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > real 0m14.137s > > user 0m12.795s > > sys 0m1.348s > > 7030 undefined > > 794 undefined_symbols > > > > (the difference on the number of undefined symbols are due to the fix for > > it to properly handle 'msi-irqs/\d+' regex) > > > > - > > > > While this series is independent from Documentation/ABI changes, it > > works best when applied from this tree, which also contain ABI fixes > > and a couple of additions of frequent missed symbols on my machine: > > > > https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3 > > I've taken all of these, but get_abi.pl seems to be stuck in an endless > loop or something. I gave up and stopped it after 14 minutes. It had > stopped printing out anything after finding all of the pci attributes > that are not documented :) It is probably not an endless loop, just there are too many vars to check on your system, which could make it really slow. The way the search algorithm works is that reduces the number of regex expressions that will be checked for a given file entry at sysfs. It does that by looking at the devnode name. For instance, when it checks for this file: /sys/bus/pci/drivers/iosf_mbi_pci/bind The logic will seek only the "What:" expressions that end with "bind". Currently, there are just two What expressions for it[1]: What: /sys/bus/fsl\-mc/drivers/.*/bind What: /sys/bus/pci/drivers/.*/bind It will then run an O(n²) algorithm to seek: foreach my $a (@names) { foreach my $w (split /\xac/, $what) { if ($a =~ m#^$w$#) { exact = 1; last; } } } Which runs quickly, when there are few regexs to seek. There are, however, some What: expressions that end with a wildcard. Those are harder to process. Right now, they're all grouped together, which makes them slower. Most of the processing time are spent on those. I'm working right now on some strategy to also speed up the search for them. Once I get something better, I'll send a patch series. -- [1] On a side note, there are currently some problems with the What: definitions for bind/unbind, as: - it doesn't match all PCI devices; - it doesn't match ACPI and other buses that also export bind/unbind. > > Anything I can do to help debug this? > There are two parameters that can help to identify the issue: a) You can add a "--show-hints" parameter. This turns on some prints that may help to identify what the script is doing. It is not really a debug option, but it helps to identify when some regexes are failing. b) You can limit the What expressions that will be parsed with: --search-string <something> You can combine both. For instance, if you want to make it a lot more verbose, you could run it as: ./scripts/get_abi.pl undefined --search-string /sys --show-hints The script will then print all regexes that will be checked, and when actually checking for the missing vars, it will print all names for a given entry at sysfs. So, if you want to know how an i2c bind has been validated, you could do: $ ./scripts/get_abi.pl undefined --search-string i2c/.*/bind --show-hints --> /sys/bus/i2c/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/driver/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/driver/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/dummy/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/dummy/bind more likely regexes: /sys/bus/fsl\-mc/drivers/.*/bind /sys/bus/pci/drivers/.*/bind --> /sys/bus/i2c/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/axp20x-i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/axp20x-i2c/bind more likely regexes: /sys/bus/fsl\-mc/drivers/.*/bind /sys/bus/pci/drivers/.*/bind --> /sys/bus/i2c/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/smbus_alert/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/smbus_alert/bind --> /sys/module/i2c_smbus/drivers/i2c:smbus_alert/bind more likely regexes: /sys/bus/fsl\-mc/drivers/.*/bind /sys/bus/pci/drivers/.*/bind --> /sys/bus/i2c/drivers/ee1004/bind --> /sys/module/ee1004/drivers/i2c:ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/ee1004/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/driver/bind more likely regexes: /sys/bus/fsl\-mc/drivers/.*/bind /sys/bus/pci/drivers/.*/bind --> /sys/bus/i2c/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/intel_soc_pmic_i2c/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/intel_soc_pmic_i2c/bind more likely regexes: /sys/bus/fsl\-mc/drivers/.*/bind /sys/bus/pci/drivers/.*/bind --> /sys/bus/i2c/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/tps68470/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/tps68470/bind more likely regexes: /sys/bus/fsl\-mc/drivers/.*/bind /sys/bus/pci/drivers/.*/bind --> /sys/bus/i2c/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-3/i2c-14/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-4/i2c-15/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0036/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-2/i2c-13/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0050/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-10/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-5/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-3/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:15.1/i2c_designware.1/i2c-1/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/drm/card1/card1-DP-1/i2c-12/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/16-0037/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-8/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-9/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:15.0/i2c_designware.0/i2c-0/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:1f.4/i2c-16/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-7/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-6/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/CHT Whiskey Cove PMIC/bind --> /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/i2c-11/subsystem/drivers/CHT Whiskey Cove PMIC/bind more likely regexes: /sys/bus/fsl\-mc/drivers/.*/bind /sys/bus/pci/drivers/.*/bind Btw, on the above example, I have already a patch addressing it (see enclosed). I intend to submit it on a newer patch series. Thanks, Mauro [PATCH] ABI: sysfs-bus-pci: add a alternative What fields There are some PCI ABI that aren't shown under: /sys/bus/pci/drivers/.../ Because they're registered with a different class. That's the case of, for instance: /sys/bus/i2c/drivers/CHT Whiskey Cove PMIC/unbind This one is not present under /sys/bus/pci: $ find /sys/bus/pci -name 'CHT Whiskey Cove PMIC' Although clearly this is provided by a PCI driver: /sys/devices/pci0000:00/0000:00:02.0/i2c-4/subsystem/drivers/CHT Whiskey Cove PMIC/unbind So, add an altertate What location in order to match bind/unbind to such devices. Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org> diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci index 1da4c8db3a9e..f4efbcb0b18c 100644 --- a/Documentation/ABI/testing/sysfs-bus-pci +++ b/Documentation/ABI/testing/sysfs-bus-pci @@ -1,4 +1,5 @@ What: /sys/bus/pci/drivers/.../bind +What: /sys/devices/pciX/.../bind Date: December 2003 Contact: linux-pci@vger.kernel.org Description: @@ -14,6 +15,7 @@ Description: (Note: kernels before 2.6.28 may require echo -n). What: /sys/bus/pci/drivers/.../unbind +What: /sys/devices/pciX/.../unbind Date: December 2003 Contact: linux-pci@vger.kernel.org Description: @@ -29,6 +31,7 @@ Description: (Note: kernels before 2.6.28 may require echo -n). What: /sys/bus/pci/drivers/.../new_id +What: /sys/devices/pciX/.../new_id Date: December 2003 Contact: linux-pci@vger.kernel.org Description: @@ -47,6 +50,7 @@ Description: # echo "8086 10f5" > /sys/bus/pci/drivers/foo/new_id What: /sys/bus/pci/drivers/.../remove_id +What: /sys/devices/pciX/.../remove_id Date: February 2009 Contact: Chris Wright <chrisw@sous-sol.org> Description:
On Tue, Sep 21, 2021 at 08:16:33PM +0200, Mauro Carvalho Chehab wrote: > Em Tue, 21 Sep 2021 18:52:42 +0200 > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu: > > > On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote: > > > Hi Greg, > > > > > > Add a new feature at get_abi.pl to optionally check for existing symbols > > > under /sys that won't match a "What:" inside Documentation/ABI. > > > > > > Such feature is very useful to detect missing documentation for ABI. > > > > > > This series brings a major speedup, plus it fixes a few border cases when > > > matching regexes that end with a ".*" or \d+. > > > > > > patch 1 changes get_abi.pl logic to handle multiple What: lines, in > > > order to make the script more robust; > > > > > > patch 2 adds the basic logic. It runs really quicky (up to 2 > > > seconds), but it doesn't use sysfs softlinks. > > > > > > Patch 3 adds support for parsing softlinks. It makes the script a > > > lot slower, making it take a couple of minutes to process the entire > > > sysfs files. It could be optimized in the future by using a graph, > > > but, for now, let's keep it simple. > > > > > > Patch 4 adds an optional parameter to allow filtering the results > > > using a regex given by the user. When this parameter is used > > > (which should be the normal usecase), it will only try to find softlinks > > > if the sysfs node matches a regex. > > > > > > Patch 5 improves the report by avoiding it to ignore What: that > > > ends with a wildcard. > > > > > > Patch 6 is a minor speedup. On a Dell Precision 5820, after patch 6, > > > results are: > > > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > > > real 2m35.563s > > > user 2m34.346s > > > sys 0m1.220s > > > 7595 undefined > > > 896 undefined_symbols > > > > > > Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3) > > > search for links by a logic which handle symlinks using BFS. It > > > also addresses a border case that was making 'msi-irqs/\d+' regex to > > > be misparsed. > > > > > > After patch 7, it is 11 times faster: > > > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > > > real 0m14.137s > > > user 0m12.795s > > > sys 0m1.348s > > > 7030 undefined > > > 794 undefined_symbols > > > > > > (the difference on the number of undefined symbols are due to the fix for > > > it to properly handle 'msi-irqs/\d+' regex) > > > > > > - > > > > > > While this series is independent from Documentation/ABI changes, it > > > works best when applied from this tree, which also contain ABI fixes > > > and a couple of additions of frequent missed symbols on my machine: > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3 > > > > I've taken all of these, but get_abi.pl seems to be stuck in an endless > > loop or something. I gave up and stopped it after 14 minutes. It had > > stopped printing out anything after finding all of the pci attributes > > that are not documented :) > > It is probably not an endless loop, just there are too many vars to > check on your system, which could make it really slow. Ah, yes, I ran it overnight and got the following: $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols real 29m39.503s user 29m37.556s sys 0m0.851s 26669 undefined 765 undefined_symbols > The way the search algorithm works is that reduces the number of regex > expressions that will be checked for a given file entry at sysfs. It > does that by looking at the devnode name. For instance, when it checks for > this file: > > /sys/bus/pci/drivers/iosf_mbi_pci/bind > > The logic will seek only the "What:" expressions that end with "bind". > Currently, there are just two What expressions for it[1]: > > What: /sys/bus/fsl\-mc/drivers/.*/bind > What: /sys/bus/pci/drivers/.*/bind > > It will then run an O(n²) algorithm to seek: > > foreach my $a (@names) { > foreach my $w (split /\xac/, $what) { > if ($a =~ m#^$w$#) { > exact = 1; > last; > } > } > } > > Which runs quickly, when there are few regexs to seek. There are, > however, some What: expressions that end with a wildcard. Those are > harder to process. Right now, they're all grouped together, which > makes them slower. Most of the processing time are spent on those. > > I'm working right now on some strategy to also speed up the search > for them. Once I get something better, I'll send a patch series. > > -- > > [1] On a side note, there are currently some problems with the What: > definitions for bind/unbind, as: > > - it doesn't match all PCI devices; > - it doesn't match ACPI and other buses that also export > bind/unbind. > > > > > Anything I can do to help debug this? > > > > There are two parameters that can help to identify the issue: > > a) You can add a "--show-hints" parameter. This turns on some > prints that may help to identify what the script is doing. > It is not really a debug option, but it helps to identify > when some regexes are failing. > > b) You can limit the What expressions that will be parsed with: > --search-string <something> > > You can combine both. For instance, if you want to make it > a lot more verbose, you could run it as: > > ./scripts/get_abi.pl undefined --search-string /sys --show-hints Let me run this and time stamp it to see where it is getting hung up on. Give it another 30 minutes :) thanks, greg k-hj
Em Wed, 22 Sep 2021 08:22:33 +0200 Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu: > On Wed, Sep 22, 2021 at 07:43:42AM +0200, Greg Kroah-Hartman wrote: > > On Tue, Sep 21, 2021 at 08:16:33PM +0200, Mauro Carvalho Chehab wrote: > > > Em Tue, 21 Sep 2021 18:52:42 +0200 > > > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu: > > > > > > > On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote: > > > > > Hi Greg, > > > > > > > > > > Add a new feature at get_abi.pl to optionally check for existing symbols > > > > > under /sys that won't match a "What:" inside Documentation/ABI. > > > > > > > > > > Such feature is very useful to detect missing documentation for ABI. > > > > > > > > > > This series brings a major speedup, plus it fixes a few border cases when > > > > > matching regexes that end with a ".*" or \d+. > > > > > > > > > > patch 1 changes get_abi.pl logic to handle multiple What: lines, in > > > > > order to make the script more robust; > > > > > > > > > > patch 2 adds the basic logic. It runs really quicky (up to 2 > > > > > seconds), but it doesn't use sysfs softlinks. > > > > > > > > > > Patch 3 adds support for parsing softlinks. It makes the script a > > > > > lot slower, making it take a couple of minutes to process the entire > > > > > sysfs files. It could be optimized in the future by using a graph, > > > > > but, for now, let's keep it simple. > > > > > > > > > > Patch 4 adds an optional parameter to allow filtering the results > > > > > using a regex given by the user. When this parameter is used > > > > > (which should be the normal usecase), it will only try to find softlinks > > > > > if the sysfs node matches a regex. > > > > > > > > > > Patch 5 improves the report by avoiding it to ignore What: that > > > > > ends with a wildcard. > > > > > > > > > > Patch 6 is a minor speedup. On a Dell Precision 5820, after patch 6, > > > > > results are: > > > > > > > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > > > > > > > real 2m35.563s > > > > > user 2m34.346s > > > > > sys 0m1.220s > > > > > 7595 undefined > > > > > 896 undefined_symbols > > > > > > > > > > Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3) > > > > > search for links by a logic which handle symlinks using BFS. It > > > > > also addresses a border case that was making 'msi-irqs/\d+' regex to > > > > > be misparsed. > > > > > > > > > > After patch 7, it is 11 times faster: > > > > > > > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > > > > > > > real 0m14.137s > > > > > user 0m12.795s > > > > > sys 0m1.348s > > > > > 7030 undefined > > > > > 794 undefined_symbols > > > > > > > > > > (the difference on the number of undefined symbols are due to the fix for > > > > > it to properly handle 'msi-irqs/\d+' regex) > > > > > > > > > > - > > > > > > > > > > While this series is independent from Documentation/ABI changes, it > > > > > works best when applied from this tree, which also contain ABI fixes > > > > > and a couple of additions of frequent missed symbols on my machine: > > > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3 > > > > > > > > I've taken all of these, but get_abi.pl seems to be stuck in an endless > > > > loop or something. I gave up and stopped it after 14 minutes. It had > > > > stopped printing out anything after finding all of the pci attributes > > > > that are not documented :) > > > > > > It is probably not an endless loop, just there are too many vars to > > > check on your system, which could make it really slow. > > > > Ah, yes, I ran it overnight and got the following: > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > real 29m39.503s > > user 29m37.556s > > sys 0m0.851s > > 26669 undefined > > 765 undefined_symbols > > > > > The way the search algorithm works is that reduces the number of regex > > > expressions that will be checked for a given file entry at sysfs. It > > > does that by looking at the devnode name. For instance, when it checks for > > > this file: > > > > > > /sys/bus/pci/drivers/iosf_mbi_pci/bind > > > > > > The logic will seek only the "What:" expressions that end with "bind". > > > Currently, there are just two What expressions for it[1]: > > > > > > What: /sys/bus/fsl\-mc/drivers/.*/bind > > > What: /sys/bus/pci/drivers/.*/bind > > > > > > It will then run an O(n²) algorithm to seek: > > > > > > foreach my $a (@names) { > > > foreach my $w (split /\xac/, $what) { > > > if ($a =~ m#^$w$#) { > > > exact = 1; > > > last; > > > } > > > } > > > } > > > > > > Which runs quickly, when there are few regexs to seek. There are, > > > however, some What: expressions that end with a wildcard. Those are > > > harder to process. Right now, they're all grouped together, which > > > makes them slower. Most of the processing time are spent on those. > > > > > > I'm working right now on some strategy to also speed up the search > > > for them. Once I get something better, I'll send a patch series. > > > > > > -- > > > > > > [1] On a side note, there are currently some problems with the What: > > > definitions for bind/unbind, as: > > > > > > - it doesn't match all PCI devices; > > > - it doesn't match ACPI and other buses that also export > > > bind/unbind. > > > > > > > > > > > Anything I can do to help debug this? > > > > > > > > > > There are two parameters that can help to identify the issue: > > > > > > a) You can add a "--show-hints" parameter. This turns on some > > > prints that may help to identify what the script is doing. > > > It is not really a debug option, but it helps to identify > > > when some regexes are failing. > > > > > > b) You can limit the What expressions that will be parsed with: > > > --search-string <something> > > > > > > You can combine both. For instance, if you want to make it > > > a lot more verbose, you could run it as: > > > > > > ./scripts/get_abi.pl undefined --search-string /sys --show-hints > > > > Let me run this and time stamp it to see where it is getting hung up on. > > Give it another 30 minutes :) > > Hm, that didn't make too much sense as to what it was stalled on. I've > attached the compressed file if you are curious. Hmm... [07:52:44] --> /sys/devices/pci0000:40/0000:40:01.3/0000:4a:00.1/iommu/amd-iommu/cap [08:07:52] --> /sys/devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:05.0/iommu/amd-iommu/cap It sounds it took quite a while handling iommu cap, which sounds weird, as it should be looking just 3 What expressions: [07:43:06] What: /sys/class/iommu/.*/amd\-iommu/cap [07:43:06] What: /sys/class/iommu/.*/intel\-iommu/cap [07:43:06] What: /sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap Maybe there was a memory starvation while running the script, causing swaps. Still, it is weird that it would happen there, as the hashes and arrays used at the script are all allocated before it starts the search logic. Here, the allocation part takes ~2 seconds. At least on my Dell Precision 5820 (12 cpu threads), the amount of memory it uses is not huge: $ /usr/bin/time -v ./scripts/get_abi.pl undefined >/dev/null Command being timed: "./scripts/get_abi.pl undefined" User time (seconds): 12.68 System time (seconds): 1.29 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.98 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 212608 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 52003 Voluntary context switches: 1 Involuntary context switches: 56 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 Unfortunately, I don't have any amd-based machine here, but I'll try to run it later on a big arm server and see how it behaves. > Anyway, this is all in my tree now, and I'll gladly take patches to make > it go faster :) Ok! Thanks, Mauro
On Wed, Sep 22, 2021 at 09:36:09AM +0200, Mauro Carvalho Chehab wrote: > Em Wed, 22 Sep 2021 08:22:33 +0200 > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu: > > > On Wed, Sep 22, 2021 at 07:43:42AM +0200, Greg Kroah-Hartman wrote: > > > On Tue, Sep 21, 2021 at 08:16:33PM +0200, Mauro Carvalho Chehab wrote: > > > > Em Tue, 21 Sep 2021 18:52:42 +0200 > > > > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu: > > > > > > > > > On Sat, Sep 18, 2021 at 11:52:10AM +0200, Mauro Carvalho Chehab wrote: > > > > > > Hi Greg, > > > > > > > > > > > > Add a new feature at get_abi.pl to optionally check for existing symbols > > > > > > under /sys that won't match a "What:" inside Documentation/ABI. > > > > > > > > > > > > Such feature is very useful to detect missing documentation for ABI. > > > > > > > > > > > > This series brings a major speedup, plus it fixes a few border cases when > > > > > > matching regexes that end with a ".*" or \d+. > > > > > > > > > > > > patch 1 changes get_abi.pl logic to handle multiple What: lines, in > > > > > > order to make the script more robust; > > > > > > > > > > > > patch 2 adds the basic logic. It runs really quicky (up to 2 > > > > > > seconds), but it doesn't use sysfs softlinks. > > > > > > > > > > > > Patch 3 adds support for parsing softlinks. It makes the script a > > > > > > lot slower, making it take a couple of minutes to process the entire > > > > > > sysfs files. It could be optimized in the future by using a graph, > > > > > > but, for now, let's keep it simple. > > > > > > > > > > > > Patch 4 adds an optional parameter to allow filtering the results > > > > > > using a regex given by the user. When this parameter is used > > > > > > (which should be the normal usecase), it will only try to find softlinks > > > > > > if the sysfs node matches a regex. > > > > > > > > > > > > Patch 5 improves the report by avoiding it to ignore What: that > > > > > > ends with a wildcard. > > > > > > > > > > > > Patch 6 is a minor speedup. On a Dell Precision 5820, after patch 6, > > > > > > results are: > > > > > > > > > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > > > > > > > > > real 2m35.563s > > > > > > user 2m34.346s > > > > > > sys 0m1.220s > > > > > > 7595 undefined > > > > > > 896 undefined_symbols > > > > > > > > > > > > Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3) > > > > > > search for links by a logic which handle symlinks using BFS. It > > > > > > also addresses a border case that was making 'msi-irqs/\d+' regex to > > > > > > be misparsed. > > > > > > > > > > > > After patch 7, it is 11 times faster: > > > > > > > > > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > > > > > > > > > real 0m14.137s > > > > > > user 0m12.795s > > > > > > sys 0m1.348s > > > > > > 7030 undefined > > > > > > 794 undefined_symbols > > > > > > > > > > > > (the difference on the number of undefined symbols are due to the fix for > > > > > > it to properly handle 'msi-irqs/\d+' regex) > > > > > > > > > > > > - > > > > > > > > > > > > While this series is independent from Documentation/ABI changes, it > > > > > > works best when applied from this tree, which also contain ABI fixes > > > > > > and a couple of additions of frequent missed symbols on my machine: > > > > > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3 > > > > > > > > > > I've taken all of these, but get_abi.pl seems to be stuck in an endless > > > > > loop or something. I gave up and stopped it after 14 minutes. It had > > > > > stopped printing out anything after finding all of the pci attributes > > > > > that are not documented :) > > > > > > > > It is probably not an endless loop, just there are too many vars to > > > > check on your system, which could make it really slow. > > > > > > Ah, yes, I ran it overnight and got the following: > > > > > > $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols > > > > > > real 29m39.503s > > > user 29m37.556s > > > sys 0m0.851s > > > 26669 undefined > > > 765 undefined_symbols > > > > > > > The way the search algorithm works is that reduces the number of regex > > > > expressions that will be checked for a given file entry at sysfs. It > > > > does that by looking at the devnode name. For instance, when it checks for > > > > this file: > > > > > > > > /sys/bus/pci/drivers/iosf_mbi_pci/bind > > > > > > > > The logic will seek only the "What:" expressions that end with "bind". > > > > Currently, there are just two What expressions for it[1]: > > > > > > > > What: /sys/bus/fsl\-mc/drivers/.*/bind > > > > What: /sys/bus/pci/drivers/.*/bind > > > > > > > > It will then run an O(n²) algorithm to seek: > > > > > > > > foreach my $a (@names) { > > > > foreach my $w (split /\xac/, $what) { > > > > if ($a =~ m#^$w$#) { > > > > exact = 1; > > > > last; > > > > } > > > > } > > > > } > > > > > > > > Which runs quickly, when there are few regexs to seek. There are, > > > > however, some What: expressions that end with a wildcard. Those are > > > > harder to process. Right now, they're all grouped together, which > > > > makes them slower. Most of the processing time are spent on those. > > > > > > > > I'm working right now on some strategy to also speed up the search > > > > for them. Once I get something better, I'll send a patch series. > > > > > > > > -- > > > > > > > > [1] On a side note, there are currently some problems with the What: > > > > definitions for bind/unbind, as: > > > > > > > > - it doesn't match all PCI devices; > > > > - it doesn't match ACPI and other buses that also export > > > > bind/unbind. > > > > > > > > > > > > > > Anything I can do to help debug this? > > > > > > > > > > > > > There are two parameters that can help to identify the issue: > > > > > > > > a) You can add a "--show-hints" parameter. This turns on some > > > > prints that may help to identify what the script is doing. > > > > It is not really a debug option, but it helps to identify > > > > when some regexes are failing. > > > > > > > > b) You can limit the What expressions that will be parsed with: > > > > --search-string <something> > > > > > > > > You can combine both. For instance, if you want to make it > > > > a lot more verbose, you could run it as: > > > > > > > > ./scripts/get_abi.pl undefined --search-string /sys --show-hints > > > > > > Let me run this and time stamp it to see where it is getting hung up on. > > > Give it another 30 minutes :) > > > > Hm, that didn't make too much sense as to what it was stalled on. I've > > attached the compressed file if you are curious. > > Hmm... > > [07:52:44] --> /sys/devices/pci0000:40/0000:40:01.3/0000:4a:00.1/iommu/amd-iommu/cap > [08:07:52] --> /sys/devices/pci0000:40/0000:40:01.1/0000:41:00.0/0000:42:05.0/iommu/amd-iommu/cap > > It sounds it took quite a while handling iommu cap, which sounds weird, as > it should be looking just 3 What expressions: > > [07:43:06] What: /sys/class/iommu/.*/amd\-iommu/cap > [07:43:06] What: /sys/class/iommu/.*/intel\-iommu/cap > [07:43:06] What: /sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap > > Maybe there was a memory starvation while running the script, causing > swaps. Still, it is weird that it would happen there, as the hashes > and arrays used at the script are all allocated before it starts the > search logic. Here, the allocation part takes ~2 seconds. No memory starvation here, this thing is a beast: $ free -h total used free shared buff/cache available Mem: 251Gi 36Gi 13Gi 402Mi 202Gi 212Gi Swap: 4.0Gi 182Mi 3.8Gi $ nproc 64 > At least on my Dell Precision 5820 (12 cpu threads), the amount of memory it > uses is not huge: > > $ /usr/bin/time -v ./scripts/get_abi.pl undefined >/dev/null > Command being timed: "./scripts/get_abi.pl undefined" > User time (seconds): 12.68 > System time (seconds): 1.29 > Percent of CPU this job got: 99% > Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.98 > Average shared text size (kbytes): 0 > Average unshared data size (kbytes): 0 > Average stack size (kbytes): 0 > Average total size (kbytes): 0 > Maximum resident set size (kbytes): 212608 > Average resident set size (kbytes): 0 > Major (requiring I/O) page faults: 0 > Minor (reclaiming a frame) page faults: 52003 > Voluntary context switches: 1 > Involuntary context switches: 56 > Swaps: 0 > File system inputs: 0 > File system outputs: 0 > Socket messages sent: 0 > Socket messages received: 0 > Signals delivered: 0 > Page size (bytes): 4096 > Exit status: 0 > > Unfortunately, I don't have any amd-based machine here, but I'll > try to run it later on a big arm server and see how it behaves. I'll run that and get back to you in 30 minutes :) thanks, greg k-h
On Wed, Sep 22, 2021 at 10:11:02AM +0200, Greg Kroah-Hartman wrote: > On Wed, Sep 22, 2021 at 09:36:09AM +0200, Mauro Carvalho Chehab wrote: > > It sounds it took quite a while handling iommu cap, which sounds weird, as > > it should be looking just 3 What expressions: > > > > [07:43:06] What: /sys/class/iommu/.*/amd\-iommu/cap > > [07:43:06] What: /sys/class/iommu/.*/intel\-iommu/cap > > [07:43:06] What: /sys/devices/pci.*.*.*.*\:.*.*/0000\:.*.*\:.*.*..*/dma/dma.*chan.*/quickdata/cap > > > > Maybe there was a memory starvation while running the script, causing > > swaps. Still, it is weird that it would happen there, as the hashes > > and arrays used at the script are all allocated before it starts the > > search logic. Here, the allocation part takes ~2 seconds. > > No memory starvation here, this thing is a beast: > $ free -h > total used free shared buff/cache available > Mem: 251Gi 36Gi 13Gi 402Mi 202Gi 212Gi > Swap: 4.0Gi 182Mi 3.8Gi > > $ nproc > 64 > > > > At least on my Dell Precision 5820 (12 cpu threads), the amount of memory it > > uses is not huge: > > > > $ /usr/bin/time -v ./scripts/get_abi.pl undefined >/dev/null > > Command being timed: "./scripts/get_abi.pl undefined" > > User time (seconds): 12.68 > > System time (seconds): 1.29 > > Percent of CPU this job got: 99% > > Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.98 > > Average shared text size (kbytes): 0 > > Average unshared data size (kbytes): 0 > > Average stack size (kbytes): 0 > > Average total size (kbytes): 0 > > Maximum resident set size (kbytes): 212608 > > Average resident set size (kbytes): 0 > > Major (requiring I/O) page faults: 0 > > Minor (reclaiming a frame) page faults: 52003 > > Voluntary context switches: 1 > > Involuntary context switches: 56 > > Swaps: 0 > > File system inputs: 0 > > File system outputs: 0 > > Socket messages sent: 0 > > Socket messages received: 0 > > Signals delivered: 0 > > Page size (bytes): 4096 > > Exit status: 0 > > > > Unfortunately, I don't have any amd-based machine here, but I'll > > try to run it later on a big arm server and see how it behaves. > > I'll run that and get back to you in 30 minutes :) $ /usr/bin/time -v ./scripts/get_abi.pl undefined > /dev/null Command being timed: "./scripts/get_abi.pl undefined" User time (seconds): 1756.94 System time (seconds): 0.76 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 29:18.94 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 228116 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 55862 Voluntary context switches: 1 Involuntary context switches: 17205 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
Hi Greg, Add a new feature at get_abi.pl to optionally check for existing symbols under /sys that won't match a "What:" inside Documentation/ABI. Such feature is very useful to detect missing documentation for ABI. This series brings a major speedup, plus it fixes a few border cases when matching regexes that end with a ".*" or \d+. patch 1 changes get_abi.pl logic to handle multiple What: lines, in order to make the script more robust; patch 2 adds the basic logic. It runs really quicky (up to 2 seconds), but it doesn't use sysfs softlinks. Patch 3 adds support for parsing softlinks. It makes the script a lot slower, making it take a couple of minutes to process the entire sysfs files. It could be optimized in the future by using a graph, but, for now, let's keep it simple. Patch 4 adds an optional parameter to allow filtering the results using a regex given by the user. When this parameter is used (which should be the normal usecase), it will only try to find softlinks if the sysfs node matches a regex. Patch 5 improves the report by avoiding it to ignore What: that ends with a wildcard. Patch 6 is a minor speedup. On a Dell Precision 5820, after patch 6, results are: $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols real 2m35.563s user 2m34.346s sys 0m1.220s 7595 undefined 896 undefined_symbols Patch 7 makes a *huge* speedup: it basically switches a linear O(n^3) search for links by a logic which handle symlinks using BFS. It also addresses a border case that was making 'msi-irqs/\d+' regex to be misparsed. After patch 7, it is 11 times faster: $ time ./scripts/get_abi.pl undefined |sort >undefined && cat undefined| perl -ne 'print "$1\n" if (m#.*/(\S+) not found#)'|sort|uniq -c|sort -nr >undefined_symbols; wc -l undefined; wc -l undefined_symbols real 0m14.137s user 0m12.795s sys 0m1.348s 7030 undefined 794 undefined_symbols (the difference on the number of undefined symbols are due to the fix for it to properly handle 'msi-irqs/\d+' regex) - While this series is independent from Documentation/ABI changes, it works best when applied from this tree, which also contain ABI fixes and a couple of additions of frequent missed symbols on my machine: https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/log/?h=get_undefined_abi_v3 - v3: - Fixed parse issues with 'msi-irqs/\d+' regex; - Added a BFS graph logic to solve symlinks at sysfs; v2: - multiple What: for the same description are now properly handled; - some special cases are now better handled; - some bugs got fixed. The full series, with the ABI changes and some ABI improvements can be found at: https://git.kernel.org/pub/scm/linux/kernel/git/mchehab/devel.git/commit/?h=get_undefined&id=1838d8fb149170f6c19feda0645d6c3157f46f4f Mauro Carvalho Chehab (7): scripts: get_abi.pl: Better handle multiple What parameters scripts: get_abi.pl: Check for missing symbols at the ABI specs scripts: get_abi.pl: detect softlinks scripts: get_abi.pl: add an option to filter undefined results scripts: get_abi.pl: don't skip what that ends with wildcards scripts: get_abi.pl: Ignore fs/cgroup sysfs nodes earlier scripts: get_abi.pl: add a graph to speedup the undefined algorithm scripts/get_abi.pl | 327 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 320 insertions(+), 7 deletions(-)