| Message ID | 20231206012441.840082-1-xujialu@vimux.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v4] gen_compile_commands.py: fix path resolve with symlinks in it | expand |
Hi, On Wed, Dec 06, 2023 at 09:24:42AM +0800, Jialu Xu wrote: > When a path contains relative symbolic links, os.path.abspath() might > not follow the symlinks and instead return the absolute path with just > the relative paths resolved, resulting in an incorrect path. > > 1. Say "drivers/hdf/" has some symlinks: > > # ls -l drivers/hdf/ > total 364 > drwxrwxr-x 2 ... 4096 ... evdev > lrwxrwxrwx 1 ... 44 ... framework -> ../../../../../../drivers/hdf_core/framework > -rw-rw-r-- 1 ... 359010 ... hdf_macro_test.h > lrwxrwxrwx 1 ... 55 ... inner_api -> ../../../../../../drivers/hdf_core/interfaces/inner_api > lrwxrwxrwx 1 ... 53 ... khdf -> ../../../../../../drivers/hdf_core/adapter/khdf/linux > -rw-r--r-- 1 ... 74 ... Makefile > drwxrwxr-x 3 ... 4096 ... wifi > > 2. One .cmd file records that: > > # head -1 ./framework/core/manager/src/.devmgr_service.o.cmd > cmd_drivers/hdf/khdf/manager/../../../../framework/core/manager/src/devmgr_service.o := ... \ > /path/to/out/drivers/hdf/khdf/manager/../../../../framework/core/manager/src/devmgr_service.c > > 3. os.path.abspath returns "/path/to/out/framework/core/manager/src/devmgr_service.c", not correct: > > # ./scripts/clang-tools/gen_compile_commands.py > INFO: Could not add line from ./framework/core/manager/src/.devmgr_service.o.cmd: File \ > /path/to/out/framework/core/manager/src/devmgr_service.c not found > > Use pathlib.Path.resolve(), which resolves the symlinks and normalizes > the paths correctly. > > # cat compile_commands.json > ... > { > "command": ... > "directory": ... > "file": "/path/to/blabla/drivers/hdf_core/framework/core/manager/src/devmgr_service.c" > }, > ... > > Reviewed-by: Nathan Chancellor <nathan@kernel.org> > Signed-off-by: Jialu Xu <xujialu@vimux.org> > --- This looks good to me. I prefer using pathlib over anything in os module, even if the behavior was the same. In this case, the pathlib behavior is better -- which is a bonus. Reviewed-by: Justin Stitt <justinstitt@google.com> > scripts/clang-tools/gen_compile_commands.py | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py > index 180952fb91c1b..99e28b7152c19 100755 > --- a/scripts/clang-tools/gen_compile_commands.py > +++ b/scripts/clang-tools/gen_compile_commands.py > @@ -11,6 +11,7 @@ import argparse > import json > import logging > import os > +from pathlib import Path > import re > import subprocess > import sys > @@ -172,8 +173,9 @@ def process_line(root_directory, command_prefix, file_path): > # by Make, so this code replaces the escaped version with '#'. > prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') > > - # Use os.path.abspath() to normalize the path resolving '.' and '..' . > - abs_path = os.path.abspath(os.path.join(root_directory, file_path)) > + # Make the path absolute, resolving all symlinks on the way and also normalizing it. > + # Convert Path object to a string because 'PosixPath' is not JSON serializable. > + abs_path = str(Path(root_directory, file_path).resolve()) > if not os.path.exists(abs_path): > raise ValueError('File %s not found' % abs_path) > return { > -- > 2.39.2 >
On Wed, Dec 6, 2023 at 10:26 AM Jialu Xu <xujialu@vimux.org> wrote: > > When a path contains relative symbolic links, os.path.abspath() might > not follow the symlinks and instead return the absolute path with just > the relative paths resolved, resulting in an incorrect path. > > 1. Say "drivers/hdf/" has some symlinks: > > # ls -l drivers/hdf/ > total 364 > drwxrwxr-x 2 ... 4096 ... evdev > lrwxrwxrwx 1 ... 44 ... framework -> ../../../../../../drivers/hdf_core/framework > -rw-rw-r-- 1 ... 359010 ... hdf_macro_test.h > lrwxrwxrwx 1 ... 55 ... inner_api -> ../../../../../../drivers/hdf_core/interfaces/inner_api > lrwxrwxrwx 1 ... 53 ... khdf -> ../../../../../../drivers/hdf_core/adapter/khdf/linux > -rw-r--r-- 1 ... 74 ... Makefile > drwxrwxr-x 3 ... 4096 ... wifi > > 2. One .cmd file records that: > > # head -1 ./framework/core/manager/src/.devmgr_service.o.cmd > cmd_drivers/hdf/khdf/manager/../../../../framework/core/manager/src/devmgr_service.o := ... \ > /path/to/out/drivers/hdf/khdf/manager/../../../../framework/core/manager/src/devmgr_service.c > > 3. os.path.abspath returns "/path/to/out/framework/core/manager/src/devmgr_service.c", not correct: > > # ./scripts/clang-tools/gen_compile_commands.py > INFO: Could not add line from ./framework/core/manager/src/.devmgr_service.o.cmd: File \ > /path/to/out/framework/core/manager/src/devmgr_service.c not found > > Use pathlib.Path.resolve(), which resolves the symlinks and normalizes > the paths correctly. > > # cat compile_commands.json > ... > { > "command": ... > "directory": ... > "file": "/path/to/blabla/drivers/hdf_core/framework/core/manager/src/devmgr_service.c" > }, > ... > > Reviewed-by: Nathan Chancellor <nathan@kernel.org> > Signed-off-by: Jialu Xu <xujialu@vimux.org> > --- > scripts/clang-tools/gen_compile_commands.py | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py > index 180952fb91c1b..99e28b7152c19 100755 > --- a/scripts/clang-tools/gen_compile_commands.py > +++ b/scripts/clang-tools/gen_compile_commands.py > @@ -11,6 +11,7 @@ import argparse > import json > import logging > import os > +from pathlib import Path > import re > import subprocess > import sys > @@ -172,8 +173,9 @@ def process_line(root_directory, command_prefix, file_path): > # by Make, so this code replaces the escaped version with '#'. > prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') > > - # Use os.path.abspath() to normalize the path resolving '.' and '..' . > - abs_path = os.path.abspath(os.path.join(root_directory, file_path)) > + # Make the path absolute, resolving all symlinks on the way and also normalizing it. > + # Convert Path object to a string because 'PosixPath' is not JSON serializable. > + abs_path = str(Path(root_directory, file_path).resolve()) > if not os.path.exists(abs_path): > raise ValueError('File %s not found' % abs_path) > return { Is there any reason why you didn't simply replace os.path.abspath() with os.path.realpath() ? This patch uses pathlib.Path() just in one place, leaving many call-sites of os.path.*() functions. If it is just a matter of your preference, you need to convert os.path.*() for consistency (as a follow-up patch). I see one more os.path.abspath() return (args.log_level, os.path.abspath(args.directory), args.output, args.ar, args.paths if len(args.paths) > 0 else [args.directory]) Does it cause a similar issue for the 'directory' field with symbolic link jungles? -- Best Regards Masahiro Yamada
>Is there any reason why you didn't simply replace >os.path.abspath() with os.path.realpath() ? I have tried it before, but obviously, I made a mistake. >This patch uses pathlib.Path() just in one place, >leaving many call-sites of os.path.*() functions. > >If it is just a matter of your preference, >you need to convert os.path.*() for consistency >(as a follow-up patch). Keep os.path.* as os.path.realpath() works. >I see one more os.path.abspath() > > return (args.log_level, > os.path.abspath(args.directory), > args.output, > args.ar, > args.paths if len(args.paths) > 0 else [args.directory]) > >Does it cause a similar issue for the 'directory' field >with symbolic link jungles? Yes, also fixed. -- Best Regards Jialu Xu
diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py index 180952fb91c1b..99e28b7152c19 100755 --- a/scripts/clang-tools/gen_compile_commands.py +++ b/scripts/clang-tools/gen_compile_commands.py @@ -11,6 +11,7 @@ import argparse import json import logging import os +from pathlib import Path import re import subprocess import sys @@ -172,8 +173,9 @@ def process_line(root_directory, command_prefix, file_path): # by Make, so this code replaces the escaped version with '#'. prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') - # Use os.path.abspath() to normalize the path resolving '.' and '..' . - abs_path = os.path.abspath(os.path.join(root_directory, file_path)) + # Make the path absolute, resolving all symlinks on the way and also normalizing it. + # Convert Path object to a string because 'PosixPath' is not JSON serializable. + abs_path = str(Path(root_directory, file_path).resolve()) if not os.path.exists(abs_path): raise ValueError('File %s not found' % abs_path) return {
When a path contains relative symbolic links, os.path.abspath() might not follow the symlinks and instead return the absolute path with just the relative paths resolved, resulting in an incorrect path. 1. Say "drivers/hdf/" has some symlinks: # ls -l drivers/hdf/ total 364 drwxrwxr-x 2 ... 4096 ... evdev lrwxrwxrwx 1 ... 44 ... framework -> ../../../../../../drivers/hdf_core/framework -rw-rw-r-- 1 ... 359010 ... hdf_macro_test.h lrwxrwxrwx 1 ... 55 ... inner_api -> ../../../../../../drivers/hdf_core/interfaces/inner_api lrwxrwxrwx 1 ... 53 ... khdf -> ../../../../../../drivers/hdf_core/adapter/khdf/linux -rw-r--r-- 1 ... 74 ... Makefile drwxrwxr-x 3 ... 4096 ... wifi 2. One .cmd file records that: # head -1 ./framework/core/manager/src/.devmgr_service.o.cmd cmd_drivers/hdf/khdf/manager/../../../../framework/core/manager/src/devmgr_service.o := ... \ /path/to/out/drivers/hdf/khdf/manager/../../../../framework/core/manager/src/devmgr_service.c 3. os.path.abspath returns "/path/to/out/framework/core/manager/src/devmgr_service.c", not correct: # ./scripts/clang-tools/gen_compile_commands.py INFO: Could not add line from ./framework/core/manager/src/.devmgr_service.o.cmd: File \ /path/to/out/framework/core/manager/src/devmgr_service.c not found Use pathlib.Path.resolve(), which resolves the symlinks and normalizes the paths correctly. # cat compile_commands.json ... { "command": ... "directory": ... "file": "/path/to/blabla/drivers/hdf_core/framework/core/manager/src/devmgr_service.c" }, ... Reviewed-by: Nathan Chancellor <nathan@kernel.org> Signed-off-by: Jialu Xu <xujialu@vimux.org> --- scripts/clang-tools/gen_compile_commands.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)