From patchwork Sat Aug 22 14:56:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731165 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5EE4B1392 for ; Sat, 22 Aug 2020 14:57:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 47CE02080D for ; Sat, 22 Aug 2020 14:57:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108278; bh=4ljmxDjZ6NN9Cod5PD3WwfQrZr4wIAWG8YJxHM7brf0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=YEfHx513CWmgYd8sqW9CUItkH4jke43JKlDKbDc0RcLWp86FyNthmYffLubJ6GNtE 7BrF/Xl4RUW+QrAkfWVuCQP2qsiQcVOOl8TPWMS/M5go7NMng5O/laSPYpAMjn+grH bXkw7SKGBI/ifWYdr2xmzzMMZa+s3JBBNNQZCEGE= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728260AbgHVO5v (ORCPT ); Sat, 22 Aug 2020 10:57:51 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47411 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728020AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVQ025434; Sat, 22 Aug 2020 23:56:21 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVQ025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108182; bh=95U6Fk7JxWZcBC6emSKUXU+RPpgimScrngM3/3sca3M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kn4xDBCCwLj7T9hZXgeNdDZx9dRv46QZyU0AzvXPdaLSOCSt5AIVBC6GvSMqXxZjb WniWd+hSDKcCrWCR6rhfdq1srppx93gY5comjtWcvDFykJq2NI1XEXe46mjGziJFL0 UY+3cHGXXt1qyIo3n1dS+Ghm+n+fOA0oHMxA9zzrlSnkREngWIUefgMnoQG2bCz97I Oi+yUPRDn0SLawTDdDhRrEeVNXT5EG/YmKPRIRN7b1hEh1XcVjT2mA7jp78TGsvwAB PqtPeZugYxm049U/Mu7hNAK7k3WNUHGIt7/ts8w9RJIunFJQKFGr/zDM8SccwzL54z Jhav+pwhUUWkQ== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 01/10] gen_compile_commands: parse only the first line of .*.cmd files Date: Sat, 22 Aug 2020 23:56:09 +0900 Message-Id: <20200822145618.1222514-2-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org After the allmodconfig build, this script takes about 5 sec on my machine. Most of the run-time is consumed for needless regex matching. We know the format of .*.cmd file; the first line is the build command. There is no need to parse the rest. With this optimization, now it runs 4 times faster. Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers Tested-by: Nick Desaulniers --- (no changes since v2) Changes in v2: - Remove the unneeded variable 'line' scripts/gen_compile_commands.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index c458696ef3a7..1bcf33a93cb9 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -125,11 +125,8 @@ def main(): filepath = os.path.join(dirpath, filename) with open(filepath, 'rt') as f: - for line in f: - result = line_matcher.match(line) - if not result: - continue - + result = line_matcher.match(f.readline()) + if result: try: entry = process_line(directory, dirpath, result.group(1), result.group(2)) From patchwork Sat Aug 22 14:56:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731147 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 393B21510 for ; Sat, 22 Aug 2020 14:57:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 20ADB207DF for ; Sat, 22 Aug 2020 14:57:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108242; bh=5yk1kkRJMSsDhsOA+5w1pBH7nd2PwF2x0bjMlmbYyfY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=ZtbghQaLVedxmEOMQ0K5sKpeR9Fp79FDRAOIJK6wX3E4X8Ox8qKoQS7yUsska/34W loPfor6UejGN4mLBlw6Eaxzb72v0V62Rh4YLnKadPVj14+uPseMQq89F0PY7Qficf5 JDx1KW6E0uJj+T23/oCjr5ZHi85EFB3GdxR+kA3c= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728207AbgHVO5U (ORCPT ); Sat, 22 Aug 2020 10:57:20 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47410 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728124AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVR025434; Sat, 22 Aug 2020 23:56:22 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVR025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108182; bh=IBZFRCC+dauuYn3v7p0fADfhiey+2jMphRtB7L0P/CE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=2Qp4fqRO02umY4vHVytB6cc/5LxtS7YxrjgHlW2Pw96JvedtvpxUsG3mN7ouGROnM 1sKl0QngqSf0hMA6M6W9ciUUS+q3vrftp7gDLNs6I+E8w0xODDv5bPZFgYZi/IQPrw czf9dQRG3is6Ekd94pxyTgbvk0Tsh8TdKRkzjS+pxROA5DmpFNYhREqoA7swV8bvsD cYfeIsva7JL47SGTBA0hQDhWtGmPUcT9A/0rui1i67MIiwopZPAlWn9Fh7TFl9BG+z S/rO0GEn8owjS7SfEAq737gVbH/Of/C5OBYiTyik3MoKtWZtmbKugTQHgW6kjLNEhf 1m+7PY06fHuVw== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 02/10] gen_compile_commands: use choices for --log_levels option Date: Sat, 22 Aug 2020 23:56:10 +0900 Message-Id: <20200822145618.1222514-3-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Use 'choices' instead of the own code to check if the given parameter is valid. I also simplified the help message because, with 'choices', --help shows the list of valid parameters: --log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL} I started the help message with a lower case, "the level of log ..." in order to be consistent with the -h option: -h, --help show this help message and exit The message "show this help ..." comes from the ArgumentParser library code, and I do not know how to change it. So, I changed our code. Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers --- (no changes since v2) Changes in v2: - New patch scripts/gen_compile_commands.py | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index 1bcf33a93cb9..535248cf2d7e 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -45,24 +45,18 @@ def parse_arguments(): 'compile_commands.json in the search directory)') parser.add_argument('-o', '--output', type=str, help=output_help) - log_level_help = ('The level of log messages to produce (one of ' + - ', '.join(_VALID_LOG_LEVELS) + '; defaults to ' + + log_level_help = ('the level of log messages to produce (defaults to ' + _DEFAULT_LOG_LEVEL + ')') - parser.add_argument( - '--log_level', type=str, default=_DEFAULT_LOG_LEVEL, - help=log_level_help) + parser.add_argument('--log_level', choices=_VALID_LOG_LEVELS, + default=_DEFAULT_LOG_LEVEL, help=log_level_help) args = parser.parse_args() - log_level = args.log_level - if log_level not in _VALID_LOG_LEVELS: - raise ValueError('%s is not a valid log level' % log_level) - directory = args.directory or os.getcwd() output = args.output or os.path.join(directory, _DEFAULT_OUTPUT) directory = os.path.abspath(directory) - return log_level, directory, output + return args.log_level, directory, output def process_line(root_directory, file_directory, command_prefix, relative_path): From patchwork Sat Aug 22 14:56:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731151 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 772921392 for ; Sat, 22 Aug 2020 14:57:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5F0B220838 for ; Sat, 22 Aug 2020 14:57:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108242; bh=zLQE8Z66rSgeD6mallEw1MI37Rg81lqra3Z89FTQE08=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=EdiZWjxoAe9gxPo7E7tIOXcpaGqokublGuMYMF1J+7dun67lBQNRAJAIy1iGT3LKl 8g+YZw55UNop3Y9OzZVXOyNphPeCSNAmOL0OSNrniqHiDzrltvobgU9Ebv8IF7K8wV WNFd88baUGp58+dmxBpskdQE95CqcQmZdBn0KVKc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728211AbgHVO5V (ORCPT ); Sat, 22 Aug 2020 10:57:21 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47429 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728173AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVS025434; Sat, 22 Aug 2020 23:56:23 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVS025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108183; bh=q+4NzfCiNDCECz9HAmB0Vnyx61RvHFvh4fBxtXHAgmk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vhSHkbteTJ4+PlHc++uEkTlryLkihsAhwH56fQSbp1JRkabaLrLoTErVW/rWy0uD9 BkoIEgk8wuUmaQNf26R50vbFOk4NYa+l+oaiqsFAItcg9DvL2+NbHwVWex3nNLKHQK r+zAjMNJxNtI/Q4/clhLypJEi1RQviUZVp2BInZM+QcWkL7zryrUlvV/qRC/jOnUqG U0+6fk3So/cyW6TTM1q9u2LME8eXzvv1hqxhyE1Y5JG9rhHy6hA8C5Z+zHUbBMO4YI eMJPMau0PMpvHriXzZOmfV0gRpLxVUblDyXNqih/ihhXzWEiNYbrgIpnrWue9l/xww GRuaVAiSuZHBQ== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 03/10] gen_compile_commands: do not support .cmd files under tools/ directory Date: Sat, 22 Aug 2020 23:56:11 +0900 Message-Id: <20200822145618.1222514-4-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org The tools/ directory uses a different build system, and the format of .cmd files is different because the tools builds run in a different work directory. Supporting two formats compilicates the script. The only loss by this change is objtool. Also, rename the confusing variable 'relative_path' because it is not necessarily a relative path. When the output directory is not the direct child of the source tree (e.g. O=foo/bar), it is an absolute path. Rename it to 'file_path'. os.path.join(root_directory, file_path) works whether the file_path is relative or not. If file_path is already absolute, it returns it as-is. I used os.path.abspath() to normalize file paths. If you run this script against the kernel built with O=foo option, the file_path contains '../' patterns. os.path.abspath() fixes up 'foo/bar/../baz' into 'foo/baz', and produces a cleaner commands_database.json. Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers --- Changes in v3: - Add a comment about why I used os.path.abspath() Changes in v2: - New patch scripts/gen_compile_commands.py | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index 535248cf2d7e..49fff0b0b385 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -59,23 +59,21 @@ def parse_arguments(): return args.log_level, directory, output -def process_line(root_directory, file_directory, command_prefix, relative_path): +def process_line(root_directory, command_prefix, file_path): """Extracts information from a .cmd line and creates an entry from it. Args: root_directory: The directory that was searched for .cmd files. Usually used directly in the "directory" entry in compile_commands.json. - file_directory: The path to the directory the .cmd file was found in. command_prefix: The extracted command line, up to the last element. - relative_path: The .c file from the end of the extracted command. - Usually relative to root_directory, but sometimes relative to - file_directory and sometimes neither. + file_path: The .c file from the end of the extracted command. + Usually relative to root_directory, but sometimes absolute. Returns: An entry to append to compile_commands. Raises: - ValueError: Could not find the extracted file based on relative_path and + ValueError: Could not find the extracted file based on file_path and root_directory or file_directory. """ # The .cmd files are intended to be included directly by Make, so they @@ -84,20 +82,14 @@ def process_line(root_directory, file_directory, command_prefix, relative_path): # by Make, so this code replaces the escaped version with '#'. prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') - cur_dir = root_directory - expected_path = os.path.join(cur_dir, relative_path) - if not os.path.exists(expected_path): - # Try using file_directory instead. Some of the tools have a different - # style of .cmd file than the kernel. - cur_dir = file_directory - expected_path = os.path.join(cur_dir, relative_path) - if not os.path.exists(expected_path): - raise ValueError('File %s not in %s or %s' % - (relative_path, root_directory, file_directory)) + # Use os.path.abspath() to normalize the path resolving '.' and '..' . + abs_path = os.path.abspath(os.path.join(root_directory, file_path)) + if not os.path.exists(abs_path): + raise ValueError('File %s not found' % abs_path) return { - 'directory': cur_dir, - 'file': relative_path, - 'command': prefix + relative_path, + 'directory': root_directory, + 'file': abs_path, + 'command': prefix + file_path, } @@ -122,7 +114,7 @@ def main(): result = line_matcher.match(f.readline()) if result: try: - entry = process_line(directory, dirpath, + entry = process_line(directory, result.group(1), result.group(2)) compile_commands.append(entry) except ValueError as err: From patchwork Sat Aug 22 14:56:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731161 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D00AC1510 for ; Sat, 22 Aug 2020 14:57:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AD1CD2078D for ; Sat, 22 Aug 2020 14:57:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108271; bh=CBF9jYr2MzUbyza3ivEvjBPhb9m4BbSEeeBMZ0QRdTc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=IBv+vI+DKr+4dhSEkV1aR+FQVxxbPCga2GMcFlji8hAUaJXEVuVm+dGzaUIFlZOqR OK8Wd+UF5RLX9NZI6uEz5/nw8dmcZgGbHzOJctFqauTuu3A3RFxPeAsKNm9LLMAzVm kogZLg56B6DfNyT5ys4MEtSrne0S8SZ7bJruu2wU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728257AbgHVO5t (ORCPT ); Sat, 22 Aug 2020 10:57:49 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47425 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728133AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVT025434; Sat, 22 Aug 2020 23:56:23 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVT025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108184; bh=xgwPl0sYIxicw/V74mC1zpDsm1aQ6gpJpcgmlt5eG3c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=v16cp4VvoYqhmKEim6nw7wMgeapmhbD/kfC+ecB/OETdOkwANPVAnS9uXYTGJw/Nc leII5AGywsBYI+Tl4eAiRJH1w0O9PydEq26Jxv/qA6/mMp5tW41bN9rl8HHw9ukEV8 AwKwA9ns4VLkWUpqDILogncjI7Vq9qBRfKbujIhhZEANgU6EggMK/CT2tVGY1vQ13I Mq7vPNmtszBYFpjZ8XT/XhXKluxKxnA8Oez44+rJJoYCnoIT+vZ3dv5sppG/FAtmNC wjIxDgB12Da5iD0CDytmzPplvRV044P7yeCPoxxX9cKTNqW+gt2cKFcDP/cS3aBhni 8l2aZVRchn1ng== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 04/10] gen_compile_commands: reword the help message of -d option Date: Sat, 22 Aug 2020 23:56:12 +0900 Message-Id: <20200822145618.1222514-5-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org I think the help message of the -d option is somewhat misleading. Path to the kernel source directory to search (defaults to the working directory) The part "kernel source directory" is the source of the confusion. Some people misunderstand as if this script did not support separate output directories. Actually, this script also works for out-of-tree builds. You can use the -d option to point to the object output directory, not to the source directory. It should match to the O= option used in the previous kernel build, and then appears in the "directory" field of compile_commands.json. Reword the help message. Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers --- Changes in v3: - Add the missing punctuation to the comment Changes in v2: - New patch scripts/gen_compile_commands.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index 49fff0b0b385..f37c1dac8db4 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -31,13 +31,13 @@ def parse_arguments(): Returns: log_level: A logging level to filter log output. - directory: The directory to search for .cmd files. + directory: The work directory where the objects were built. output: Where to write the compile-commands JSON file. """ usage = 'Creates a compile_commands.json database from kernel .cmd files' parser = argparse.ArgumentParser(description=usage) - directory_help = ('Path to the kernel source directory to search ' + directory_help = ('specify the output directory used for the kernel build ' '(defaults to the working directory)') parser.add_argument('-d', '--directory', type=str, help=directory_help) From patchwork Sat Aug 22 14:56:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731149 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57A3116B1 for ; Sat, 22 Aug 2020 14:57:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3E595207DF for ; Sat, 22 Aug 2020 14:57:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108242; bh=iyNWXyADCAMrwuK4pGERc+sNjuLS+oSPR+1fnSw+zlo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=sWuiSeT4kkpXHfjciEQwAd/r4zbb++ehF45v/oTJ2OEM9wNsPrpbWUEyr3LTXt8Rt O90VoZg9RhP2b4QlmnkdkrpVJVVNJlFh0OsKpiogDhofeBRsPZzwITKADQsouhRqGh 8crGjECV+jPW/MiJ4Z8suEggD7BAtjW2CxQfIUP8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728209AbgHVO5V (ORCPT ); Sat, 22 Aug 2020 10:57:21 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47424 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728129AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVU025434; Sat, 22 Aug 2020 23:56:24 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVU025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108185; bh=kQajBz1G8qgjw+0MUJ0nLzYzwlOyQl9s1CrXywysVeA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vVURP5tFck6wTrk+tHt9VgorM49hz3tpp9tlXMa3TTq/tlPVJ0pG6BxeKc6BN0ZYd aMLUO7uhDeFf5Vb8btlg0HItIdVY04qqIWczH42rNBUw1A0lLOKxT5EzHyWLlg5MYV xUdEZpMZt3RnUy11CpbDkFEunCegtTSnwf91albn0hWqDpCj6xlrmR1ND/k3eQ+Hoh Asca0qIOpasw8LMoL00yibcSF64p1ggahnNOLqwO7Mk5goKGnpwPp2Z5hEbuDumVkn xz6VFXS8NOI3jnke/0WpTkcIW705gTLT1hqNOBppWkZ9rjFwf/H9LFkKOfU6JFVMg7 IDnMRkctY88+g== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 05/10] gen_compile_commands: make -o option independent of -d option Date: Sat, 22 Aug 2020 23:56:13 +0900 Message-Id: <20200822145618.1222514-6-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Change the -o option independent of the -d option, which is I think clearer behavior. Some people may like to use -d to specify a separate output directory, but still output the compile_commands.py in the source directory (unless the source tree is read-only) because it is the default location Clang Tools search for the compilation database. Also, move the default parameter to the default= argument of the .add_argument(). Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers --- (no changes since v2) Changes in v2: - New patch scripts/gen_compile_commands.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index f37c1dac8db4..71a0630ae188 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -39,11 +39,13 @@ def parse_arguments(): directory_help = ('specify the output directory used for the kernel build ' '(defaults to the working directory)') - parser.add_argument('-d', '--directory', type=str, help=directory_help) + parser.add_argument('-d', '--directory', type=str, default='.', + help=directory_help) - output_help = ('The location to write compile_commands.json (defaults to ' - 'compile_commands.json in the search directory)') - parser.add_argument('-o', '--output', type=str, help=output_help) + output_help = ('path to the output command database (defaults to ' + + _DEFAULT_OUTPUT + ')') + parser.add_argument('-o', '--output', type=str, default=_DEFAULT_OUTPUT, + help=output_help) log_level_help = ('the level of log messages to produce (defaults to ' + _DEFAULT_LOG_LEVEL + ')') @@ -52,11 +54,9 @@ def parse_arguments(): args = parser.parse_args() - directory = args.directory or os.getcwd() - output = args.output or os.path.join(directory, _DEFAULT_OUTPUT) - directory = os.path.abspath(directory) - - return args.log_level, directory, output + return (args.log_level, + os.path.abspath(args.directory), + args.output) def process_line(root_directory, command_prefix, file_path): From patchwork Sat Aug 22 14:56:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731145 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F9811510 for ; Sat, 22 Aug 2020 14:57:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EBBC62078D for ; Sat, 22 Aug 2020 14:57:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108234; bh=7tU2JSwdAxQ3AStTHU9D1r/ibInF+JSBrnPYfQDWEzg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=14GXZ5GJL3BtWwTUL2+eBmV8Kdc6Trzzg6DS4SZv76ZQrMd0suOQaDbmf/wTYSgEI 2lXRgCqsrRuTF8kt7OHCM3unabJQMw1vjG8kA8SRpeC5To1sZ4SjK9N8uf512+LzaY K5qniOgQOSuSHIt+1U5fGn6/j1QXhlUjr6HqLSfM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728192AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47418 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728019AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVV025434; Sat, 22 Aug 2020 23:56:25 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVV025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108185; bh=jH2tLSORRhlXDuP+7ApdPBfQ1QixQf9c+4qwpQLm8xk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XCcHsUJFon+sjkDt/sF4cmBipK+M3z/uIpLS97rIiAkegFB/PmFLCO9s5n3J0M284 xAEpaVQ5VVF89hevHo9UOPOh7aQnhYAQWigyPMoUbeY420NAt2pFqrBnMuDehvvKLa U4wTGK5tizgqR+nB3Zly+6Qhswx1nj6aR38oyJT0n2OEdTkEpzascAzoEHHrIkx1yw c/KzjmsIPt5fjDqBcho33KO5IXp5z6zQ/KvRgTbk6ndnP0Vt+Yfh2P6UY8DHsSjQeb SFneOkEnH5qn5sxeDsyeKN90E8i6VMruuDSBnCw35rzRjkIaczSXmllJ4+AgLffAn2 vi7f/IJeSfpgQ== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 06/10] gen_compile_commands: move directory walk to a generator function Date: Sat, 22 Aug 2020 23:56:14 +0900 Message-Id: <20200822145618.1222514-7-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Currently, this script walks under the specified directory (default to the current directory), then parses all .cmd files found. Split it into a separate helper function because the next commit will add more helpers to pick up .cmd files associated with given file(s). There is no point to build and return a huge list at once. I used a generator so it works in the for-loop with less memory. Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers --- Changes in v3: - Add the missing punctuation in the comment Changes in v2: - New patch scripts/gen_compile_commands.py | 44 ++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 12 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index 71a0630ae188..e45f17be8817 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -33,6 +33,7 @@ def parse_arguments(): log_level: A logging level to filter log output. directory: The work directory where the objects were built. output: Where to write the compile-commands JSON file. + paths: The list of directories to handle to find .cmd files. """ usage = 'Creates a compile_commands.json database from kernel .cmd files' parser = argparse.ArgumentParser(description=usage) @@ -56,7 +57,28 @@ def parse_arguments(): return (args.log_level, os.path.abspath(args.directory), - args.output) + args.output, + [args.directory]) + + +def cmdfiles_in_dir(directory): + """Generate the iterator of .cmd files found under the directory. + + Walk under the given directory, and yield every .cmd file found. + + Args: + directory: The directory to search for .cmd files. + + Yields: + The path to a .cmd file. + """ + + filename_matcher = re.compile(_FILENAME_PATTERN) + + for dirpath, _, filenames in os.walk(directory): + for filename in filenames: + if filename_matcher.match(filename): + yield os.path.join(dirpath, filename) def process_line(root_directory, command_prefix, file_path): @@ -95,31 +117,29 @@ def process_line(root_directory, command_prefix, file_path): def main(): """Walks through the directory and finds and parses .cmd files.""" - log_level, directory, output = parse_arguments() + log_level, directory, output, paths = parse_arguments() level = getattr(logging, log_level) logging.basicConfig(format='%(levelname)s: %(message)s', level=level) - filename_matcher = re.compile(_FILENAME_PATTERN) line_matcher = re.compile(_LINE_PATTERN) compile_commands = [] - for dirpath, _, filenames in os.walk(directory): - for filename in filenames: - if not filename_matcher.match(filename): - continue - filepath = os.path.join(dirpath, filename) - with open(filepath, 'rt') as f: + for path in paths: + cmdfiles = cmdfiles_in_dir(path) + + for cmdfile in cmdfiles: + with open(cmdfile, 'rt') as f: result = line_matcher.match(f.readline()) if result: try: - entry = process_line(directory, - result.group(1), result.group(2)) + entry = process_line(directory, result.group(1), + result.group(2)) compile_commands.append(entry) except ValueError as err: logging.info('Could not add line from %s: %s', - filepath, err) + cmdfile, err) with open(output, 'wt') as f: json.dump(compile_commands, f, indent=2, sort_keys=True) From patchwork Sat Aug 22 14:56:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731155 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B703A1510 for ; Sat, 22 Aug 2020 14:57:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 996042078D for ; Sat, 22 Aug 2020 14:57:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108269; bh=FSzpB+IprUy583vSaiJkx6eIK40Giy77a9wktrK7cbk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=VCeVhWeyahxgWU9dSV0rkk7ukMCq6TeUKGbxdZ8vrZattZ/aKyZwNPmTRgD6FUFo0 Em0pJcY/eyeRJBJAl9UQ+ehf+MK9X16D0xzbnlvjurDgR23K2qbtia1YgxQOYGOQx+ o4dLPI1Kz9mB6FJEX+QF7Qs/BsSylMeKTU1eb/qw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728135AbgHVO5p (ORCPT ); Sat, 22 Aug 2020 10:57:45 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47432 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728177AbgHVO5N (ORCPT ); Sat, 22 Aug 2020 10:57:13 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVW025434; Sat, 22 Aug 2020 23:56:26 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVW025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108186; bh=xK0TccZgu0kMI4oYgleYpG0IMmmzu3uu8KTK04GIMN0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QbN6B9GsoDqA/CiNLvNTihVKHlARD9K0ftDask49e2/yXJey4b8pBDxnz/2pkSRpN ZB7XeOC6Ik3ijdtFdRcjGlTRBNpmfliGEWf1r7BNP8Jt/TubI/TVNniE/h7yj5rPwm y71B1xPP1bQpU9zdn+hqs3RMd/eTwAD6sVPcC57aUIscsIcgLMV9ol1VAZk4FSDO6E /wZBgWMUL8KsqaurHB5+6UHNO525fLKSox/Gop7xwKw6WdchVuntlQtGuu4crr4mY/ GJwiw8nCAhSS8hk8uwtpfKkuklvMyDXWrsythZ71ix4ZAYT7nsX5Yp/q/y/qGk/UiC HdNNoCllQY21g== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 07/10] gen_compile_commands: support *.o, *.a, modules.order in positional argument Date: Sat, 22 Aug 2020 23:56:15 +0900 Message-Id: <20200822145618.1222514-8-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org This script currently searches the specified directory for .cmd files. One drawback is it may contain stale .cmd files after you rebuild the kernel several times without 'make clean'. This commit supports *.o, *.a, and modules.order as positional parameters. If such files are given, they are parsed to collect associated .cmd files. I added a generator helper for each of them. This feature is useful to get the list of active .cmd files from the last build, and will be used by the next commit to wire up the compile_commands.json rule to the Makefile. Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers --- Changes in v3: - Use 'llvm-ar' instead of 'ar' for the default of -a option - Fix the corrupted comment block Changes in v2: - Separate the file parser into generator functions - Use 'obj' instead of 'object' because 'object' is a built-in function - I think using 'file' is OK because it is not a built-in function in Python3 (https://docs.python.org/3/library/functions.html) Anyway, the variable 'file' is no longer used in this version - Keep the previous work-flow to allow to search the given directory scripts/gen_compile_commands.py | 100 ++++++++++++++++++++++++++++++-- 1 file changed, 96 insertions(+), 4 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index e45f17be8817..f370375b2f70 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -12,6 +12,7 @@ import json import logging import os import re +import subprocess _DEFAULT_OUTPUT = 'compile_commands.json' _DEFAULT_LOG_LEVEL = 'WARNING' @@ -32,8 +33,9 @@ def parse_arguments(): Returns: log_level: A logging level to filter log output. directory: The work directory where the objects were built. + ar: Command used for parsing .a archives. output: Where to write the compile-commands JSON file. - paths: The list of directories to handle to find .cmd files. + paths: The list of files/directories to handle to find .cmd files. """ usage = 'Creates a compile_commands.json database from kernel .cmd files' parser = argparse.ArgumentParser(description=usage) @@ -53,12 +55,21 @@ def parse_arguments(): parser.add_argument('--log_level', choices=_VALID_LOG_LEVELS, default=_DEFAULT_LOG_LEVEL, help=log_level_help) + ar_help = 'command used for parsing .a archives' + parser.add_argument('-a', '--ar', type=str, default='llvm-ar', help=ar_help) + + paths_help = ('directories to search or files to parse ' + '(files should be *.o, *.a, or modules.order). ' + 'If nothing is specified, the current directory is searched') + parser.add_argument('paths', type=str, nargs='*', help=paths_help) + args = parser.parse_args() return (args.log_level, os.path.abspath(args.directory), args.output, - [args.directory]) + args.ar, + args.paths if len(args.paths) > 0 else [args.directory]) def cmdfiles_in_dir(directory): @@ -81,6 +92,73 @@ def cmdfiles_in_dir(directory): yield os.path.join(dirpath, filename) +def to_cmdfile(path): + """Return the path of .cmd file used for the given build artifact + + Args: + Path: file path + + Returns: + The path to .cmd file + """ + dir, base = os.path.split(path) + return os.path.join(dir, '.' + base + '.cmd') + + +def cmdfiles_for_o(obj): + """Generate the iterator of .cmd files associated with the object + + Yield the .cmd file used to build the given object + + Args: + obj: The object path + + Yields: + The path to .cmd file + """ + yield to_cmdfile(obj) + + +def cmdfiles_for_a(archive, ar): + """Generate the iterator of .cmd files associated with the archive. + + Parse the given archive, and yield every .cmd file used to build it. + + Args: + archive: The archive to parse + + Yields: + The path to every .cmd file found + """ + for obj in subprocess.check_output([ar, '-t', archive]).decode().split(): + yield to_cmdfile(obj) + + +def cmdfiles_for_modorder(modorder): + """Generate the iterator of .cmd files associated with the modules.order. + + Parse the given modules.order, and yield every .cmd file used to build the + contained modules. + + Args: + modorder: The modules.order file to parse + + Yields: + The path to every .cmd file found + """ + with open(modorder) as f: + for line in f: + ko = line.rstrip() + base, ext = os.path.splitext(ko) + if ext != '.ko': + sys.exit('{}: module path must end with .ko'.format(ko)) + mod = base + '.mod' + # The first line of *.mod lists the objects that compose the module. + with open(mod) as m: + for obj in m.readline().split(): + yield to_cmdfile(obj) + + def process_line(root_directory, command_prefix, file_path): """Extracts information from a .cmd line and creates an entry from it. @@ -117,7 +195,7 @@ def process_line(root_directory, command_prefix, file_path): def main(): """Walks through the directory and finds and parses .cmd files.""" - log_level, directory, output, paths = parse_arguments() + log_level, directory, output, ar, paths = parse_arguments() level = getattr(logging, log_level) logging.basicConfig(format='%(levelname)s: %(message)s', level=level) @@ -127,7 +205,21 @@ def main(): compile_commands = [] for path in paths: - cmdfiles = cmdfiles_in_dir(path) + # If 'path' is a directory, handle all .cmd files under it. + # Otherwise, handle .cmd files associated with the file. + # Most of built-in objects are linked via archives (built-in.a or lib.a) + # but some objects are linked to vmlinux directly. + # Modules are listed in modules.order. + if os.path.isdir(path): + cmdfiles = cmdfiles_in_dir(path) + elif path.endswith('.o'): + cmdfiles = cmdfiles_for_o(path) + elif path.endswith('.a'): + cmdfiles = cmdfiles_for_a(path, ar) + elif path.endswith('modules.order'): + cmdfiles = cmdfiles_for_modorder(path) + else: + sys.exit('{}: unknown file type'.format(path)) for cmdfile in cmdfiles: with open(cmdfile, 'rt') as f: From patchwork Sat Aug 22 14:56:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731163 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 109241510 for ; Sat, 22 Aug 2020 14:57:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E9D09207DF for ; Sat, 22 Aug 2020 14:57:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108273; bh=iBQ7lhjQWnmKWa3eDocdPtlZUCTNXghVjr7OPms/VH8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=cgEOEohbT0ZGu7UKoS3d/2QEe5ucro74RKg9vyV+oMhJdq/86IQQmzv1iJzppMgrV rafSPSUtSM1etGi31qHmjHhARdeW3+/qv+TAEQOudNy2x222L3GzKb+s3DeMtzmYM4 5t/qbZyn+bT7uVqgsNJYHS8CmJKlierH7gd3Wmdg= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728258AbgHVO5v (ORCPT ); Sat, 22 Aug 2020 10:57:51 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47430 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728176AbgHVO5M (ORCPT ); Sat, 22 Aug 2020 10:57:12 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVX025434; Sat, 22 Aug 2020 23:56:26 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVX025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108187; bh=phlZWnRlrnA8K0o/P5ReuSSQU8VkFPSdv5LbHcJmxTo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QJxYVV9QPb7lOdI/NaJJMgtfP6t7FBPGpC+nccIVHWvGPodWXjwxZV3UJpouVRumI 59hjhGAfHShFTzzyNMSUpqIdgZEiNjFmKE5jbXU6j4mArWVDJ95d419EdPD+OyZMOx zRs6FayGlm822jYlTCQBkCSUhh3LduSSnQ7KZFy39u3polz2nKeMNFit7mE62nBpFS DahfawnJ32vRaDqOK6LD1+OTrXKdWx18yG1JZctmmyehKR8W6dc70EyGQdEFG1seVz anDap2um4kOPl/Gc6QMfgA9F1iW6xMB23wwKF73tHRyuvOK4lRyViVW3cLxvLAZIWW iStGBP8DH6OkA== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , Michal Marek , linux-kernel@vger.kernel.org Subject: [PATCH v3 08/10] kbuild: wire up the build rule of compile_commands.json to Makefile Date: Sat, 22 Aug 2020 23:56:16 +0900 Message-Id: <20200822145618.1222514-9-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org Currently, you need to manually run scripts/gen_compile_commands.py to create compile_commands.json. It parses all the .*.cmd files found under the specified directory. If you rebuild the kernel over again without 'make clean', .*.cmd files from older builds will create stale entries in compile_commands.json. This commit wires up the compile_commands.json rule to Makefile, and makes it parse only the .*.cmd files involved in the current build. Pass $(KBUILD_VMLINUX_OBJS), $(KBUILD_VMLINUX_LIBS), and modules.order to the script. The objects or archives linked to vmlinux are listed in $(KBUILD_VMLINUX_OBJS) or $(KBUILD_VMLINUX_LIBS). All the modules are listed in modules.order. You can create compile_commands.json from Make: $ make -j$(nproc) CC=clang compile_commands.json You can also build vmlinux, modules, and compile_commands.json all together in a single command: $ make -j$(nproc) CC=clang all compile_commands.json It works for M= builds as well. In this case, compile_commands.json is created in the top directory of the external module. This is convenient, but it has a drawback; the coverage of the compile_commands.json is reduced because only the objects linked to vmlinux or modules are handled. For example, the following C files are not included in the compile_commands.json: - Decompressor source files (arch/*/boot/) - VDSO source files - C files used to generate intermediates (e.g. kernel/bounds.c) - Standalone host programs I think it is fine for most developers because our main interest is the kernel-space code. If you want to cover all the compiled C files, please build the kernel, then run the script manually as you did before: $ make clean # if you want to remove stale .cmd files [optional] $ make -j$(nproc) CC=clang $ scripts/gen_compile_commands.py Here is a note for out-of-tree builds. 'make compile_commands.json' works with O= option, but please notice compile_commands.json is created in the object tree instead of the source tree. Some people may want to have compile_commands.json in the source tree because Clang Tools searches for it through all parent paths of the first input source file. However, you cannot do this for O= builds. Kbuild should never generate any build artifact in the source tree when O= is given because the source tree might be read-only. Any write attempt to the source tree is monitored and the violation may be reported. See the commit log of 8ef14c2c41d9. So, the only possible way is to create compile_commands.json in the object tree, then specify '-p ' when you use clang-check, clang-tidy, etc. Signed-off-by: Masahiro Yamada Acked-by: Nick Desaulniers --- (no changes since v1) Makefile | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 9cac6fde3479..65ed336a6de1 100644 --- a/Makefile +++ b/Makefile @@ -635,7 +635,7 @@ endif # in addition to whatever we do anyway. # Just "make" or "make all" shall build modules as well -ifneq ($(filter all modules nsdeps,$(MAKECMDGOALS)),) +ifneq ($(filter all modules nsdeps %compile_commands.json,$(MAKECMDGOALS)),) KBUILD_MODULES := 1 endif @@ -1464,7 +1464,8 @@ endif # CONFIG_MODULES # Directories & files removed with 'make clean' CLEAN_FILES += include/ksym vmlinux.symvers \ - modules.builtin modules.builtin.modinfo modules.nsdeps + modules.builtin modules.builtin.modinfo modules.nsdeps \ + compile_commands.json # Directories & files removed with 'make mrproper' MRPROPER_FILES += include/config include/generated \ @@ -1698,9 +1699,12 @@ KBUILD_MODULES := 1 build-dirs := $(KBUILD_EXTMOD) PHONY += modules -modules: descend +modules: $(MODORDER) $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost +$(MODORDER): descend + @: + PHONY += modules_install modules_install: _emodinst_ _emodinst_post @@ -1714,8 +1718,12 @@ PHONY += _emodinst_post _emodinst_post: _emodinst_ $(call cmd,depmod) +compile_commands.json: $(extmod-prefix)compile_commands.json +PHONY += compile_commands.json + clean-dirs := $(KBUILD_EXTMOD) -clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps +clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps \ + $(KBUILD_EXTMOD)/compile_commands.json PHONY += help help: @@ -1828,6 +1836,19 @@ nsdeps: export KBUILD_NSDEPS=1 nsdeps: modules $(Q)$(CONFIG_SHELL) $(srctree)/scripts/nsdeps +# Clang Tooling +# --------------------------------------------------------------------------- + +quiet_cmd_gen_compile_commands = GEN $@ + cmd_gen_compile_commands = $(PYTHON3) $< -a $(AR) -o $@ $(filter-out $<, $(real-prereqs)) + +$(extmod-prefix)compile_commands.json: scripts/gen_compile_commands.py \ + $(if $(KBUILD_EXTMOD),,$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS)) \ + $(if $(CONFIG_MODULES), $(MODORDER)) FORCE + $(call if_changed,gen_compile_commands) + +targets += $(extmod-prefix)compile_commands.json + # Scripts to check various things for consistency # --------------------------------------------------------------------------- From patchwork Sat Aug 22 14:56:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731157 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E52911392 for ; Sat, 22 Aug 2020 14:57:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C3C272078D for ; Sat, 22 Aug 2020 14:57:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108269; bh=RrzpRfmgtD7P1y4aujWiCzCuDDtkcATAv6yS7R2jfjg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=cn6ZQUzMb139PBBIhVzW0yArJDssULgRcp3abYPuhvVoC+wjIyC3koEX5CZ0N1HeI VO844WWk/jf2Od8HjAcvRvWXVvhW6AmmVbtiFwPs0cZODkMBYA1vxMcC/vhNwjtDh5 RkbRJH18PeQpY11H3vI45qkrbgCjbSFFEIoWpC5E= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728207AbgHVO5q (ORCPT ); Sat, 22 Aug 2020 10:57:46 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47438 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728189AbgHVO5N (ORCPT ); Sat, 22 Aug 2020 10:57:13 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVY025434; Sat, 22 Aug 2020 23:56:27 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVY025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108188; bh=gGF1fPLTpTUcjBBymKMSRSRXlTzNhlWRVku6EwQu2/0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C9+Lh+l4D0h2JrO7yHcAGRans6Y85EQbBH9Oy6eIepJkRn2DY2tvHqhB2EFgc6k9/ Zf2Xo61a4gZ4nGv0tGbLTLY6xrsFJSBH+PU060XcXNI+IX7otMrluh+VIb9ZDGuAn3 sPXZXVK6WiDlbSat15k8Nh1K2oa8PSW8XUMUkcC5XC547DDX5YKqWzzGRsk5TSE9r8 k9XG6LQYypfLjJ8D5suTW1coFqrsM116apQzvG6LvgBBIq/XVtBBiyhoi8YTfjgyDk fmW7TyuI8b4BMnZX5Ki6liaZ/h8VtJpIYeEhXj9sbF9ODyBw9IHCmeOGjgx6ZZDpLx iTU8NTzd2GN0g== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 09/10] gen_compile_commands: remove the warning about too few .cmd files Date: Sat, 22 Aug 2020 23:56:17 +0900 Message-Id: <20200822145618.1222514-10-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org This warning was useful when users previously needed to manually build the kernel and run this script. Now you can simply do 'make compile_commands.json', which updates all the necessary build artifacts and automatically creates the compilation database. There is no more worry for a mistake like "Oh, I forgot to build the kernel". Now, this warning is rather annoying. You can create compile_commands.json for an external module: $ make M=/path/to/your/external/module compile_commands.json Then, this warning is displayed since there are usually less than 300 files in a single module. Signed-off-by: Masahiro Yamada Reviewed-by: Nick Desaulniers --- Changes in v3: - New patch scripts/gen_compile_commands.py | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/scripts/gen_compile_commands.py b/scripts/gen_compile_commands.py index f370375b2f70..1de745577e6d 100755 --- a/scripts/gen_compile_commands.py +++ b/scripts/gen_compile_commands.py @@ -21,11 +21,6 @@ _FILENAME_PATTERN = r'^\..*\.cmd$' _LINE_PATTERN = r'^cmd_[^ ]*\.o := (.* )([^ ]*\.c)$' _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] -# A kernel build generally has over 2000 entries in its compile_commands.json -# database. If this code finds 300 or fewer, then warn the user that they might -# not have all the .cmd files, and they might need to compile the kernel. -_LOW_COUNT_THRESHOLD = 300 - def parse_arguments(): """Sets up and parses command-line arguments. @@ -236,11 +231,6 @@ def main(): with open(output, 'wt') as f: json.dump(compile_commands, f, indent=2, sort_keys=True) - count = len(compile_commands) - if count < _LOW_COUNT_THRESHOLD: - logging.warning( - 'Found %s entries. Have you compiled the kernel?', count) - if __name__ == '__main__': main() From patchwork Sat Aug 22 14:56:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 11731153 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3D9851510 for ; Sat, 22 Aug 2020 14:57:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 201AE2080D for ; Sat, 22 Aug 2020 14:57:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1598108258; bh=EnPx5ed5nFQQ48F75ZzKaPerYrv74qCef+3eKRc5OiQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=c/aFUqqbkIhsn4BX5zBkXm4nzSye4sRyL0ZSHpSYEmqKNMgcm4TjwjS+0AHUB8NpG 8IQRGd7L63Y4x+NbmGCB3EZNYuKxzWA99CEy65gm4g0TDvCQvJ/LaOddEimc+tCVfk fcxeXcVoay8tSheoWPUWqeqE4DkPPYO//9GiZYZE= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728173AbgHVO5f (ORCPT ); Sat, 22 Aug 2020 10:57:35 -0400 Received: from conuserg-09.nifty.com ([210.131.2.76]:47703 "EHLO conuserg-09.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728124AbgHVO5Z (ORCPT ); Sat, 22 Aug 2020 10:57:25 -0400 Received: from oscar.flets-west.jp (softbank126090211135.bbtec.net [126.90.211.135]) (authenticated) by conuserg-09.nifty.com with ESMTP id 07MEuKVZ025434; Sat, 22 Aug 2020 23:56:28 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-09.nifty.com 07MEuKVZ025434 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1598108189; bh=may07TN/E9M6QqtFSP+c/0evMKANYT+YPi6JrwTRamU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=klQH9Dq9Z5FPYoPHNq8maIWQ7TSrwQP9xYPlHVCCzT6FTbZ4jqyv/rDR1Isr8ZTPS lEzg+B/SyH6KlusVTZGR3R2q14MV7ef2ycxchbLAuInnvdPsbODJVdKeBcMKw0TSWz 7UUh3RUnpqwSE8bmDrX7auxhNSQQu4v2xRsXIAQDJASYBNzQjn4J98aEWGJL82X/Wv iYiDpihoQYqEm+ce98+6N6GSg8AaDRZ6VyrJyvXn7GaMi76T5JcUPvjlAa7gRvIf7L Z7rvVt1K0OsaKXP2A76qgt23CPtn2Q+BHPtf4uOMg47bLd7EJlDMBG1PwnSubwPZ7o NrGKSt/1gS7kw== X-Nifty-SrcIP: [126.90.211.135] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Nathan Huckleberry , Nick Desaulniers , Tom Roeder , clang-built-linux@googlegroups.com, Lukas Bulwahn , Masahiro Yamada , Michal Marek , linux-kernel@vger.kernel.org Subject: [PATCH v3 10/10] Makefile: Add clang-tidy and static analyzer support to makefile Date: Sat, 22 Aug 2020 23:56:18 +0900 Message-Id: <20200822145618.1222514-11-masahiroy@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200822145618.1222514-1-masahiroy@kernel.org> References: <20200822145618.1222514-1-masahiroy@kernel.org> MIME-Version: 1.0 Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org From: Nathan Huckleberry This patch adds clang-tidy and the clang static-analyzer as make targets. The goal of this patch is to make static analysis tools usable and extendable by any developer or researcher who is familiar with basic c++. The current static analysis tools require intimate knowledge of the internal workings of the static analysis. Clang-tidy and the clang static analyzers expose an easy to use api and allow users unfamiliar with clang to write new checks with relative ease. ===Clang-tidy=== Clang-tidy is an easily extendable 'linter' that runs on the AST. Clang-tidy checks are easy to write and understand. A check consists of two parts, a matcher and a checker. The matcher is created using a domain specific language that acts on the AST (https://clang.llvm.org/docs/LibASTMatchersReference.html). When AST nodes are found by the matcher a callback is made to the checker. The checker can then execute additional checks and issue warnings. Here is an example clang-tidy check to report functions that have calls to local_irq_disable without calls to local_irq_enable and vice-versa. Functions flagged with __attribute((annotation("ignore_irq_balancing"))) are ignored for analysis. (https://reviews.llvm.org/D65828) ===Clang static analyzer=== The clang static analyzer is a more powerful static analysis tool that uses symbolic execution to find bugs. Currently there is a check that looks for potential security bugs from invalid uses of kmalloc and kfree. There are several more general purpose checks that are useful for the kernel. The clang static analyzer is well documented and designed to be extensible. (https://clang-analyzer.llvm.org/checker_dev_manual.html) (https://github.com/haoNoQ/clang-analyzer-guide/releases/download/v0.1/clang-analyzer-guide-v0.1.pdf) The main draw of the clang tools is how accessible they are. The clang documentation is very nice and these tools are built specifically to be easily extendable by any developer. They provide an accessible method of bug-finding and research to people who are not overly familiar with the kernel codebase. Signed-off-by: Nathan Huckleberry Reviewed-by: Nick Desaulniers Tested-by: Nick Desaulniers Tested-by: Lukas Bulwahn Signed-off-by: Masahiro Yamada --- (no changes since v1) MAINTAINERS | 1 + Makefile | 20 ++++- .../{ => clang-tools}/gen_compile_commands.py | 0 scripts/clang-tools/run-clang-tools.py | 74 +++++++++++++++++++ 4 files changed, 93 insertions(+), 2 deletions(-) rename scripts/{ => clang-tools}/gen_compile_commands.py (100%) create mode 100755 scripts/clang-tools/run-clang-tools.py diff --git a/MAINTAINERS b/MAINTAINERS index deaafb617361..19b916dbc796 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4247,6 +4247,7 @@ W: https://clangbuiltlinux.github.io/ B: https://github.com/ClangBuiltLinux/linux/issues C: irc://chat.freenode.net/clangbuiltlinux F: Documentation/kbuild/llvm.rst +F: scripts/clang-tools/ K: \b(?i:clang|llvm)\b CLEANCACHE API diff --git a/Makefile b/Makefile index 65ed336a6de1..9ece191d8d51 100644 --- a/Makefile +++ b/Makefile @@ -635,7 +635,7 @@ endif # in addition to whatever we do anyway. # Just "make" or "make all" shall build modules as well -ifneq ($(filter all modules nsdeps %compile_commands.json,$(MAKECMDGOALS)),) +ifneq ($(filter all modules nsdeps %compile_commands.json clang-%,$(MAKECMDGOALS)),) KBUILD_MODULES := 1 endif @@ -1577,6 +1577,8 @@ help: @echo ' export_report - List the usages of all exported symbols' @echo ' headerdep - Detect inclusion cycles in headers' @echo ' coccicheck - Check with Coccinelle' + @echo ' clang-analyzer - Check with clang static analyzer' + @echo ' clang-tidy - Check with clang-tidy' @echo '' @echo 'Tools:' @echo ' nsdeps - Generate missing symbol namespace dependencies' @@ -1842,13 +1844,27 @@ nsdeps: modules quiet_cmd_gen_compile_commands = GEN $@ cmd_gen_compile_commands = $(PYTHON3) $< -a $(AR) -o $@ $(filter-out $<, $(real-prereqs)) -$(extmod-prefix)compile_commands.json: scripts/gen_compile_commands.py \ +$(extmod-prefix)compile_commands.json: scripts/clang-tools/gen_compile_commands.py \ $(if $(KBUILD_EXTMOD),,$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS)) \ $(if $(CONFIG_MODULES), $(MODORDER)) FORCE $(call if_changed,gen_compile_commands) targets += $(extmod-prefix)compile_commands.json +PHONY += clang-tidy clang-analyzer + +ifdef CONFIG_CC_IS_CLANG +quiet_cmd_clang_tools = CHECK $< + cmd_clang_tools = $(PYTHON3) $(srctree)/scripts/clang-tools/run-clang-tools.py $@ $< + +clang-tidy clang-analyzer: $(extmod-prefix)compile_commands.json + $(call cmd,clang_tools) +else +clang-tidy clang-analyzer: + @echo "$@ requires CC=clang" >&2 + @false +endif + # Scripts to check various things for consistency # --------------------------------------------------------------------------- diff --git a/scripts/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py similarity index 100% rename from scripts/gen_compile_commands.py rename to scripts/clang-tools/gen_compile_commands.py diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py new file mode 100755 index 000000000000..fa7655c7cec0 --- /dev/null +++ b/scripts/clang-tools/run-clang-tools.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) Google LLC, 2020 +# +# Author: Nathan Huckleberry +# +"""A helper routine run clang-tidy and the clang static-analyzer on +compile_commands.json. +""" + +import argparse +import json +import multiprocessing +import os +import subprocess +import sys + + +def parse_arguments(): + """Set up and parses command-line arguments. + Returns: + args: Dict of parsed args + Has keys: [path, type] + """ + usage = """Run clang-tidy or the clang static-analyzer on a + compilation database.""" + parser = argparse.ArgumentParser(description=usage) + + type_help = "Type of analysis to be performed" + parser.add_argument("type", + choices=["clang-tidy", "clang-analyzer"], + help=type_help) + path_help = "Path to the compilation database to parse" + parser.add_argument("path", type=str, help=path_help) + + return parser.parse_args() + + +def init(l, a): + global lock + global args + lock = l + args = a + + +def run_analysis(entry): + # Disable all checks, then re-enable the ones we want + checks = "-checks=-*," + if args.type == "clang-tidy": + checks += "linuxkernel-*" + else: + checks += "clang-analyzer-*" + p = subprocess.run(["clang-tidy", "-p", args.path, checks, entry["file"]], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + cwd=entry["directory"]) + with lock: + sys.stderr.buffer.write(p.stdout) + + +def main(): + args = parse_arguments() + + lock = multiprocessing.Lock() + pool = multiprocessing.Pool(initializer=init, initargs=(lock, args)) + # Read JSON data into the datastore variable + with open(args.path, "r") as f: + datastore = json.load(f) + pool.map(run_analysis, datastore) + + +if __name__ == "__main__": + main()