From patchwork Wed Aug 31 20:57:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27DC3ECAAD1 for ; Wed, 31 Aug 2022 20:58:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231645AbiHaU6S (ORCPT ); Wed, 31 Aug 2022 16:58:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230456AbiHaU6J (ORCPT ); Wed, 31 Aug 2022 16:58:09 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BFE6EA171 for ; Wed, 31 Aug 2022 13:58:05 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id c7so13140031wrp.11 for ; Wed, 31 Aug 2022 13:58:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=whP68Gd/EiQj+TzRQqsLhcuwk4nFviheC/Zx1O4SZQI=; b=THFcQyhyAr1i41FaxWsUGlhJAaDORZ4zhopOzlY9cyeIynNOK9tuvwAbgUbuByC7cu KhaiyOj/5CLZUsc5TGelv0ZdgcR2nePZJ4pp10s/l6jPR3Euv0CIho9lO1Jz3Nq3uGBY nrpNJD6PTOoU0GVRAV1SJbn7nv7G2a6IvKqY5xmxQkzORU9BzL79RPaCbVmapxdmn9pa aZakteSG7kA9dNB4RBH4I0VWEmie9NaKrEes8479i8pSALAkJAivUbFs53suhUeLXtoW Ty2TDy1oNru4giUE47KqHjdLLm9ygwGWbm/Vq8lmzmb+Gfg0QlppDYy6C0zxMGBDxs9Y Wp5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=whP68Gd/EiQj+TzRQqsLhcuwk4nFviheC/Zx1O4SZQI=; b=CvtBO0Z+tBwhtOWP5Oa3RkQah8dchsf2Qs13PA55bA7bNjxqpqgcTXOnbJsxM+jgBP RyLuRnXxsQlRJ8ry+S7muFm2qFi7DBx8r5PEyLdOsSMQ6Y6JJo4eWy/ZDK9yCxfIEGzg jUANVOf3nwu8feWZZTEXKIvgsqYEofhDybtvZE2JEo+fS2Y+F0IeESqf3dCZjvczcfn8 N/NOeWqlJfnee/TCSg5qFQHCZJs7KhR7CfkohSJlWnBmUpyH8wzpMhgFfzq8/RyJUdb1 mOs9zRQE021zH0/Sx+2oAtm61nd2boMOPTpSTfqLhwHeZCjuw51P7EboYyObu/s2rPk3 AIWg== X-Gm-Message-State: ACgBeo0/l+6665lculSQmfqaxEsmXDeza5uPbQLQZHRKFivCsctNMkEf +xn39Dr+fJt1WKJAK9/Rg820mT7XRxQ9vw== X-Google-Smtp-Source: AA6agR5gqugSutnYGEDwsfxMYbBsWhoDR0Z0BaTfscN4zX2NfSHgG5SNrlcjgtf6kH1/yXJulcGanQ== X-Received: by 2002:a5d:64c9:0:b0:226:dfa0:3fa7 with SMTP id f9-20020a5d64c9000000b00226dfa03fa7mr7569322wri.350.1661979483173; Wed, 31 Aug 2022 13:58:03 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:02 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 1/9] cocci rules: remove unused "F" metavariable from pending rule Date: Wed, 31 Aug 2022 22:57:48 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix an issue with a rule added in 9b45f499818 (object-store: prepare has_{sha1, object}_file to handle any repo, 2018-11-13). We've been spewing out this warning into our $@.log since that rule was added: warning: rule starting on line 21: metavariable F not used in the - or context code We should do a better job of scouring our coccinelle log files for such issues, but for now let's fix this as a one-off. Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/the_repository.pending.cocci | 1 - 1 file changed, 1 deletion(-) diff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci index 072ea0d9228..747d382ff5f 100644 --- a/contrib/coccinelle/the_repository.pending.cocci +++ b/contrib/coccinelle/the_repository.pending.cocci @@ -20,7 +20,6 @@ expression E; @@ expression E; -expression F; @@ - has_object_file_with_flags( + repo_has_object_file_with_flags(the_repository, From patchwork Wed Aug 31 20:57:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34E29ECAAD1 for ; Wed, 31 Aug 2022 20:58:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231874AbiHaU6M (ORCPT ); Wed, 31 Aug 2022 16:58:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231660AbiHaU6J (ORCPT ); Wed, 31 Aug 2022 16:58:09 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EB53D3EFC for ; Wed, 31 Aug 2022 13:58:06 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id n23-20020a7bc5d7000000b003a62f19b453so236544wmk.3 for ; Wed, 31 Aug 2022 13:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Xcd0u5sX46uyp5gvRct9QL9P6aQDzvPVWI6B40FZSe8=; b=P2lDQ8jG6aNxrQYFcgPYiFqE0w3VttoVxPtXS868csygmBE0RPrYuB9Y/6/IdFeM9P r29nwr0f+NeJPwPSfDY+KulC/4BWfyxKxkplaYxGB56sby+LAQ02yM/7S8ZWhkrUQp5d xeBBf7tcCM+Gg+6Wt3Ald/WrcTJmIt/j0P6sxjs0Ci5PNqrk0P337DeVIg6EtNHXIcbr +DNigTs7/CfnbGcrAUOisup5IPVdkdJLagXumEce6jRGENqZ9NWsaFwtN16qMTZKiQmW COviQyMcR6HkoYMZY9soizxz5u07xM59oVG8ZM/qoZSLLq/bFjq7pFwrmobA9J9KKpvo UWWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Xcd0u5sX46uyp5gvRct9QL9P6aQDzvPVWI6B40FZSe8=; b=5iW3WkX24Qgye43/lINKiy+K7g+qEUYmCuq//qVKRpCNT83hhzQAc3F6lA6NNuMi+B eBjrfXYdPeIN9ar9vUJgAVxtZYdGcbJoQOQpAnD7WRAxsXlhegrZn92AhmhpIAlvDwlU 9pKfjzarNuw7G1l1FLhaQ4hQe7E/XrHwa/3p53Sv613XVhvUjeo1bkGQfijWcaBOVdqa y6lAyVuwNkzozMbKA9IFtM3mjniJvjn6eCkwHE9eElh0qf57ElL3NA7b1bs8RZ2qbdWf XQySV9Rus9VsnqqKaz1dgj5EGaAIntmfSxyD1IiAAnzc7zH2wWuDmgNDnYBDXVhEqEPz XYlA== X-Gm-Message-State: ACgBeo0hMOu7lDGyHcRI+Rup1Rpfe63NgAysbWW3Bi3FmNCXir02fE5w yNWyLUE7rtB0yqRaHTgMLld4fscC7AJjhg== X-Google-Smtp-Source: AA6agR6Zzl43FtbbaunBLMysqgxEshHe4RbYRObA6j5Qzllv/xu7VRpg88vTsGSX1tWevQZ1x6wtuw== X-Received: by 2002:a1c:4c03:0:b0:3a5:d65c:c1e7 with SMTP id z3-20020a1c4c03000000b003a5d65cc1e7mr3170954wmf.4.1661979484628; Wed, 31 Aug 2022 13:58:04 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:03 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 2/9] Makefile: add ability to TAB-complete cocci *.patch rules Date: Wed, 31 Aug 2022 22:57:49 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Declare the contrib/coccinelle/.cocci.patch rules in such a way as to allow TAB-completion, and slightly optimize the Makefile by cutting down on the number of $(wildcard) in favor of defining "coccicheck" and "coccicheck-pending" in terms of the same incrementally filtered list. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index eac30126e29..42a70eeaef4 100644 --- a/Makefile +++ b/Makefile @@ -3139,9 +3139,15 @@ check: $(GENERATED_H) exit 1; \ fi +COCCI = $(wildcard contrib/coccinelle/*.cocci) +COCCI_PATCHES = $(addsuffix .patch,$(COCCI)) +COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES)) +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) + COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -%.cocci.patch: %.cocci $(COCCI_SOURCES) +$(COCCI_PATCHES): $(COCCI_SOURCES) +$(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \ limit=; \ @@ -3178,11 +3184,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test -coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci))) +coccicheck: $(COCCICHECK) # See contrib/coccinelle/README coccicheck-pending: coccicheck-test -coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci)) +coccicheck-pending: $(COCCICHECK_PENDING) .PHONY: coccicheck coccicheck-pending @@ -3450,7 +3456,7 @@ profile-clean: cocciclean: $(RM) -r .build/contrib/coccinelle - $(RM) contrib/coccinelle/*.cocci.patch* + $(RM) $(COCCI_PATCHES)* clean: profile-clean coverage-clean cocciclean $(RM) -r .build From patchwork Wed Aug 31 20:57:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87384ECAAD1 for ; Wed, 31 Aug 2022 20:58:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232003AbiHaU6Q (ORCPT ); Wed, 31 Aug 2022 16:58:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231672AbiHaU6J (ORCPT ); Wed, 31 Aug 2022 16:58:09 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15D70DA3F0 for ; Wed, 31 Aug 2022 13:58:08 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id s23so7975908wmj.4 for ; Wed, 31 Aug 2022 13:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Kdy5vFeElnRH1nV4Cp8wiUhmD094GQ6MJvqpTruDVec=; b=f030PgT//Dk6bVTPodRoV360iZoZoZad1lTCVCQfryUwAgTLNu/IE770vqEzcOLOjb AbRRBle5E3FidbUIscgXtCKdKWDa8IC70dxPjs/l6WvNfeBvGiR9EoDSLrb2d8z+Vkz+ FAxpkOgsh/jmUTEwmPoOdT1LpkR4Vc2JK8ticbvx8CYas57aXcYpPSDtQrORG0ZPd8RJ EHU9UKg2em4UlDmplwyWPwYrxPejA6O6CTNLv2cnj4j1TD3VFE75qymamN30OWSIBHRM azfmYt174I/uiH08e4KVK1m+wbEaw7K+RGLpkksA0CZKHCrn333/4k+53PGxsOP+2zA4 +Rrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Kdy5vFeElnRH1nV4Cp8wiUhmD094GQ6MJvqpTruDVec=; b=co1VY595EsvXpg5kAOOhXsXYhIDGX6LklRRAuhnZXTvsDSogN6ujhTv1Qx/Tkd0FFU xxU2SsoE19o7ZUGws52gNMNVys7NFwH4m94LIK0NtL/VQMomD9iPFRjXI/9jCfs5smUj TZjkOrhGIZs6Cwaa52dyFYlrxsWg1Il6QiAVG2q91w+g7932+qtFLCQZh48spUQcydAS PILqqGBFsQIxZ1xyqBmJDZ/IVakpp/ADddZzS/gLuQD4fw0ZD4Gb8R2TCMvDtxBt/BD9 xA519CU+vp9PuWc4fwe29ZRmzIeaplFzI24p11uPofhkncP2V3OClXCT84s893dOX6MS zAKg== X-Gm-Message-State: ACgBeo23vrW9RZCsgvM0YF6nA6uZnDMR9ptJg5ytdTPeYzHfDTNKcTRG hjtIYY7qR0CNyCy6QFAONd/urgPgWMyg+w== X-Google-Smtp-Source: AA6agR6bRbuerNmTG0CtAHJhdHLTyN5rrZww+lSEXdtQ4NMaGaEHnohqMAK2w0Rwg7K/yLApXJmobw== X-Received: by 2002:a05:600c:a46:b0:3a6:9c49:b751 with SMTP id c6-20020a05600c0a4600b003a69c49b751mr3115272wmq.169.1661979486280; Wed, 31 Aug 2022 13:58:06 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:04 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 3/9] Makefile: have "coccicheck" re-run if flags change Date: Wed, 31 Aug 2022 22:57:50 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix an issue with the "coccicheck" family of rules that's been here since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike e.g. "make grep.o" we wouldn't re-run it when $(SPATCH) or $(SPATCH_FLAGS) changed. To test new flags we needed to first do a "make cocciclean". This now uses the same (copy/pasted) pattern as other "DEFINES" rules. As a result we'll re-run properly. This can be demonstrated e.g. on the issue noted in [1]: $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1 [...] SPATCH contrib/coccinelle/xcalloc.cocci $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS="--all-includes --recursive-includes" * new spatch flags SPATCH contrib/coccinelle/xcalloc.cocci SPATCH result: contrib/coccinelle/xcalloc.cocci.patch $ 1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/ Signed-off-by: Ævar Arnfjörð Bjarmason --- .gitignore | 1 + Makefile | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/.gitignore b/.gitignore index 80b530bbed2..e67085f6918 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ /GIT-PERL-HEADER /GIT-PYTHON-VARS /GIT-SCRIPT-DEFINES +/GIT-SPATCH-DEFINES /GIT-USER-AGENT /GIT-VERSION-FILE /bin-wrappers/ diff --git a/Makefile b/Makefile index 42a70eeaef4..55ae3a2feda 100644 --- a/Makefile +++ b/Makefile @@ -1300,6 +1300,18 @@ SANITIZE_ADDRESS = SPATCH_FLAGS = --all-includes SPATCH_BATCH_SIZE = 1 +# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change +TRACK_SPATCH_DEFINES = +TRACK_SPATCH_DEFINES += $(SPATCH) +TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) +TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) +GIT-SPATCH-DEFINES: FORCE + @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ + if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ + echo >&2 " * new spatch flags"; \ + echo "$$FLAGS" >GIT-SPATCH-DEFINES; \ + fi + include config.mak.uname -include config.mak.autogen -include config.mak @@ -3146,6 +3158,7 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) +$(COCCI_PATCHES): GIT-SPATCH-DEFINES $(COCCI_PATCHES): $(COCCI_SOURCES) $(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ @@ -3169,6 +3182,7 @@ $(COCCI_PATCHES): %.patch: % fi COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) +$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci From patchwork Wed Aug 31 20:57:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D753ECAAD1 for ; Wed, 31 Aug 2022 20:58:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232027AbiHaU6X (ORCPT ); Wed, 31 Aug 2022 16:58:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231771AbiHaU6J (ORCPT ); Wed, 31 Aug 2022 16:58:09 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 027EEDC090 for ; Wed, 31 Aug 2022 13:58:09 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id i188-20020a1c3bc5000000b003a7b6ae4eb2so234004wma.4 for ; Wed, 31 Aug 2022 13:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=DrsUosIIpIajqDEOwhBDnv/TgRiUvuDA2kKSf4E3zFk=; b=fVgdSJQJu58rCBowtiWFskU3j6hmBfzZaT3JC7t6+BP1hKF4icSg7YVz1a0LYbuXee sO9Vela05sd40G+0xVwdx5K00IZpmG3jHZi9EpoLf8m5UyOJZXPlU8akf1bbz8VKj/ot 1wca+fWvscvrCcJou6iBxO6Tek4AszSxZOrMoji0oILwf8a/psJNa0s13NlH1ZSqaXOR VUL6SNUPbsObJ07fkcekuVDawAlZIBHyujT7TOWPCSQVzAP8joY00xUJ3dRW7EcD7sb7 aMk2CDWwv+Ug0zwzY2ICZmt9sQBvI2pgx1FyPdg22bLq9xmqWuk1TMX0xuWpiCqrDm8l w7JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=DrsUosIIpIajqDEOwhBDnv/TgRiUvuDA2kKSf4E3zFk=; b=JRMHLx7fmD/gZRCb6b/ho9J4um722cNE9q/HzArnto9vjPPcy0oCvp+2r09Q5CUh+S /NSdtrMx5RKh/TmdCxD9elMG8swdIUGEXo3nrPobeKy1wZGME1KIGqRVc+M4HMrdwThZ 7l1CAAvmush6Hjd0fBSDAFrAsGHltkz2zzRHQfQnSqp3KJpQoUVdptI8oweDcudAUgqw 0cxM143eamoqze28VlMmONEmTYWz0Cmozdfzdei4p99/isQlJqQmzKG2VnzhLQhah5U3 Lwd/AD7U5vFyTSC+UCr84ucSeFz6K9ttYhi1AyRgAfA5Ulh6myjyPH+aPvzHyIrx/jGA w8Tg== X-Gm-Message-State: ACgBeo2kdCYcKlS/Cb90C/F4jARHMZGeqGHJrO8tsdrgNoIXKpVDtbke 8PU0EV44n354B3e762PqdqD0vumZiDdmCw== X-Google-Smtp-Source: AA6agR4UxzsICGgAWqXJ9K30So4VcUzzDFN/5xQ1QX14aSbLGnpG/edX918GWfO3SNISWJJkHm9Jhg== X-Received: by 2002:a7b:c4d5:0:b0:3a6:161b:4d77 with SMTP id g21-20020a7bc4d5000000b003a6161b4d77mr3007288wmk.87.1661979487247; Wed, 31 Aug 2022 13:58:07 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:06 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 4/9] Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" heading Date: Wed, 31 Aug 2022 22:57:51 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Split off the "; setting[...]" part of the comment added in In 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06), and restore what we had before that, which was a comment indicating that variables for the "coccicheck" target were being set here. When 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it left no natural place to add a new comment about other flags that preceded it. As subsequent commits will add such comments we need to split the existing comment up. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 55ae3a2feda..fa2e2193e99 100644 --- a/Makefile +++ b/Makefile @@ -1294,10 +1294,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer SANITIZE_LEAK = SANITIZE_ADDRESS = -# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will +# For the 'coccicheck' target +SPATCH_FLAGS = --all-includes +# Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_FLAGS = --all-includes SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change From patchwork Wed Aug 31 20:57:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4C58ECAAD4 for ; Wed, 31 Aug 2022 20:58:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232034AbiHaU60 (ORCPT ); Wed, 31 Aug 2022 16:58:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40386 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231837AbiHaU6L (ORCPT ); Wed, 31 Aug 2022 16:58:11 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB115D398C for ; Wed, 31 Aug 2022 13:58:09 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id az24-20020a05600c601800b003a842e4983cso294248wmb.0 for ; Wed, 31 Aug 2022 13:58:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=huE2oZFXgMg3JaRAnVD36R6fKzt6QOzSgs5fCZyEW8A=; b=oCjd4zUWOCHmerHL/5BXs+7OKgmH8vcdkHHIxKwBQ57cUa3KZf+/Vsor1/k5cKEtoa dcopnyKcvccRusuPYozNe/ZR1uAldiW+eBKD8Uv44YJOCkcgM5uYHcWgKJq2oLNbEDt9 Kzl0Z3vLYcVycYUX2DBHynfH27kZZEZ5Uq7/Rsk7fGI42bPNfuUYRVTUBBh5SwXqVsst J/JVAdo19VHFSyZqf4ZuDhzlgGSaGB7K9LtHsTMmf79qMavXl/8fwqGcBqie4F4hz3zm zkhmwaD8Wtf1UQhgdpQRcAZBtkGN5CbCVv5Dr+IuPO+oiwg9sBzhDzViYcKXcWJRmjQ2 ADsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=huE2oZFXgMg3JaRAnVD36R6fKzt6QOzSgs5fCZyEW8A=; b=MlQ4mcK/wP+Tl5oZ9Zoq8eQA8Z7EDrvj09oldtVy5s7v4uvfjMqimBGo7+vzi7n4DB vVPpTfcUqLYxtAAaiD6FTDILhbQBRiNEtXslwBXTQtxyXIXIw8pU60rL60y5QBELN9KO nIzQ4WLPF72dsU6125cuJGo1j4/BL3L8MTCtdS9c0mKRNBMICqBoben3VgMyuefQud1f XABa5qClRs2a1iHgm7kzH4+ofQ9U8xaI06p3Cy2IglUnGjrzZpfi/B/+r4C/k1VKC63f Oktwo9TVNO0IZi4CfTaa/w3uU6/UCYNcMRZoA5/FM0TyfE94+uh0ZlWX5+oYbQYK1co3 Q6bg== X-Gm-Message-State: ACgBeo0UEdzKIx+JINBLZ5rxjlUI85zKfZkmRvFuOoyphHYRImAmur6n 4GouJdQ3GqgVW8trkoGvg1gqmxI3Lil4Jg== X-Google-Smtp-Source: AA6agR7mgHnmRbD9hV/i0C1fessgEw1TepPV6kc5UDCPPYGk/2g4mfI10vzYJzK5rrJ+WxVr9NwVaw== X-Received: by 2002:a1c:7708:0:b0:3a5:5543:cec4 with SMTP id t8-20020a1c7708000000b003a55543cec4mr3136676wmi.47.1661979488094; Wed, 31 Aug 2022 13:58:08 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:07 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 5/9] cocci: split off include-less "tests" from SPATCH_FLAGS Date: Wed, 31 Aug 2022 22:57:52 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Amend the "coccicheck-test" rule added in f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) to stop using "--all-includes". The flags we'll need for the tests are different than the ones we'll need for our main source code. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index fa2e2193e99..143c012b3be 100644 --- a/Makefile +++ b/Makefile @@ -1296,6 +1296,7 @@ SANITIZE_ADDRESS = # For the 'coccicheck' target SPATCH_FLAGS = --all-includes +SPATCH_TEST_FLAGS = # Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. @@ -1305,6 +1306,7 @@ SPATCH_BATCH_SIZE = 1 TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) +TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ @@ -3188,7 +3190,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci $(call mkdir_p_parent_template) - $(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \ + $(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_TEST_FLAGS) \ --very-quiet --no-show-diff \ --sp-file $< -o $@ \ $(@:.build/%.res=%.c) && \ From patchwork Wed Aug 31 20:57:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40C0FECAAD5 for ; Wed, 31 Aug 2022 20:58:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232058AbiHaU62 (ORCPT ); Wed, 31 Aug 2022 16:58:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231881AbiHaU6M (ORCPT ); Wed, 31 Aug 2022 16:58:12 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7C71D3EFC for ; Wed, 31 Aug 2022 13:58:10 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id d12-20020a05600c34cc00b003a83d20812fso276015wmq.1 for ; Wed, 31 Aug 2022 13:58:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=W1RwiYz6EWFBsHYila6NAStlnA/WTwounioX2F6DTIQ=; b=VG00ufZDrp+r0vb2nnDecmj3A63ur97azzYE2mfXx+KgZWjW9kbVduprgia4vz6NUJ /5SFIkjVp6h2riVNojGxze6aDH49N2IxoD1d/2+PW1Sv+woqDJsDI9TTC35HjZOiXEAx I/nbpwvMUE/ES2sVC0DSLIhcD6IzjIlMbJ4UVxwYwSttDAOAQHs0VgIbYVOxy/+WHqqi fEgYYKo1epESX/kVNZpA9yjIPFTGkgUVKDJ+N3T5cudRmd5Y4zpVc6fvEA8fAGjaY6K5 0Z1f8bEDbXpENS3uh70YcgvT3dT7bTWmj+fZOEdqUmC1l9Wr9VfL9jy+CVE2hAjhyTtm qsXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=W1RwiYz6EWFBsHYila6NAStlnA/WTwounioX2F6DTIQ=; b=71CvVgsPe8RyOWM2xpdrim2hMvmJJRVbAz2DRnkHrsQBox8Xg5Fl1/cB3UYO3bkzJa 771T9fupQuXTGBXpp5gvm4h6e6+gsAlM/XbjesxhDj9WEeLgr6zVIOM7eE8Cz7ZQ1YJW K3eI+NyCcXMsbSRiWM8/rNWj2YISNJRGAwAWPjcowpvXySwbW/3/oow7rYu8TRk05Mmp 3rHCWY9XdeDN9fcrp5GVUNJOmvqnLcp9rVHuqzlp3Ka2wBCqNLvycoqXCMDklN78B80B JldEQLLrV5zy2BuIGinOuMVOaBDRCpKp6pklNIcijJ9QduLTNiSz1yqp8OWRvVdhPw5v edXA== X-Gm-Message-State: ACgBeo3EE4eRBC2PC7GGcKX+4TWvzcyf4flJuSrPjFwZwk9uP6DMvYJn LOZbq6X54/lXJzFs5kKFSVJt5yXupGYKnA== X-Google-Smtp-Source: AA6agR7oIXFGdazekIHl9Yks0gnpcyLpVqHrl2m4rEL8MAwVTvFUwwulQfx0WyRe9Nkg/aOTYP8t0w== X-Received: by 2002:a05:600c:4f89:b0:3a6:243d:3bbe with SMTP id n9-20020a05600c4f8900b003a6243d3bbemr3102735wmq.16.1661979489116; Wed, 31 Aug 2022 13:58:09 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:08 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 6/9] cocci: split off "--all-includes" from SPATCH_FLAGS Date: Wed, 31 Aug 2022 22:57:53 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Per the rationale in 7b63ea57500 (Makefile: remove mandatory "spatch" arguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that are truly mandatory, such as "--sp-file" and "--patch .". The "--all-includes" flag is also critical, but per [1] we might want to ad-hoc tweak it occasionally for testing or one-offs. But being unable to set e.g. SPATCH_FLAGS="--verbose-parsing" without breaking how our "spatch" works isn't ideal, i.e. before this we'd need to know about the default include flags, and specify: SPATCH_FLAGS="--all-includes --verbose-parsing". If we were then to change the default include flag (e.g. to "--recursive-includes") in the future any such one-off commands would need to be correspondingly updated. Let's instead leave the SPATCH_FLAGS for the user, while creating a new SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include strategy itself. 1. https://lore.kernel.org/git/20220823095733.58685-1-szeder.dev@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 143c012b3be..662f950e075 100644 --- a/Makefile +++ b/Makefile @@ -1295,7 +1295,8 @@ SANITIZE_LEAK = SANITIZE_ADDRESS = # For the 'coccicheck' target -SPATCH_FLAGS = --all-includes +SPATCH_INCLUDE_FLAGS = --all-includes +SPATCH_FLAGS = SPATCH_TEST_FLAGS = # Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. @@ -1305,6 +1306,7 @@ SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) +TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) @@ -3172,6 +3174,7 @@ $(COCCI_PATCHES): %.patch: % fi; \ if ! echo $(COCCI_SOURCES) | xargs $$limit \ $(SPATCH) $(SPATCH_FLAGS) \ + $(SPATCH_INCLUDE_FLAGS) \ --sp-file $< --patch . \ >$@+ 2>$@.log; \ then \ From patchwork Wed Aug 31 20:57:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0AF3AECAAD5 for ; Wed, 31 Aug 2022 20:58:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232050AbiHaU6d (ORCPT ); Wed, 31 Aug 2022 16:58:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40442 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231939AbiHaU6O (ORCPT ); Wed, 31 Aug 2022 16:58:14 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 605B1D398C for ; Wed, 31 Aug 2022 13:58:12 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id r204-20020a1c44d5000000b003a84b160addso248527wma.2 for ; Wed, 31 Aug 2022 13:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=c4Glt1x6uv6Q9oNnGTYdqOAoPRMxxmYs1A14yd4Pntc=; b=V9PbZnOynAYMVXnVsRtzxBpXraPg+3yhfKY+ZkskEUBFumAl9CxhVo4iX7q4D5GTMW fZNQBhr3VTiI42fVsdofBZFAyr5rmsW+AfMTVkSq6WSlZHC3x8YFV6k7Q2pE20B1ZEMg cOtj5XdZA70Zy8v/JucN1TYpMc+Es52N142h0feI4PQ8P53JjM8CSot7Rc4CBdkTeM6s xECp/X02XiAv5A4oYvhIVy31KrjAfEUHo1Nxwyd9JP5wTSPqZywOg7ta57a5CmpWdCJr mZvLzYIjdfPu5nmaTe3+ZylrN9KNbFEaltz9i8/6PZ1Nfrw4t/csF0C03QjMpnC25OmB rFSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=c4Glt1x6uv6Q9oNnGTYdqOAoPRMxxmYs1A14yd4Pntc=; b=IaikSFUdktVFv3k17p+PLT+GLMfW9AyuRE9c79p4ecffTsmIifqHBeiQKyMlnBZwgB m0wPRLINaqg8gSBG2Ju9akHsgQzmAk09AX0gA4ANZ3p4QjVeWWly7iTx8YxcAw1UP5uz A0Ct4OyRmCqelLLclA3gG8fzn701X2ibeEK+2AQn8UrhoW0Gbco5Z2NzouPcv/GzIxlM Hyz2f+y1EwihWhU435bjRjY/0wRHs6ijtzsZ3hTd+2UUNoqSx5efPRcwim0Pzb249IeT rNGo/zpZves4PcEQ4UldO4vL84QveX4HYXvpIQD40zRAa4AosylkrKt6hW51z4Mbl+4x Sbcw== X-Gm-Message-State: ACgBeo3rGJApsgyLi351htNd+0o7WdHphwhCMciR42RHl/qmoSbfXYco UFF1P3qq0uF/E3eMh588c7TJl0rAjMpEBQ== X-Google-Smtp-Source: AA6agR7fkM6gxQ+8M3dxjbJN21U4WflU6k5IU6UMWUUynXapa1D0QK2LbFaubyiizJNopqPZFiinlw== X-Received: by 2002:a05:600c:502c:b0:3a8:41e9:cd32 with SMTP id n44-20020a05600c502c00b003a841e9cd32mr3017403wmr.177.1661979490230; Wed, 31 Aug 2022 13:58:10 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:09 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 7/9] cocci: make "coccicheck" rule incremental Date: Wed, 31 Aug 2022 22:57:54 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Optimize the very slow "coccicheck" target to take advantage of incremental rebuilding, and fix outstanding dependency problems with the existing rule. The rule is now faster both on the initial run as we can make better use of GNU make's parallelism than the old ad-hoc combination of make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the "--jobs" argument to "spatch(1)". It also makes us *much* faster when incrementally building, it's now viable to "make coccicheck" as topic branches are merged down. The rule didn't use FORCE (or its equivalents) before, so a: make coccicheck make coccicheck Would report nothing to do on the second iteration. But all of our patch output depended on all $(COCCI_SOURCES) files, therefore e.g.: make -W grep.c coccicheck Would do a full re-run, i.e. a a change in a single file would force us to do a full re-run. The reason for this (not the initial rationale, but my analysis) is: * Since we create a single "*.cocci.patch+" we don't know where to pick up where we left off, or how to incrementally merge e.g. a "grep.c" change with an existing *.cocci.patch. * We've been carrying forward the dependency on the *.c files since 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was initially added as a sort of poor man's dependency discovery. As we don't include other *.c files depending on other *.c files has always been broken, as could be trivially demonstrated e.g. with: make coccicheck make -W strbuf.h coccicheck However, depending on the corresponding *.c files has been doing something, namely that *if* an API change modified both *.c and *.h files we'd catch the change to the *.h we care about via the *.c being changed. For API changes that happened only via *.h files we'd do the wrong thing before this change, but e.g. for function additions (not "static inline" ones) catch the *.h change by proxy. Now we'll instead: * Create a / pair in the .build directory, E.g. for swap.cocci and grep.c we'll create .build/contrib/coccinelle/swap.cocci.patch/grep.c. That file is the diff we'll apply for that - combination, if there's no changes to me made (the common case) it'll be an empty file. * Our generated *.patch file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple "cat $^" of all of all of the / files for a given . In the case discussed above of "grep.c" being changed we'll do the full "cat" every time, so they resulting *.cocci.patch will always be correct and up-to-date, even if it's "incrementally updated". See 1cc0425a27c (Makefile: have "make pot" not "reset --hard", 2022-05-26) for another recent rule that used that technique. As before we'll: * End up generating a contrib/coccinelle/swap.cocci.patch, if we "fail" by creating a non-empty patch we'll still exit with a zero exit code. Arguably we should move to a more Makefile-native way of doing this, i.e. fail early, and if we want all of the "failed" changes we can use "make -k", but as the current "ci/run-static-analysis.sh" expects us to behave this way let's keep the existing behavior of exhaustively discovering all cocci changes, and only failing if spatch itself errors out. Further implementation details & notes: * Before this change running "make coccicheck" would by default end up pegging just one CPU at the very end for a while, usually as we'd finish whichever *.cocci rule was the most expensive. This could be mitigated by combining "make -jN" with SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06). But doing so required careful juggling, as e.g. setting both to 4 would yield 16 workers. There will be cases where getting rid of "SPATCH_BATCH_SIZE" makes things worse, but a from-scratch "make coccicheck" with the default of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference) is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed the CPU more work in a less staggered way. * Getting rid of "SPATCH_BATCH_SIZE" particularly helps in cases where the default of 1 yields parallelism under "make coccicheck", but then running e.g.: make -W contrib/coccinelle/swap.cocci coccicheck I.e. before that would use only one CPU core, until the user remembered to adjust "SPATCH_BATCH_SIZE" differently than the setting that makes sense when doing a non-incremental run of "make coccicheck". Outstanding issues & future work: * We could get rid of "--all-includes" in favor of manually specifying a list of includes to give to "spatch(1)". As noted upthread of [1] a naïve removal of "--all-includes" will result in broken *.cocci patches, but if we know the exhaustive list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to re-scan for them, we could grab the headers to include from the .depend.d/.o.d and supply them with the "--include" option to spatch(1).q 1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 83 +++++++++++++++++++++++++++++++++++++----------------- shared.mak | 3 +- 2 files changed, 59 insertions(+), 27 deletions(-) diff --git a/Makefile b/Makefile index 662f950e075..562820301c4 100644 --- a/Makefile +++ b/Makefile @@ -1298,10 +1298,6 @@ SANITIZE_ADDRESS = SPATCH_INCLUDE_FLAGS = --all-includes SPATCH_FLAGS = SPATCH_TEST_FLAGS = -# Setting SPATCH_BATCH_SIZE higher will -# usually result in less CPU usage at the cost of higher peak memory. -# Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = @@ -1309,7 +1305,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH) TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) -TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ @@ -3156,36 +3151,72 @@ check: $(GENERATED_H) exit 1; \ fi +# It's expensive to compute the many=many rules below, only eval them +# on $(MAKECMDGOALS) that match these $(COCCI_RULES) +COCCI_RULES = +COCCI_RULES += cocci% +COCCI_RULES += contrib/coccinelle/% +COCCI_RULES += .build/contrib/coccinelle/% + COCCI = $(wildcard contrib/coccinelle/*.cocci) +COCCI_NAMES = $(COCCI:contrib/coccinelle/%.cocci=%) COCCI_PATCHES = $(addsuffix .patch,$(COCCI)) COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES)) -COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -$(COCCI_PATCHES): GIT-SPATCH-DEFINES -$(COCCI_PATCHES): $(COCCI_SOURCES) -$(COCCI_PATCHES): %.patch: % - $(QUIET_SPATCH) \ - if test $(SPATCH_BATCH_SIZE) = 0; then \ - limit=; \ - else \ - limit='-n $(SPATCH_BATCH_SIZE)'; \ - fi; \ - if ! echo $(COCCI_SOURCES) | xargs $$limit \ - $(SPATCH) $(SPATCH_FLAGS) \ - $(SPATCH_INCLUDE_FLAGS) \ - --sp-file $< --patch . \ - >$@+ 2>$@.log; \ +.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) + $(call mkdir_p_parent_template) + $(QUIET_GEN) >$@ + +define cocci-rule + +## Rule for .build/$(1).patch/$(2); Params: +# $(1) = e.g. "free.cocci" +# $(2) = e.g. "grep.c" +COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) +.build/$(1).patch/$(2): GIT-SPATCH-DEFINES +.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES +.build/$(1).patch/$(2): $(1) +.build/$(1).patch/$(2): .build/$(1).patch/% : % + $$(call mkdir_p_parent_template) + $$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \ + $$(SPATCH_INCLUDE_FLAGS) \ + --sp-file $(1) --patch . $$< \ + >$$@ 2>$$@.log; \ then \ - cat $@.log; \ + echo "ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:"; \ + cat $$@.log; \ exit 1; \ - fi; \ - mv $@+ $@; \ - if test -s $@; \ + fi +endef + +define cocci-matrix-2 + +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s))) +endef +define cocci-matrix-1 +$(foreach c,$(COCCI),$(call cocci-matrix-2,$(c))) +endef + +ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),) +$(eval $(call cocci-matrix-1)) +endif + +define spatch-rule + +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) + $(QUIET_SPATCH_M)cat $$^ >$$@ && \ + if test -s $$@; \ then \ - echo ' ' SPATCH result: $@; \ + echo ' ' SPATCH result: $$@; \ fi +endef + +ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),) +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n)))) +endif COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES @@ -3476,7 +3507,7 @@ profile-clean: cocciclean: $(RM) -r .build/contrib/coccinelle - $(RM) $(COCCI_PATCHES)* + $(RM) $(COCCI_PATCHES) clean: profile-clean coverage-clean cocciclean $(RM) -r .build diff --git a/shared.mak b/shared.mak index 33f43edbf9a..d9fdd6b4484 100644 --- a/shared.mak +++ b/shared.mak @@ -69,8 +69,9 @@ ifndef V QUIET_SP = @echo ' ' SP $<; QUIET_HDR = @echo ' ' HDR $(<:hcc=h); QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_SPATCH = @echo ' ' SPATCH $@; QUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%); + QUIET_SPATCH_M = @echo ' ' SPATCH MERGE $$@; ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; From patchwork Wed Aug 31 20:57:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E95AECAAD1 for ; Wed, 31 Aug 2022 20:58:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231959AbiHaU63 (ORCPT ); Wed, 31 Aug 2022 16:58:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231936AbiHaU6O (ORCPT ); Wed, 31 Aug 2022 16:58:14 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C639DDC0A4 for ; Wed, 31 Aug 2022 13:58:12 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id m16so19773336wru.9 for ; Wed, 31 Aug 2022 13:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=aVntDdu/R532+e6kQTUwlzZ9vCj8nZYJDEwRHN1bdRE=; b=p8ZV/+BOzwQV9cR0bxRCzk0N9/LKS5JHuTfdO6EIgqIVA7m8YO/wPc7g7vEUVzbMpI H97NZvdpr7UBqHo0OGbfRdODUpRVBOsAw08eahsGg2Nnh76II6LWCQv4KsNozR/ILhFv MmiQ/mzZCprrKpu4cbsQKYXP3ZUtkdqH5hIam/K8cWHFnMBqE4FFQYkOY930fAEItlAP Tzwvk5Dp/nW0lOrdijCdj5icdrCuChQdnBlAajQJmQvI+DshhFl1PzO8Ly2yxbdxHWGK skbsJb4Rw7SFYahbVsBXrycOzXHCA3//GuP2RVC3/L3dLUsdVub3cVhr6D8CRXlNOJDX LOQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=aVntDdu/R532+e6kQTUwlzZ9vCj8nZYJDEwRHN1bdRE=; b=fn49DvRIy4ApsStQYMS6B+mz7vGqfYbAK6JnrQf7RFsa5din9RBU+mHG7DoDNaVXtM hUXh+3Rgko9ZgpiRFsKm+gSShunhSYExMgfpVtHlra2GBORdaseJdMAXAWNv+n7Q7QuE k8C12p7kVuhnM9GAGkkMavAOenTXuwpxihkav29aKLxd5F2H9XO11w/s0IamS2ZNN/ow aWSeHuR12lXFNkrNi51dhPGD7X6bccuyNcC9eGtfMRNVRyMEgjWMUTn9KHnpbHUIBhER tG55svbfohGZ9JOF0/HSQvGH3C9BfQnmjfizePXNS8M16FACgLvFfa6FVcAMmr7Mb+w0 7Xig== X-Gm-Message-State: ACgBeo2HbYtOQ6js3smdpI14I64UYfJzCUcoyX1KKhZu9nmmgVx/eIMU JPJo8l1UaqyDKJqrxi/5qkm6ppdUgzt1dQ== X-Google-Smtp-Source: AA6agR76Nl9PjMck6ChYqjJqiwrerr2onObSZ1y/fN7GhG2XiK1rOr5KEvs396+282vr4CUuMVVPXQ== X-Received: by 2002:a5d:6a85:0:b0:226:d8d9:20ab with SMTP id s5-20020a5d6a85000000b00226d8d920abmr10001354wru.514.1661979490934; Wed, 31 Aug 2022 13:58:10 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:10 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 8/9] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES Date: Wed, 31 Aug 2022 22:57:55 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Improve the incremental rebuilding support of "coccicheck" by piggy-backing on the computed dependency information of the corresponding *.o file, rather than rebuilding all / pairs if either their corresponding file changes, or if any header changes. This in effect uses the same method that the "sparse" target was made to use in c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23), except that the dependency on the *.o file isn't a hard one, we check with $(wildcard) if the *.o file exists, and if so we'll depend on it. This means that the common case of: make make coccicheck Will benefit from incremental rebuilding, now changing e.g. a header will only re-run "spatch" on those those *.c files that make use of it: By depending on the *.o we piggy-back on COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that for the *.sp files. E.g.: make contrib/coccinelle/free.cocci.patch make -W column.h contrib/coccinelle/free.cocci.patch Will take around 15 seconds for the second command on my 8 core box if I didn't run "make" beforehand to create the *.o files. But around 2 seconds if I did and we have those "*.o" files. Notes about the approach of piggy-backing on *.o for dependencies: * It *is* a trade-off since we'll pay the extra cost of running the C compiler, but we're probably doing that anyway. The compiler is much faster than "spatch", so even though we need to re-compile the *.o to create the dependency info for the *.c for "spatch" it's faster (especially if using "ccache"). * There *are* use-cases where some would like to have *.o files around, but to have the "make coccicheck" ignore them. See: https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/ For those users a: make make coccicheck SPATCH_USE_O_DEPENDENCIES= Will avoid considering the *.o files. * If that *.o file doesn't exist we'll depend on an intermediate file of ours which in turn depends on $(FOUND_H_SOURCES). This covers both an initial build, or where "coccicheck" is run without running "all" beforehand, and because we run "coccicheck" on e.g. files in compat/* that we don't know how to build unless the requisite flag was provided to the Makefile. Most of the runtime of "incremental" runs is now spent on various compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and therefore conflate whether we *can* compile an object and generate dependency information for it with whether we'd like to link it into our binary. Before this change the distinction didn't matter, but now one way to make this even faster on incremental builds would be to peel those concerns apart so that we can see that e.g. compat/mmap.c doesn't depend on column.h. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 562820301c4..9e2ef8f0dc4 100644 --- a/Makefile +++ b/Makefile @@ -3170,14 +3170,20 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ +SPATCH_USE_O_DEPENDENCIES = yes +ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no) +SPATCH_USE_O_DEPENDENCIES = +endif + define cocci-rule ## Rule for .build/$(1).patch/$(2); Params: # $(1) = e.g. "free.cocci" # $(2) = e.g. "grep.c" +# $(3) = e.g. "grep.o" COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) .build/$(1).patch/$(2): GIT-SPATCH-DEFINES -.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES +.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) .build/$(1).patch/$(2): $(1) .build/$(1).patch/$(2): .build/$(1).patch/% : % $$(call mkdir_p_parent_template) @@ -3194,7 +3200,7 @@ endef define cocci-matrix-2 -$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s))) +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o))) endef define cocci-matrix-1 $(foreach c,$(COCCI),$(call cocci-matrix-2,$(c))) From patchwork Wed Aug 31 20:57:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12961422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 578ADECAAD1 for ; Wed, 31 Aug 2022 20:58:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232052AbiHaU6e (ORCPT ); Wed, 31 Aug 2022 16:58:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231991AbiHaU6P (ORCPT ); Wed, 31 Aug 2022 16:58:15 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09253F2410 for ; Wed, 31 Aug 2022 13:58:14 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id az27so19804225wrb.6 for ; Wed, 31 Aug 2022 13:58:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=bx45SRo/L19bzgeqWcj1onO/DutGAnmIgp6nsnVuDrU=; b=H8GWyaqxXaRc69tQQC22iAVANLPLmBDBL2MK2RTfvY6MqSupAl929me/l311AXwFjc Kx9f8oKKCUbTmLSyIdsCK6DFEPbrIdNMgdJ9tjlEZOmh4pagFSHgUTUGzb4a6ll+mMfW wZSSfjuQVQx6OSAgMtkb9EUwHWxszByq8tmURgbo8NYR4LXeTGX+TXCNenfHZspmZmTo Mi6394DFvitLRyqJBb92Gsxgauv2h8CM/Jy8uvo1TNZAKzCIM4dtYqHviEDTKLxHOfvn loWItASDFk5/RZNpxjtHsKWmWuNwD1PsKZ98zftrsmV8lSY5R74hgKGM9PyUpt1fXpjO tQRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=bx45SRo/L19bzgeqWcj1onO/DutGAnmIgp6nsnVuDrU=; b=6xJYkCwGQyG6JQNNh9T+960zF4umiWCBglggOsmVmwgnacO7eJg0jMD2xe7GPapqkM N2RHNd0lw7MLpGBAtTh0Dmih17VMWCt2+Dwce85E9y7wjUq6PWZG5BaF2tpDJ012UohH mx3BV/0K9vAAc/dTgVwu6MsSQJn3GrNldm7zidRcpt8CwrxO7fMLMETqXDA4LyfT5wmT 52LZFzCd3jJRp3yfO1yctDsWcOBNFR9ZOOVQw18fhehjhzmUfT0hhNa+f/G/ZbVMpvBz KWPusOJqsbFTXchNhn86Y7m2pGhkGHiLJWt5MgaQtuLcNZyYgx+UqvBbY8tCGsW3RuXs IriQ== X-Gm-Message-State: ACgBeo1rGFUSKykEFD3s9ntorao+zo3/fy3hOFcz0t/79GQeHj6AXkf8 3UjUeva/hHdjErJTdh8+9SUhtW0zkb9rtQ== X-Google-Smtp-Source: AA6agR6DAaBiBsc5Cdi7TxhJFEcXdgxuvUEBjbhYjtbZLHkANPE+fKrnB+bzQEtDBN6KdSqvgZ/xew== X-Received: by 2002:a5d:460a:0:b0:225:86c6:6e1 with SMTP id t10-20020a5d460a000000b0022586c606e1mr12834253wrq.117.1661979492066; Wed, 31 Aug 2022 13:58:12 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002253af82fa7sm15766955wrr.9.2022.08.31.13.58.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 13:58:11 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 9/9] spatchcache: add a ccache-alike for "spatch" Date: Wed, 31 Aug 2022 22:57:56 +0200 Message-Id: X-Mailer: git-send-email 2.37.3.1420.g76f8a3d556c In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a rather trivial "spatchcache", with this running e.g.: make cocciclean make contrib/coccinelle/free.cocci.patch \ SPATCH=contrib/coccicheck/spatchcache \ SPATCH_FLAGS=--very-quiet Is cut down from ~20s to ~5s on my system. Much of that is either fixable shell overhead, or the around 40 files we "CANTCACHE" (see the implementation). This uses "redis" as a cache by default, but it's configurable. See the embedded documentation. This is *not* like ccache in that we won't cache failed spatch invocations, or those where spatch suggests changes for us. Those cases are so rare that I didn't think it was worth the bother, by far the most common case is that it has no suggested changes. We'll also refuse to cache any "spatch" invocation that has output on stderr, which means that "--very-quiet" must be added to "SPATCH_FLAGS". Because we narrow the cache to that we don't need to save away stdout, stderr & the exit code. We simply cache the cases where we had no suggested changes. Another benchmark is to compare this with the previous SPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running: make clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0 Would take 33s, but with the preceding changes running without this "spatchcache" is slightly slower, or around 35s: make clean; time make contrib/coccinelle/array.cocci.patch Now doing the same with SPATCH=contrib/coccinelle/spatchcache will take around 6s, but we'll need to compile the *.o files first to take full advantage of it (which can be fast with "ccache"): make clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache 1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/ Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/spatchcache | 234 +++++++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+) create mode 100755 contrib/coccinelle/spatchcache diff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache new file mode 100755 index 00000000000..f0b7b24e0ae --- /dev/null +++ b/contrib/coccinelle/spatchcache @@ -0,0 +1,234 @@ +#!/bin/sh +# +# spatchcache: a poor-man's "ccache"-alike for "spatch" in git.git +# +# This caching command relies on the peculiarities of the Makefile +# driving "spatch" in git.git, in particular if we invoke: +# +# make +# make coccicheck SPATCH_FLAGS=--very-quiet +# +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with +# "gcc" and "clang") write e.g. a .depend/grep.o.d for grep.c, when we +# compile grep.o. +# +# The .depend/grep.o.d will have the full header dependency tree of +# grep.c, and we can thus cache the output of "spatch" by: +# +# 1. Hashing all of those files +# 2. Hashing our source file, and the *.cocci rule we're +# applying +# 3. Running spatch, if suggests no changes (by far the common +# case) we invoke "spatchCache.getCmd" and +# "spatchCache.setCmd" with a hash SHA-256 to ask "does this +# ID have no changes" or "say that ID had no changes> +# 4. If no "spatchCache.{set,get}Cmd" is specified we'll use +# "redis-cli" and maintain a SET called "spatch-cache". Set +# appropriate redis memory policies to keep it from growing +# out of control. +# +# This along with the general incremental "make" support for +# "contrib/coccinelle" makes it viable to (re-)run coccicheck +# e.g. when merging integration branches. +# +# Note that the "--very-quiet" flag is currently critical. The cache +# will refuse to cache anything that has output on STDERR (which might +# be errors from spatch). The STDERR (and exit code) could in +# principle be cached (as with ccache), but then the simple structure +# in the Redis cache would need to change, so just supply +# "--very-quiet" for now. +# +# To use this, simply set SPATCH to +# contrib/coccinelle/spatchcache. Then optionally set: +# +# [spatchCache] +# # Optional: path to a custom spatch +# spatch = ~/g/coccicheck/spatch.opt +# +# As well as this trace config (debug implies trace): +# +# trace = false +# debug = false +# +# Setting "trace" to "true" allows for seeing when we have a cache HIT +# or MISS. To debug whether the cache is working do that, and run e.g.: +# +# redis-cli FLUSHALL +# +# grep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c +# 600 CANTCACHE +# 7365 MISS +# 7365 SET +# +# A subsequent "make cocciclean && make coccicheck" should then have +# all "HIT"'s and "CANTCACHE"'s. + +set -e + +## Our own configuration & options +debug=$(git config --bool "spatchCache.debug") +if test "$debug" != "true" +then + debug= +fi +if test -n "$debug" +then + set -x +fi + +trace=$(git config --bool "spatchCache.trace") +if test "$trace" != "true" +then + trace= +fi +if test -n "$debug" +then + # debug implies trace + trace=true +fi + +trace_it () { + if test -z "$trace" + then + return + fi + echo "$@" >&2 +} + +spatch=$(git config --path "spatchCache.spatch" || :) +if test -n "$spatch" +then + if test -n "$debug" + then + trace_it "custom spatchCache.spatch='$spatch'" + fi +else + spatch=spatch +fi + +set=$(git config spatchCache.setCmd || :) +get=$(git config spatchCache.getCmd || :) + +## Parse spatch()-like command-line for caching info +arg_sp= +arg_file= +args="$@" +spatch_opts() { + while test $# != 0 + do + arg_file="$1" + case "$1" in + --sp-file) + arg_sp="$2" + ;; + esac + shift + done +} +spatch_opts "$@" +if ! test -f "$arg_file" +then + arg_file= +fi + +hash_for_cache() { + # Parameters that should affect the cache + echo "spatch=$spatch" + echo "args=$args" + echo + + # Our target file and its dependencies + git hash-object "$1" "$2" $(grep -E -o '^[^:]+:$' "$3" | tr -d ':') +} + +# Sanity checks +if ! test -f "$arg_sp" && ! test -f "$arg_file" +then + echo $0: no idea how to cache "$@" >&2 + exit 128 +fi + +# Main logic +d=$(dirname "$arg_file") +b=$(basename "$arg_file") +bwoext="${b%.c}" +dep="$d/.depend/$bwoext.o.d" + +if ! test -f "$dep" +then + trace_it "$0: CANTCACHE have no '$dep' for '$arg_file'!" + exec "$spatch" "$@" +fi + +if test -n "$debug" +then + trace_it "$0: The full cache input for '$arg_sp' '$arg_file' '$dep'" + hash_for_cache "$arg_sp" "$arg_file" "$dep" >&2 +fi +sum=$(hash_for_cache "$arg_sp" "$arg_file" "$dep" | git hash-object --stdin) + +trace_it "$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'" + +getret= +if test -z "$get" +then + if test $(redis-cli SISMEMBER spatch-cache "$sum") = 1 + then + getret=0 + else + getret=1 + fi +else + $set "$sum" + getret=$? +fi + +if test "$getret" = 0 +then + trace_it "$0: HIT for '$arg_file' with '$arg_sp'" + exit 0 +else + trace_it "$0: MISS: for '$arg_file' with '$arg_sp'" +fi + +out="$(mktemp)" +err="$(mktemp)" + +set +e +"$spatch" "$@" >"$out" 2>"$err" +ret=$? +set -e + +if test $ret = 0 && ! test -s "$out" && ! test -s "$err" +then + rm -f "$out" "$err" + + trace_it "$0: SET: for '$arg_file' with '$arg_sp'" + + setret= + if test -z "$set" + then + if test $(redis-cli SADD spatch-cache "$sum") = 1 + then + setret=0 + else + setret=1 + fi + else + "$set" "$sum" + setret=$? + fi + + if test "$setret" != 0 + then + echo "FAILED to set '$sum' in cache!" >&2 + exit 128 + fi +else + trace_it "$0: NOCACHE (have changes): for '$arg_file' with '$arg_sp'" + cat "$out" + cat "$err" >&2 + rm -f "$out" "$err" + exit "$ret" +fi +rm -f "$out" "$err"