From patchwork Fri Oct 14 15:31:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53235C433FE for ; Fri, 14 Oct 2022 15:32:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230458AbiJNPcR (ORCPT ); Fri, 14 Oct 2022 11:32:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59652 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230273AbiJNPbj (ORCPT ); Fri, 14 Oct 2022 11:31:39 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FC145B516 for ; Fri, 14 Oct 2022 08:31:35 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id b4so8169698wrs.1 for ; Fri, 14 Oct 2022 08:31:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lLbmUgtl+mF/6MvyNI8rejJ8ApT7XS3QJDqdENaeE5I=; b=Z+gZVgErVGVPxyvQsr7dTPl5bXprI4izcnph367zsod9Jz9xX0hqawPMGDJdJQTt0u ydciEb89H4V5p6zhk0UulMpDRbUiDEKv6Xx/s/h8X6EzcE0vnHnus4RS8F3YUHT3z9/1 BoU/rBk+tTF9Q64+ax59+N+9IHHZQekVpKBTKaRQjyRc+HBGWqqTzZBds50mUGHkZLre ucqi++eDE0IxdM760vkhhFZeT10YB2/k/tfZR0Eizo3E6TSDd+3jmNmVre4wf5GlRwJm bcGrenG/6RRyVYa9HQvPsUXyv5W8ypLc36INL4bl7DxJxD+HFy4wRUFIdF//zfjf2Ivc zdRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lLbmUgtl+mF/6MvyNI8rejJ8ApT7XS3QJDqdENaeE5I=; b=VvMvj49tNK8y4ydsfuh/8Xk9rMZcPlqd6cj/7+8y5eMhhYNMy+dW5PWSpvPGrz9iL5 kdmoYHVrccvE7lykf/87TllIZNhicELENy+hO7i08ALtMFuQIGj/D2wtGSloLUSLO25k Lg5NevrqJkPfsUwBRvQAAyksFxNIHvOGD8aUgvX4ev5EbPNXTxCoIX4C1b3ssf08uJvc ftv1VZ2DvgbasTem+DFLgRcEbKqj/fZbPpuLrpkutHsqt6QD/zcIb366qCS9lC2NYrn/ FGq7pkrpsHXmsX2wRo/2up1WxUWpwbKOKoVOqpP89pMaDDac8WumDfJK8QcSczHNdPqL Gvdw== X-Gm-Message-State: ACrzQf00at27iYmntQ6VKZnKkGJO4PiuqhzBv/WQJDKH2yLX0av61ffy n6Pay7VRN1v8pJfiXr5ay64TdqPbfpNerA== X-Google-Smtp-Source: AMsMyM4OUjewNF4xf4eSukW8abvnRdMQlWm/6pVr41iucECIL9DjwZ1Rfa2AdQsp6bomymfDy4vFkw== X-Received: by 2002:a05:6000:1845:b0:22f:bfca:6439 with SMTP id c5-20020a056000184500b0022fbfca6439mr3826444wri.298.1665761492269; Fri, 14 Oct 2022 08:31:32 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:31 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 01/11] Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T) Date: Fri, 14 Oct 2022 17:31:17 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) we abbreviated "_TEST" to "_T" to have it align with the rest of the "="'s above it. Subsequent commits will add more QUIET_SPATCH_* variables, so let's stop abbreviating this, and indent it in preparation for adding more of these variables. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 2 +- shared.mak | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 6bfb62cbe94..f641adab2fd 100644 --- a/Makefile +++ b/Makefile @@ -3164,7 +3164,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci $(call mkdir_p_parent_template) - $(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \ + $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \ --very-quiet --no-show-diff \ --sp-file $< -o $@ \ $(@:.build/%.res=%.c) && \ diff --git a/shared.mak b/shared.mak index 33f43edbf9a..96b06acc455 100644 --- a/shared.mak +++ b/shared.mak @@ -69,8 +69,10 @@ ifndef V QUIET_SP = @echo ' ' SP $<; QUIET_HDR = @echo ' ' HDR $(<:hcc=h); QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; - QUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%); + +## Used in "Makefile": SPATCH + QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_SPATCH_TEST = @echo ' ' SPATCH TEST $(@:.build/%=%); ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; From patchwork Fri Oct 14 15:31:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6328EC433FE for ; Fri, 14 Oct 2022 15:32:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230487AbiJNPcT (ORCPT ); Fri, 14 Oct 2022 11:32:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230176AbiJNPbl (ORCPT ); Fri, 14 Oct 2022 11:31:41 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 881611D2F5B for ; Fri, 14 Oct 2022 08:31:36 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id w18so8120860wro.7 for ; Fri, 14 Oct 2022 08:31:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JD8xRBw9YOj6NbUOLtIRD7AfmY+vNrx4gJNIXPJPBDQ=; b=igP13j2v02gWGDAUiNn13/cuhEggXSF8gVdt4RKPD7VfLob0dHhSevgMbDsIDXg+qW FCq6HmO5QUoO2KC3MiTdgKMp2B9MPxCFT71h5Bk6uJukyOPfm6Q5M2+CvuI4zC5oxOCH WXVcdTZfGvql5flGranT6dAmOALRBFBZLbKlpG/fn/trFGzRUork+RbixMkulXcJ4A6I M/7AYzIxSIsR2YfMySvYj5TV+GE4SATOBVwKAyrKMuoJ7BfskawFyIXLyTHXY8wOFY9G RBc2rU3YILU3799yGBPLvzCQNmq4M+hvN3j8UnnrC6pnz5m2X/vT2sdsNECjCa+gvDxw zjyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JD8xRBw9YOj6NbUOLtIRD7AfmY+vNrx4gJNIXPJPBDQ=; b=PHzSEIhXRy3coqB+d60T5MpWZZm392W/u4rqtWVzGyHUr7hLa8UTvGkllKSgyxWirN vCIEhZwviUE/RjA9JN0qaitOefGGBc+6Kp+WFhfJpInB4jpc71tesY0O7USNKIij61Jj YXld7zSs0LmviA6ck+MVy43t3zemSYKOEUi03enBZeHb7ggOcyxyhsNpyh/DGA3x56Ch SPpHVFZ0AVtBV5P6iwCHJS/wmkLXkuEPVkZ2CANxhxGP7emKi1Sc0LH88HGp7EGwt4MR NChTOGAw26ml5oMsSRqKt1Orkk3WJ8nxDEnZ10A4fUif4uXE/Yr4RDmbyINZkq3P3ipr f/iA== X-Gm-Message-State: ACrzQf2HF54qWgNr8or3WfKEIpGsR+H7ZmE+ATkZNBnHmUJuHYhiLYEE XLA13FX/mjYHC2kh326Mls8RFc0e8Gy7ag== X-Google-Smtp-Source: AMsMyM7taBj7edjhG04PmANs0+zK8EOFu5SBFuSGopcTXezSqmkippU/ckaj+Jj7gULuvAv8ZPPZXA== X-Received: by 2002:a5d:4748:0:b0:22e:32c1:d11e with SMTP id o8-20020a5d4748000000b0022e32c1d11emr3863389wrs.672.1665761493902; Fri, 14 Oct 2022 08:31:33 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:32 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 02/11] cocci rules: remove unused "F" metavariable from pending rule Date: Fri, 14 Oct 2022 17:31:18 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix an issue with a rule added in 9b45f499818 (object-store: prepare has_{sha1, object}_file to handle any repo, 2018-11-13). We've been spewing out this warning into our $@.log since that rule was added: warning: rule starting on line 21: metavariable F not used in the - or context code We should do a better job of scouring our coccinelle log files for such issues, but for now let's fix this as a one-off. Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/the_repository.pending.cocci | 1 - 1 file changed, 1 deletion(-) diff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci index 072ea0d9228..747d382ff5f 100644 --- a/contrib/coccinelle/the_repository.pending.cocci +++ b/contrib/coccinelle/the_repository.pending.cocci @@ -20,7 +20,6 @@ expression E; @@ expression E; -expression F; @@ - has_object_file_with_flags( + repo_has_object_file_with_flags(the_repository, From patchwork Fri Oct 14 15:31:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35C8FC433FE for ; Fri, 14 Oct 2022 15:32:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231176AbiJNPcY (ORCPT ); Fri, 14 Oct 2022 11:32:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230484AbiJNPbn (ORCPT ); Fri, 14 Oct 2022 11:31:43 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 464EC1D52C8 for ; Fri, 14 Oct 2022 08:31:37 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id bp11so8115919wrb.9 for ; Fri, 14 Oct 2022 08:31:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ol2ZmjzxwxF4VR7SkQQMjvXiqxpqml71A92pk7pG/Ww=; b=WVnn14JaFt7jtnVanIJmcZpwDHg1GKy58kooRIYKGwJ5QbDerzIvPAJruURNUk5DeX rNxSegCvgcZu/4egcQ5AOVI10rPL+6qvAaJtiI+hVaR6tI/UKGU9LslODBsy5jcUsdrN mB5OxX/802VHwA72KlHS+w+qrnrynP/fxloro7RsGfUfrnG0mqSk7vEN7Pi1FYOu6/EO JfUp7S1ZjOQo7edGzJ6+fVmk8KXgf1xGorIIMLu/2OJqpJfT4AkW0y2+eAXhVdBdzASg rNfS71knnpLh4o4O2l70OPxSPpwYWQFPmaDxhQEBun5jJ15tdptPWKH3NNlnummfbtFo Xfbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ol2ZmjzxwxF4VR7SkQQMjvXiqxpqml71A92pk7pG/Ww=; b=3hRcl95rs1KdFRyNx+uhlEYzhULaa7bkDlAMkDCQ/nM6FdcUYX/7aLm3SVGYTeWP0w RPi2qw4ELScKqprJaYhslzxaABzUnXtE6QNEhbqfqiz5YLXaF78zRp1LrZ9Qv9wAbr4a U5q3nAIOq21IRD+6ein7yEMaXidTeAdlguYDolj3qwHSR1hBRBYvSCQclue/zeKL8RFi +FWBq6a45uXi7QrgrnLbDlVL4+gE9VMKGG1vEcNGMIz5ASO5wiYHzPvh+X+H646cVVJe L4YAG/9RF2OasYgmRKmek8hdgKiksBpea5P9Cv1t2ftvEJ5kmROoWagz3SQhKLo3MhGA C+ow== X-Gm-Message-State: ACrzQf2J5hCexRw7Gs9dVW0MLb5se9e7zOesszaUqvNMwtH5n0gxOYUd +XnzWgI2CjSXWcbF4F4EMm6fqpkjyNLgmg== X-Google-Smtp-Source: AMsMyM6csYGO/sCppHOhAl04mFBTTnhb7gcc23rBPeKVgZEA7UjaSH2vHjrsdgxQaQV0/vdsw1pQjQ== X-Received: by 2002:a05:6000:112:b0:22e:58ce:40d4 with SMTP id o18-20020a056000011200b0022e58ce40d4mr3807484wrx.193.1665761494962; Fri, 14 Oct 2022 08:31:34 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:34 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 03/11] Makefile: add ability to TAB-complete cocci *.patch rules Date: Fri, 14 Oct 2022 17:31:19 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Declare the contrib/coccinelle/.cocci.patch rules in such a way as to allow TAB-completion, and slightly optimize the Makefile by cutting down on the number of $(wildcard) in favor of defining "coccicheck" and "coccicheck-pending" in terms of the same incrementally filtered list. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index f641adab2fd..4647f317ede 100644 --- a/Makefile +++ b/Makefile @@ -3136,9 +3136,20 @@ check: $(GENERATED_H) exit 1; \ fi +COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) +COCCI_RULES = $(COCCI_GLOB) + +COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES)) +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) + +COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch) +COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) + COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -%.cocci.patch: %.cocci $(COCCI_SOURCES) +COCCI_PATCHES = $(COCCI_RULES:%=%.patch) +$(COCCI_PATCHES): $(COCCI_SOURCES) +$(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \ limit=; \ @@ -3175,11 +3186,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test -coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci))) +coccicheck: $(COCCICHECK_PATCHES) # See contrib/coccinelle/README coccicheck-pending: coccicheck-test -coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci)) +coccicheck-pending: $(COCCICHECK_PATCHES_PENDING) .PHONY: coccicheck coccicheck-pending From patchwork Fri Oct 14 15:31:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4476EC43217 for ; Fri, 14 Oct 2022 15:32:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231178AbiJNPcb (ORCPT ); Fri, 14 Oct 2022 11:32:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230494AbiJNPbq (ORCPT ); Fri, 14 Oct 2022 11:31:46 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45D601974C5 for ; Fri, 14 Oct 2022 08:31:38 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id j16so8136191wrh.5 for ; Fri, 14 Oct 2022 08:31:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=na5KVCc787YYRRVbEjsF0Jc3yy2yFna5hjPBXuQ7J30=; b=nMiySAH8ZuoIBfeKRihWFkZ5MSAmJ5E+v3JHhqpfPegk+UPwV/vYn8gzExZ5jTMK6q miBvGy/WftlcPb/x9BNaKnztuDBee745Q0j1DChOru+mmZS/mbFHCB4rXXNGgyuAiQdX n2LeKWlx03QJy1qZWx3GU7W9Vr/bMiAbfS2AK5DZQV7XN0ZGl6phJDbSWa3C9jhbnt+3 PVbyCnE5zpPOLXCwWhUYrqKSkJY0vdbu8v2ZvzbdOQNptG6tdB/NLX22PqOGCYppZdBB yH0eENeyzGa+8Wv0xp15/+71AAdKgvPvLSOqHI/Ml2xd9CliCAOH1GvkrhCOQRgiEDc+ k8/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=na5KVCc787YYRRVbEjsF0Jc3yy2yFna5hjPBXuQ7J30=; b=Gj+YUySn38IOWpeG6rlGsiPuNVDxCr2uflrEH+ZetyQdRNfV+R+N85gvDEHu9y06Zq m0mZA0OVdbZmDRVx+U7zupxkddG/sbwUxmna+cFrt63CgNqt/mfl3eoZ3umVH6JK9LVU NxLXe6Awo35njuvicYfcEOQZo2iHuvIOQaFZCI7tAYrkIGY4xetxyVdTQGqWjlw7aS7s Tuzycfc3Pl6bnDAjZaeiISYQNihMsoVS1PxW5ZxALsbSAOCamDbbDcBDNcstZ1MeH4J9 rXEIY9h6KjXHZ1Pa+6IPYo38gBKuhkw2ZGwzotaaPilv+sD3hjjNvsLL/8NZ3K0R6BJH 5uHw== X-Gm-Message-State: ACrzQf0h0bDD5LgVBEtxNSMRoWBW10iwn7Lezbsnov6X6canZR92Cmia ENZYjE56zExu7maTMnA49o1ejbQQpXRfRQ== X-Google-Smtp-Source: AMsMyM6/kf56qDHqCeHVIM9MXHE3ZelsvAHVqzlkmVnBk5D+WtP/cql71xV3y4jy2uUiDwgjeZGpaw== X-Received: by 2002:adf:c747:0:b0:231:65ef:2fe5 with SMTP id b7-20020adfc747000000b0023165ef2fe5mr3750430wrh.594.1665761496133; Fri, 14 Oct 2022 08:31:36 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:35 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 04/11] Makefile: have "coccicheck" re-run if flags change Date: Fri, 14 Oct 2022 17:31:20 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix an issue with the "coccicheck" family of rules that's been here since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike e.g. "make grep.o" we wouldn't re-run it when $(SPATCH) or $(SPATCH_FLAGS) changed. To test new flags we needed to first do a "make cocciclean". This now uses the same (copy/pasted) pattern as other "DEFINES" rules. As a result we'll re-run properly. This can be demonstrated e.g. on the issue noted in [1]: $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1 [...] SPATCH contrib/coccinelle/xcalloc.cocci $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS="--all-includes --recursive-includes" * new spatch flags SPATCH contrib/coccinelle/xcalloc.cocci SPATCH result: contrib/coccinelle/xcalloc.cocci.patch $ 1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/ Signed-off-by: Ævar Arnfjörð Bjarmason --- .gitignore | 1 + Makefile | 15 +++++++++++++++ 2 files changed, 16 insertions(+) diff --git a/.gitignore b/.gitignore index 62720c6135d..7cce39270d4 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,7 @@ /GIT-PERL-HEADER /GIT-PYTHON-VARS /GIT-SCRIPT-DEFINES +/GIT-SPATCH-DEFINES /GIT-USER-AGENT /GIT-VERSION-FILE /bin-wrappers/ diff --git a/Makefile b/Makefile index 4647f317ede..d62f2311107 100644 --- a/Makefile +++ b/Makefile @@ -1304,6 +1304,18 @@ SANITIZE_ADDRESS = SPATCH_FLAGS = --all-includes SPATCH_BATCH_SIZE = 1 +# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change +TRACK_SPATCH_DEFINES = +TRACK_SPATCH_DEFINES += $(SPATCH) +TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) +TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) +GIT-SPATCH-DEFINES: FORCE + @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ + if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ + echo >&2 " * new spatch flags"; \ + echo "$$FLAGS" >GIT-SPATCH-DEFINES; \ + fi + include config.mak.uname -include config.mak.autogen -include config.mak @@ -3148,6 +3160,7 @@ COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) COCCI_PATCHES = $(COCCI_RULES:%=%.patch) +$(COCCI_PATCHES): GIT-SPATCH-DEFINES $(COCCI_PATCHES): $(COCCI_SOURCES) $(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ @@ -3171,6 +3184,7 @@ $(COCCI_PATCHES): %.patch: % fi COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) +$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci @@ -3457,6 +3471,7 @@ profile-clean: $(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs))) cocciclean: + $(RM) GIT-SPATCH-DEFINES $(RM) -r .build/contrib/coccinelle $(RM) contrib/coccinelle/*.cocci.patch* From patchwork Fri Oct 14 15:31:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F332C433FE for ; Fri, 14 Oct 2022 15:32:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231169AbiJNPcV (ORCPT ); Fri, 14 Oct 2022 11:32:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230060AbiJNPbn (ORCPT ); Fri, 14 Oct 2022 11:31:43 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50DD21D52D1 for ; Fri, 14 Oct 2022 08:31:38 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id bv10so8125299wrb.4 for ; Fri, 14 Oct 2022 08:31:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eRt+/yoI9E45KVGiTW5R8rH/AgC0ajGEGQ00fVmHVMs=; b=kNPiJ/y77ym2saVcqBxE54aka86IOmM2O7xCk3eOjHgtqu8uodj5TdLX1rjhWq9SyJ 4qRZuSIL4wGSLLhN+BL2KvPPGuww5MjBXWxh0WHvGZO7HZHxFP3BBMbMVPrhNXwrjPs4 8lkfHTLQJeudymJfzpZsr3RRwkUkSMsABQHFxDETy/Nsv4v1OnxvfjkLgal9s+5JDyi1 9l9DJbMUALPWax0Qkt4T7qNZcYT834DpjiHSywKxTHSPGen2to6tipL+KEbMTg09iLHG fa25UBH9BXi13ZtV2LIZx5TPv/2LYzisK39T6qNpSYqSwbFxK91QFYFR0GrJEx5df/ci z5cA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eRt+/yoI9E45KVGiTW5R8rH/AgC0ajGEGQ00fVmHVMs=; b=gSv4WmXG416r8sAuWcoj9iNZS1aXS2fQJumjD6XMmlz4wCUioGFtIPN22MvNmpZesQ oO5UE25Efi/XqPc/gP3VNdD0KLoi/1L7BnoCekFPO9z1476jlmJhtsDPzrwZaa2Ql+wu LxtCCi4wDbr5q5DHkZTC3dMw/2zUjJvxF8bywkKPgECRTlUg0S5e3VZ2JwJjsefGasxT niZSwwD/TvS4MXbCgPNAdyVu9p3wguseyhAMVgK0y3E3M135KSx+uoXTMv4hpceos9O1 rW8rWNKzWlaRhrHBImglpyt0akuxyli6zutXu8876jmzovohIw08wfRQzEE4ZJFYNsBz 4U5w== X-Gm-Message-State: ACrzQf3jsIeXlb9NI/ngY9dKlI1XKZIwI4RYKejs0JgA6ZfgEkK07osx u/Ujs1TDFwzVOilFZarCq5lHJ22qdgSGJw== X-Google-Smtp-Source: AMsMyM6GOR0zzrftTxRqnCu/KK5NVS4Jh4JFDrQYmlBLVWutODowYtEFWFhwBFAgduOHnYGTAyPh6w== X-Received: by 2002:adf:f804:0:b0:22e:5a9a:15d9 with SMTP id s4-20020adff804000000b0022e5a9a15d9mr3844586wrp.390.1665761497329; Fri, 14 Oct 2022 08:31:37 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:36 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 05/11] Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" heading Date: Fri, 14 Oct 2022 17:31:21 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Split off the "; setting[...]" part of the comment added in In 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06), and restore what we had before that, which was a comment indicating that variables for the "coccicheck" target were being set here. When 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it left no natural place to add a new comment about other flags that preceded it. As subsequent commits will add such comments we need to split the existing comment up. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index d62f2311107..2a106f633b9 100644 --- a/Makefile +++ b/Makefile @@ -1298,10 +1298,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer SANITIZE_LEAK = SANITIZE_ADDRESS = -# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will +# For the 'coccicheck' target +SPATCH_FLAGS = --all-includes +# Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_FLAGS = --all-includes SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change From patchwork Fri Oct 14 15:31:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C25E5C4332F for ; Fri, 14 Oct 2022 15:32:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231183AbiJNPcd (ORCPT ); Fri, 14 Oct 2022 11:32:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230500AbiJNPbr (ORCPT ); Fri, 14 Oct 2022 11:31:47 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 768AA6585 for ; Fri, 14 Oct 2022 08:31:41 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id bk15so8089082wrb.13 for ; Fri, 14 Oct 2022 08:31:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hicRwkmhJQ9CHXVMhfSDNiha+q3HTlz5GM0evgAbHY8=; b=XcFiLnZwwNfEGvgjSbzMlUJe5bXkQqpNsJjTMCsW1q0TgPSIclZuNmzB9VdHA8zMvK 5i6qD3Em4NDML8xn4bjD+0nnoDwfeIpG2mjYB+TYD5x7TTbMS4pvywoeJsKwwpjhwkxx itRHbwKUE7+x29WJQKcKM7AOJ5+Gb0dza2cyMjREraG7QEm0oRHvY8i5EuO9iBt/J5mk TAzxIXfFhXH1K0jM/VsLt7Iry9yxPGx6BkEZOJ6Eno7ULzp53a1ubVE9jI7q2/HZ3y3H 6DlhwUcDxk3A2E7P5NSY1+RNQqIJcsaCqtRDOXP47liM0rJVkzodwayPpy0qjorCJ5LO Zcjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hicRwkmhJQ9CHXVMhfSDNiha+q3HTlz5GM0evgAbHY8=; b=XG8YsGD9e85+sm74WMk1n6Axxu/IVV00gq7Fed4BnrVaf4ZZG4SzLjaQvUgiEswbL8 eP3BcePCpE9TXRAGbOorkWi0IwMM/fU7RIz7GPmkFMlpQ04gQsHKU+qw/8U0QBK3uMQj 6cVp5x88tvoifu8cF9Q9ob4giJSvIBZHJpKl5VX4K1xeNpXVrMVqDfLKPLPBzeyH2NS7 W392pLuKAwM7+kB9oN8g4hLAFTUPNyocm2NV+rBo/doTYQXqP92+L0KwYJRPAS/ZINio jQ2GWZ4NHhQowvU+gv6zNOJdQbRUMggCrfTRoAPo4cOlAIY/x8FQwGABHz3X1oUmD+Cz Ye2w== X-Gm-Message-State: ACrzQf2U8gLL5ARv2Vq3dHQlscCl/NzVNDG2jDGzLF2v/WzC0W5NKpjp 2uzlk8iS7aWcuCtjG0HNoEYUMuvrgda6kg== X-Google-Smtp-Source: AMsMyM6cMXv0f0LIsY5rTCNSmhdRITHVCX4GkHlPLZbAG+ZjKkyvt+Ys7c638JOWz22IiO9HQY+cSA== X-Received: by 2002:a05:6000:1051:b0:22e:5d8a:d3d5 with SMTP id c17-20020a056000105100b0022e5d8ad3d5mr3806445wrx.476.1665761498633; Fri, 14 Oct 2022 08:31:38 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:37 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 06/11] cocci: split off include-less "tests" from SPATCH_FLAGS Date: Fri, 14 Oct 2022 17:31:22 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Amend the "coccicheck-test" rule added in f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) to stop using "--all-includes". The flags we'll need for the tests are different than the ones we'll need for our main source code. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 2a106f633b9..535dece5d48 100644 --- a/Makefile +++ b/Makefile @@ -1300,6 +1300,7 @@ SANITIZE_ADDRESS = # For the 'coccicheck' target SPATCH_FLAGS = --all-includes +SPATCH_TEST_FLAGS = # Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. @@ -1309,6 +1310,7 @@ SPATCH_BATCH_SIZE = 1 TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) +TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ @@ -3190,7 +3192,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci $(call mkdir_p_parent_template) - $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \ + $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \ --very-quiet --no-show-diff \ --sp-file $< -o $@ \ $(@:.build/%.res=%.c) && \ From patchwork Fri Oct 14 15:31:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C889C4332F for ; Fri, 14 Oct 2022 15:32:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230484AbiJNPc2 (ORCPT ); Fri, 14 Oct 2022 11:32:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230495AbiJNPbq (ORCPT ); Fri, 14 Oct 2022 11:31:46 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34BC281103 for ; Fri, 14 Oct 2022 08:31:42 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id bp11so8116249wrb.9 for ; Fri, 14 Oct 2022 08:31:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Xe56/qlZnntAEt3Jsb5FEbGmYbsBNvTRubi0B8DX2p0=; b=B38B6tboRIufuQFQKAmWXOScay9Qwlf84eVTCDVX7lbSwpLVELO5q6gzqtwGWAv1No CtEdVPUGFFulrLwAS8dhCycnYW0YHOeMwmzqmVFMdCGQElLWMoIRo79kHnGczhRxFSTe 88zcnvqVeKKuN+IXFfXXVp8haLMzsFrmHpZNJCnvaCmvWC58OsbLxCBP8bF/FRDaiQz9 Kl+1N9RbQXojFTGlzYfcfIj9twVewDBnozg0+JqKPZH1jUaMyQFie5AsPZMDtDZhA4WJ Rfe45ybdrV1/OlloKSdMOt0smwpiUshxO3F2M6RF6Q13cSenjv/62wvd3U9qF4BlIJWw FbcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xe56/qlZnntAEt3Jsb5FEbGmYbsBNvTRubi0B8DX2p0=; b=L3smHeEAiUrwB0AK9efn2DcMho5gx1AA/Oq2cwWHRvy6jBR71V5wV4C5Z2SoDcFHXu fsMXD9kT+8W3AoPnolw0ZXEFmW0bl7l1npBbqToLccepHySGWy/OSBAye/MnSZpJo1Cy efM3qKTnNwEkFl0nGeB7SQr/gACV7jrXdYfr5mn8gSvUiI3FM9wXteyPVs0loSO4KWxi wf31QzreZg/F/Vp4MMKOWokhiHlNEiWK41dzhOf/CSFaKraoH09HLT7NqGtLd4isr7HG DcTsPJbMW3+9X4amxA2YB0k4M6NMZb6qNqV7SxO8WwsH0fRn6HTY56IW96/nbuFqlNs4 /GHw== X-Gm-Message-State: ACrzQf0ufWGV8YgP/c8dHLGFk74nDabqycRecSF0dFF+fu3qx0u2CurR ol8heFM+Bn2y8KoYckqo7qZIeXbvsMQcPA== X-Google-Smtp-Source: AMsMyM5+Xqh7PADTLie5OCZnkovrWASnZMDw8Hrm1X7zfBB08z1JqM158p+U0oeRUJmgIqNwdx24ow== X-Received: by 2002:a5d:64ab:0:b0:231:44c1:32b0 with SMTP id m11-20020a5d64ab000000b0023144c132b0mr3809419wrp.255.1665761499911; Fri, 14 Oct 2022 08:31:39 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:38 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 07/11] cocci: split off "--all-includes" from SPATCH_FLAGS Date: Fri, 14 Oct 2022 17:31:23 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Per the rationale in 7b63ea57500 (Makefile: remove mandatory "spatch" arguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that are truly mandatory, such as "--sp-file" and "--patch .". The "--all-includes" flag is also critical, but per [1] we might want to ad-hoc tweak it occasionally for testing or one-offs. But being unable to set e.g. SPATCH_FLAGS="--verbose-parsing" without breaking how our "spatch" works isn't ideal, i.e. before this we'd need to know about the default include flags, and specify: SPATCH_FLAGS="--all-includes --verbose-parsing". If we were then to change the default include flag (e.g. to "--recursive-includes") in the future any such one-off commands would need to be correspondingly updated. Let's instead leave the SPATCH_FLAGS for the user, while creating a new SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include strategy itself. 1. https://lore.kernel.org/git/20220823095733.58685-1-szeder.dev@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 535dece5d48..f79697053bc 100644 --- a/Makefile +++ b/Makefile @@ -1299,7 +1299,8 @@ SANITIZE_LEAK = SANITIZE_ADDRESS = # For the 'coccicheck' target -SPATCH_FLAGS = --all-includes +SPATCH_INCLUDE_FLAGS = --all-includes +SPATCH_FLAGS = SPATCH_TEST_FLAGS = # Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. @@ -1309,6 +1310,7 @@ SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) +TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) @@ -3174,6 +3176,7 @@ $(COCCI_PATCHES): %.patch: % fi; \ if ! echo $(COCCI_SOURCES) | xargs $$limit \ $(SPATCH) $(SPATCH_FLAGS) \ + $(SPATCH_INCLUDE_FLAGS) \ --sp-file $< --patch . \ >$@+ 2>$@.log; \ then \ From patchwork Fri Oct 14 15:31:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE7CAC433FE for ; Fri, 14 Oct 2022 15:32:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231184AbiJNPcf (ORCPT ); Fri, 14 Oct 2022 11:32:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230501AbiJNPbr (ORCPT ); Fri, 14 Oct 2022 11:31:47 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA75357889 for ; Fri, 14 Oct 2022 08:31:43 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id f11so8116097wrm.6 for ; Fri, 14 Oct 2022 08:31:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lGasDuuovclq1lyhg2BBHz5Oh2ttJTfofjq5hBIEsME=; b=KJQjLiL5DIJZkWY/uLgfjNXTtSB3PUqb5l/ez/yad9zljrF01qD7mImA/wNmeoSq8y QLZWPUFgkeg2lBRKxrlWYcyjWInGj2HLK7EtmgtkEPa2oMEKfG5pQpJ62biEs7cA4/jP s/gCO9s4aFgpJTvywjbYEsBZnhDZCCwtfUXNhQVT7rNE1VkIXMfRut1nEl9jVHepKwke ZCfIYzC0Thvy7rikulHT+qFdd2VwCUYjNtMsa4Ma4H1K5zs13C4rd2tBTXu+1WlMO0wj D5wxkAUf6DITtzhuxJpxxzFKwPuPm/B3EJXMpYPtdD/yMBQP01HGce+MZgSKFNUGMpZe 4RVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lGasDuuovclq1lyhg2BBHz5Oh2ttJTfofjq5hBIEsME=; b=4/ny1M0bbRzyp3EKDrc8U/M5KeMBghDv8E0smDCpfnzwOo9YnQUtVNJqbiOgLCOlpo PxHzJtOQnV6qi987/ZYlSzbRnDa3IglU5vyhZ3nhCzCbohe8eJ1+OpxAATkicVykB4xl g6LN/w00z5lNaw6pSQ9QazbbLiqcOhpo4dYgKP/rLPq1TsD5ITIyzRKjbzcPXjSRYT2R XdDzv3CQ1zoAId+sBfiGLMPc3Nuq7hRWqlj20wTvCccFXubN9pyI/8L+hIE26GHThA8c WZvKZ3Rwgx2BOmOFqONc+o0HH0cFHbYeweORRFUwZ6uK3rG9SixEClwvACJgvRjTnaEP jnFg== X-Gm-Message-State: ACrzQf1/5C1ajwo1bnhe3bZVjkV750UNDOlmonVkh5aLM9ZBsoN6lo2Y OVPkjoqk+gIp0fb7CbtRXnYRxI63isUyDw== X-Google-Smtp-Source: AMsMyM7s5FPD74mlLIY1L+sykSgap7D7JTxhCNPAmouDADfCdKSimkfvPdkPMmBwtCiJf7UwzCFRPA== X-Received: by 2002:adf:d1ea:0:b0:22e:33f9:bcc1 with SMTP id g10-20020adfd1ea000000b0022e33f9bcc1mr3728647wrd.535.1665761501325; Fri, 14 Oct 2022 08:31:41 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:40 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 08/11] cocci: make "coccicheck" rule incremental Date: Fri, 14 Oct 2022 17:31:24 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Optimize the very slow "coccicheck" target to take advantage of incremental rebuilding, and fix outstanding dependency problems with the existing rule. The rule is now faster both on the initial run as we can make better use of GNU make's parallelism than the old ad-hoc combination of make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the "--jobs" argument to "spatch(1)". It also makes us *much* faster when incrementally building, it's now viable to "make coccicheck" as topic branches are merged down. The rule didn't use FORCE (or its equivalents) before, so a: make coccicheck make coccicheck Would report nothing to do on the second iteration. But all of our patch output depended on all $(COCCI_SOURCES) files, therefore e.g.: make -W grep.c coccicheck Would do a full re-run, i.e. a a change in a single file would force us to do a full re-run. The reason for this (not the initial rationale, but my analysis) is: * Since we create a single "*.cocci.patch+" we don't know where to pick up where we left off, or how to incrementally merge e.g. a "grep.c" change with an existing *.cocci.patch. * We've been carrying forward the dependency on the *.c files since 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was initially added as a sort of poor man's dependency discovery. As we don't include other *.c files depending on other *.c files has always been broken, as could be trivially demonstrated e.g. with: make coccicheck make -W strbuf.h coccicheck However, depending on the corresponding *.c files has been doing something, namely that *if* an API change modified both *.c and *.h files we'd catch the change to the *.h we care about via the *.c being changed. For API changes that happened only via *.h files we'd do the wrong thing before this change, but e.g. for function additions (not "static inline" ones) catch the *.h change by proxy. Now we'll instead: * Create a / pair in the .build directory, E.g. for swap.cocci and grep.c we'll create .build/contrib/coccinelle/swap.cocci.patch/grep.c. That file is the diff we'll apply for that - combination, if there's no changes to me made (the common case) it'll be an empty file. * Our generated *.patch file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple "cat $^" of all of all of the / files for a given . In the case discussed above of "grep.c" being changed we'll do the full "cat" every time, so they resulting *.cocci.patch will always be correct and up-to-date, even if it's "incrementally updated". See 1cc0425a27c (Makefile: have "make pot" not "reset --hard", 2022-05-26) for another recent rule that used that technique. As before we'll: * End up generating a contrib/coccinelle/swap.cocci.patch, if we "fail" by creating a non-empty patch we'll still exit with a zero exit code. Arguably we should move to a more Makefile-native way of doing this, i.e. fail early, and if we want all of the "failed" changes we can use "make -k", but as the current "ci/run-static-analysis.sh" expects us to behave this way let's keep the existing behavior of exhaustively discovering all cocci changes, and only failing if spatch itself errors out. Further implementation details & notes: * Before this change running "make coccicheck" would by default end up pegging just one CPU at the very end for a while, usually as we'd finish whichever *.cocci rule was the most expensive. This could be mitigated by combining "make -jN" with SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06). There will be cases where getting rid of "SPATCH_BATCH_SIZE" makes things worse, but a from-scratch "make coccicheck" with the default of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference) is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed the CPU more work in a less staggered way. * Getting rid of "SPATCH_BATCH_SIZE" particularly helps in cases where the default of 1 yields parallelism under "make coccicheck", but then running e.g.: make -W contrib/coccinelle/swap.cocci coccicheck I.e. before that would use only one CPU core, until the user remembered to adjust "SPATCH_BATCH_SIZE" differently than the setting that makes sense when doing a non-incremental run of "make coccicheck". * Before the "make coccicheck" rule would have to clean "contrib/coccinelle/*.cocci.patch*", since we'd create "*+" and "*.log" files there. Now those are created in .build/contrib/coccinelle/, which is covered by the "cocciclean" rule already. Outstanding issues & future work: * We could get rid of "--all-includes" in favor of manually specifying a list of includes to give to "spatch(1)". As noted upthread of [1] a naïve removal of "--all-includes" will result in broken *.cocci patches, but if we know the exhaustive list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to re-scan for them, we could grab the headers to include from the .depend.d/.o.d and supply them with the "--include" option to spatch(1).q 1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 84 +++++++++++++++++++++++++++++++++++++----------------- shared.mak | 5 +++- 2 files changed, 62 insertions(+), 27 deletions(-) diff --git a/Makefile b/Makefile index f79697053bc..4e1da7206db 100644 --- a/Makefile +++ b/Makefile @@ -1302,10 +1302,6 @@ SANITIZE_ADDRESS = SPATCH_INCLUDE_FLAGS = --all-includes SPATCH_FLAGS = SPATCH_TEST_FLAGS = -# Setting SPATCH_BATCH_SIZE higher will -# usually result in less CPU usage at the cost of higher peak memory. -# Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = @@ -1313,7 +1309,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH) TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) -TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ @@ -3155,39 +3150,75 @@ check: $(GENERATED_H) COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) COCCI_RULES = $(COCCI_GLOB) +COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%) COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES)) +COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) +COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch) + +# It's expensive to compute the many=many rules below, only eval them +# on $(MAKECMDGOALS) that match these $(COCCI_RULES) +COCCI_RULES_GLOB = +COCCI_RULES_GLOB += cocci% +COCCI_RULES_GLOB += .build/contrib/coccinelle/% +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES) +COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING) +COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS)) COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch) COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -COCCI_PATCHES = $(COCCI_RULES:%=%.patch) -$(COCCI_PATCHES): GIT-SPATCH-DEFINES -$(COCCI_PATCHES): $(COCCI_SOURCES) -$(COCCI_PATCHES): %.patch: % - $(QUIET_SPATCH) \ - if test $(SPATCH_BATCH_SIZE) = 0; then \ - limit=; \ - else \ - limit='-n $(SPATCH_BATCH_SIZE)'; \ - fi; \ - if ! echo $(COCCI_SOURCES) | xargs $$limit \ - $(SPATCH) $(SPATCH_FLAGS) \ - $(SPATCH_INCLUDE_FLAGS) \ - --sp-file $< --patch . \ - >$@+ 2>$@.log; \ +.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) + $(call mkdir_p_parent_template) + $(QUIET_GEN) >$@ + +define cocci-rule + +## Rule for .build/$(1).patch/$(2); Params: +# $(1) = e.g. "free.cocci" +# $(2) = e.g. "grep.c" +COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) +.build/$(1).patch/$(2): GIT-SPATCH-DEFINES +.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES +.build/$(1).patch/$(2): $(1) +.build/$(1).patch/$(2): .build/$(1).patch/% : % + $$(call mkdir_p_parent_template) + $$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \ + $$(SPATCH_INCLUDE_FLAGS) \ + --sp-file $(1) --patch . $$< \ + >$$@ 2>$$@.log; \ then \ - cat $@.log; \ + echo "ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:"; \ + cat $$@.log; \ exit 1; \ - fi; \ - mv $@+ $@; \ - if test -s $@; \ + fi +endef + +define cocci-matrix + +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s))) +endef + +ifdef COCCI_GOALS +$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c)))) +endif + +define spatch-rule + +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) + $(QUIET_SPATCH_CAT_TMPL)cat $$^ >$$@ && \ + if test -s $$@; \ then \ - echo ' ' SPATCH result: $@; \ + echo ' ' SPATCH result: $$@; \ fi +endef + +ifdef COCCI_GOALS +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n)))) +endif COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES @@ -3479,7 +3510,8 @@ profile-clean: cocciclean: $(RM) GIT-SPATCH-DEFINES $(RM) -r .build/contrib/coccinelle - $(RM) contrib/coccinelle/*.cocci.patch* + $(RM) $(COCCICHECK_PATCHES) + $(RM) $(COCCICHECK_PATCHES_PENDING) clean: profile-clean coverage-clean cocciclean $(RM) -r .build diff --git a/shared.mak b/shared.mak index 96b06acc455..5ccd6889fcb 100644 --- a/shared.mak +++ b/shared.mak @@ -71,9 +71,12 @@ ifndef V QUIET_RC = @echo ' ' RC $@; ## Used in "Makefile": SPATCH - QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_SPATCH = @echo ' ' SPATCH $@; QUIET_SPATCH_TEST = @echo ' ' SPATCH TEST $(@:.build/%=%); +## Used in "Makefile": SPATCH_*TMPL (quoted for use in "define"'s) + QUIET_SPATCH_CAT_TMPL = @echo ' ' SPATCH CAT $$$$^ \>$$@; + ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; QUIET_XMLTO = @echo ' ' XMLTO $@; From patchwork Fri Oct 14 15:31:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98BB9C4332F for ; Fri, 14 Oct 2022 15:32:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231128AbiJNPcx (ORCPT ); Fri, 14 Oct 2022 11:32:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230520AbiJNPbx (ORCPT ); Fri, 14 Oct 2022 11:31:53 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB353F708D for ; Fri, 14 Oct 2022 08:31:46 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id bk15so8089577wrb.13 for ; Fri, 14 Oct 2022 08:31:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wGmsNyij+p/pGl1CpEikcPadeNmFGDkLJ7eI/3Yc/4Q=; b=OL7/Djra6f+rLDYgFeL5rIpePNDBa8u5AaOeaYfgqBkEK/8dv70dDbwcJ7Jj+mEJTJ 3doI4yDA1M4QMNTnXhwov1lvkug1lhYALlkxHAbg80wFzXGH2UgyY3SbxGsZMI2MR9r4 imq10KsWalfsQGrMPvE1wBIuOIbsxRKZ9wOMTrZOWwi5rx8ZlFXUL5rnfN/IsKvgs/6F GAe5M6rlAq96Sl/i4M+byvfEqnQkS9StsukLNqYZMcb5ggPLLdlGXb3N8XUGaxdkTeNp 4H+izoItPjLqbGDQHWiG5eEvG7ubOSo/1puwufeN0Polg0NXfI41TeiEiRBV2RPJkIHq AtgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wGmsNyij+p/pGl1CpEikcPadeNmFGDkLJ7eI/3Yc/4Q=; b=P6aFZQ89jjZijsi6EnqNxioKEtCauAK9mfWwgJSZZKmc7UtGzWK1bW707p5suRDfYu PBjtqRy9D0HqicLFOB7pu9TXMsRlngP4933a8Lmx0890H+L5mk9XYOIf1ujVtGH0FdfR 6ElbBo4n5xJaK2mFJ19Ry3w/PwMALarBFf03PT7G/uUq6RQcRoAqCyT7yRhwhJP/SqEk 6wNOYOkbaLCBXGeVCJFP+MEY0DDodB7eB/NfhiEQsuWxJQUzSIcZ0wrgaBlDpmLLpTSe ZjZZuHki+6mGq5FKIC4dg+nT0bC+ElWM/eq/x43t2S4dBlQNpJhzEIb8Qh6Wg8oan3zu UtHA== X-Gm-Message-State: ACrzQf3tZDtZcWtkFBYddjTCWQV9o7KP6QUO8jTJUHShjtwBL9M0lb7i VrfQBVmvDxn9b0+LRzcCjNNCpSHU8tjTQQ== X-Google-Smtp-Source: AMsMyM4DPdV84hVZLQFPObI+AisvPHb79h0S6io8gn00qy+yQovSdiwPwgLxbI9N3rOeKlV+/NTZWw== X-Received: by 2002:a5d:4150:0:b0:22f:f9f6:cca1 with SMTP id c16-20020a5d4150000000b0022ff9f6cca1mr3923637wrq.510.1665761504275; Fri, 14 Oct 2022 08:31:44 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:41 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 09/11] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES Date: Fri, 14 Oct 2022 17:31:25 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Improve the incremental rebuilding support of "coccicheck" by piggy-backing on the computed dependency information of the corresponding *.o file, rather than rebuilding all / pairs if either their corresponding file changes, or if any header changes. This in effect uses the same method that the "sparse" target was made to use in c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23), except that the dependency on the *.o file isn't a hard one, we check with $(wildcard) if the *.o file exists, and if so we'll depend on it. This means that the common case of: make make coccicheck Will benefit from incremental rebuilding, now changing e.g. a header will only re-run "spatch" on those those *.c files that make use of it: By depending on the *.o we piggy-back on COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that for the *.sp files. E.g.: make contrib/coccinelle/free.cocci.patch make -W column.h contrib/coccinelle/free.cocci.patch Will take around 15 seconds for the second command on my 8 core box if I didn't run "make" beforehand to create the *.o files. But around 2 seconds if I did and we have those "*.o" files. Notes about the approach of piggy-backing on *.o for dependencies: * It *is* a trade-off since we'll pay the extra cost of running the C compiler, but we're probably doing that anyway. The compiler is much faster than "spatch", so even though we need to re-compile the *.o to create the dependency info for the *.c for "spatch" it's faster (especially if using "ccache"). * There *are* use-cases where some would like to have *.o files around, but to have the "make coccicheck" ignore them. See: https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/ For those users a: make make coccicheck SPATCH_USE_O_DEPENDENCIES= Will avoid considering the *.o files. * If that *.o file doesn't exist we'll depend on an intermediate file of ours which in turn depends on $(FOUND_H_SOURCES). This covers both an initial build, or where "coccicheck" is run without running "all" beforehand, and because we run "coccicheck" on e.g. files in compat/* that we don't know how to build unless the requisite flag was provided to the Makefile. Most of the runtime of "incremental" runs is now spent on various compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and therefore conflate whether we *can* compile an object and generate dependency information for it with whether we'd like to link it into our binary. Before this change the distinction didn't matter, but now one way to make this even faster on incremental builds would be to peel those concerns apart so that we can see that e.g. compat/mmap.c doesn't depend on column.h. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 4e1da7206db..bd3b5c21d6f 100644 --- a/Makefile +++ b/Makefile @@ -1303,6 +1303,13 @@ SPATCH_INCLUDE_FLAGS = --all-includes SPATCH_FLAGS = SPATCH_TEST_FLAGS = +# If *.o files are present, have "coccicheck" depend on them, with +# COMPUTE_HEADER_DEPENDENCIES this will speed up the common-case of +# only needing to re-generate coccicheck results for the users of a +# given API if it's changed, and not all files in the project. If +# COMPUTE_HEADER_DEPENDENCIES=no this will be unset too. +SPATCH_USE_O_DEPENDENCIES = YesPlease + # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) @@ -3175,14 +3182,18 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ +ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no) +SPATCH_USE_O_DEPENDENCIES = +endif define cocci-rule ## Rule for .build/$(1).patch/$(2); Params: # $(1) = e.g. "free.cocci" # $(2) = e.g. "grep.c" +# $(3) = e.g. "grep.o" COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) .build/$(1).patch/$(2): GIT-SPATCH-DEFINES -.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES +.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) .build/$(1).patch/$(2): $(1) .build/$(1).patch/$(2): .build/$(1).patch/% : % $$(call mkdir_p_parent_template) @@ -3199,7 +3210,7 @@ endef define cocci-matrix -$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s))) +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o))) endef ifdef COCCI_GOALS From patchwork Fri Oct 14 15:31:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A110C433FE for ; Fri, 14 Oct 2022 15:33:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231195AbiJNPdG (ORCPT ); Fri, 14 Oct 2022 11:33:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231153AbiJNPcJ (ORCPT ); Fri, 14 Oct 2022 11:32:09 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10C3BEAC9F for ; Fri, 14 Oct 2022 08:31:58 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id bp11so8117509wrb.9 for ; Fri, 14 Oct 2022 08:31:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WyHp44ea796Bm+h62F+npYKatJbISFsFdh6TPLTwNLo=; b=Z7olGB/t+iqhxhFLjtnXQsp+DVGdsNtrbA/uWiGeIYImXqrI6p9zanSjY/H2LB1bur s20FYBpx8vlCoct9IA5oEax3HK4AO6UUcE8x66ORPGMJAMfm6u3fBNPdExBe126ZtO/s BsRiKMl8MpEln+Fmt2x/iV5d2ZDp3Ja3vgZ+ECJL9b4RgnjlbQVaNfwtHQEsNFkP09P3 opGVv4nKGGu0baVK6koopDcCj2pe5a5G+IStgzO1NiIFrI3YjXydjmjDY67Mr9tDE0ov nLprLHqOj5jnNfon2Y5JiRXiUCT9+z40okVuFXF5U5YG6vDbUz9Lp4S3owVvDhY++FPu bvSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WyHp44ea796Bm+h62F+npYKatJbISFsFdh6TPLTwNLo=; b=zpw2nqkpXiIbrpg/65DNFV6cIwlchQQX9TGlXj1KHxwX3t2+egMWbixV08SJQFUtZC lyxg8xjz684dSBFxqgJVli2AnsJeiwwG3FFa3l26QuSEZXUJ5fX1O9MdMaw8Mp/JX2zU +fbbOAnimDNeyWgBDc5OoqENJgOo5+AzQykdOq8jp0r2d/ibPndFX0NrOFxknKMnBReh 2WnuzJA6UBwNoXdzNx4mkZA26JYag5ANePrKY/VRlV8tmxZG6kxrxIA1QE3l8rcrpWy6 7BTfssD8t+OhCGJTfhGhgCal/ATU/KdqSf/82WegdK/gc6DVDCXPAsveDum89x5EP7x9 QaHw== X-Gm-Message-State: ACrzQf0y1CdwNxub7Uy4/Ul87FLrUIX7q7dy3i079EyU0m4b9zuZkUJZ wDqY8N+k59LB/Yn2nkHxH+vbJPRqtTCxZw== X-Google-Smtp-Source: AMsMyM5FoeL18dDljsXRN1ksuPwkyVur9Zx+uk3pvyXGHBM6XwRsok5Oe5aUM3gmvB4XgONAJX9PcA== X-Received: by 2002:adf:e4d1:0:b0:232:eb2:68bc with SMTP id v17-20020adfe4d1000000b002320eb268bcmr3716554wrm.231.1665761506084; Fri, 14 Oct 2022 08:31:46 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:44 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 10/11] cocci: run against a generated ALL.cocci Date: Fri, 14 Oct 2022 17:31:26 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The preceding commits to make the "coccicheck" target incremental made it slower in some cases. As an optimization let's not have the many=many mapping of <*.cocci>=<*.[ch]>, but instead concat the <*.cocci> into an ALL.cocci, and then run one-to-many ALL.cocci=<*.[ch]>. A "make coccicheck" is now around 2x as fast as it was on "master", and around 1.5x as fast as the preceding change to make the run incremental: $ git hyperfine -L rev origin/master,HEAD~,HEAD -p 'make clean' 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' -r 3 Benchmark 1: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'origin/master Time (mean ± σ): 4.258 s ± 0.015 s [User: 27.432 s, System: 1.532 s] Range (min … max): 4.241 s … 4.268 s 3 runs Benchmark 2: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD~ Time (mean ± σ): 5.365 s ± 0.079 s [User: 36.899 s, System: 1.810 s] Range (min … max): 5.281 s … 5.436 s 3 runs Benchmark 3: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD Time (mean ± σ): 2.725 s ± 0.063 s [User: 14.796 s, System: 0.233 s] Range (min … max): 2.667 s … 2.792 s 3 runs Summary 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD' ran 1.56 ± 0.04 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'origin/master' 1.97 ± 0.05 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD~' This can be turned off with SPATCH_CONCAT_COCCI, but as the beneficiaries of "SPATCH_CONCAT_COCCI=" would mainly be those developing the *.cocci rules themselves, let's leave this optimization on by default. For more information see my "Optimizing *.cocci rules by concat'ing them" (<220901.8635dbjfko.gmgdl@evledraar.gmail.com>) on the cocci@inria.fr mailing list. This potentially changes the results of our *.cocci rules, but as noted in that discussion it should be safe for our use. We don't name rules, or if we do their names don't conflict across our *.cocci files. To the extent that we'd have any inter-dependencies between rules this doesn't make that worse, as we'd have them now if we ran "make coccicheck", applied the results, and would then have (due to hypothetical interdependencies) suggested changes on the subsequent "make coccicheck". Our "coccicheck-test" target makes use of the ALL.cocci when running tests, e.g. when testing unused.{c,out} we test it against ALL.cocci, not unused.cocci. We thus assert (to the extent that we have test coverage) that this concatenation doesn't change the expected results of running these rules. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 40 ++++++++++++++++++++++++++++++++--- contrib/coccinelle/.gitignore | 1 + shared.mak | 1 + 3 files changed, 39 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index bd3b5c21d6f..0dc25a2c7ad 100644 --- a/Makefile +++ b/Makefile @@ -1310,6 +1310,25 @@ SPATCH_TEST_FLAGS = # COMPUTE_HEADER_DEPENDENCIES=no this will be unset too. SPATCH_USE_O_DEPENDENCIES = YesPlease +# Set SPATCH_CONCAT_COCCI to concatenate the contrib/cocci/*.cocci +# files into a single contrib/cocci/ALL.cocci before running +# "coccicheck". +# +# Pros: +# +# - Speeds up a one-shot run of "make coccicheck", as we won't have to +# parse *.[ch] files N times for the N *.cocci rules +# +# Cons: +# +# - Will make incremental development of *.cocci slower, as +# e.g. changing strbuf.cocci will re-run all *.cocci. +# +# - Makes error and performance analysis harder, as rules will be +# applied from a monolithic ALL.cocci, rather than +# e.g. strbuf.cocci. +SPATCH_CONCAT_COCCI = YesPlease + # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) @@ -3155,9 +3174,10 @@ check: $(GENERATED_H) exit 1; \ fi +COCCI_GEN_ALL = contrib/coccinelle/ALL.cocci COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) -COCCI_RULES = $(COCCI_GLOB) -COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%) +COCCI_NAMES = $(sort ALL $(COCCI_GLOB:contrib/coccinelle/%.cocci=%)) +COCCI_RULES = $(filter-out $(COCCI_GEN_ALL),$(COCCI_GLOB)) COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES)) COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) @@ -3171,6 +3191,7 @@ COCCI_RULES_GLOB += cocci% COCCI_RULES_GLOB += .build/contrib/coccinelle/% COCCI_RULES_GLOB += $(COCCICHECK_PATCHES) COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING) +COCCI_RULES_GLOB += $(COCCI_GEN_ALL) COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS)) COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch) @@ -3182,6 +3203,10 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ +$(COCCI_GEN_ALL): $(COCCICHECK) + $(call mkdir_p_parent_template) + $(QUIET_SPATCH_CAT)cat $^ >$@ + ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no) SPATCH_USE_O_DEPENDENCIES = endif @@ -3214,7 +3239,7 @@ $(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o))) endef ifdef COCCI_GOALS -$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c)))) +$(eval $(foreach c,$(COCCI_RULES) $(COCCI_GEN_ALL),$(call cocci-matrix,$(c)))) endif define spatch-rule @@ -3235,7 +3260,11 @@ COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res +ifdef SPATCH_CONCAT_COCCI +$(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : $(COCCI_GEN_ALL) +else $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci +endif $(call mkdir_p_parent_template) $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \ --very-quiet --no-show-diff \ @@ -3248,7 +3277,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test +ifdef SPATCH_CONCAT_COCCI +coccicheck: contrib/coccinelle/ALL.cocci.patch +else coccicheck: $(COCCICHECK_PATCHES) +endif # See contrib/coccinelle/README coccicheck-pending: coccicheck-test @@ -3523,6 +3556,7 @@ cocciclean: $(RM) -r .build/contrib/coccinelle $(RM) $(COCCICHECK_PATCHES) $(RM) $(COCCICHECK_PATCHES_PENDING) + $(RM) $(COCCI_GEN_ALL) clean: profile-clean coverage-clean cocciclean $(RM) -r .build diff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore index d3f29646dc3..2709d98eb91 100644 --- a/contrib/coccinelle/.gitignore +++ b/contrib/coccinelle/.gitignore @@ -1 +1,2 @@ +/ALL.cocci *.patch* diff --git a/shared.mak b/shared.mak index 5ccd6889fcb..66dcf6768db 100644 --- a/shared.mak +++ b/shared.mak @@ -73,6 +73,7 @@ ifndef V ## Used in "Makefile": SPATCH QUIET_SPATCH = @echo ' ' SPATCH $@; QUIET_SPATCH_TEST = @echo ' ' SPATCH TEST $(@:.build/%=%); + QUIET_SPATCH_CAT = @echo ' ' SPATCH CAT $$^ \>$@; ## Used in "Makefile": SPATCH_*TMPL (quoted for use in "define"'s) QUIET_SPATCH_CAT_TMPL = @echo ' ' SPATCH CAT $$$$^ \>$$@; From patchwork Fri Oct 14 15:31:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13007140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 369ABC4332F for ; Fri, 14 Oct 2022 15:32:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231140AbiJNPcz (ORCPT ); Fri, 14 Oct 2022 11:32:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230525AbiJNPb4 (ORCPT ); Fri, 14 Oct 2022 11:31:56 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00CAE188132 for ; Fri, 14 Oct 2022 08:31:50 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id e18so3307050wmq.3 for ; Fri, 14 Oct 2022 08:31:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rip9TeiOFqEr2uDlzarNPVqeDi4tOTTXJOg/3PaLDTQ=; b=MJgVd2CuSzruncy6TMCg7fvgVZN42/Mwun1jPHLEGIDWP+E3b/o8V34hI+2sZNiOlS ghnnFWPrUwP9/nmlfpGojIwaH/slzpktTgocVj4qWSuZO+NsHDS3dqoolPpyN0ERSCO9 0Gkosus8Yhy9e5yEQ6UJ/nfNEHCEBAIul0kdcn7lV2lsWpsZKcZu0zCIUoLbkEwYnZex Veef6zkti+tTefE7keWMl3iKN58fMRiZ0teBOdcskzYXk3AEVxweWDxwfrg47DIJnM7c pTvec+Vc5JFMvKLx8fcqRD/+I62Llr3u0qMHItzSXh6ySf4oBoHRrvVcCg76aLsxyVjl uw4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rip9TeiOFqEr2uDlzarNPVqeDi4tOTTXJOg/3PaLDTQ=; b=ADH18/g9YsPH8IU070A6ddw85wfe49+hKuZCgVud7edTpAH5VwrXTQxyvoEuhYj2g5 HE0TP7wiUfQLPkSGiXpXVJdNewp+Mm2p8h+2v/zFroJEwz1StEjMYoIWZP64lHe49U4Z VJqE7YqEdHT+NWfRU9NHr1nrkSd6Pm7LZg2g25Lpa9KlO/K/CDeMpQG7+wGS8rMycN2D WnbB3/819KMDrapuXwaIyHhK5pznBtyO3d31H7OLkQL3wq3BdAiDGulOaPQ3fX35hdPl Fc/Rb00Zy8+xK6y8R+4a05ykZddbDT7Es/7q7GuhEm6m/30sno5IDwMIHHfJKJm5O5A8 tj6g== X-Gm-Message-State: ACrzQf1NRcF7F9oYd80DiOmQ94I6qlZKTx+X1HsHgk/1IQHwPpR7jysy RBRMB+S/RG1YD8jjdJq9Y5EzWWLxFTvHlQ== X-Google-Smtp-Source: AMsMyM5I8AHJMnoQ+x7UvxYXmHQ29rLZy6usWGH5IwWUfZ9ylTPpD8Gzw6dyOstg9GHkUeH4Sc6hPg== X-Received: by 2002:a1c:ed11:0:b0:3b4:d3e1:bec with SMTP id l17-20020a1ced11000000b003b4d3e10becmr4053192wmh.196.1665761507347; Fri, 14 Oct 2022 08:31:47 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id dn7-20020a05600c654700b003b47ff307e1sm2219053wmb.31.2022.10.14.08.31.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Oct 2022 08:31:46 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v3 11/11] spatchcache: add a ccache-alike for "spatch" Date: Fri, 14 Oct 2022 17:31:27 +0200 Message-Id: X-Mailer: git-send-email 2.38.0.1092.g8c0298861b0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a rather trivial "spatchcache", with this running e.g.: make cocciclean make contrib/coccinelle/free.cocci.patch \ SPATCH=contrib/coccicheck/spatchcache \ SPATCH_FLAGS=--very-quiet Is cut down from ~20s to ~5s on my system. Much of that is either fixable shell overhead, or the around 40 files we "CANTCACHE" (see the implementation). This uses "redis" as a cache by default, but it's configurable. See the embedded documentation. This is *not* like ccache in that we won't cache failed spatch invocations, or those where spatch suggests changes for us. Those cases are so rare that I didn't think it was worth the bother, by far the most common case is that it has no suggested changes. We'll also refuse to cache any "spatch" invocation that has output on stderr, which means that "--very-quiet" must be added to "SPATCH_FLAGS". Because we narrow the cache to that we don't need to save away stdout, stderr & the exit code. We simply cache the cases where we had no suggested changes. Another benchmark is to compare this with the previous SPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running: make clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0 Would take 33s, but with the preceding changes running without this "spatchcache" is slightly slower, or around 35s: make clean; time make contrib/coccinelle/array.cocci.patch Now doing the same with SPATCH=contrib/coccinelle/spatchcache will take around 6s, but we'll need to compile the *.o files first to take full advantage of it (which can be fast with "ccache"): make clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache 1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/ Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/spatchcache | 272 +++++++++++++++++++++++++++++++++ 1 file changed, 272 insertions(+) create mode 100755 contrib/coccinelle/spatchcache diff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache new file mode 100755 index 00000000000..7ec0dfcb1e4 --- /dev/null +++ b/contrib/coccinelle/spatchcache @@ -0,0 +1,272 @@ +#!/bin/sh +# +# spatchcache: a poor-man's "ccache"-alike for "spatch" in git.git +# +# This caching command relies on the peculiarities of the Makefile +# driving "spatch" in git.git, in particular if we invoke: +# +# make +# make coccicheck SPATCH_FLAGS=--very-quiet +# +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with +# "gcc" and "clang") write e.g. a .depend/grep.o.d for grep.c, when we +# compile grep.o. +# +# The .depend/grep.o.d will have the full header dependency tree of +# grep.c, and we can thus cache the output of "spatch" by: +# +# 1. Hashing all of those files +# 2. Hashing our source file, and the *.cocci rule we're +# applying +# 3. Running spatch, if suggests no changes (by far the common +# case) we invoke "spatchCache.getCmd" and +# "spatchCache.setCmd" with a hash SHA-256 to ask "does this +# ID have no changes" or "say that ID had no changes> +# 4. If no "spatchCache.{set,get}Cmd" is specified we'll use +# "redis-cli" and maintain a SET called "spatch-cache". Set +# appropriate redis memory policies to keep it from growing +# out of control. +# +# This along with the general incremental "make" support for +# "contrib/coccinelle" makes it viable to (re-)run coccicheck +# e.g. when merging integration branches. +# +# Note that the "--very-quiet" flag is currently critical. The cache +# will refuse to cache anything that has output on STDERR (which might +# be errors from spatch), but see spatchCache.cacheWhenStderr below. +# +# The STDERR (and exit code) could in principle be cached (as with +# ccache), but then the simple structure in the Redis cache would need +# to change, so just supply "--very-quiet" for now. +# +# To use this, simply set SPATCH to +# contrib/coccinelle/spatchcache. Then optionally set: +# +# [spatchCache] +# # Optional: path to a custom spatch +# spatch = ~/g/coccicheck/spatch.opt +# +# As well as this trace config (debug implies trace): +# +# cacheWhenStderr = true +# trace = false +# debug = false +# +# The ".depend/grep.o.d" can also be customized, as a string that will +# be eval'd, it has access to a "$dirname" and "$basename": +# +# [spatchCache] +# dependFormat = "$dirname/.depend/${basename%.c}.o.d" +# +# Setting "trace" to "true" allows for seeing when we have a cache HIT +# or MISS. To debug whether the cache is working do that, and run e.g.: +# +# redis-cli FLUSHALL +# +# grep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c +# 600 CANTCACHE +# 7365 MISS +# 7365 SET +# +# A subsequent "make cocciclean && make coccicheck" should then have +# all "HIT"'s and "CANTCACHE"'s. +# +# The "spatchCache.cacheWhenStderr" option is critical when using +# spatchCache.{trace,debug} to debug whether something is set in the +# cache, as we'll write to the spatch logs in .build/* we'd otherwise +# always emit a NOCACHE. + +set -e + +## Our own configuration & options +debug=$(git config --bool "spatchCache.debug") +if test "$debug" != "true" +then + debug= +fi +if test -n "$debug" +then + set -x +fi + +trace=$(git config --bool "spatchCache.trace") +if test "$trace" != "true" +then + trace= +fi +if test -n "$debug" +then + # debug implies trace + trace=true +fi + +cacheWhenStderr=$(git config --bool "spatchCache.cacheWhenStderr") +if test "$cacheWhenStderr" != "true" +then + cacheWhenStderr= +fi + +trace_it () { + if test -z "$trace" + then + return + fi + echo "$@" >&2 +} + +spatch=$(git config --path "spatchCache.spatch" || :) +if test -n "$spatch" +then + if test -n "$debug" + then + trace_it "custom spatchCache.spatch='$spatch'" + fi +else + spatch=spatch +fi + +dependFormat='$dirname/.depend/${basename%.c}.o.d' +dependFormatCfg=$(git config "spatchCache.dependFormat" || :) +if test -n "$dependFormatCfg" +then + dependFormat="$dependFormatCfg" +fi + +set=$(git config spatchCache.setCmd || :) +get=$(git config spatchCache.getCmd || :) + +## Parse spatch()-like command-line for caching info +arg_sp= +arg_file= +args="$@" +spatch_opts() { + while test $# != 0 + do + arg_file="$1" + case "$1" in + --sp-file) + arg_sp="$2" + ;; + esac + shift + done +} +spatch_opts "$@" +if ! test -f "$arg_file" +then + arg_file= +fi + +hash_for_cache() { + # Parameters that should affect the cache + echo "args=$args" + echo "config spatchCache.spatch=$spatch" + echo "config spatchCache.debug=$debug" + echo "config spatchCache.trace=$trace" + echo "config spatchCache.cacheWhenStderr=$cacheWhenStderr" + echo + + # Our target file and its dependencies + git hash-object "$1" "$2" $(grep -E -o '^[^:]+:$' "$3" | tr -d ':') +} + +# Sanity checks +if ! test -f "$arg_sp" && ! test -f "$arg_file" +then + echo $0: no idea how to cache "$@" >&2 + exit 128 +fi + +# Main logic +dirname=$(dirname "$arg_file") +basename=$(basename "$arg_file") +eval "dep=$dependFormat" + +if ! test -f "$dep" +then + trace_it "$0: CANTCACHE have no '$dep' for '$arg_file'!" + exec "$spatch" "$@" +fi + +if test -n "$debug" +then + trace_it "$0: The full cache input for '$arg_sp' '$arg_file' '$dep'" + hash_for_cache "$arg_sp" "$arg_file" "$dep" >&2 +fi +sum=$(hash_for_cache "$arg_sp" "$arg_file" "$dep" | git hash-object --stdin) + +trace_it "$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'" + +getret= +if test -z "$get" +then + if test $(redis-cli SISMEMBER spatch-cache "$sum") = 1 + then + getret=0 + else + getret=1 + fi +else + $set "$sum" + getret=$? +fi + +if test "$getret" = 0 +then + trace_it "$0: HIT for '$arg_file' with '$arg_sp'" + exit 0 +else + trace_it "$0: MISS: for '$arg_file' with '$arg_sp'" +fi + +out="$(mktemp)" +err="$(mktemp)" + +set +e +"$spatch" "$@" >"$out" 2>>"$err" +ret=$? +cat "$out" +cat "$err" >&2 +set -e + +nocache= +if test $ret != 0 +then + nocache="exited non-zero: $ret" +elif test -s "$out" +then + nocache="had patch output" +elif test -z "$cacheWhenStderr" && test -s "$err" +then + nocache="had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)" +fi + +if test -n "$nocache" +then + trace_it "$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'" + exit "$ret" +fi + +trace_it "$0: SET: for '$arg_file' with '$arg_sp'" + +setret= +if test -z "$set" +then + if test $(redis-cli SADD spatch-cache "$sum") = 1 + then + setret=0 + else + setret=1 + fi +else + "$set" "$sum" + setret=$? +fi + +if test "$setret" != 0 +then + echo "FAILED to set '$sum' in cache!" >&2 + exit 128 +fi + +exit "$ret"