From patchwork Tue Nov 1 22:35:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5320AC433FE for ; Tue, 1 Nov 2022 22:36:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231313AbiKAWgy (ORCPT ); Tue, 1 Nov 2022 18:36:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231273AbiKAWgh (ORCPT ); Tue, 1 Nov 2022 18:36:37 -0400 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6387D201AA for ; Tue, 1 Nov 2022 15:36:03 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id ud5so40815298ejc.4 for ; Tue, 01 Nov 2022 15:36:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9i0ODwq7p4KVBBc2IQQkiYby8hh3z3ZxvRjg5+jaQ7k=; b=Y6U85o1TGc9QwQsQ+hd0ubkheR7pQ3es6lbTjf5kCV4uqLKRejYa9RpRgPeSS7PAPQ 8xzB3JEo98PcGAUMTVnRSryc1Hla7uABrUnAFJk5sMzY/CvuvRV16V+40dW90C3sdLVb uUptad2gECZPwavzhzq4SdCIScpxlEJDdsATKZ6aZwF0UQ8no3fTxg53XmoD6tL+s4w9 GxK6G3TYkICyeGUSWi9SCagqQsRdH3sG5snu4uTy7o1j497+N54b0zMQQw2cZEUAYfH8 LNb5RprsU53JioDeD832VKRkSgzuIR+OcfG5xS2M5NdyGENzyJkQ80fZK0adzgZpA05O SqXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9i0ODwq7p4KVBBc2IQQkiYby8hh3z3ZxvRjg5+jaQ7k=; b=nNuLwoy2mr4F5gsiT3uYU7qGHYV2wPwBsCJn6V6gKxbtHz6X8N52MpL7nAf///fit9 8K4Azk2TzvjueoL8d6B5ZczCZ0V7mI8J4y4sWBNJ/gvPUddmAqStkc9rrq1yt0OFrLEu 0LhOypBfm5Qdw6q9PbDpLOCSH+VwDC80NTxgDY+S+d7osTnwpWEmGlo8TPFLSOC04oxE /5RwMh2j2IpSJiAlKK3niHWmY8QT9L+E6WVFrVS9nl4dmAdNMlpDfLwc2koLY5F86wwG Rn5T5zuuzeqhiWVoeW1oc8jL6JdZAdKV1gFAfH1VV1m42rUDm+VJ6AxO88gzA0933vcV pTyA== X-Gm-Message-State: ACrzQf2g7eaw6Bar1kjZ/Z2p0MxakT39WqglpaCYt1Osv17oBfW07iHL fh3Wy5N2C+HQkgtOrBg33LNFteMaxpI4rQ== X-Google-Smtp-Source: AMsMyM7upA2UVDaWBYEex0vM402/I6RNCJUeR3Zi9prDa/ry5n+yr+PBAPbBIGwroID6b685JOu9+w== X-Received: by 2002:a17:907:760e:b0:7ad:e4c9:dc79 with SMTP id jx14-20020a170907760e00b007ade4c9dc79mr8346617ejc.308.1667342161713; Tue, 01 Nov 2022 15:36:01 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:00 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 01/13] Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T) Date: Tue, 1 Nov 2022 23:35:43 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) we abbreviated "_TEST" to "_T" to have it align with the rest of the "="'s above it. Subsequent commits will add more QUIET_SPATCH_* variables, so let's stop abbreviating this, and indent it in preparation for adding more of these variables. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 2 +- shared.mak | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 4927379184c..b07ee35e17d 100644 --- a/Makefile +++ b/Makefile @@ -3172,7 +3172,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci $(call mkdir_p_parent_template) - $(QUIET_SPATCH_T)$(SPATCH) $(SPATCH_FLAGS) \ + $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \ --very-quiet --no-show-diff \ --sp-file $< -o $@ \ $(@:.build/%.res=%.c) && \ diff --git a/shared.mak b/shared.mak index 33f43edbf9a..96b06acc455 100644 --- a/shared.mak +++ b/shared.mak @@ -69,8 +69,10 @@ ifndef V QUIET_SP = @echo ' ' SP $<; QUIET_HDR = @echo ' ' HDR $(<:hcc=h); QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; - QUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%); + +## Used in "Makefile": SPATCH + QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_SPATCH_TEST = @echo ' ' SPATCH TEST $(@:.build/%=%); ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; From patchwork Tue Nov 1 22:35:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFBB9C4332F for ; Tue, 1 Nov 2022 22:36:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231349AbiKAWg5 (ORCPT ); Tue, 1 Nov 2022 18:36:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231296AbiKAWgi (ORCPT ); Tue, 1 Nov 2022 18:36:38 -0400 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53EA6201BB for ; Tue, 1 Nov 2022 15:36:04 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id f27so40782442eje.1 for ; Tue, 01 Nov 2022 15:36:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wEsGcDbioj3TrSYABwZLxNQdhqiyZAxuNg/HCrBStho=; b=oGAOKJVXUbzuu1Q3b9gF7p/P/j7M5lGQNXgpMEvzEksih0Rgmw7t2HXhGZrapjd1LA iU9K8wwRxBlSzqrRAUDkWqh6YF1pem7tlJG/lVcdi4t+oMFYJnnIdC9y4W+8SGgtQUGR f0H1KdmZ3VMm826cRcP8QK1SrvQuH5pQ50qWw2dZEmgutPFdCAST9tcpe7821EtbsBSb fOLbeSkxyMcwwjUVNGhPuQ4enOwOBTvgUXuJPW85VWAk+mFBlCKH8Ry/98F8vLwnyTmc 3TtgLm2ejsqJRmSX+W8mghyxNv9Qtr5dW4BZ3cjClMVk7mNFtWDmHfZUhYyb6WT2MM8+ 3eIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wEsGcDbioj3TrSYABwZLxNQdhqiyZAxuNg/HCrBStho=; b=5+fFi4m2Jpu1Gx8PaPAjz1zp133S2OKHOBuDKCE/eRxE+vUDYDx1C3C6JJklWhXARj bEF6VIebbIpnuxPDxdQi1DYoTlALADLExJbxRBNHxlCXViZxYTzhO/PXsRzFKMFUqJUN PTKjtZiW3aLvtJ6JjyfFER1Cy7omgYNQcmjQXJWs2ueDdj6jFlTbDhylUrkQuMRhFgB9 VmabeHf3nkkkCu2/f4V7zH+0R7BmgKQQTcIbHqE/vme3ZMyIrYpkpwvU51VZOL1gx7ap /GxhNthzkzoiUEaz4r7kdJ+BrLgzGy8KmhkOWzX00HCYNmqpwc3Lo7EYmNK3zLBJjooZ Q5NQ== X-Gm-Message-State: ACrzQf0rCzNacDIK7JkIiFumjdK6/R8yeERi3He8wQwF4baw5avdM7OC BQCV3Rb/9xE5VluTBCd9Xr1dHkuCC1/qzg== X-Google-Smtp-Source: AMsMyM7Yo8J0AqXk8Fn0HT/Ffo+Hpb+2xid+SRQQfYiBCwQErBWTq3wLF/1giAwY+r3Sb7c6eas13A== X-Received: by 2002:a17:906:8a6f:b0:780:96b4:d19e with SMTP id hy15-20020a1709068a6f00b0078096b4d19emr20317058ejc.624.1667342162571; Tue, 01 Nov 2022 15:36:02 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:01 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 02/13] cocci rules: remove unused "F" metavariable from pending rule Date: Tue, 1 Nov 2022 23:35:44 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix an issue with a rule added in 9b45f499818 (object-store: prepare has_{sha1, object}_file to handle any repo, 2018-11-13). We've been spewing out this warning into our $@.log since that rule was added: warning: rule starting on line 21: metavariable F not used in the - or context code We should do a better job of scouring our coccinelle log files for such issues, but for now let's fix this as a one-off. Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/the_repository.pending.cocci | 1 - 1 file changed, 1 deletion(-) diff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci index 072ea0d9228..747d382ff5f 100644 --- a/contrib/coccinelle/the_repository.pending.cocci +++ b/contrib/coccinelle/the_repository.pending.cocci @@ -20,7 +20,6 @@ expression E; @@ expression E; -expression F; @@ - has_object_file_with_flags( + repo_has_object_file_with_flags(the_repository, From patchwork Tue Nov 1 22:35:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C43AC4332F for ; Tue, 1 Nov 2022 22:37:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231405AbiKAWg7 (ORCPT ); Tue, 1 Nov 2022 18:36:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231297AbiKAWgj (ORCPT ); Tue, 1 Nov 2022 18:36:39 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E95A201BD for ; Tue, 1 Nov 2022 15:36:05 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id y14so40787638ejd.9 for ; Tue, 01 Nov 2022 15:36:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hz75qdjgmxuIXxRyP7RyUMm4W1pPF0rrghOokrjmf9Y=; b=SfvJY1exRXdrrbpw8N9jY4rYv9wNfLpVKxPp5gqeIxtzwdPcib3zNOE/pwM9HQd5qQ W0GY0Ky9DXcIYyMPaOOxeeClki5BhlVN9U5hBclrkyfyyNTSwTKIhVb72tRaFZps9HBz quTiKPbiFsChbzwLe/jsTLzY4XvxOGDPzG6nVJX1Z2m4tLSaVXrm6W2AteoY3kpENLEc ODYdexHzHzYsJZtRf9B/IRHLjdhUd16euPh7xlm5DIwiPZNCDVjhO8lFDgfetK78n8+l Aiu6c2Lan0cezLeHQMDwM+Iu3pFhSipYZiEXqDvViyxu8/2xCVbAY/ly4mLOb5pqFy6b WuOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hz75qdjgmxuIXxRyP7RyUMm4W1pPF0rrghOokrjmf9Y=; b=e8mtes5OGUtdwyBcOt9F7b/V/61iAVBIhOqw8lcL21ZPKddE9+BOP7vqSo6L6b5wDU bWtpbEp8U3NJz3G7BflP0mg+sFrEwwnOPXGyx/MPBPiXfVvl9gLUfNvZFLQUolk6gqta x6ggY5I8N3TRnlQLkAvDLoda6iGBPuO/47TLFWPoo29UxZUrYWTJ3MvJjh2A4EQy0CX8 5KEY76ln9azsy7eKvTbqXx8Xy6PS9wgOyUnjj4Useq1qQIeyaNwz8fuMX7s7I7eL7O7c IJ0hlN0vzJni9Vbs/7IgJch5920XDdAIRSALHkZOImp0n4B9/pJZbhcxRilxrYXLBMne 24rQ== X-Gm-Message-State: ACrzQf0fKminyMjRSu4K831K22NFtEV5nVrU0kktKlvcvjj0nqblKEUS wt993CBqu02xz+MQcIvAwHp3Zhe6h27apA== X-Google-Smtp-Source: AMsMyM7nUNYxXoU7Bphrw4NF6Zl2KCOEQnwW/Gc9MLvL6S5M2yJBUHKUFOexx2MTiKneRxJPQAHf5A== X-Received: by 2002:a17:906:3b48:b0:7ad:fa67:1daa with SMTP id h8-20020a1709063b4800b007adfa671daamr1990376ejf.653.1667342163518; Tue, 01 Nov 2022 15:36:03 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:02 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 03/13] Makefile: add ability to TAB-complete cocci *.patch rules Date: Tue, 1 Nov 2022 23:35:45 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Declare the contrib/coccinelle/.cocci.patch rules in such a way as to allow TAB-completion, and slightly optimize the Makefile by cutting down on the number of $(wildcard) in favor of defining "coccicheck" and "coccicheck-pending" in terms of the same incrementally filtered list. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index b07ee35e17d..828d332096a 100644 --- a/Makefile +++ b/Makefile @@ -3144,9 +3144,20 @@ check: $(GENERATED_H) exit 1; \ fi +COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) +COCCI_RULES = $(COCCI_GLOB) + +COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES)) +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) + +COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch) +COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) + COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -%.cocci.patch: %.cocci $(COCCI_SOURCES) +COCCI_PATCHES = $(COCCI_RULES:%=%.patch) +$(COCCI_PATCHES): $(COCCI_SOURCES) +$(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \ limit=; \ @@ -3183,11 +3194,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test -coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci))) +coccicheck: $(COCCICHECK_PATCHES) # See contrib/coccinelle/README coccicheck-pending: coccicheck-test -coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci)) +coccicheck-pending: $(COCCICHECK_PATCHES_PENDING) .PHONY: coccicheck coccicheck-pending From patchwork Tue Nov 1 22:35:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ED2CC433FE for ; Tue, 1 Nov 2022 22:37:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231414AbiKAWhC (ORCPT ); Tue, 1 Nov 2022 18:37:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231339AbiKAWgk (ORCPT ); Tue, 1 Nov 2022 18:36:40 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B91E1A3B5 for ; Tue, 1 Nov 2022 15:36:06 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id t25so40792358ejb.8 for ; Tue, 01 Nov 2022 15:36:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8mzFEpdaqsDCo8f2gxQ7sl29y3emxaFEH+mHNyQ34cE=; b=Mp21wHW+URjliteZxnRUo8jUKBDxJviG6jI8bQcIJY9a8irTMj2ocWLq4UFFgLI8yq RJAmbACyaVfybyaafdMhm5Z2FCcjVEI9JsvPqxZja3f6MZ967HkMZGyHlCs++SnJs9pv pm8nmzMETkYFeJ0AsUmPWH17PB4//qAO2nADpHaVvXsUqJb06qmEdKlX0uxN+9xnxucQ MZpWklz7PU8+cGnxJNh2v8WsIdms+MJeM2djrYACFPR01mSIX6uXcPICrDabOFujwXkS mxYrESPRyLzT586rZwopYCWeP6fvblpBV0yjPfGxddSeXNmmlpUpqIFaHiu09qE/qjZS fdng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8mzFEpdaqsDCo8f2gxQ7sl29y3emxaFEH+mHNyQ34cE=; b=KFmDVlgTeaP1jzfAz1AwFhFR2f4PzoMl0VM9hJJQIYxV5fxnBsMZhK5/ONbVFJlXQx 0AK7Xds6MVJRW0WLScRjLSaalaUkeLdRNaR2yI6ceiE5z/qTFLJQ4Q5/PX2HJHpZQoSA gcTunSOCHWJDhttk02uRUvR+TI5vEs1WxiHYBMEUiuZGaAxoXsr45HFE6E4K4XJfeobQ cZPLob9WvtatSnKqlq6lCpmKEfhnZYA14ajFXsB+WEHks/BpUhFOL48LzaZjic/tsrJW +EI52GEuy84lbJzrvh0KsEtXTHUEXa+Kozt56oDDpUmnIvHbdkF4PoeTdIUe1XfUbyJP 9ZAQ== X-Gm-Message-State: ACrzQf0zB8ccMr2TfuquULQUauAY5z8Ezq5H3Zwm8ok2/UN4WGxBxSwE ipihfHfeBG3HE4q4ZEWtevp82DHonDerXg== X-Google-Smtp-Source: AMsMyM4uhDFiuL//YHmxqRMlDgM/brtXd+DRDkM4z2cBTpnRNjdvxOGVH2lz/TXHUtAFghy8AGMU7Q== X-Received: by 2002:a17:906:9b8e:b0:791:8f57:6845 with SMTP id dd14-20020a1709069b8e00b007918f576845mr20998243ejc.174.1667342164452; Tue, 01 Nov 2022 15:36:04 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:03 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 04/13] Makefile: have "coccicheck" re-run if flags change Date: Tue, 1 Nov 2022 23:35:46 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix an issue with the "coccicheck" family of rules that's been here since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike e.g. "make grep.o" we wouldn't re-run it when $(SPATCH) or $(SPATCH_FLAGS) changed. To test new flags we needed to first do a "make cocciclean". This now uses the same (copy/pasted) pattern as other "DEFINES" rules. As a result we'll re-run properly. This can be demonstrated e.g. on the issue noted in [1]: $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1 [...] SPATCH contrib/coccinelle/xcalloc.cocci $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS="--all-includes --recursive-includes" * new spatch flags SPATCH contrib/coccinelle/xcalloc.cocci SPATCH result: contrib/coccinelle/xcalloc.cocci.patch $ 1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/ Signed-off-by: Ævar Arnfjörð Bjarmason --- .gitignore | 1 + Makefile | 15 +++++++++++++++ 2 files changed, 16 insertions(+) diff --git a/.gitignore b/.gitignore index cb0231fb401..0832f1da77b 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ /GIT-PERL-HEADER /GIT-PYTHON-VARS /GIT-SCRIPT-DEFINES +/GIT-SPATCH-DEFINES /GIT-USER-AGENT /GIT-VERSION-FILE /bin-wrappers/ diff --git a/Makefile b/Makefile index 828d332096a..d81b0931ddd 100644 --- a/Makefile +++ b/Makefile @@ -1308,6 +1308,18 @@ SANITIZE_ADDRESS = SPATCH_FLAGS = --all-includes SPATCH_BATCH_SIZE = 1 +# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change +TRACK_SPATCH_DEFINES = +TRACK_SPATCH_DEFINES += $(SPATCH) +TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) +TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) +GIT-SPATCH-DEFINES: FORCE + @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ + if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ + echo >&2 " * new spatch flags"; \ + echo "$$FLAGS" >GIT-SPATCH-DEFINES; \ + fi + include config.mak.uname -include config.mak.autogen -include config.mak @@ -3156,6 +3168,7 @@ COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) COCCI_PATCHES = $(COCCI_RULES:%=%.patch) +$(COCCI_PATCHES): GIT-SPATCH-DEFINES $(COCCI_PATCHES): $(COCCI_SOURCES) $(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ @@ -3179,6 +3192,7 @@ $(COCCI_PATCHES): %.patch: % fi COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) +$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci @@ -3465,6 +3479,7 @@ profile-clean: $(RM) $(addsuffix *.gcno,$(addprefix $(PROFILE_DIR)/, $(object_dirs))) cocciclean: + $(RM) GIT-SPATCH-DEFINES $(RM) -r .build/contrib/coccinelle $(RM) contrib/coccinelle/*.cocci.patch* From patchwork Tue Nov 1 22:35:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45CE2C4332F for ; Tue, 1 Nov 2022 22:37:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229933AbiKAWhH (ORCPT ); Tue, 1 Nov 2022 18:37:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230427AbiKAWgl (ORCPT ); Tue, 1 Nov 2022 18:36:41 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D4711B789 for ; Tue, 1 Nov 2022 15:36:07 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id k2so40811194ejr.2 for ; Tue, 01 Nov 2022 15:36:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Cv7Ha4jPIZFzajFvVVbIPnGzkWE2fHrj1dJoi+RdFQo=; b=T5d1NofFa4GM5S7ssKcSDE1uIOeaoV28Vg6svrJtgcWKEGGnQKV6YZWsFhU4C5m9v/ ds+sXMdV1oNi+JMmjxTMWelC+CkDk5faBG1mv6vRp4QeMPzFAUzBfuQbwRX13WA/+Hu0 nkYdeyPfpZQfb0V0US3ZT7r4pRi04wAbtVhN9Z3e2cfTH4dycobvjdnSMxFfqHLDrzlU jkjeA2trFGGgfTNIrkIJfy/rrU3qLKOTezW0wtEKJIy+KTdzUvcBF12UXmfseytUY7C7 ie6vSPwMeV8RAQDtrWUfbVpRIhe1NXLZvd9pqfM00oI80DIbFTqsTgw23tKaoUs+Baik OX0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Cv7Ha4jPIZFzajFvVVbIPnGzkWE2fHrj1dJoi+RdFQo=; b=3fOc1hx8ToGpTMGTwEr7KASXv1xvaPXPM1WrC8crqSl7XhzD/lthT4jkM5MvUnalmS CZG4GuzPdbQ7rrR18WcsK8KiiuTF5946uYFM2EmvuleCnZ7CShF0uzH2X4geEYodPU6R Kmt6vbpzWeVB697VjykM6A0tqwpCtXYVon8r7auKN58uDSSe7wJdvFS6Y6P7VejsWgXY j3Chy3J9LtxtyXnix63Dqb0BvxKryhwSjy7wmfaK5c9+XjlSsII8tCrwfZfgmDEt+fhX fS/8JFiuGFmZLSxA98nXOCp9TKCfxLCAxVwuVtg488TQtrDj9MMKbJqWPnG9AJ7lYa3Q vYzw== X-Gm-Message-State: ACrzQf25Rb4bvapnQvM5Eo6XpP5LQ13Nw2FP9Zs7SDwt8F3pdDpiT8mv Y2xrUXlwfGIIzbfLonFY/ypaTGaLBk42zQ== X-Google-Smtp-Source: AMsMyM6BmcTT1qNlfrcpjXoWkqUoD4ZoFCl+YDO1ybi6vyzR04dgvCxlUkzAgJ37dB8rS7zNaRWQLA== X-Received: by 2002:a17:906:3fd2:b0:78d:b793:5ef9 with SMTP id k18-20020a1709063fd200b0078db7935ef9mr20030416ejj.496.1667342165461; Tue, 01 Nov 2022 15:36:05 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:04 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 05/13] Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" heading Date: Tue, 1 Nov 2022 23:35:47 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Split off the "; setting[...]" part of the comment added in In 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06), and restore what we had before that, which was a comment indicating that variables for the "coccicheck" target were being set here. When 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it left no natural place to add a new comment about other flags that preceded it. As subsequent commits will add such comments we need to split the existing comment up. The wrapping for the "SPATCH_BATCH_SIZE" is now a bit odd, but minimizes the diff size. As a subsequent commit will remove that feature altogether this is worth it. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index d81b0931ddd..8c4751e1bd6 100644 --- a/Makefile +++ b/Makefile @@ -1302,10 +1302,11 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer SANITIZE_LEAK = SANITIZE_ADDRESS = -# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will +# For the 'coccicheck' target +SPATCH_FLAGS = --all-includes +# Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_FLAGS = --all-includes SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change From patchwork Tue Nov 1 22:35:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 042CDC4332F for ; Tue, 1 Nov 2022 22:37:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231418AbiKAWhE (ORCPT ); Tue, 1 Nov 2022 18:37:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231166AbiKAWgl (ORCPT ); Tue, 1 Nov 2022 18:36:41 -0400 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47398647A for ; Tue, 1 Nov 2022 15:36:07 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id f27so40782816eje.1 for ; Tue, 01 Nov 2022 15:36:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ToZjVXMHPb/KCSianxlKaWCBzKTx89aKBrYNYdCa1vU=; b=cFTIfgEGsAGXccICx8EvBqUynQKPA0Q5ZGPXixgxwrK48E86V+6mMW3nZsW/clJPqJ GiOWmuDl+vDhO4I5Qn/oa7raZRaANQiPTGceEFYrKTAGi+q8RJlO6oP8XRZFf0OkRDvb SD5hwGbmoxcvZkE7XudTUsosNZZB9c/LlkVoJ7+L0bPvOkvtpomRhRu7sdD1u4aBEHuq sVywEgbob+F5etT7noRvWSNFfH3ebkOM0RtUsWi8Jp8m7aTaAlXS7FBMNF5HHAVSYBXe jTg5GtmYDrVzQhvZ+IckYNxNjKiW4rtbSJWc0PFTSIL2Zl6LiMaGdEgmXVEjlD0xuTOz ++sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ToZjVXMHPb/KCSianxlKaWCBzKTx89aKBrYNYdCa1vU=; b=kx13JWKU6C7TIlLSDL5uO3HoQbSq/Wa/zjZiIzpV5G4934C+ql8jZ7ZslefVeQ/7cA fNTGphNn9Y2+uOGk1rRhWgn+Aj/FVe7hdpYYCqqYTLArjcmYlaVEerZsGMhD8QOyAQCK utG9VvCPIOqRwJX6uUA4X9T4/b64yZMf2UY+uf8FptijQk+oWQCwpH1wU+dVwQj02W84 F1A5nvITbWwFyX6TCslVkscWG1wkgzIFypEBZgB0XbIKpBY50/qGSY2XW0cvBFKHjUnL lh6HUuAgkHeaR0LrhtWDt4Tups+D0yXjPeDmDK38pRqiPkq2jGvmXokNnfqKKOl15fFT ZtjA== X-Gm-Message-State: ACrzQf3rKsk2BCfKO7CbXzv+HS9KeXjY2Q3MAZJw1fzZw7JMlExfLcpc RCS/++lLKyT5BO0JpddvJU+OixY3HG+iPw== X-Google-Smtp-Source: AMsMyM4O7+1j796YBLm1TYW7veMCxS0sRnPVbt+owEUBsB5N5ssMKS0DLzpYXsdamSHNjPWfmDG5MA== X-Received: by 2002:a17:907:70b:b0:740:ef93:2ffb with SMTP id xb11-20020a170907070b00b00740ef932ffbmr19640361ejb.93.1667342166536; Tue, 01 Nov 2022 15:36:06 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:05 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 06/13] cocci: split off include-less "tests" from SPATCH_FLAGS Date: Tue, 1 Nov 2022 23:35:48 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Amend the "coccicheck-test" rule added in f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) to stop using "--all-includes". The flags we'll need for the tests are different than the ones we'll need for our main source code. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 8c4751e1bd6..6785e8175b5 100644 --- a/Makefile +++ b/Makefile @@ -1304,6 +1304,7 @@ SANITIZE_ADDRESS = # For the 'coccicheck' target SPATCH_FLAGS = --all-includes +SPATCH_TEST_FLAGS = # Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. @@ -1313,6 +1314,7 @@ SPATCH_BATCH_SIZE = 1 TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) +TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ @@ -3198,7 +3200,7 @@ $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci $(call mkdir_p_parent_template) - $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_FLAGS) \ + $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \ --very-quiet --no-show-diff \ --sp-file $< -o $@ \ $(@:.build/%.res=%.c) && \ From patchwork Tue Nov 1 22:35:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8065C4332F for ; Tue, 1 Nov 2022 22:37:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231368AbiKAWhJ (ORCPT ); Tue, 1 Nov 2022 18:37:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231319AbiKAWgm (ORCPT ); Tue, 1 Nov 2022 18:36:42 -0400 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4128D1EC7E for ; Tue, 1 Nov 2022 15:36:08 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id ud5so40815848ejc.4 for ; Tue, 01 Nov 2022 15:36:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xExwtgC3Cf4QzawXzrNZJGVKj2Rmiwedkn0aG/AC7+U=; b=iveiPrGkiLA+6xvJ1gqF4rU1v1j51OyYaQMBkzf/NGk21klWX0nmpjb7BTgVdO/3qO ki+111MPdFpA33GMeLQTclyaRB65HkC2auTKZIVzdWjdnYwe1/ivDJlIs26Yb4+CCjk2 NH/0iRzaYa87XW7MgTruIkJYYChYnZjbhxpPC6bgXMqcVCWMS9JGqy0Q9RDb2Ug9JRRJ 7zxvbBIijdUS+B0N0K3dwX+S4lWraDXDK1Am2HSsjnjgbYxCCOZlib34i5rmjXGNJ5UB a3WXAIAwTt3NYvhJRJlHEjfksUne/A3/79RjNq12cL/3PAUN2L/sIML+LerYuRnye+MF +QMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xExwtgC3Cf4QzawXzrNZJGVKj2Rmiwedkn0aG/AC7+U=; b=rQf5fnpy4cQgIbZVSzaB9LJus16+ruIWEBrGHhTucedklf8mJCRq6YiQkXtV/F5FlA hMgACC+kvTexu9bDT9ZL0Vww3YFtLBcKcqNO4HixS9dzAASUaKKFSjrvtAe+eP3jqIJu 5QO7YncjqA0ApPWapiq7Y679qWf3+4khaJisQArQ2TsfyphKCRehU6ji72m4HAQLH9U+ vWwG4TyXQ41Q3HrcD9RQCpMsllru8GOYTJ1Qx/8EOZ33NF/WI35s1eN1oKFy3o2j2a3C EapO8G1s54X26jVgYrl7R65RJL6R0/9yGswsVAB7LN9N8Is8EdxUqzuYRwUk7mpJpW52 HvZg== X-Gm-Message-State: ACrzQf254sI1sveOIGXgukq5QbPvKDS1KUWqLdIBU6db5E5fTekmL/tr 8H/+slQOAenhaNie+T1r7uq/AHMYNWCH1A== X-Google-Smtp-Source: AMsMyM7zfaog+3L5pdppjUvBJhh6enQUbTJGtX4h1xam2rZpQiCLi5ZkgVNrfcKgJK9/LdcOAwmKIQ== X-Received: by 2002:a17:907:1dc7:b0:7ad:83d5:6abd with SMTP id og7-20020a1709071dc700b007ad83d56abdmr19599639ejc.412.1667342167466; Tue, 01 Nov 2022 15:36:07 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:06 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 07/13] cocci: split off "--all-includes" from SPATCH_FLAGS Date: Tue, 1 Nov 2022 23:35:49 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Per the rationale in 7b63ea57500 (Makefile: remove mandatory "spatch" arguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that are truly mandatory, such as "--sp-file" and "--patch .". The "--all-includes" flag is also critical, but per [1] we might want to ad-hoc tweak it occasionally for testing or one-offs. But being unable to set e.g. SPATCH_FLAGS="--verbose-parsing" without breaking how our "spatch" works isn't ideal, i.e. before this we'd need to know about the default include flags, and specify: SPATCH_FLAGS="--all-includes --verbose-parsing". If we were then to change the default include flag (e.g. to "--recursive-includes") in the future any such one-off commands would need to be correspondingly updated. Let's instead leave the SPATCH_FLAGS for the user, while creating a new SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include strategy itself. 1. https://lore.kernel.org/git/20220823095733.58685-1-szeder.dev@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 6785e8175b5..25f0c7e837d 100644 --- a/Makefile +++ b/Makefile @@ -1303,7 +1303,8 @@ SANITIZE_LEAK = SANITIZE_ADDRESS = # For the 'coccicheck' target -SPATCH_FLAGS = --all-includes +SPATCH_INCLUDE_FLAGS = --all-includes +SPATCH_FLAGS = SPATCH_TEST_FLAGS = # Setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. @@ -1313,6 +1314,7 @@ SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) +TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) @@ -3182,6 +3184,7 @@ $(COCCI_PATCHES): %.patch: % fi; \ if ! echo $(COCCI_SOURCES) | xargs $$limit \ $(SPATCH) $(SPATCH_FLAGS) \ + $(SPATCH_INCLUDE_FLAGS) \ --sp-file $< --patch . \ >$@+ 2>$@.log; \ then \ From patchwork Tue Nov 1 22:35:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA250C433FE for ; Tue, 1 Nov 2022 22:37:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231425AbiKAWhX (ORCPT ); Tue, 1 Nov 2022 18:37:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231327AbiKAWgo (ORCPT ); Tue, 1 Nov 2022 18:36:44 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 616C715822 for ; Tue, 1 Nov 2022 15:36:10 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id ud5so40815961ejc.4 for ; Tue, 01 Nov 2022 15:36:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FCSVIOui7p3T+WTxDXizMVjFmNc+mJdGOMuuJJatyc8=; b=FuZjUbz9d7aHf0YDIED75Xof+EA4DAi/dxMbppcsKHd0TzUVdyOFBg98SXhL48v7Ub pMSD1zZDNnWj1MGI7XIYpgRaOvHFZnALVad1zfBq9+8tzU0rLpVJxwSc6AHha2Yz14Uk UINjmug9CcBI940TkVclgoGmhTGfgHtLZzRnA/2O3anxyUdUVos4NqwNvOP+ACVZh0kU ksY8H0YjX2RnQCGDft8uGO61ckqGeHXYdNkK5A+a1RVB2Dt10I/t+RfJQK6I2tryO/5r F7sDZ8+iZtthKIOwGFrbCQnRArCN7+UWZrbGfYI/t3EBIMj81eZsGyXza83U5tSRggxe 5trg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FCSVIOui7p3T+WTxDXizMVjFmNc+mJdGOMuuJJatyc8=; b=KLMvOQnQMUQKoumMMoFeyjN1bWtufWU1SLnCCn+xYdtgv2//m2k4Xfw8PT63PeTeBm hnzM7wa+hjjOE+n85VsWODvIy3Kd0ACGVGk5iTXKFi4sxCIzSht8/2P/zfvOfPHTzJDo aLLAbvqHKkOTcW29fdR7ZcW+o5CdOpFSD2Z68CN8tVs6pbX3P369POIE1CnFrj/lbxqz MJQMp/1IviRcEZSm1m5Bertp7Hh5hDgBd8NqKV0Wykb9/neQ0ucJmhwG3c5lb+VXGnIy mZr2vl4BPImX54Xqeb6uHP1CtFh8z9k4RxQpV1V67mlSi3o7bduHbh4mDezBnxougu07 1Vfw== X-Gm-Message-State: ACrzQf0JR5hvuOPALEFJ+cZ00RAmoIZbWzvHbo7o3U0amTy8KxMaFWtg eZqbS7BPC3yx5mMG8BMuKeOBChDt1MvkbQ== X-Google-Smtp-Source: AMsMyM446OHZAcPbWuY91H89OTuGWYQeel1fEXgLB0R3aubuuWIlcImqe2ne1x8SxIWRMt6Py3iPxg== X-Received: by 2002:a17:906:3406:b0:7ad:a8c3:585d with SMTP id c6-20020a170906340600b007ada8c3585dmr19839131ejb.573.1667342168436; Tue, 01 Nov 2022 15:36:08 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:07 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 08/13] cocci: make "coccicheck" rule incremental Date: Tue, 1 Nov 2022 23:35:50 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Optimize the very slow "coccicheck" target to take advantage of incremental rebuilding, and fix outstanding dependency problems with the existing rule. The rule is now faster both on the initial run as we can make better use of GNU make's parallelism than the old ad-hoc combination of make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the "--jobs" argument to "spatch(1)". It also makes us *much* faster when incrementally building, it's now viable to "make coccicheck" as topic branches are merged down. The rule didn't use FORCE (or its equivalents) before, so a: make coccicheck make coccicheck Would report nothing to do on the second iteration. But all of our patch output depended on all $(COCCI_SOURCES) files, therefore e.g.: make -W grep.c coccicheck Would do a full re-run, i.e. a a change in a single file would force us to do a full re-run. The reason for this (not the initial rationale, but my analysis) is: * Since we create a single "*.cocci.patch+" we don't know where to pick up where we left off, or how to incrementally merge e.g. a "grep.c" change with an existing *.cocci.patch. * We've been carrying forward the dependency on the *.c files since 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was initially added as a sort of poor man's dependency discovery. As we don't include other *.c files depending on other *.c files has always been broken, as could be trivially demonstrated e.g. with: make coccicheck make -W strbuf.h coccicheck However, depending on the corresponding *.c files has been doing something, namely that *if* an API change modified both *.c and *.h files we'd catch the change to the *.h we care about via the *.c being changed. For API changes that happened only via *.h files we'd do the wrong thing before this change, but e.g. for function additions (not "static inline" ones) catch the *.h change by proxy. Now we'll instead: * Create a / pair in the .build directory, E.g. for swap.cocci and grep.c we'll create .build/contrib/coccinelle/swap.cocci.patch/grep.c. That file is the diff we'll apply for that - combination, if there's no changes to me made (the common case) it'll be an empty file. * Our generated *.patch file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple "cat $^" of all of all of the / files for a given . In the case discussed above of "grep.c" being changed we'll do the full "cat" every time, so they resulting *.cocci.patch will always be correct and up-to-date, even if it's "incrementally updated". See 1cc0425a27c (Makefile: have "make pot" not "reset --hard", 2022-05-26) for another recent rule that used that technique. As before we'll: * End up generating a contrib/coccinelle/swap.cocci.patch, if we "fail" by creating a non-empty patch we'll still exit with a zero exit code. Arguably we should move to a more Makefile-native way of doing this, i.e. fail early, and if we want all of the "failed" changes we can use "make -k", but as the current "ci/run-static-analysis.sh" expects us to behave this way let's keep the existing behavior of exhaustively discovering all cocci changes, and only failing if spatch itself errors out. Further implementation details & notes: * Before this change running "make coccicheck" would by default end up pegging just one CPU at the very end for a while, usually as we'd finish whichever *.cocci rule was the most expensive. This could be mitigated by combining "make -jN" with SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06). There will be cases where getting rid of "SPATCH_BATCH_SIZE" makes things worse, but a from-scratch "make coccicheck" with the default of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference) is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed the CPU more work in a less staggered way. * Getting rid of "SPATCH_BATCH_SIZE" particularly helps in cases where the default of 1 yields parallelism under "make coccicheck", but then running e.g.: make -W contrib/coccinelle/swap.cocci coccicheck I.e. before that would use only one CPU core, until the user remembered to adjust "SPATCH_BATCH_SIZE" differently than the setting that makes sense when doing a non-incremental run of "make coccicheck". * Before the "make coccicheck" rule would have to clean "contrib/coccinelle/*.cocci.patch*", since we'd create "*+" and "*.log" files there. Now those are created in .build/contrib/coccinelle/, which is covered by the "cocciclean" rule already. Outstanding issues & future work: * We could get rid of "--all-includes" in favor of manually specifying a list of includes to give to "spatch(1)". As noted upthread of [1] a naïve removal of "--all-includes" will result in broken *.cocci patches, but if we know the exhaustive list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to re-scan for them, we could grab the headers to include from the .depend.d/.o.d and supply them with the "--include" option to spatch(1).q 1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 81 ++++++++++++++++++++++++----------- contrib/coccinelle/.gitignore | 2 +- shared.mak | 3 +- 3 files changed, 58 insertions(+), 28 deletions(-) diff --git a/Makefile b/Makefile index 25f0c7e837d..4bedff26285 100644 --- a/Makefile +++ b/Makefile @@ -1306,10 +1306,6 @@ SANITIZE_ADDRESS = SPATCH_INCLUDE_FLAGS = --all-includes SPATCH_FLAGS = SPATCH_TEST_FLAGS = -# Setting SPATCH_BATCH_SIZE higher will -# usually result in less CPU usage at the cost of higher peak memory. -# Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = @@ -1317,7 +1313,6 @@ TRACK_SPATCH_DEFINES += $(SPATCH) TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) -TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ @@ -3163,6 +3158,7 @@ check: $(GENERATED_H) COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) COCCI_RULES = $(COCCI_GLOB) +COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%) COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES)) COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) @@ -3170,32 +3166,65 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch) COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) +# It's expensive to compute the many=many rules below, only eval them +# on $(MAKECMDGOALS) that match these $(COCCI_RULES) +COCCI_RULES_GLOB = +COCCI_RULES_GLOB += cocci% +COCCI_RULES_GLOB += .build/contrib/coccinelle/% +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES) +COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING) +COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS)) + COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -COCCI_PATCHES = $(COCCI_RULES:%=%.patch) -$(COCCI_PATCHES): GIT-SPATCH-DEFINES -$(COCCI_PATCHES): $(COCCI_SOURCES) -$(COCCI_PATCHES): %.patch: % - $(QUIET_SPATCH) \ - if test $(SPATCH_BATCH_SIZE) = 0; then \ - limit=; \ - else \ - limit='-n $(SPATCH_BATCH_SIZE)'; \ - fi; \ - if ! echo $(COCCI_SOURCES) | xargs $$limit \ - $(SPATCH) $(SPATCH_FLAGS) \ - $(SPATCH_INCLUDE_FLAGS) \ - --sp-file $< --patch . \ - >$@+ 2>$@.log; \ +.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) + $(call mkdir_p_parent_template) + $(QUIET_GEN) >$@ + +define cocci-rule + +## Rule for .build/$(1).patch/$(2); Params: +# $(1) = e.g. "free.cocci" +# $(2) = e.g. "grep.c" +COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) +.build/$(1).patch/$(2): GIT-SPATCH-DEFINES +.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES +.build/$(1).patch/$(2): $(1) +.build/$(1).patch/$(2): .build/$(1).patch/% : % + $$(call mkdir_p_parent_template) + $$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \ + $$(SPATCH_INCLUDE_FLAGS) \ + --sp-file $(1) --patch . $$< \ + >$$@ 2>$$@.log; \ then \ - cat $@.log; \ + echo "ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:"; \ + cat $$@.log; \ exit 1; \ - fi; \ - mv $@+ $@; \ - if test -s $@; \ + fi +endef + +define cocci-matrix + +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s))) +endef + +ifdef COCCI_GOALS +$(eval $(foreach c,$(COCCI_RULES),$(call cocci-matrix,$(c)))) +endif + +define spatch-rule + +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) + $$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \ + if test -s $$@; \ then \ - echo ' ' SPATCH result: $@; \ + echo ' ' SPATCH result: $$@; \ fi +endef + +ifdef COCCI_GOALS +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n)))) +endif COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES @@ -3487,7 +3516,7 @@ profile-clean: cocciclean: $(RM) GIT-SPATCH-DEFINES $(RM) -r .build/contrib/coccinelle - $(RM) contrib/coccinelle/*.cocci.patch* + $(RM) contrib/coccinelle/*.cocci.patch clean: profile-clean coverage-clean cocciclean $(RM) -r .build diff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore index d3f29646dc3..1d45c0a40c8 100644 --- a/contrib/coccinelle/.gitignore +++ b/contrib/coccinelle/.gitignore @@ -1 +1 @@ -*.patch* +*.patch diff --git a/shared.mak b/shared.mak index 96b06acc455..f437073e48c 100644 --- a/shared.mak +++ b/shared.mak @@ -71,8 +71,9 @@ ifndef V QUIET_RC = @echo ' ' RC $@; ## Used in "Makefile": SPATCH - QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_SPATCH = @echo ' ' SPATCH $@; QUIET_SPATCH_TEST = @echo ' ' SPATCH TEST $(@:.build/%=%); + QUIET_SPATCH_CAT = @echo ' ' SPATCH CAT $$^ \>$@; ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; From patchwork Tue Nov 1 22:35:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C35C4332F for ; Tue, 1 Nov 2022 22:37:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231386AbiKAWh0 (ORCPT ); Tue, 1 Nov 2022 18:37:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231334AbiKAWgp (ORCPT ); Tue, 1 Nov 2022 18:36:45 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70CF62034E for ; Tue, 1 Nov 2022 15:36:11 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id b2so40797510eja.6 for ; Tue, 01 Nov 2022 15:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YxbYKbELPp7cXOUSqr2Kpjy5MCUwtMDRTybnIkab66A=; b=Eg09kla29pjKKRLqNUOT9vTnfr57YApFPE+gLoWbKPnxFKwoqSWN2IHaHBoIBD8kWy nqA6cSaE1fWrIYTlyTtKtWaRyQ9XvE0voyFkBIBh3idVKQF/9JoiMa34HRkSx73KorQe YxSBa5fTL63BTYRG+8jjPlpy2NT+ikhm5vZp7NL/P2u1Ss9ilu55Xr4g3lOfAhHAS97F aFOwbOVwLeTmfHqWGXskRbGj9fGc69pchm3Je9b3pm+6FKbWWeV1Kz5qlQ3WoZuh8ust 0fThOY4evwQY3RwYHenR/fqsh+q2yOsRNkKaoKwVkjSCJZLV6z/hyouzxGQsK4SFMb8m 2fgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YxbYKbELPp7cXOUSqr2Kpjy5MCUwtMDRTybnIkab66A=; b=60pHvkV4HbvAxILAfHH0WPNzxKCDXxuWCaazwdwiKV1Xl8r56UT3JE9VtqqzjCqoZp 92QbEH4Yah3UdwL/e9LkQhRyDhsRB/ObucieYiuEKz91CR+5k3+zOkNh0I/SNLFujVf1 0ZOw3MFykHtA1++4LzNjdXpbwzWWer4K/6eb1YpjlpzMd+XbgE6QPaPVHaH5eRFZbD7o +dgt8YxK3dYQRWa1umv3uD/U7ZNDXEZZeLNVIsxDRCJPVjXCrZT5G8bB8aWjMDnPCeHn sefMfbOiYVVYVv2S/0ySgrJg0yYXFMovmFOw6ygxNj6tDP5hwu7HO0U+XBs08kEjsMZP iquA== X-Gm-Message-State: ACrzQf00MYGrYS+/Y1TIaxcMx/SRGQBOx4chCvWOLzMqfL1DhkVJgv/o ZHxWjDwzeOJcBSNG6/jyRY1mWHxLNBWT7g== X-Google-Smtp-Source: AMsMyM5X9K0avmgDbYISB2rcebFmKVI5GMhjPU7kqhR4Iqh2KnbpL+jeGIH2wBgUxAKQJoVLKrazxg== X-Received: by 2002:a17:907:7ea0:b0:7ad:ef1e:3bc6 with SMTP id qb32-20020a1709077ea000b007adef1e3bc6mr4746342ejc.690.1667342169577; Tue, 01 Nov 2022 15:36:09 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:08 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 09/13] cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES Date: Tue, 1 Nov 2022 23:35:51 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Improve the incremental rebuilding support of "coccicheck" by piggy-backing on the computed dependency information of the corresponding *.o file, rather than rebuilding all / pairs if either their corresponding file changes, or if any header changes. This in effect uses the same method that the "sparse" target was made to use in c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23), except that the dependency on the *.o file isn't a hard one, we check with $(wildcard) if the *.o file exists, and if so we'll depend on it. This means that the common case of: make make coccicheck Will benefit from incremental rebuilding, now changing e.g. a header will only re-run "spatch" on those those *.c files that make use of it: By depending on the *.o we piggy-back on COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that for the *.sp files. E.g.: make contrib/coccinelle/free.cocci.patch make -W column.h contrib/coccinelle/free.cocci.patch Will take around 15 seconds for the second command on my 8 core box if I didn't run "make" beforehand to create the *.o files. But around 2 seconds if I did and we have those "*.o" files. Notes about the approach of piggy-backing on *.o for dependencies: * It *is* a trade-off since we'll pay the extra cost of running the C compiler, but we're probably doing that anyway. The compiler is much faster than "spatch", so even though we need to re-compile the *.o to create the dependency info for the *.c for "spatch" it's faster (especially if using "ccache"). * There *are* use-cases where some would like to have *.o files around, but to have the "make coccicheck" ignore them. See: https://lore.kernel.org/git/20220826104312.GJ1735@szeder.dev/ For those users a: make make coccicheck SPATCH_USE_O_DEPENDENCIES= Will avoid considering the *.o files. * If that *.o file doesn't exist we'll depend on an intermediate file of ours which in turn depends on $(FOUND_H_SOURCES). This covers both an initial build, or where "coccicheck" is run without running "all" beforehand, and because we run "coccicheck" on e.g. files in compat/* that we don't know how to build unless the requisite flag was provided to the Makefile. Most of the runtime of "incremental" runs is now spent on various compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and therefore conflate whether we *can* compile an object and generate dependency information for it with whether we'd like to link it into our binary. Before this change the distinction didn't matter, but now one way to make this even faster on incremental builds would be to peel those concerns apart so that we can see that e.g. compat/mmap.c doesn't depend on column.h. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 15 +++++++++++++-- contrib/coccinelle/README | 16 ++++++++++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 4bedff26285..c7c96d284dd 100644 --- a/Makefile +++ b/Makefile @@ -1307,6 +1307,13 @@ SPATCH_INCLUDE_FLAGS = --all-includes SPATCH_FLAGS = SPATCH_TEST_FLAGS = +# If *.o files are present, have "coccicheck" depend on them, with +# COMPUTE_HEADER_DEPENDENCIES this will speed up the common-case of +# only needing to re-generate coccicheck results for the users of a +# given API if it's changed, and not all files in the project. If +# COMPUTE_HEADER_DEPENDENCIES=no this will be unset too. +SPATCH_USE_O_DEPENDENCIES = YesPlease + # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) @@ -3181,14 +3188,18 @@ COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ +ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no) +SPATCH_USE_O_DEPENDENCIES = +endif define cocci-rule ## Rule for .build/$(1).patch/$(2); Params: # $(1) = e.g. "free.cocci" # $(2) = e.g. "grep.c" +# $(3) = e.g. "grep.o" COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) .build/$(1).patch/$(2): GIT-SPATCH-DEFINES -.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES +.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) .build/$(1).patch/$(2): $(1) .build/$(1).patch/$(2): .build/$(1).patch/% : % $$(call mkdir_p_parent_template) @@ -3205,7 +3216,7 @@ endef define cocci-matrix -$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(1),$(s))) +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o))) endef ifdef COCCI_GOALS diff --git a/contrib/coccinelle/README b/contrib/coccinelle/README index f0e80bd7f03..09ea8298e19 100644 --- a/contrib/coccinelle/README +++ b/contrib/coccinelle/README @@ -41,3 +41,19 @@ There are two types of semantic patches: This allows to expose plans of pending large scale refactorings without impacting the bad pattern checks. + +Git-specific tips & things to know about how we run "spatch": + + * The "make coccicheck" will piggy-back on + "COMPUTE_HEADER_DEPENDENCIES". If you've built a given object file + the "coccicheck" target will consider its depednency to decide if + it needs to re-run on the corresponding source file. + + This means that a "make coccicheck" will re-compile object files + before running. This might be unexpected, but speeds up the run in + the common case, as e.g. a change to "column.h" won't require all + coccinelle rules to be re-run against "grep.c" (or another file + that happens not to use "column.h"). + + To disable this behavior use the "SPATCH_USE_O_DEPENDENCIES=NoThanks" + flag. From patchwork Tue Nov 1 22:35:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CF03C4332F for ; Tue, 1 Nov 2022 22:37:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231437AbiKAWhZ (ORCPT ); Tue, 1 Nov 2022 18:37:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231329AbiKAWgo (ORCPT ); Tue, 1 Nov 2022 18:36:44 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7095A1F9EB for ; Tue, 1 Nov 2022 15:36:11 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id y14so40788295ejd.9 for ; Tue, 01 Nov 2022 15:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6AiDSlOSCsG1WOcpWmNBiOt/HYN32uZVq4wOAkmWFu8=; b=FWF++rA2NQCSCrV1Lgzxka2OoNoVeis8vfSm2wiNmYjNrvlLQ+XqqkWxhvXlfRUdhg Y45FahkYDTpq1RhjYeAUyR3jF0W7a0lKpelCjbkqP3tiKIG6SdXczjXuVe13Pr6w4oMy MtUQ9cGM9bkU+yyeZ28YlKMpXZl//K6NALRhC1Kl18wZR+Xq6OA0PP4HzB24Ytdcr+jG gBVdIVHZ2aYofNlXVr4nWhaRKYoyyYyfcKZw9Nny+jepprySw2rIIpW6C8wdjHMSZZKS MAgGI9N7eVG7zigd/DAHVO0jim36NK7rTGvidU9yKQ7rDk3LuGa9FwUmii7OueHsgRPg v8Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6AiDSlOSCsG1WOcpWmNBiOt/HYN32uZVq4wOAkmWFu8=; b=fPqFziEHdJt9XP+o/9eaA3cxfO/yN88sBcWforsVQZ5rXDldzsc375v0EY0mpcR9iJ /16PGXYuwJdREiTD1EtpZuH5htqknKl6KqQaFnfIQaua7HZsQIxdw3ApHnGXJUycaa4H n9FWXHWYNvh64QkBYomzoU6I8iVnEjk9w/UrNstaqDGttm5c3BzdpvLejDdi78gOosQG O82+Qnh+TxH4uBROStOMkNgClEQUkxL/zXewGWno1v1lzLciHPixJ6cTTEHCA6dJsp+l b+Qlr9pY7EYidiB+ZKdfcE/sWzn3KAzD/Z9/x9YHqe3X9K+EFFnMIJEWTPjmkcM5devF mi0g== X-Gm-Message-State: ACrzQf1U7X4m8GYIRFfkMEJT8+fWVOZB5k3QUX3x5ZJuhwdxFV0ht87j dNkgJ369GOWuBoa+GO6UEqKJT5TfND9OfQ== X-Google-Smtp-Source: AMsMyM7dtlvF/iJZDdCqxiQGFIjCMffeuM3xZBGmL29jr8Ya76BEs4TmpaJczEAWOEtWTI4Lc+3lFA== X-Received: by 2002:a17:906:4717:b0:7ad:c606:349f with SMTP id y23-20020a170906471700b007adc606349fmr14916938ejq.214.1667342170618; Tue, 01 Nov 2022 15:36:10 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:09 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 10/13] Makefile: copy contrib/coccinelle/*.cocci to build/ Date: Tue, 1 Nov 2022 23:35:52 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the "coccinelle" rule so that we first copy the *.cocci source in e.g. "contrib/coccinelle/strbuf.cocci" to ".build/contrib/coccinelle/strbuf.cocci" before operating on it. For now this serves as a rather pointless indirection, but prepares us for the subsequent commit where we'll be able to inject generated *.cocci files. Having the entire dependency tree live inside .build/* simplifies both the globbing we'd need to do, and any "clean" rules. It will also help for future targets which will want to act on the generated patches or the logs, e.g. targets to alert if we can't parse certain files (or, less so than usual) with "spatch", and e.g. a replacement for "ci/run-static-analysis.sh". Such a replacement won't care about placing the patches in the in-tree, only whether they're "OK" (and about the diff). Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 27 +++++++++++++++++++++------ shared.mak | 1 + 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index c7c96d284dd..44c906b65d5 100644 --- a/Makefile +++ b/Makefile @@ -3164,8 +3164,11 @@ check: $(GENERATED_H) fi COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) -COCCI_RULES = $(COCCI_GLOB) -COCCI_NAMES = $(COCCI_RULES:contrib/coccinelle/%.cocci=%) +COCCI_RULES_TRACKED = $(COCCI_GLOB:%=.build/%) +COCCI_RULES = +COCCI_RULES += $(COCCI_RULES_TRACKED) +COCCI_NAMES = +COCCI_NAMES += $(COCCI_RULES:.build/contrib/coccinelle/%.cocci=%) COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES)) COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) @@ -3173,6 +3176,9 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) COCCICHECK_PATCHES = $(COCCICHECK:%=%.patch) COCCICHECK_PATCHES_PENDING = $(COCCICHECK_PENDING:%=%.patch) +COCCICHECK_PATCHES_INTREE = $(COCCICHECK_PATCHES:.build/%=%) +COCCICHECK_PATCHES_PENDING_INTREE = $(COCCICHECK_PATCHES_PENDING:.build/%=%) + # It's expensive to compute the many=many rules below, only eval them # on $(MAKECMDGOALS) that match these $(COCCI_RULES) COCCI_RULES_GLOB = @@ -3180,10 +3186,16 @@ COCCI_RULES_GLOB += cocci% COCCI_RULES_GLOB += .build/contrib/coccinelle/% COCCI_RULES_GLOB += $(COCCICHECK_PATCHES) COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING) +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_INTREE) +COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_PENDING_INTREE) COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS)) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) +$(COCCI_RULES_TRACKED): .build/% : % + $(call mkdir_p_parent_template) + $(QUIET_CP)cp $< $@ + .build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ @@ -3197,7 +3209,7 @@ define cocci-rule # $(1) = e.g. "free.cocci" # $(2) = e.g. "grep.c" # $(3) = e.g. "grep.o" -COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) +COCCI_$(1:.build/contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) .build/$(1).patch/$(2): GIT-SPATCH-DEFINES .build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) .build/$(1).patch/$(2): $(1) @@ -3225,12 +3237,15 @@ endif define spatch-rule -contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) +.build/contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) $$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \ if test -s $$@; \ then \ echo ' ' SPATCH result: $$@; \ fi +contrib/coccinelle/$(1).cocci.patch: .build/contrib/coccinelle/$(1).cocci.patch + $$(QUIET_CP)cp $$< $$@ + endef ifdef COCCI_GOALS @@ -3254,11 +3269,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test -coccicheck: $(COCCICHECK_PATCHES) +coccicheck: $(COCCICHECK_PATCHES_INTREE) # See contrib/coccinelle/README coccicheck-pending: coccicheck-test -coccicheck-pending: $(COCCICHECK_PATCHES_PENDING) +coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE) .PHONY: coccicheck coccicheck-pending diff --git a/shared.mak b/shared.mak index f437073e48c..a34b66c926d 100644 --- a/shared.mak +++ b/shared.mak @@ -60,6 +60,7 @@ ifndef V QUIET_AR = @echo ' ' AR $@; QUIET_LINK = @echo ' ' LINK $@; QUIET_BUILT_IN = @echo ' ' BUILTIN $@; + QUIET_CP = @echo ' ' CP $< $@; QUIET_LNCP = @echo ' ' LN/CP $@; QUIET_XGETTEXT = @echo ' ' XGETTEXT $@; QUIET_MSGINIT = @echo ' ' MSGINIT $@; From patchwork Tue Nov 1 22:35:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D115C433FE for ; Tue, 1 Nov 2022 22:37:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231443AbiKAWha (ORCPT ); Tue, 1 Nov 2022 18:37:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231340AbiKAWgp (ORCPT ); Tue, 1 Nov 2022 18:36:45 -0400 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B74C1F9D6 for ; Tue, 1 Nov 2022 15:36:13 -0700 (PDT) Received: by mail-ed1-x533.google.com with SMTP id 21so23873142edv.3 for ; Tue, 01 Nov 2022 15:36:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KBcBjeXGFmcicrIVyeWKgUrc3Kdu7+IRrkI/jEaZZaw=; b=ESkunLfaXYiLW4czKuGPHvVxtLyESVv/tw1+CizHtfUWUn9sVCxrJDA3QC3ZpqP5gO v0moYRP0Gp3PQ2sevX1pbi1Muu4pl74tBJppGf8Or7X2gAYF74u8c20a6barwFIdLBqK IPLtKIg6TWkgB8ASb7ZQ26pU4VAHiN/oetgIzCw5HRQwJ5Ao7FEQAVOOzfKEkAJJft6D Qq1ouQGXg7SeUBh/k10nza7+NoUj6mAi8QbMUFoI70dpkvuCoDF9TkcT38MiRJKADFly CjPTTrftj9OxYug/7qlhbijn2OJo7762oui8t3Voo7F8W/tBSk7+P5vZjVCpONHRwimb 2WtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KBcBjeXGFmcicrIVyeWKgUrc3Kdu7+IRrkI/jEaZZaw=; b=775L/6Bu6Z38mhMcde389vqH34INuuzf7MOD+iFUZLElI5qc5pSV8SCO7cVIJx3ULS aTqfhs7IiXc9hdL2iwWLVbFMmTq2vmoRl32WpDVhvQ0Ruso6xsApu32aIaYr61knzKqV 5btLKKh3/28HrhxjjsHUb+LKvghefo0mYqA+9DJs64AfdlnMafnrM+SBp+yDH6ig9LZJ G5vnk+vCtpSBCNVMduVs2jXEylD8Q2BAXYgKowVRMfmCSsh4U4/xPAMNOkGYI0ISzZ46 zQ509aeGTPgwJZyKNL5xA7Q661YoVUUJ+Vj0uQziQ2Od5ibenfWmy+2VRJTR0WADFM/P c8wQ== X-Gm-Message-State: ACrzQf2zxZ+knmaYumtMjU2PU72vhXXfM+Pg5H57hYmjlgjl/Xdg3aL8 QXn6q4crhn8wxGk1WYcmC4foC68xIddhEg== X-Google-Smtp-Source: AMsMyM4tUUsUP/bu9T6E8EWTNzZJx/2vzMUp8dnbn5dg+GTqiDvTDnMTWq82T/okYyC0/YOQGCd6OA== X-Received: by 2002:a05:6402:3789:b0:461:3ae6:8d73 with SMTP id et9-20020a056402378900b004613ae68d73mr21522513edb.229.1667342171493; Tue, 01 Nov 2022 15:36:11 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:10 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 11/13] cocci rules: remove 's from rules that don't need them Date: Tue, 1 Nov 2022 23:35:53 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The in the part of the coccinelle syntax[1] is for our purposes there to declares if we have inter-dependencies between different rules. But such 's must be unique within a given semantic patch file. As we'll be processing a concatenated version of our rules in the subsequent commit let's remove these names. They weren't being used for the semantic patches themselves, and equated to a short comment about the rule. Both the filename and context of the rules makes it clear what they're doing, so we're not gaining anything from keeping these. Retaining them goes against recommendations that "contrib/coccinelle/README" will be making in the subsequent commit. This leaves only one named rule in our sources, where it's needed for a " <-> " relationship: $ git -P grep '^@ ' -- contrib/coccinelle/ contrib/coccinelle/swap.cocci:@ swap @ contrib/coccinelle/swap.cocci:@ extends swap @ 1. https://coccinelle.gitlabpages.inria.fr/website/docs/main_grammar.html Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/hashmap.cocci | 2 +- contrib/coccinelle/preincr.cocci | 2 +- contrib/coccinelle/strbuf.cocci | 2 +- contrib/coccinelle/swap.cocci | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/contrib/coccinelle/hashmap.cocci b/contrib/coccinelle/hashmap.cocci index d69e120ccff..c5dbb4557b5 100644 --- a/contrib/coccinelle/hashmap.cocci +++ b/contrib/coccinelle/hashmap.cocci @@ -1,4 +1,4 @@ -@ hashmap_entry_init_usage @ +@@ expression E; struct hashmap_entry HME; @@ diff --git a/contrib/coccinelle/preincr.cocci b/contrib/coccinelle/preincr.cocci index 7fe1e8d2d9a..ae42cb07302 100644 --- a/contrib/coccinelle/preincr.cocci +++ b/contrib/coccinelle/preincr.cocci @@ -1,4 +1,4 @@ -@ preincrement @ +@@ identifier i; @@ - ++i > 1 diff --git a/contrib/coccinelle/strbuf.cocci b/contrib/coccinelle/strbuf.cocci index 0970d98ad72..5f06105df6d 100644 --- a/contrib/coccinelle/strbuf.cocci +++ b/contrib/coccinelle/strbuf.cocci @@ -1,4 +1,4 @@ -@ strbuf_addf_with_format_only @ +@@ expression E; constant fmt !~ "%"; @@ diff --git a/contrib/coccinelle/swap.cocci b/contrib/coccinelle/swap.cocci index a0934d1fdaf..522177afb66 100644 --- a/contrib/coccinelle/swap.cocci +++ b/contrib/coccinelle/swap.cocci @@ -1,4 +1,4 @@ -@ swap_with_declaration @ +@@ type T; identifier tmp; T a, b; From patchwork Tue Nov 1 22:35:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6D45C4332F for ; Tue, 1 Nov 2022 22:37:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231417AbiKAWhb (ORCPT ); Tue, 1 Nov 2022 18:37:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231343AbiKAWgp (ORCPT ); Tue, 1 Nov 2022 18:36:45 -0400 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C8231E3D4 for ; Tue, 1 Nov 2022 15:36:13 -0700 (PDT) Received: by mail-ej1-x633.google.com with SMTP id f27so40783319eje.1 for ; Tue, 01 Nov 2022 15:36:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J3vZr7n5XXkRoy11pMjNo2PlAotPPHJIuwh/129yPus=; b=RUPZyFj8eSTmqrsSdfH5MilUCvXKIepoYjDEQjiNlxX/sSmqYjlbWRePSNeT8CD0ZH xMs1CKeArndDgffMfE2Y2WhhBaTtJ9c6Xzhx0bzOECjRyPophZZmGKtFdc30JFrc4dsE o5hIfmn+pLCp4qZp1JR8OiRKMZFdNeC8cyImjAb/9buMT++m3E3f/9c52/lChk1erDRB KMDQUoXRAfIVL11f3JTlYMsebTZm8GdKDx30MGNnl88IB9OnSgbourhIL6QurSsWsRYw avZmHvfqpTWLS1z4yCVjLJ02PMktRx2Gm4RJgpi4nHWQbPbb20cYAssSa8ksj7EceULS z6vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J3vZr7n5XXkRoy11pMjNo2PlAotPPHJIuwh/129yPus=; b=XsegMxVxXJyIVRRGGlXzlWCqRTM3td9mNhn2bZ8sVH9nvaXjB8+pp1GQ00OXC0ltYv oUe1tnuVQg9BsrRtLwr/Ej53pxQwixw5QF6Z2th3np6Sh3h6CAaK7Z56Yrr8hQ5G/hiW KtS+oYSOlPxDG4pvPCKxAMACLES3no0PzRlZjwM/FuLCC9QSRINgSL/BWmXIr+Y8T+5K PA+7QncWLrJYFl56vnzhorPqd7Jdav/c43hxGHt7nvL7w7TlchV5moXTifoJ6o2lBVn5 8pkr4mW8ODSmcp2mwSX5X1aG+kiSf7rJcnMgTZV/329iYLzOXJSQGB8Fk5QllUxIpddz j26A== X-Gm-Message-State: ACrzQf1gKRq5AuWNbhyk8McEvrNVGPf+i59z3az+dphCpzTQ4gCT8U0n 6BFOyT5+xkIiljf+R0aP1ThkyElyDyVSFQ== X-Google-Smtp-Source: AMsMyM6UHTwfaRE6H2dm/orTBIvI+LOdO8ASZ7GwIai291IuRY0fI/xVkjzEJxGC6o0M2bYG/iQHxA== X-Received: by 2002:a17:907:2be9:b0:7a1:11a9:1334 with SMTP id gv41-20020a1709072be900b007a111a91334mr19916154ejc.131.1667342172340; Tue, 01 Nov 2022 15:36:12 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:11 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 12/13] cocci: run against a generated ALL.cocci Date: Tue, 1 Nov 2022 23:35:54 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The preceding commits to make the "coccicheck" target incremental made it slower in some cases. As an optimization let's not have the many=many mapping of <*.cocci>=<*.[ch]>, but instead concat the <*.cocci> into an ALL.cocci, and then run one-to-many ALL.cocci=<*.[ch]>. A "make coccicheck" is now around 2x as fast as it was on "master", and around 1.5x as fast as the preceding change to make the run incremental: $ git hyperfine -L rev origin/master,HEAD~,HEAD -p 'make clean' 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' -r 3 Benchmark 1: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'origin/master Time (mean ± σ): 4.258 s ± 0.015 s [User: 27.432 s, System: 1.532 s] Range (min … max): 4.241 s … 4.268 s 3 runs Benchmark 2: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD~ Time (mean ± σ): 5.365 s ± 0.079 s [User: 36.899 s, System: 1.810 s] Range (min … max): 5.281 s … 5.436 s 3 runs Benchmark 3: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD Time (mean ± σ): 2.725 s ± 0.063 s [User: 14.796 s, System: 0.233 s] Range (min … max): 2.667 s … 2.792 s 3 runs Summary 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD' ran 1.56 ± 0.04 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'origin/master' 1.97 ± 0.05 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD~' This can be turned off with SPATCH_CONCAT_COCCI, but as the beneficiaries of "SPATCH_CONCAT_COCCI=" would mainly be those developing the *.cocci rules themselves, let's leave this optimization on by default. For more information see my "Optimizing *.cocci rules by concat'ing them" (<220901.8635dbjfko.gmgdl@evledraar.gmail.com>) on the cocci@inria.fr mailing list. This potentially changes the results of our *.cocci rules, but as noted in that discussion it should be safe for our use. We don't name rules, or if we do their names don't conflict across our *.cocci files. To the extent that we'd have any inter-dependencies between rules this doesn't make that worse, as we'd have them now if we ran "make coccicheck", applied the results, and would then have (due to hypothetical interdependencies) suggested changes on the subsequent "make coccicheck". Our "coccicheck-test" target makes use of the ALL.cocci when running tests, e.g. when testing unused.{c,out} we test it against ALL.cocci, not unused.cocci. We thus assert (to the extent that we have test coverage) that this concatenation doesn't change the expected results of running these rules. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 38 ++++++++++++++++++++++++++++++++++++++ contrib/coccinelle/README | 13 +++++++++++++ 2 files changed, 51 insertions(+) diff --git a/Makefile b/Makefile index 44c906b65d5..36cefe3b1d6 100644 --- a/Makefile +++ b/Makefile @@ -1314,6 +1314,29 @@ SPATCH_TEST_FLAGS = # COMPUTE_HEADER_DEPENDENCIES=no this will be unset too. SPATCH_USE_O_DEPENDENCIES = YesPlease +# Set SPATCH_CONCAT_COCCI to concatenate the contrib/cocci/*.cocci +# files into a single contrib/cocci/ALL.cocci before running +# "coccicheck". +# +# Pros: +# +# - Speeds up a one-shot run of "make coccicheck", as we won't have to +# parse *.[ch] files N times for the N *.cocci rules +# +# Cons: +# +# - Will make incremental development of *.cocci slower, as +# e.g. changing strbuf.cocci will re-run all *.cocci. +# +# - Makes error and performance analysis harder, as rules will be +# applied from a monolithic ALL.cocci, rather than +# e.g. strbuf.cocci. To work around this either undefine this, or +# generate a specific patch, e.g. this will always use strbuf.cocci, +# not ALL.cocci: +# +# make contrib/coccinelle/strbuf.cocci.patch +SPATCH_CONCAT_COCCI = YesPlease + # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) @@ -3163,9 +3186,12 @@ check: $(GENERATED_H) exit 1; \ fi +COCCI_GEN_ALL = .build/contrib/coccinelle/ALL.cocci COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) COCCI_RULES_TRACKED = $(COCCI_GLOB:%=.build/%) +COCCI_RULES_TRACKED_NO_PENDING = $(filter-out %.pending.cocci,$(COCCI_RULES_TRACKED)) COCCI_RULES = +COCCI_RULES += $(COCCI_GEN_ALL) COCCI_RULES += $(COCCI_RULES_TRACKED) COCCI_NAMES = COCCI_NAMES += $(COCCI_RULES:.build/contrib/coccinelle/%.cocci=%) @@ -3200,6 +3226,10 @@ $(COCCI_RULES_TRACKED): .build/% : % $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ +$(COCCI_GEN_ALL): $(COCCI_RULES_TRACKED_NO_PENDING) + $(call mkdir_p_parent_template) + $(QUIET_SPATCH_CAT)cat $^ >$@ + ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no) SPATCH_USE_O_DEPENDENCIES = endif @@ -3256,7 +3286,11 @@ COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res +ifdef SPATCH_CONCAT_COCCI +$(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : $(COCCI_GEN_ALL) +else $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci +endif $(call mkdir_p_parent_template) $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \ --very-quiet --no-show-diff \ @@ -3269,7 +3303,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test +ifdef SPATCH_CONCAT_COCCI +coccicheck: contrib/coccinelle/ALL.cocci.patch +else coccicheck: $(COCCICHECK_PATCHES_INTREE) +endif # See contrib/coccinelle/README coccicheck-pending: coccicheck-test diff --git a/contrib/coccinelle/README b/contrib/coccinelle/README index 09ea8298e19..09b72bfd4e7 100644 --- a/contrib/coccinelle/README +++ b/contrib/coccinelle/README @@ -57,3 +57,16 @@ Git-specific tips & things to know about how we run "spatch": To disable this behavior use the "SPATCH_USE_O_DEPENDENCIES=NoThanks" flag. + + * To speed up our rules the "make coccicheck" target will by default + concatenate all of the *.cocci files here into an "ALL.cocci", and + apply it to each source file. + + This makes the run faster, as we don't need to run each rule + against each source file. See the Makefile for further discussion, + this behavior can be disabled with "SPATCH_CONCAT_COCCI=". + + But since they're concatenated any in the (e.g. "@ + my_name", v.s. anonymous "@@") needs to be unique across all our + *.cocci files. You should only need to name rules if other rules + depend on them (currently only one rule is named). From patchwork Tue Nov 1 22:35:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13027585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37757C433FE for ; Tue, 1 Nov 2022 22:37:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231445AbiKAWhg (ORCPT ); Tue, 1 Nov 2022 18:37:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231381AbiKAWgq (ORCPT ); Tue, 1 Nov 2022 18:36:46 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 684781E3F9 for ; Tue, 1 Nov 2022 15:36:14 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id k2so40811940ejr.2 for ; Tue, 01 Nov 2022 15:36:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j0OIYTPPOX5DG9M04sU21hEG+QEyuRUiTTyMCrFvxSk=; b=djBNMpJtmav0cK3syI5iKTrmpfRFRkX3wZx24JPuFOD2eni0+JKh7QxvQPjUy0yTpz pAzkxH/Zu83exkMaZzGcuyrfksQ2fXSv183ND7zOBnrZhn/tcr4jy5EUPYG9nofqL249 bkn34BM2B1rGCLXA7Og7TobSB7GmLQHS6wgMQOdjBPqZtjklWC6pnPJwmtGvL/Ulp4Dq PUToqN0tSj31XPuIoBHE3y7icztd3QMh6gSbzsw4T49Wdy02KZADJH68fj3jE7nYpB2E HJ6MsPzgvopjv2Vza60DhZiOCvV3ezFYI6KcGFQuPfOLb+kDmbcvjDx45Ka7zz6nQrE5 LIYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j0OIYTPPOX5DG9M04sU21hEG+QEyuRUiTTyMCrFvxSk=; b=T0wA7THYQg/w9LVBHlR0RoAMgbsg5RbaRHgJr1Xje2ARkEiUaFrkkm0j123RV4dntf CIPXo157R3i8Dqr9aADMQ3fFiCUQsVinwuXnv8qFb8xZH3SbCtdu3y4Az0br4uV+STlO ePNgYJNJwQMM2euzPbOSyyVfzN+FuFf1FpWiJRiDCLjSCyG9a2+XqgRfKjbH3rlJoGfa LCgJgGVcRqGvZcg3Ptp9ciiDJtA5IbasOvtwkJUtVUZssBPVbrBu8Py98J550B/DPsdx wYvhrEBmTaSJSX3XgFJiDlDbCrIbom6DQXml/XpPy4HuPGQuQ31YCEzwYhRPUEout9Z3 rm0A== X-Gm-Message-State: ACrzQf1kgxv0BQao0Lka2wyHeNUXoAwuchmS403a++uxL4Ocbt5gpMMF cwoddFLa5GBpBCPa78NA8Nu8ygfKqWZa8A== X-Google-Smtp-Source: AMsMyM4Jae3X183xcu4H27yYM3rUkLchREVOoBMfwGgsxygRvtMAKyREbOSKouxAUtphue6w+hmmIg== X-Received: by 2002:a17:907:9729:b0:78d:9fb4:16e0 with SMTP id jg41-20020a170907972900b0078d9fb416e0mr19975548ejc.455.1667342173471; Tue, 01 Nov 2022 15:36:13 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id rp7-20020a170906d96700b0078dd4c89781sm4693845ejb.35.2022.11.01.15.36.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 15:36:12 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v5 13/13] spatchcache: add a ccache-alike for "spatch" Date: Tue, 1 Nov 2022 23:35:55 +0100 Message-Id: X-Mailer: git-send-email 2.38.0.1280.g8136eb6fab2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a rather trivial "spatchcache", with this running e.g.: make cocciclean make contrib/coccinelle/free.cocci.patch \ SPATCH=contrib/coccicheck/spatchcache \ SPATCH_FLAGS=--very-quiet Is cut down from ~20s to ~5s on my system. Much of that is either fixable shell overhead, or the around 40 files we "CANTCACHE" (see the implementation). This uses "redis" as a cache by default, but it's configurable. See the embedded documentation. This is *not* like ccache in that we won't cache failed spatch invocations, or those where spatch suggests changes for us. Those cases are so rare that I didn't think it was worth the bother, by far the most common case is that it has no suggested changes. We'll also refuse to cache any "spatch" invocation that has output on stderr, which means that "--very-quiet" must be added to "SPATCH_FLAGS". Because we narrow the cache to that we don't need to save away stdout, stderr & the exit code. We simply cache the cases where we had no suggested changes. Another benchmark is to compare this with the previous SPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running: make clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0 Would take 33s, but with the preceding changes running without this "spatchcache" is slightly slower, or around 35s: make clean; time make contrib/coccinelle/array.cocci.patch Now doing the same with SPATCH=contrib/coccinelle/spatchcache will take around 6s, but we'll need to compile the *.o files first to take full advantage of it (which can be fast with "ccache"): make clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache 1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/ Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/README | 20 +++ contrib/coccinelle/spatchcache | 304 +++++++++++++++++++++++++++++++++ 2 files changed, 324 insertions(+) create mode 100755 contrib/coccinelle/spatchcache diff --git a/contrib/coccinelle/README b/contrib/coccinelle/README index 09b72bfd4e7..d1daa1f6263 100644 --- a/contrib/coccinelle/README +++ b/contrib/coccinelle/README @@ -70,3 +70,23 @@ Git-specific tips & things to know about how we run "spatch": my_name", v.s. anonymous "@@") needs to be unique across all our *.cocci files. You should only need to name rules if other rules depend on them (currently only one rule is named). + + * To speed up incremental runs even more use the "spatchcache" tool + in this directory as your "SPATCH". It aimns to be a "ccache" for + coccinelle, and piggy-backs on "COMPUTE_HEADER_DEPENDENCIES". + + It caches in Redis by default, see it source for a how-to. + + In one setup with a primed cache "make coccicheck" followed by a + "make clean && make" takes around 10s to run, but 2m30s with the + default of "SPATCH_CONCAT_COCCI=Y". + + With "SPATCH_CONCAT_COCCI=" the total runtime is around ~6m, sped + up to ~1m with "spatchcache". + + Most of the 10s (or ~1m) being spent on re-running "spatch" on + files we couldn't cache, as we didn't compile them (in contrib/* + and compat/* mostly). + + The absolute times will differ for you, but the relative speedup + from caching should be on that order. diff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache new file mode 100755 index 00000000000..29e9352d8a2 --- /dev/null +++ b/contrib/coccinelle/spatchcache @@ -0,0 +1,304 @@ +#!/bin/sh +# +# spatchcache: a poor-man's "ccache"-alike for "spatch" in git.git +# +# This caching command relies on the peculiarities of the Makefile +# driving "spatch" in git.git, in particular if we invoke: +# +# make +# # See "spatchCache.cacheWhenStderr" for why "--very-quiet" is +# # used +# make coccicheck SPATCH_FLAGS=--very-quiet +# +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with +# "gcc" and "clang") write e.g. a .depend/grep.o.d for grep.c, when we +# compile grep.o. +# +# The .depend/grep.o.d will have the full header dependency tree of +# grep.c, and we can thus cache the output of "spatch" by: +# +# 1. Hashing all of those files +# 2. Hashing our source file, and the *.cocci rule we're +# applying +# 3. Running spatch, if suggests no changes (by far the common +# case) we invoke "spatchCache.getCmd" and +# "spatchCache.setCmd" with a hash SHA-256 to ask "does this +# ID have no changes" or "say that ID had no changes> +# 4. If no "spatchCache.{set,get}Cmd" is specified we'll use +# "redis-cli" and maintain a SET called "spatch-cache". Set +# appropriate redis memory policies to keep it from growing +# out of control. +# +# This along with the general incremental "make" support for +# "contrib/coccinelle" makes it viable to (re-)run coccicheck +# e.g. when merging integration branches. +# +# Note that the "--very-quiet" flag is currently critical. The cache +# will refuse to cache anything that has output on STDERR (which might +# be errors from spatch), but see spatchCache.cacheWhenStderr below. +# +# The STDERR (and exit code) could in principle be cached (as with +# ccache), but then the simple structure in the Redis cache would need +# to change, so just supply "--very-quiet" for now. +# +# To use this, simply set SPATCH to +# contrib/coccinelle/spatchcache. Then optionally set: +# +# [spatchCache] +# # Optional: path to a custom spatch +# spatch = ~/g/coccicheck/spatch.opt +# +# As well as this trace config (debug implies trace): +# +# cacheWhenStderr = true +# trace = false +# debug = false +# +# The ".depend/grep.o.d" can also be customized, as a string that will +# be eval'd, it has access to a "$dirname" and "$basename": +# +# [spatchCache] +# dependFormat = "$dirname/.depend/${basename%.c}.o.d" +# +# Setting "trace" to "true" allows for seeing when we have a cache HIT +# or MISS. To debug whether the cache is working do that, and run e.g.: +# +# redis-cli FLUSHALL +# +# grep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c +# 600 CANTCACHE +# 7365 MISS +# 7365 SET +# +# A subsequent "make cocciclean && make coccicheck" should then have +# all "HIT"'s and "CANTCACHE"'s. +# +# The "spatchCache.cacheWhenStderr" option is critical when using +# spatchCache.{trace,debug} to debug whether something is set in the +# cache, as we'll write to the spatch logs in .build/* we'd otherwise +# always emit a NOCACHE. +# +# Reading the config can make the command much slower, to work around +# this the config can be set in the environment, with environment +# variable name corresponding to the config key. "default" can be used +# to use whatever's the script default, e.g. setting +# spatchCache.cacheWhenStderr=true and deferring to the defaults for +# the rest is: +# +# export GIT_CONTRIB_SPATCHCACHE_DEBUG=default +# export GIT_CONTRIB_SPATCHCACHE_TRACE=default +# export GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR=true +# export GIT_CONTRIB_SPATCHCACHE_SPATCH=default +# export GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT=default +# export GIT_CONTRIB_SPATCHCACHE_SETCMD=default +# export GIT_CONTRIB_SPATCHCACHE_GETCMD=default + +set -e + +env_or_config () { + env="$1" + shift + if test "$env" = "default" + then + # Avoid expensive "git config" invocation + return + elif test -n "$env" + then + echo "$env" + else + git config $@ || : + fi +} + +## Our own configuration & options +debug=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_DEBUG" --bool "spatchCache.debug") +if test "$debug" != "true" +then + debug= +fi +if test -n "$debug" +then + set -x +fi + +trace=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_TRACE" --bool "spatchCache.trace") +if test "$trace" != "true" +then + trace= +fi +if test -n "$debug" +then + # debug implies trace + trace=true +fi + +cacheWhenStderr=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR" --bool "spatchCache.cacheWhenStderr") +if test "$cacheWhenStderr" != "true" +then + cacheWhenStderr= +fi + +trace_it () { + if test -z "$trace" + then + return + fi + echo "$@" >&2 +} + +spatch=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_SPATCH" --path "spatchCache.spatch") +if test -n "$spatch" +then + if test -n "$debug" + then + trace_it "custom spatchCache.spatch='$spatch'" + fi +else + spatch=spatch +fi + +dependFormat='$dirname/.depend/${basename%.c}.o.d' +dependFormatCfg=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT" "spatchCache.dependFormat") +if test -n "$dependFormatCfg" +then + dependFormat="$dependFormatCfg" +fi + +set=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_SETCMD" "spatchCache.setCmd") +get=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_GETCMD" "spatchCache.getCmd") + +## Parse spatch()-like command-line for caching info +arg_sp= +arg_file= +args="$@" +spatch_opts() { + while test $# != 0 + do + arg_file="$1" + case "$1" in + --sp-file) + arg_sp="$2" + ;; + esac + shift + done +} +spatch_opts "$@" +if ! test -f "$arg_file" +then + arg_file= +fi + +hash_for_cache() { + # Parameters that should affect the cache + echo "args=$args" + echo "config spatchCache.spatch=$spatch" + echo "config spatchCache.debug=$debug" + echo "config spatchCache.trace=$trace" + echo "config spatchCache.cacheWhenStderr=$cacheWhenStderr" + echo + + # Our target file and its dependencies + git hash-object "$1" "$2" $(grep -E -o '^[^:]+:$' "$3" | tr -d ':') +} + +# Sanity checks +if ! test -f "$arg_sp" && ! test -f "$arg_file" +then + echo $0: no idea how to cache "$@" >&2 + exit 128 +fi + +# Main logic +dirname=$(dirname "$arg_file") +basename=$(basename "$arg_file") +eval "dep=$dependFormat" + +if ! test -f "$dep" +then + trace_it "$0: CANTCACHE have no '$dep' for '$arg_file'!" + exec "$spatch" "$@" +fi + +if test -n "$debug" +then + trace_it "$0: The full cache input for '$arg_sp' '$arg_file' '$dep'" + hash_for_cache "$arg_sp" "$arg_file" "$dep" >&2 +fi +sum=$(hash_for_cache "$arg_sp" "$arg_file" "$dep" | git hash-object --stdin) + +trace_it "$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'" + +getret= +if test -z "$get" +then + if test $(redis-cli SISMEMBER spatch-cache "$sum") = 1 + then + getret=0 + else + getret=1 + fi +else + $set "$sum" + getret=$? +fi + +if test "$getret" = 0 +then + trace_it "$0: HIT for '$arg_file' with '$arg_sp'" + exit 0 +else + trace_it "$0: MISS: for '$arg_file' with '$arg_sp'" +fi + +out="$(mktemp)" +err="$(mktemp)" + +set +e +"$spatch" "$@" >"$out" 2>>"$err" +ret=$? +cat "$out" +cat "$err" >&2 +set -e + +nocache= +if test $ret != 0 +then + nocache="exited non-zero: $ret" +elif test -s "$out" +then + nocache="had patch output" +elif test -z "$cacheWhenStderr" && test -s "$err" +then + nocache="had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)" +fi + +if test -n "$nocache" +then + trace_it "$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'" + exit "$ret" +fi + +trace_it "$0: SET: for '$arg_file' with '$arg_sp'" + +setret= +if test -z "$set" +then + if test $(redis-cli SADD spatch-cache "$sum") = 1 + then + setret=0 + else + setret=1 + fi +else + "$set" "$sum" + setret=$? +fi + +if test "$setret" != 0 +then + echo "FAILED to set '$sum' in cache!" >&2 + exit 128 +fi + +exit "$ret"