From patchwork Thu Aug 25 14:36:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12954839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96D0BC64991 for ; Thu, 25 Aug 2022 14:36:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242172AbiHYOgf (ORCPT ); Thu, 25 Aug 2022 10:36:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242076AbiHYOgZ (ORCPT ); Thu, 25 Aug 2022 10:36:25 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AF92AA4F4 for ; Thu, 25 Aug 2022 07:36:24 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id a4so24928984wrq.1 for ; Thu, 25 Aug 2022 07:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=1yC5Wv1NsWdozjXB4QyYZ+Pc30xfRmVSE0XFWPKkjIo=; b=Rlz+XABr110kkjESOBZ4JhehCIa5MYpHo9gNE6h9yDCof1GiGRmL1YE412owaFrzCH ye5cMsKOmO74fCRra05eXdwBKFMxRmO3k45NGjTbpG4mTmTLfbmLP43KMM7zW4BSZdjD +FMBfnyK1JrqDggBbxxZTKWiP2drMDgaop9jBoFlC4DiS2k6grpsh1J4yZoQJm+RjimA prPIYC1K87ZEI8a7k5WaiELaPjhqsPvFVlDFSehgfv1eIEDPrYOlchjiF6v2W/c9mS0c wUkUmJuk8/hp5FLGrlsOzvk4A+a3YD2syqvJEB15Ht8ZkuwoNGTxKvDbfQOf9QTe8uKj p2dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=1yC5Wv1NsWdozjXB4QyYZ+Pc30xfRmVSE0XFWPKkjIo=; b=mJeh2t1i3ILC74QqJTdWzBBjquMatwVThtgpgtSU5VYAFBer8xow3GU8Pw/BbOphSk 7rZhryTbdA/maafTz62nPQACeYYZ0gQ0y1CJ+rq88aeLW69HQSxtEPxJohz2GFcbtr+U nHsW43D6mSrA7jw9U0u7oirkAuBiESEZ/6BN9tYNblijPIz2nMKJnev+9CPUSWdtBAJ8 Fw5g17nughqPVJ/WKjy1j1KiCS09+7JRagnMRHAsu6jKKRgHPfahZayvL1K+KXj9gozM e4eWUm3n5e6a171kvnGJVNjSZmKorkzgzGNU6oCjWHeeif0YpEXZ8/TE9geleG/2Mfql LoQw== X-Gm-Message-State: ACgBeo2JKYJMidoDp7p/PC0OY0fQo7hJRXvWuOllYs/0WbqGBxWjMNwV Va73PP7hvhFDGNQzzI0vVBTNacyx5GA= X-Google-Smtp-Source: AA6agR5O8NiGSWclGQbn1SYet6cdROuayt569bEjCQ3gTE69SbT6+VDFeu7rYDaGTHI/snVJfoR8FA== X-Received: by 2002:a05:6000:1c0c:b0:225:569c:dd2c with SMTP id ba12-20020a0560001c0c00b00225569cdd2cmr2445826wrb.628.1661438182266; Thu, 25 Aug 2022 07:36:22 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id m27-20020a056000181b00b0022549ac786asm14418027wrh.47.2022.08.25.07.36.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 07:36:21 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH 1/5] Makefile: add ability to TAB-complete cocci *.patch rules Date: Thu, 25 Aug 2022 16:36:13 +0200 Message-Id: X-Mailer: git-send-email 2.37.2.1279.g64dec4e13cf In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Declare the contrib/coccinelle/.cocci.patch rules in such a way as to allow TAB-completion, and slightly optimize the Makefile by cutting down on the number of $(wildcard) in favor of defining "coccicheck" and "coccicheck-pending" in terms of the same incrementally filtered list. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index e8adeb09f1c..bd8faac61ee 100644 --- a/Makefile +++ b/Makefile @@ -3135,9 +3135,15 @@ check: $(GENERATED_H) exit 1; \ fi +COCCI = $(wildcard contrib/coccinelle/*.cocci) +COCCI_PATCHES = $(addsuffix .patch,$(COCCI)) +COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES)) +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) + COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -%.cocci.patch: %.cocci $(COCCI_SOURCES) +$(COCCI_PATCHES): $(COCCI_SOURCES) +$(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ if test $(SPATCH_BATCH_SIZE) = 0; then \ limit=; \ @@ -3174,11 +3180,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test -coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/coccinelle/*.cocci))) +coccicheck: $(COCCICHECK) # See contrib/coccinelle/README coccicheck-pending: coccicheck-test -coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci)) +coccicheck-pending: $(COCCICHECK_PENDING) .PHONY: coccicheck coccicheck-pending @@ -3446,7 +3452,7 @@ profile-clean: cocciclean: $(RM) -r .build/contrib/coccinelle - $(RM) contrib/coccinelle/*.cocci.patch* + $(RM) $(COCCI_PATCHES)* clean: profile-clean coverage-clean cocciclean $(RM) -r .build From patchwork Thu Aug 25 14:36:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12954840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBE6AC65C0D for ; Thu, 25 Aug 2022 14:36:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242076AbiHYOgh (ORCPT ); Thu, 25 Aug 2022 10:36:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242056AbiHYOga (ORCPT ); Thu, 25 Aug 2022 10:36:30 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 017B4AB189 for ; Thu, 25 Aug 2022 07:36:25 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id n17so277935wrm.4 for ; Thu, 25 Aug 2022 07:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=N5jFCS2rph2mfSaem0PO14uWeAUqXg8pBdj6jplhKtM=; b=f+gAAiKsFNRPeJfFlvuYzeCb4gcaBgnpvPHEW1lLyGUGVqKX0PV6Ud2K9gZoqGQKNO gFiwgxU77THZu61hbw5pjynMfd4OxHxfUvcabtbJxwW5yFRdnMF8lRn6Hl+sRyyFl5jU ERT7nnDBTS58+MzxNN85TplSpt9ZFDxYQiRuT70cSi8Tm7pvbTk8Au7WCBuwJeXwzSj0 OEe8OR/kDoYpIBxXZwwhbTQ5xJBHcroVuo/IPTzRObMgBiliP5fBFleXXoB8eRSp1586 laKklRWJFaNaOe432efprRNLG+DIsDUBqdzmShAl2vXhba+Igs/QLmjJlRBD5xRRBO9O /G7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=N5jFCS2rph2mfSaem0PO14uWeAUqXg8pBdj6jplhKtM=; b=i6QpuhdEzysrxPjAhhHTBtJRZjf9vV9B+5PuFE/+behYc9iPJalzug6ot4YO0xGDXi C8IUhupgfLqWlJVRxhNa6C+Nh86vSNost9xgnMhinqo8RKecmDashdq1jdk6W02tpGW4 nMA/YiYcqcMHbHX+CpaSr9ExOkhIJqUtcASCDvjbCtOnXH08nPj3gr2IaVwHHMMpp3h3 8gEl2rily3Z+6Df+w9XJK6/0xR+sdwaHuLkqfluD5uFIHRUhW/SbSWmWJuRja1TQYJ32 +iwTDHc1NRML2xo7eaLAol8AVr6lUNRv7xryBvxfRLm3SUqn2rqXkaujDLasEsHKV0dV kd9w== X-Gm-Message-State: ACgBeo08sd/N09eYtoQMZZh4UofIvL5j367jR/nMnaKxR5ZKk9g+f/nE pC5biFH8rDjpqfdGQ1BdtqbhOVGL8js= X-Google-Smtp-Source: AA6agR4dRQnmnXyPiTJOPuE6Rl50El3w3OVKSmN3mpJNTtZz2acdkyzfScfK0Q3Uq6iQXxLwdddUHg== X-Received: by 2002:a5d:6e8e:0:b0:220:5fa1:d508 with SMTP id k14-20020a5d6e8e000000b002205fa1d508mr2658839wrz.337.1661438183199; Thu, 25 Aug 2022 07:36:23 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id m27-20020a056000181b00b0022549ac786asm14418027wrh.47.2022.08.25.07.36.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 07:36:22 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH 2/5] Makefile: have "coccicheck" re-run if flags change Date: Thu, 25 Aug 2022 16:36:14 +0200 Message-Id: X-Mailer: git-send-email 2.37.2.1279.g64dec4e13cf In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Fix an issue with the "coccicheck" family of rules that's been here since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike e.g. "make grep.o" we wouldn't re-run it when $(SPATCH) or $(SPATCH_FLAGS) changed. To test new flags we needed to first do a "make clean". This now uses the same (copy/pasted) pattern as other "DEFINES" rules. As a result we'll re-run properly. This can be demonstrated e.g. on the issue noted in [1]: $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1 [...] SPATCH contrib/coccinelle/xcalloc.cocci $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS="--all-includes --recursive-includes" * new spatch flags SPATCH contrib/coccinelle/xcalloc.cocci SPATCH result: contrib/coccinelle/xcalloc.cocci.patch $ 1. https://lore.kernel.org/git/20220823095602.GC1735@szeder.dev/ Signed-off-by: Ævar Arnfjörð Bjarmason --- .gitignore | 1 + Makefile | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/.gitignore b/.gitignore index 42fd7253b44..a32ad64d4c9 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ /GIT-PERL-HEADER /GIT-PYTHON-VARS /GIT-SCRIPT-DEFINES +/GIT-SPATCH-DEFINES /GIT-USER-AGENT /GIT-VERSION-FILE /bin-wrappers/ diff --git a/Makefile b/Makefile index bd8faac61ee..749eb9b25e3 100644 --- a/Makefile +++ b/Makefile @@ -1296,6 +1296,18 @@ SANITIZE_ADDRESS = SPATCH_FLAGS = --all-includes SPATCH_BATCH_SIZE = 1 +# Rebuild 'coccicheck' if $(SPATCH), its flags etc. change +TRACK_SPATCH_DEFINES = +TRACK_SPATCH_DEFINES += $(SPATCH) +TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) +TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) +GIT-SPATCH-DEFINES: FORCE + @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ + if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ + echo >&2 " * new spatch flags"; \ + echo "$$FLAGS" >GIT-SPATCH-DEFINES; \ + fi + include config.mak.uname -include config.mak.autogen -include config.mak @@ -3142,6 +3154,7 @@ COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) +$(COCCI_PATCHES): GIT-SPATCH-DEFINES $(COCCI_PATCHES): $(COCCI_SOURCES) $(COCCI_PATCHES): %.patch: % $(QUIET_SPATCH) \ @@ -3165,6 +3178,7 @@ $(COCCI_PATCHES): %.patch: % fi COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) +$(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci From patchwork Thu Aug 25 14:36:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12954842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D009AC28D13 for ; Thu, 25 Aug 2022 14:36:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242207AbiHYOgm (ORCPT ); Thu, 25 Aug 2022 10:36:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241756AbiHYOgc (ORCPT ); Thu, 25 Aug 2022 10:36:32 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BA0AAB41B for ; Thu, 25 Aug 2022 07:36:26 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id u14so24875973wrq.9 for ; Thu, 25 Aug 2022 07:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=iNx9qDpUivbgZzWrle8/Z/64ZS7TKvnxLiw3/bKo2Bs=; b=aTbxXea9hPWH1tfYoo3DoAWvAW0NgOwfbENSsl/6LPZCeNMQiMAIXgzXj51Z58nWwa CYy5bBCV/NcQ5C0hKzNYpSUv6HFBUTdAyGpoKaEoKT9r02pnA4tuE4875T1p8+aqx/n3 OnaA5x2GAsqWr8aH/9DELB36D0SSYsOaVJQq4a9i5kG5syz2u9zI+pSYg8I4+m30nog8 YK56DY8DCA07KGBIY/fC3jd99mxskccm7iGz3EKANe180XgBa78KhLN+bAppG+CQRbXu YWuB/Tz/fVzbDE72uWbvzN8msJ00tCUkwG5GrBDr6Hy0ptC/+7UeLtwV8/kTrqqMCIK4 pFeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=iNx9qDpUivbgZzWrle8/Z/64ZS7TKvnxLiw3/bKo2Bs=; b=B3siJxfeHXm4ZniJ0J2IwnqE3XlaMFPOIewPp/kBize2QZOnjU47LOf0WrSHxtDhgG MtDiV3g+SqCg2ZFU+KMP8Vk+dVIUsz6l37CnCGfo1Dat9D7enrC9OLRA/WyAeeDSM3pA vuxWU/GL4gNwLRtp/31aFFaa2kWtfBRFEMurucg+/cXjYt5nNhpS9NSfMX5Nty8ji4cT bTZicB1oQAKXeEhnwK+vLijfXsMgHMc689uhSm/uy8jrd7x7oCDkh4Fvbjpq+dvc2lXD B8zyhyjnCXitmG7C/cyfuLm7kVJaT5wMJ9PKHidQR2/yZ6eKCdf2XR/uwiixqqfoMxdv GFcg== X-Gm-Message-State: ACgBeo1FeJZMZH6ObANV5wRzOk2pMOdzlKAR9U1qErZZyINvuZbqLr0v 8cjXtDNyI12g2uNRF+bhdeGyVv6xAjHeJg== X-Google-Smtp-Source: AA6agR6LFqgVQAltolOX+ZBfx/IRaN+/e7hI0vDByOuHE7QyDqNVIsmKmgwF6DTu7RnZ0/+LCp4bXw== X-Received: by 2002:adf:fb8d:0:b0:225:4d57:17a6 with SMTP id a13-20020adffb8d000000b002254d5717a6mr2517527wrr.251.1661438184082; Thu, 25 Aug 2022 07:36:24 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id m27-20020a056000181b00b0022549ac786asm14418027wrh.47.2022.08.25.07.36.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 07:36:23 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH 3/5] cocci: make "coccicheck" rule incremental Date: Thu, 25 Aug 2022 16:36:15 +0200 Message-Id: X-Mailer: git-send-email 2.37.2.1279.g64dec4e13cf In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Optimize the very slow "coccicheck" target to take advantage of incremental rebuilding, and fix outstanding bugs and dependency problems with the existing rule. The rule is now faster both on the initial run as we can make better use of GNU make's parallelism than the old ad-hoc combination of make's parallelism combined with $(SPATCH_BATCH_SIZE) and "xargs -n". It also makes us *much* faster when incrementally building, it's now viable to "make coccicheck" as topic branches are merged down. The rule didn't use FORCE (or its equivalents) before, so a: make coccicheck make coccicheck Would report nothing to do on the second iteration. But all of our patch output depended on all $(COCCI_SOURCES) files, therefore e.g.: make -W grep.c coccicheck Would be the same as doing a full re-run. A change in a single file would force us to do a full re-run. The reason for this (not the initial rationale, but my analysis) is: * Since we create a single "*.cocci.patch+" we don't know where to pick up where we left off. Note that (per [1]) we've had a race condition since 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06) which might result in us producing corrupt patches to to the concurrent appending to "$@+" in the pre-image. That bug is now fixed. * We've been carrying forward the dependency on the *.c files since 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was initially added as a sort of poor man's dependency discovery. As we don't include other *.c files depending on other *.c files has always been broken, as could be trivially demonstrated e.g. with: make coccicheck make -W strbuf.h coccicheck However, depending on the corresponding *.c files has been doing something, namely that *if* an API change modified both *.c and *.h files we'd catch the change to the *.h we care about via the *.c being changed. For API changes that happened only via *.h files we'd do the wrong thing before this change, but e.g. for function additions (not "static inline" ones) catch the *.h change by proxy. Now we'll instead: * Create a / pair in the .build directory, E.g. for swap.cocci and grep.c we'll create .build/contrib/coccinelle/swap.cocci.patch/grep.c. That file is the diff we'll apply for that - combination, if there's no changes to me made (the common case) it'll be an empty file. * Our generated *.patch file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple "cat $^" of all of all of the / files for a given . See 1cc0425a27c (Makefile: have "make pot" not "reset --hard", 2022-05-26) for another recent rule that used that technique. * We depend on a .build/.ok file, which in turn depends on our . This emulates the previous "test -s" test, but in a more Makefile-native way (e.g. -k works as expected). As noted above this change fixes existing dependency issues, e.g. changing strbuf.h will now trigger a re-build. Our default is to depend on $(FOUND_H_SOURCES). This makes incremental re-building possible, but it's still fairly slow, as modifying one header will cause a full re-build of everything. Which is why we'll not depend on $(FOUND_H_SOURCES) but the *.o file corresponding to the *.c file, if it exists already. This means that we can do: make all make coccicheck make -W column.h coccicheck By depending on the *.o we piggy-back on COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that for the *.sp files. E.g.: make contrib/coccinelle/free.cocci.patch make -W column.h contrib/coccinelle/free.cocci.patch Will take around 15 seconds for the second command on my 8 core box if I didn't run "make" beforehand to create the *.o files. But around 2 seconds if I did and we have those "*.o" files. Notes about the approach of piggy-backing on *.o for dependencies: * It *is* a trade-off since we'll pay the extra cost of running the C compiler, but we're probably doing that anyway. The compiler is much faster than "spatch", so even though we need to re-compile the *.o to create the dependency info for the *.c for "spatch" it's faster (especially if using "ccache"). * If that *.o file doesn't exist we'll depend on an intermediate file of ours which in turn depends on $(FOUND_H_SOURCES). This covers both an initial build, or where "coccicheck" is run without running "all" beforehand, and because we run "coccicheck" on e.g. files in compat/* that we don't know how to build unless the requisite flag was provided to the Makefile. Most of the runtime of "incremental" runs is now spent on various compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and therefore conflate whether we *can* compile an object and generate dependency information for it with whether we'd like to link it into our binary. Before this change the distinction didn't matter, but now one way to make this even faster on incremental builds would be to peel those concerns apart so that we can see that e.g. compat/mmap.c doesn't depend on column.h. Further implementation details & notes: * We can take better advantage of parallelism, while making sure that we don't racily append to the contrib/coccinelle/swap.cocci.patch file from multiple workers. Before this change running "make coccicheck" would by default end up pegging just one CPU at the very end for a while, usually as we'd finish whichever *.cocci rule was the most expensive. This could be mitigated by combining "make -jN" with SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06). But doing so required careful juggling, as e.g. setting both to 4 would yield 16 workers. There will be cases where getting rid of "SPATCH_BATCH_SIZE" makes things worse, but a from-scratch "make coccicheck" with the default of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference) is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed the CPU more work in a less staggered way. * Getting rid of "SPATCH_BATCH_SIZE" particularly helps in cases where the default of 1 yields parallelism under "make coccicheck", but then running e.g.: make -W contrib/coccinelle/swap.cocci coccicheck I.e. before that would use only one CPU core, until the user remembered to adjust "SPATCH_BATCH_SIZE" differently than the setting that makes sense when doing a non-incremental run of "make coccicheck". * The change here to "contrib/coccinelle/the_repository.pending.cocci" is because we've been spewing out this warning into our $@.log since that rule was added: warning: rule starting on line 21: metavariable F not used in the - or context code It's being fixed here because we'll now notice on "make coccicheck-pending", since we're not redirecting all of STDERR to a log file, instead we use the "--very-quiet" option. Outstanding issues & future work: * We could get rid of "--all-includes" in favor of manually specifying a list of includes to give to "spatch(1)". As noted upthread of [2] a naïve removal of "--all-includes" will result in broken *.cocci patches, but if we know the exhaustive list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to re-scan for them, we could grab the headers to include from the .depend.d/.o.d and supply them with the "--include" option to spatch(1).q * We can generalize COMPUTE_HEADER_DEPENDENCIES to spew out the list of *.h dependencies without making a *.o object, which is slightly faster, and would also benefit the "make sparse" target. We'd still need to run the compiler, it would just have less work to do. 1. https://lore.kernel.org/git/877dlwotjc.fsf@evledraar.gmail.com/ 2. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 92 +++++++++++++------ .../coccinelle/the_repository.pending.cocci | 1 - shared.mak | 4 +- 3 files changed, 66 insertions(+), 31 deletions(-) diff --git a/Makefile b/Makefile index 749eb9b25e3..a374204aaca 100644 --- a/Makefile +++ b/Makefile @@ -1290,17 +1290,12 @@ SP_EXTRA_FLAGS = -Wno-universal-initializer SANITIZE_LEAK = SANITIZE_ADDRESS = -# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will -# usually result in less CPU usage at the cost of higher peak memory. -# Setting it to 0 will feed all files in a single spatch invocation. SPATCH_FLAGS = --all-includes -SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = TRACK_SPATCH_DEFINES += $(SPATCH) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) -TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ if test x"$$FLAGS" != x"`cat GIT-SPATCH-DEFINES 2>/dev/null`" ; then \ @@ -3147,35 +3142,74 @@ check: $(GENERATED_H) exit 1; \ fi +# It's expensive to compute the many=many rules below, only eval them +# on $(MAKECMDGOALS) that match these $(COCCI_RULES) +COCCI_RULES = +COCCI_RULES += cocci% +COCCI_RULES += contrib/coccinelle/% +COCCI_RULES += .build/contrib/coccinelle/% + COCCI = $(wildcard contrib/coccinelle/*.cocci) +COCCI_NAMES = $(COCCI:contrib/coccinelle/%.cocci=%) COCCI_PATCHES = $(addsuffix .patch,$(COCCI)) +COCCI_PATCHES_OK = $(COCCI_PATCHES:%=.build/%.ok) COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES)) -COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) +COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok) +COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -$(COCCI_PATCHES): GIT-SPATCH-DEFINES -$(COCCI_PATCHES): $(COCCI_SOURCES) -$(COCCI_PATCHES): %.patch: % - $(QUIET_SPATCH) \ - if test $(SPATCH_BATCH_SIZE) = 0; then \ - limit=; \ - else \ - limit='-n $(SPATCH_BATCH_SIZE)'; \ - fi; \ - if ! echo $(COCCI_SOURCES) | xargs $$limit \ - $(SPATCH) $(SPATCH_FLAGS) \ - --sp-file $< --patch . \ - >$@+ 2>$@.log; \ +.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) + $(call mkdir_p_parent_template) + $(QUIET_GEN) >$@ + +define cocci-rule + +## Rule for .build/$(1).patch/$(2); Params: +# 1 = $(1) +# 2 = $(2) +COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) +.build/$(1).patch/$(2): GIT-SPATCH-DEFINES +.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) +.build/$(1).patch/$(2): $(1) +.build/$(1).patch/$(2): .build/$(1).patch/% : % + $$(call mkdir_p_parent_template) + $$(QUIET_SPATCH)$$(SPATCH) $$(SPATCH_FLAGS) \ + --very-quiet \ + --sp-file $(1) --patch . \ + $$(SPATCH_PATCH_FLAGS) $$< >$$@ +endef + +define cocci-matrix-2 + +$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o))) +endef +define cocci-matrix-1 +$(foreach c,$(COCCI),$(call cocci-matrix-2,$(c))) +endef + +ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),) +$(eval $(call cocci-matrix-1)) +endif + +define spatch-rule + +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) + $(QUIET_SPATCH_M)cat $$^ >$$@ +endef + +$(COCCI_PATCHES_OK): .build/%.ok : % + $(QUIET_SPATCH_R)if test -s $<; \ then \ - cat $@.log; \ + cat $< | sed -e 's/^/ /'; \ exit 1; \ - fi; \ - mv $@+ $@; \ - if test -s $@; \ - then \ - echo ' ' SPATCH result: $@; \ - fi + fi && \ + >$@ + +ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),) +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n)))) +endif COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES @@ -3194,11 +3228,11 @@ $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test -coccicheck: $(COCCICHECK) +coccicheck: $(COCCICHECK_OK) # See contrib/coccinelle/README coccicheck-pending: coccicheck-test -coccicheck-pending: $(COCCICHECK_PENDING) +coccicheck-pending: $(COCCICHECK_PENDING_OK) .PHONY: coccicheck coccicheck-pending @@ -3466,7 +3500,7 @@ profile-clean: cocciclean: $(RM) -r .build/contrib/coccinelle - $(RM) $(COCCI_PATCHES)* + $(RM) $(COCCI_PATCHES) clean: profile-clean coverage-clean cocciclean $(RM) -r .build diff --git a/contrib/coccinelle/the_repository.pending.cocci b/contrib/coccinelle/the_repository.pending.cocci index 072ea0d9228..747d382ff5f 100644 --- a/contrib/coccinelle/the_repository.pending.cocci +++ b/contrib/coccinelle/the_repository.pending.cocci @@ -20,7 +20,6 @@ expression E; @@ expression E; -expression F; @@ - has_object_file_with_flags( + repo_has_object_file_with_flags(the_repository, diff --git a/shared.mak b/shared.mak index 33f43edbf9a..c0e88308069 100644 --- a/shared.mak +++ b/shared.mak @@ -69,8 +69,10 @@ ifndef V QUIET_SP = @echo ' ' SP $<; QUIET_HDR = @echo ' ' HDR $(<:hcc=h); QUIET_RC = @echo ' ' RC $@; - QUIET_SPATCH = @echo ' ' SPATCH $<; + QUIET_SPATCH = @echo ' ' SPATCH $@; QUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%); + QUIET_SPATCH_M = @echo ' ' SPATCH MERGE $$@; + QUIET_SPATCH_R = @echo ' ' SPATCH RESULT $<; ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; From patchwork Thu Aug 25 14:36:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12954841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F399AC28D13 for ; Thu, 25 Aug 2022 14:36:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242192AbiHYOgj (ORCPT ); Thu, 25 Aug 2022 10:36:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242094AbiHYOgb (ORCPT ); Thu, 25 Aug 2022 10:36:31 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 163F1AB4C3 for ; Thu, 25 Aug 2022 07:36:27 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id az27so4877237wrb.6 for ; Thu, 25 Aug 2022 07:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=8LT3wa/OX9D8faFizjEvyVbonccUPvwf6feJR3+miMQ=; b=PyRZhUUuONrs7NOzY6FS3I47mh8L7nOmSmE3dH5Dx8DsJwUpVFeoi1G8Pc6dkXe1jT /1sH8cBPauNiq7qEC1Opqx+1EbgzAOo5vb2DN8vVCumnfneYlpUiMyuR7Xw/BFDn/xlp vwCFH+1v76FA3SDDWfTqUgg3nsZr2R6rStwci8QNGw/8OBtk57U5Mdk650kOzbjtYdie wqbISOpIlhQMruNG6YojNr1xQ/JbYYakCdxK1J2cHoc8MD2w1HoA9Mv57sjj5enQnqlo c+cySTPSF2XWaKodofpBUlnbnmIrEY1eBT0QuI9sZ7+5JGv9x8bDE2Y5WDHgrQjYPu4H 6ELg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8LT3wa/OX9D8faFizjEvyVbonccUPvwf6feJR3+miMQ=; b=BYCC7+egau4Zk0+mqMQVRspS9pmrXeQelFUDeM/BIHUc9iuqFu93BK7k1IOQFz0rbX EwiLdjYm/mb88mtjmleX5FNQFzzqqfVMxJH4yqqxoa/8ULChHVqPzpQl4TrZCKz6NbTj em3NngIEToJy0OnoR37heIwsEksVbW8CTMvCl6mOCqkNsW83rQVmT5BuqxNXJ4JIQsFx goUATRt0YE2M43Qa10NLxsrlVUbw/ubPWNsTNPAxyvZ9+/27BFvcQiRZZtSPHq0ySOrh n5TvjPCll26ZWB+i1xR1jU0H3bkh4TaZCYHosztm2zVmIpFT3aTG/nQOQYqrCei1FWsN M1CA== X-Gm-Message-State: ACgBeo2vS7GdXmi1NmZqPMwAAK4yTCK2FvROK4xgyMLcvRbY1HheUfZV c3tM3WjxvqOntUOBqeos0UhjDRj6V7HupQ== X-Google-Smtp-Source: AA6agR5S9H4t/0rgQsSnSedby1Ypk8e5Q7NLbuoU0M+QGHb5FF1Or7FvJG/9270LgIhidFYKBx+sJA== X-Received: by 2002:a05:6000:178f:b0:221:7dcb:7cbf with SMTP id e15-20020a056000178f00b002217dcb7cbfmr2570201wrg.58.1661438184846; Thu, 25 Aug 2022 07:36:24 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id m27-20020a056000181b00b0022549ac786asm14418027wrh.47.2022.08.25.07.36.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 07:36:24 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH 4/5] cocci: make incremental compilation even faster Date: Thu, 25 Aug 2022 16:36:16 +0200 Message-Id: X-Mailer: git-send-email 2.37.2.1279.g64dec4e13cf In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Make the incremental compilation of "make coccicheck" even faster, at the cost of losing dependency discovery for a small whitelist of *.c files which we can't create *.o files for. This will break obscure edge cases since e.g.: time make -W block-sha1/sha1.h coccicheck Would correctly re-build before, as block-sha1/sha1.c uses block-sha1/sha1.h, but as it and these other $(USE_TINY_FOUND_H_SOURCES) files depended on the union of all such headers (and more, all of $(FOUND_H_SOURCES)) the above command would take ~10 seconds before. As noted in the preceding commit these obscure cases were broken before the preceding commit fixed them. I think it makes sense to make a correctness v.s. performance trade-off here. By doing this it becomes viable to e.g. run "make coccicheck" in the same loop that merges various topic branches down (along with "ccache" sped-up compilation). Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index a374204aaca..9410a587fc0 100644 --- a/Makefile +++ b/Makefile @@ -3160,6 +3160,16 @@ COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok) COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) +USE_TINY_FOUND_H_SOURCES = +USE_TINY_FOUND_H_SOURCES += compat/% +USE_TINY_FOUND_H_SOURCES += block-sha1/% +USE_TINY_FOUND_H_SOURCES += ppc/% + +TINY_FOUND_H_SOURCES = +TINY_FOUND_H_SOURCES += cache.h +TINY_FOUND_H_SOURCES += git-compat-util.h +TINY_FOUND_H_SOURCES += strbuf.h + .build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ @@ -3171,7 +3181,7 @@ define cocci-rule # 2 = $(2) COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) .build/$(1).patch/$(2): GIT-SPATCH-DEFINES -.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) +.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES)) .build/$(1).patch/$(2): $(1) .build/$(1).patch/$(2): .build/$(1).patch/% : % $$(call mkdir_p_parent_template) From patchwork Thu Aug 25 14:36:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12954843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ABA2C64990 for ; Thu, 25 Aug 2022 14:36:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242217AbiHYOgq (ORCPT ); Thu, 25 Aug 2022 10:36:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242109AbiHYOgc (ORCPT ); Thu, 25 Aug 2022 10:36:32 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4C6FABD48 for ; Thu, 25 Aug 2022 07:36:27 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id k9so24906973wri.0 for ; Thu, 25 Aug 2022 07:36:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Pus+R7DGKkj2ltQJo3J9Ko2zdcKTe7fI3iSNnQXvMWY=; b=imIFiQAi+dQTweNDarM9Z5FcW1IwifJjSIq7Hg6O3zJdxCJvBmaJZOH5bB3MLvemRr Jik7YkgJ0UQ557FJBLo+94zkeII0iFf157QQIafAV4/8gjiqloMRGtmt9lwvvPmu1wGU qlpv1Y5sDllCexMTTsV4qnBGJYgb0MIWRITzYwKg5lZEP8+tGQqtWuPTioVPy8a9gsl7 pKyngA1rZkqPHz9z/iw+8dXWPooWBNQBh1+VQD8XUu+OJfh96P3GWV+yO0pL1LsM5P5v 1ig9KDWOtk60877/NswBOTTMmICO+eLN2Gn3c4LGbe7/2YMgyfoZP1To0mpUEQ+FaFdN f0dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Pus+R7DGKkj2ltQJo3J9Ko2zdcKTe7fI3iSNnQXvMWY=; b=qTJHsjb6GAK9w5LdBuQb27uUMX5/Id0QagTHuAPLFSKyFJjstqfjr7KjuTyPawUkwX f1T+1BxU3suBHDDhkpPzjhVaBcMGVGGGnTEE+Tg+lNv68rbWRbJqulz5bIqUPuP+I4sM hQWjzM2rviZI7/xcQhU51EHgq99PIzN86neBIg2avBQ4ySvYkfgotnQ/5K4jqVdYql4U fwUsR7R1VM5sFz71hLaS0g4ttGu6kGkCIJ9XNWVXZ5u9hzlFqfk5mky/1uFm5JGxbepY h9T2boQYC+R7T9HIZbDGtsk9Hkjc+ty0JxscrHXaEEKyeaeH8YJhrqthuJpozG4MFOMg kY4A== X-Gm-Message-State: ACgBeo0Vk+IzPjQh0tCRQ2m6oB7scUh8lm/fl5T35y9tJQtXKuxqoMub f3qK/pkyZs74/2s+Q0uUq0hXqQ9pzDu3lw== X-Google-Smtp-Source: AA6agR6Um+VyUbbeP27nOXznvsCiN0sRK3ogecEnnnWQKmT1ldlNPJxGuQY448q/ByA8R+3rkZb83w== X-Received: by 2002:a05:6000:11c8:b0:225:3316:6957 with SMTP id i8-20020a05600011c800b0022533166957mr2503632wrx.218.1661438185793; Thu, 25 Aug 2022 07:36:25 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id m27-20020a056000181b00b0022549ac786asm14418027wrh.47.2022.08.25.07.36.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 07:36:25 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , =?utf-8?b?w4Z2YXIg?= =?utf-8?b?QXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH 5/5] spatchcache: add a ccache-alike for "spatch" Date: Thu, 25 Aug 2022 16:36:17 +0200 Message-Id: X-Mailer: git-send-email 2.37.2.1279.g64dec4e13cf In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a rather trivial "spatchcache", with this running e.g.: make clean make contrib/coccinelle/free.cocci.patch Is cut down from ~20s to ~3s on my system (and much of it is fixable shell overhead). This uses "redis" as a cache by default, but it's configurable. See the embedded documentation. This is *not* like ccache in that we won't cache failed spatch invocations, or those where spatch suggests changes for us. Those cases are so rare that I didn't think it was worth the bother, by far the most common case is that it has no suggested changes. Because we narrow the cache to that we don't need to save away stdout, stderr & the exit code. We simply cache the cases where we had no suggested changes. Another benchmark is to compare this with the previous SPATCH_BATCH_SIZE=N, as noted in [1]. Before this (on my 8 core system) running: make clean; time make contrib/coccinelle/array.cocci.patch SPATCH_BATCH_SIZE=0 Would take 33s, but with the preceding changes running without this "spatchcache" is slightly slower, or around 35s: make clean; time make contrib/coccinelle/array.cocci.patch Now doing the same with SPATCH=contrib/coccinelle/spatchcache will take around 6s, but we'll need to compile the *.o files first to take full advantage of it (which can be fast with "ccache"): make clean; make; time make contrib/coccinelle/array.cocci.patch SPATCH=contrib/coccinelle/spatchcache 1. https://lore.kernel.org/git/YwdRqP1CyUAzCEn2@coredump.intra.peff.net/ Signed-off-by: Ævar Arnfjörð Bjarmason --- contrib/coccinelle/spatchcache | 217 +++++++++++++++++++++++++++++++++ 1 file changed, 217 insertions(+) create mode 100755 contrib/coccinelle/spatchcache diff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache new file mode 100755 index 00000000000..f4f44700cfc --- /dev/null +++ b/contrib/coccinelle/spatchcache @@ -0,0 +1,217 @@ +#!/bin/sh +# +# spatchcache: a poor-man's "ccache"-alike for "spatch" in git.git +# +# This caching command relies on the peculiarities of the Makefile +# driving "spatch" in git.git, in particular if we invoke: +# +# make +# make coccicheck +# +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with +# "gcc" and "clang") write e.g. a .depend/grep.o.d for grep.c, when we +# compile grep.o. +# +# The .depend/grep.o.d will have the full header dependency tree of +# grep.c, and we can thus cache the output of "spatch" by: +# +# 1. Hashing all of those files +# 2. Hashing our source file, and the *.cocci rule we're +# applying +# 3. Running spatch, if suggests no changes (by far the common +# case) we invoke "spatchCache.getCmd" and +# "spatchCache.setCmd" with a hash SHA-256 to ask "does this +# ID have no changes" or "say that ID had no changes> +# 4. If no "spatchCache.{set,get}Cmd" is specified we'll use +# "redis-cli" and maintain a SET called "spatch-cache". Set +# appropriate redis memory policies to keep it from growing +# out of control. +# +# This along with the general incremental "make" support for +# "contrib/coccinelle" makes it viable to (re-)run coccicheck +# e.g. when merging integration branches. +# +# To use this, simply set SPATCH to +# contrib/coccinelle/spatchcache. Then optionally set: +# +# [spatchCache] +# # path to a custom spatch +# spatch = ~/g/coccicheck/spatch.opt +# +# As well as this trace config (debug implies trace): +# +# trace = false +# debug = false +# +# Setting "trace" to "true" allows for seeing when we have a cache HIT +# or MISS. + +set -e + +## Our own configuration & options +debug=$(git config --bool "spatchCache.debug") +if test "$debug" != "true" +then + debug= +fi +if test -n "$debug" +then + set -x +fi + +trace=$(git config --bool "spatchCache.trace") +if test "$trace" != "true" +then + trace= +fi +if test -n "$debug" +then + # debug implies trace + trace=true +fi + +trace_it () { + if test -z "$trace" + then + return + fi + echo "$@" >&2 +} + +spatch=$(git config --path "spatchCache.spatch" || :) +if test -n "$spatch" +then + if test -n "$debug" + then + trace_it "custom spatchCache.spatch='$spatch'" + fi +else + spatch=spatch +fi + +set=$(git config spatchCache.setCmd || :) +get=$(git config spatchCache.getCmd || :) + +## Parse spatch()-like command-line for caching info +arg_sp= +arg_file= +args="$@" +spatch_opts() { + while test $# != 0 + do + arg_file="$1" + case "$1" in + --sp-file) + arg_sp="$2" + ;; + esac + shift + done +} +spatch_opts "$@" +if ! test -f "$arg_file" +then + arg_file= +fi + +hash_for_cache() { + # Parameters that should affect the cache + echo "spatch=$spatch" + echo "args=$args" + echo + + # Our target file and its dependencies + git hash-object "$1" "$2" $(grep -E -o '^[^:]+:$' "$3" | tr -d ':') +} + +# Sanity checks +if ! test -f "$arg_sp" && ! test -f "$arg_file" +then + echo $0: no idea how to cache "$@" >&2 + exit 128 +fi + +# Main logic +d=$(dirname "$arg_file") +b=$(basename "$arg_file") +bwoext="${b%.c}" +dep="$d/.depend/$bwoext.o.d" + +if ! test -f "$dep" +then + trace_it "$0: have no '$dep' for '$arg_file', can't cache!" + exec "$spatch" "$@" +fi + +if test -n "$debug" +then + trace_it "$0: The full cache input for '$arg_sp' '$arg_file' '$dep'" + hash_for_cache "$arg_sp" "$arg_file" "$dep" >&2 +fi +sum=$(hash_for_cache "$arg_sp" "$arg_file" "$dep" | git hash-object --stdin) + +trace_it "$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'" + +getret= +if test -z "$get" +then + if test $(redis-cli SISMEMBER spatch-cache "$sum") = 1 + then + getret=0 + else + getret=1 + fi +else + $set "$sum" + getret=$? +fi + +if test "$getret" = 0 +then + trace_it "$0: HIT for '$arg_file' with '$arg_sp'" + exit 0 +else + trace_it "$0: MISS: for '$arg_file' with '$arg_sp'" +fi + +out="$(mktemp)" +err="$(mktemp)" + +set +e +"$spatch" "$@" >"$out" 2>"$err" +ret=$? +set -e + +if test $ret = 0 && ! test -s "$out" && ! test -s "$err" +then + rm -f "$out" "$err" + + trace_it "$0: SET: for '$arg_file' with '$arg_sp'" + + setret= + if test -z "$set" + then + if test $(redis-cli SADD spatch-cache "$sum") = 1 + then + setret=0 + else + setret=1 + fi + else + "$set" "$sum" + setret=$? + fi + + if test "$setret" != 0 + then + echo "FAILED to set '$sum' in cache!" >&2 + exit 128 + fi +else + trace_it "$0: NO CACHE (have changes): for '$arg_file' with '$arg_sp'" + cat "$out" + cat "$err" >&2 + rm -f "$out" "$err" + exit "$ret" +fi +rm -f "$out" "$err"