From patchwork Tue Mar 2 20:51:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12112987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59C88C433E9 for ; Wed, 3 Mar 2021 06:51:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 051E064ED9 for ; Wed, 3 Mar 2021 06:51:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355387AbhCCGfI (ORCPT ); Wed, 3 Mar 2021 01:35:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1840023AbhCBUwC (ORCPT ); Tue, 2 Mar 2021 15:52:02 -0500 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 312B2C061756 for ; Tue, 2 Mar 2021 12:51:22 -0800 (PST) Received: by mail-ed1-x52e.google.com with SMTP id h10so26953561edl.6 for ; Tue, 02 Mar 2021 12:51:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fUCmw+8TTz/rP3fAWMHG5cJN7leU0O27qXW8cEHfW+M=; b=PzKCphL3AXcsllMOdIqRSqYG2S5aoI65KfV8Ua6XsDTFMCszYuCMoOYOnZajea9CpN TzQJ/KsYzBeXSjopj2OBfWWpK3Jo2KJePxV4Uc7fS3+Sk/nXFRDZJeqW+SVxd+V1Q+bZ sA4wl5va+aasYp3afCGO2GNrkKm5tGGptWPsd2V13VXEyxoxlvDmoEh5ezN6WT9l06/A uPBp0Dcfb9a3gXhlal/6rWgFgoJHeRu07uJD3qItqLTOerl9Y+VXXtjBknfXHnAdjF2c yZOzq90e+T3xUiq6HVFnN0Tmmu9XJVMcl+HVY8e3qnaw2n7EZtDWWSyMn8IthS+Yh7gT Yu6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fUCmw+8TTz/rP3fAWMHG5cJN7leU0O27qXW8cEHfW+M=; b=WmvwmDcoWCNs0B7Gpoc1AnEiKMD4uqdD+E1DAyOYbZBg8iZb8i6y3j5b4JM3a/7VEX wnyXrNleq4VC2VA/PmFLCdmK5ENRzeD3F1j3Wx2iVOvFkdGnFvHG8hbBG8Bu6pjcjFr/ lWxwG6u5VYmQMV544ci9bQF9mm+hrVtiwuvWDIQR8E4gmn2745S8zP+1F90VoY/D41cD OuenLnzn2sjjrgmM70zh3uiubYe+39zw10G7MI/cqNP573uJGnXnM4jV3Rag/GJSMCbb fctCA2GdW0OA+yEYaAONoYMOrzMGINzmcRe3Jgx6mm2Z21HvN9aYyI7xYZmw4sYmTAw3 onZQ== X-Gm-Message-State: AOAM531o6S5N0/FqcgpZv8xkzi/txc5oXhdXqC9WlDn0DBpBEKRmtINc CjbSfgvUZjGumCrIaXQp1EGtoGtaTSyBYQ== X-Google-Smtp-Source: ABdhPJyTOc6qzD8hp/L6+HYo+4/BAvTfwuejogd5fiwWJyCG9GY+Ru8nGSqYY2/Y0zWhaDh0Uh4ZUw== X-Received: by 2002:a50:e80c:: with SMTP id e12mr22648728edn.229.1614718278759; Tue, 02 Mar 2021 12:51:18 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id s2sm19231513edt.35.2021.03.02.12.51.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Mar 2021 12:51:18 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?q?Ren=C3=A9_Scharfe?= , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , Eric Wong , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH] Makefile: fix bugs in coccicheck and speed it up Date: Tue, 2 Mar 2021 21:51:03 +0100 Message-Id: <20210302205103.12230-1-avarab@gmail.com> X-Mailer: git-send-email 2.31.0.rc0.126.g04f22c5b82 In-Reply-To: <20190506234334.GA13296@sigill.intra.peff.net> References: <20190506234334.GA13296@sigill.intra.peff.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org I've often wondered why "make coccicheck" takes so long. This change speeds it up by ~2x and makes it use much less memory. Or a reduction of a max of around ~2GB per-process (under the old SPATCH_BATCH_SIZE=0) to around ~200MB. Running the full "make coccicheck" now takes ~50 seconds with -j8 on my machine, v.s. ~2x of that before. I've got 64GB of memory on that machine, or it would be much slower. Why has it been so slow? Because I think we've always been running it in entirely the wrong mode for what we wanted, and much of the previous fixing of this target has involved re-arranging the deck chairs on that particular Titanic. What we really want to do with coccicheck is to do search/replacements in all our *.c and *.h files. This is now what we do, and we'll process a default of 64 files at a time. What we were doing before was processing all our *.c files, and for each of those *.c files we'd recursively look around for includes and see if we needed to search/replace in those too. That we did that dates back to [1] when we were only processing *.c files, and it was always very redundant. We'd e.g. visit the likes of strbuf.h lots of times since it's widely used as an include. Then in the most recent attempt to optimize coccicheck in [2] this anti-pattern finally turned into a bug. Namely: before this change, if your coccicheck rule applied to e.g. making a change in strbuf.h itself we'd get *lots* of duplicate hunks applying the exact same change, as concurrent spatch processes invoked by xargs raced one another. In one instance I ended up with 27 copies of the same hunk in a strbuf.patch. Setting SPATCH_BATCH_SIZE=0 and processing all the files in one giant batch mitigated this. I suspect the author of [2] either mostly ran in that mode, or didn't test on changes that impacted widely used header files. So since we're going to want to process all our *.c and *.h let's just do that, and drop --all-includes for --no-includes. It's not spatch's job to find our sources, we're doing that. If someone is manually tweaking COCCI_SOURCES they can just tweak SPATCH_FLAGS too. I'm entirely removing SPATCH_BATCH_SIZE. If you want to tweak it you can tweak SPATCH_XARGS_FLAGS to e.g. "-n 256", or "-P 4 -n 128". But in my testing it isn't worth it to tweak SPATCH_XARGS_FLAGS for a full "make coccicheck". I'm also the whole "cat $@.log" introduced in [3]. Since we don't call this in a loop anymore (and xargs will early-exit) we can just rely on standard V=1 for debugging issues. 1. a9a884aea5 (coccicheck: use --all-includes by default, 2016-09-30) 2. 960154b9c1 (coccicheck: optionally batch spatch invocations, 2019-05-06) 3. f5c2bc2b96 (Makefile: detect errors in running spatch, 2017-03-10) Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/Makefile b/Makefile index dd08b4ced0..2108df8913 100644 --- a/Makefile +++ b/Makefile @@ -1195,11 +1195,20 @@ PTHREAD_CFLAGS = SPARSE_FLAGS ?= SP_EXTRA_FLAGS = -Wno-universal-initializer -# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will -# usually result in less CPU usage at the cost of higher peak memory. -# Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_FLAGS = --all-includes --patch . -SPATCH_BATCH_SIZE = 1 +SPATCH_FLAGS = --no-includes --patch . +# For the 'coccicheck' target; Tweaking SPATCH_XARGS_FLAGS is +# generally not neccesary with a top-level -jN. +# +# To get concurrency when targeting a single +# contrib/coccinelle/%.patch use e.g. "-P" if your xargs(1) supports +# it: +# +# make contrib/coccinelle/strbuf.cocci.patch SPATCH_XARGS_FLAGS="-P 8 -n 64" +# +# Or a combination of the two: +# +# make -j4 coccicheck SPATCH_XARGS_FLAGS="-P 2 -n 64" +SPATCH_XARGS_FLAGS = include config.mak.uname -include config.mak.autogen @@ -2852,24 +2861,18 @@ check: config-list.h command-list.h exit 1; \ fi -FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) +FOUND_C_SOURCES = $(filter %.c %.h,$(shell $(FIND_SOURCE_FILES))) COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) $(QUIET_SPATCH) \ - if test $(SPATCH_BATCH_SIZE) = 0; then \ - limit=; \ - else \ - limit='-n $(SPATCH_BATCH_SIZE)'; \ - fi; \ - if ! echo $(COCCI_SOURCES) | xargs $$limit \ - $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ - >$@+ 2>$@.log; \ - then \ - cat $@.log; \ - exit 1; \ - fi; \ - mv $@+ $@; \ + $(RM) $@+ $@.log && \ + echo $(COCCI_SOURCES) | \ + xargs \ + -n 32 $(SPATCH_XARGS_FLAGS) \ + $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ + >>$@+ 2>>$@.log && \ + mv $@+ $@ && \ if test -s $@; \ then \ echo ' ' SPATCH result: $@; \ From patchwork Fri Mar 5 17:07:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12118923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 377F3C43332 for ; Fri, 5 Mar 2021 17:08:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D79065094 for ; Fri, 5 Mar 2021 17:08:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231440AbhCERHy (ORCPT ); Fri, 5 Mar 2021 12:07:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231377AbhCERHq (ORCPT ); Fri, 5 Mar 2021 12:07:46 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0A23C061574 for ; Fri, 5 Mar 2021 09:07:45 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id l22so2087683wme.1 for ; Fri, 05 Mar 2021 09:07:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C8E2PK7+SrZZjDhlBRMGUCZd7jdq6n99b5VyzNRE4LM=; b=N1hgc3ldfioG79Wd1QrAp2nhUS3kw3wT9G+NKDNtzpeotcIkm1hgyHktTM4W2WIUrL TrzCYqeew0kMsEaqu7/Pn9TxyHIzh+OirHrWEI2QB/rCOh2111PFRuVF4jkDsmP1tPB3 O3iKcqECmA+4DFBwjIK1FoRxc2hOk7MNSKK2nmruI6+OAmfTHk8puXV4389vVH6hP/Rd Dt+Y/FGuVEMrRveaXk/VbLaLA05DqNTslkUgX3d8OEVqTiICEtAwCgDiD1JlEb+d1s5E P+01Th3h6ClAHrJJDh72cdEGvy8Oh8mOauHERk8nGvwE971lBIpSRu3Cp7EdTsn0E4qL PXvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C8E2PK7+SrZZjDhlBRMGUCZd7jdq6n99b5VyzNRE4LM=; b=kv2i5wrZ+POu8hZmaMFBHwJ7tA7Qt0dEItkPWLZzCETfTtX4vvKOfpT7phqx2F/+kJ Um6QJ1gi0ChkUQC9Spddgbpr7NsniRcXhISzpJHxfKZBVE5sM7aotXGevtJxvMoGuri8 W58/rgepeMfN1HeJD1nc71+5eydRdmW1bpEnvskgdP0MEX88YWVSZFZPNoQ/2nhcNt7q u2SsSK+Hi+kLKJ/cqJ8vQ0ckxJtHTgKX5xwQcCObObwxZQ0ZITYqfj3VrfvP5ShxuIV6 mYcuKxNqnW7qtf5vhGg0pAxk2UH7wPxD73J4rTM+iCs/sw2kOF7l1N3uHcOqrHSCoKV5 U7Lw== X-Gm-Message-State: AOAM5316zj4lbyBHw5aTFHd05hsh9jM2x7xbus3mIizxpxUHDPs3BSiE vCNxOdzNeXllL6w/RLA+rFwpR9iFZO7CPA== X-Google-Smtp-Source: ABdhPJzVkqHLWfxdVYSpzlyAVRxQRNSl+Fn7ZOEeelEaNEqZ3jqniJk8oPpOPo4IdMzGJz8+AGqDTQ== X-Received: by 2002:a7b:cf2f:: with SMTP id m15mr9677760wmg.177.1614964064028; Fri, 05 Mar 2021 09:07:44 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i4sm5115328wmq.12.2021.03.05.09.07.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Mar 2021 09:07:43 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?q?Ren=C3=A9_Scharfe?= , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , Eric Wong , Denton Liu , =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Date: Fri, 5 Mar 2021 18:07:22 +0100 Message-Id: <20210305170724.23859-3-avarab@gmail.com> X-Mailer: git-send-email 2.31.0.rc0.126.g04f22c5b82 In-Reply-To: <20210302205103.12230-1-avarab@gmail.com> References: <20210302205103.12230-1-avarab@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the coccicheck target to run on all of our *.c and *.h files with --no-includes, instead of only on the *.c files with --all-includes. This speeds it up significantly and reduces its memory usage, since it doesn't need to parse N includes for every file it visits. See [1] for a discussion thread about this commit with some timings for details, but briefly: This change speeds it up by ~2x and makes it use much less memory. Or a reduction of a max of around ~2GB per-process (under the old SPATCH_BATCH_SIZE=0) to around ~200MB. Looking at the history of the coccicheck target I think we've always been running it in the wrong mode for what we wanted to achieve: - When it was added in 63f0a758a06 (add coccicheck make target, 2016-09-15) it explicitly processed only the %.c files. - Then in a9a884aea57 (coccicheck: use --all-includes by default, 2016-09-30) it started processing the %.h files by looking around for its own includes. Let's instead just point it to both our *.c and *.h files, then there's no need to have it recursively look around for included files to change. Setting --no-includes would not work if we expected to set COCCI_SOURCES to a subset of our source files, but that's not what we're doing here. If anyone manually tweaks COCCI_SOURCES they'll now need to tweak SPATCH_FLAGS too. The speed and correctness we gain is worth that small trade-off. Using --no-includes also fixes a subtle bug introduced in 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06) with duplicate hunks being written to the generated *.patch files. This is because that change altered a file-at-a-time for-loop to an invocation of "xargs -n X". This would not matter for most other programs, but it matters for spatch. This is because each spatch invocation will maintain shared lock files in /tmp, check if files being parsed were changed etc. I haven't dug into why exactly, but it's easy to reproduce the issue[2]. The issue goes away entirely if we just use --no-includes, which as noted above would have made sense even without that issue. 1. https://lore.kernel.org/git/20210302205103.12230-1-avarab@gmail.com/ 2. A session showing racy spatch under xargs -n X: $ cat test.cocci @@ expression E1; @@ - strbuf_avail(E1) + strbuf_has(E1) $ for i in 1 2 4 16 64 128 512 do echo with xargs -n $i: && echo *.c | xargs -n $i spatch --sp-file \ test.cocci --all-includes --patch . 2>/dev/null | \ grep -F +++ | sort | uniq -c done with xargs -n 1: 1 +++ b/convert.c 1 +++ b/strbuf.c with xargs -n 2: 1 +++ b/convert.c 1 +++ b/strbuf.c with xargs -n 4: 1 +++ b/convert.c 1 +++ b/strbuf.c with xargs -n 16: 1 +++ b/convert.c 1 +++ b/strbuf.c 2 +++ b/strbuf.h with xargs -n 64: 1 +++ b/convert.c 1 +++ b/strbuf.c 2 +++ b/strbuf.h with xargs -n 128: 1 +++ b/convert.c 1 +++ b/strbuf.c 2 +++ b/strbuf.h with xargs -n 512: 1 +++ b/convert.c 1 +++ b/strbuf.c 1 +++ b/strbuf.h Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index f881b558c44..798a0517131 100644 --- a/Makefile +++ b/Makefile @@ -1196,7 +1196,8 @@ SPARSE_FLAGS ?= SP_EXTRA_FLAGS = -Wno-universal-initializer # For the 'coccicheck' target -SPATCH_FLAGS = --all-includes --patch . +SPATCH_FLAGS = --no-includes --patch . + # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. @@ -2853,7 +2854,7 @@ check: config-list.h command-list.h exit 1; \ fi -FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES))) +FOUND_C_SOURCES = $(filter %.c %.h,$(shell $(FIND_SOURCE_FILES))) COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) From patchwork Fri Mar 5 17:07:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12118925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F160DC4332E for ; Fri, 5 Mar 2021 17:08:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 70C0D6508F for ; Fri, 5 Mar 2021 17:08:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231434AbhCERHy (ORCPT ); Fri, 5 Mar 2021 12:07:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231378AbhCERHq (ORCPT ); Fri, 5 Mar 2021 12:07:46 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DA76C061574 for ; Fri, 5 Mar 2021 09:07:46 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id l22so2087726wme.1 for ; Fri, 05 Mar 2021 09:07:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X9bYxnWT1eHHWGWfysmgHlFH2lu/AG24plq/+MsIMw4=; b=qKlYe+AUhBib2+cdt/NK+yp35TH0F6DM2mHG7bzmDBdBXpsmnglxzZx+15gAF3Vlqp vr2T+BlmMMQGeLIXn9j0ysBLyahVaqQb4bu3DoyhArDQSjYve+hcF/fzEm9NnKX8lkIL bVLKbbutPRC2MfeIirLMIReCYNbQhBTbp56y32gAfWmYp3sXO34LYOHoBoMjCoRC9Lg7 dk1Z9KRvCsbXW7G8b6cZBzZ6FTrVKEAwtUUbnNfdvXPhxMy26uJjSFmQ8+s+iIinuBaY msIDknRu2m1l2AZWUcBRN42QGuhs1NDRHtCZHqC3uXur140BWJSglaDloQ7v8kymnTL7 qoOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X9bYxnWT1eHHWGWfysmgHlFH2lu/AG24plq/+MsIMw4=; b=JrbL6ekxC4UePlHmYcIsDq8s1ndix8sYANSYE6uLwI4R3om332x4Rj/rUHR74SRY7C HW2+YAjextZW+zGGUkA6wszkJ5OVlNSj1yP9l115++es98i6I2KoRMuBHznlptsUBriq NM1D+hXvYAH2vb8JOuC+twbh9E12ZpRFh16L+6WgwaKNQBaw1Iydof54aw891Jj0LI4w aj0ULLXEmIK8E0avDvxG9SOGL1TAAeZ97uWgMUx/4s7/5RCiyXwNYnLFmHDDPtHIJ+L+ Z2FdPx+BhpoQpjSrJJLe5d5fuNHG41yXQIKeUytLoNEol+McBMEtGhW/92y4ip3uYSWe rBdg== X-Gm-Message-State: AOAM532UeTvL170D9lYaFz0fHUhDrq7sFYGfS56WJsKYdqYaht9HK1TV TOJu4mCY2/UIYPxLHOKrPqbGFWmh6oqW9w== X-Google-Smtp-Source: ABdhPJzonipVzCL1MC8JXu6E/e1zB1vbEjx+gBxwQIBx3s+vnAnYb9HyaxSW7Z2TRIwkd8Zg+1/Meg== X-Received: by 2002:a1c:e341:: with SMTP id a62mr10098798wmh.152.1614964064919; Fri, 05 Mar 2021 09:07:44 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i4sm5115328wmq.12.2021.03.05.09.07.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Mar 2021 09:07:44 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?q?Ren=C3=A9_Scharfe?= , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , Eric Wong , Denton Liu , =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 3/4] Makefile/coccicheck: allow for setting xargs concurrency Date: Fri, 5 Mar 2021 18:07:23 +0100 Message-Id: <20210305170724.23859-4-avarab@gmail.com> X-Mailer: git-send-email 2.31.0.rc0.126.g04f22c5b82 In-Reply-To: <20210302205103.12230-1-avarab@gmail.com> References: <20210302205103.12230-1-avarab@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Extend the SPATCH_BATCH_SIZE facility added in 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06) and bcb4edf7af7 (coccicheck: make batch size of 0 mean "unlimited", 2019-05-08) to allow for both setting SPATCH_BATCH_SIZE=N, and also to have a more advanced SPATCH_XARGS argument. The reason to replace the "$$limit" is that now you actually see under V=1 what argument your program will get invoked with. The reason for the "9999" limit is that if you e.g. try to define an "$(XARGS)" which is conditionally an empty string or not depending on this setting then e.g.: echo $(FILES) | $(XARGS) $(XARGS_FLAGS) $(SPATCH) Over multiple lines with "\" will error out. I think just setting it to "xargs -n 9999" as a trivial workaround is the best solution here. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 798a0517131..9fa715e4037 100644 --- a/Makefile +++ b/Makefile @@ -1203,6 +1203,25 @@ SPATCH_FLAGS = --no-includes --patch . # Setting it to 0 will feed all files in a single spatch invocation. SPATCH_BATCH_SIZE = 1 +# For the 'coccicheck' target; SPATCH_XARGS can be used to manually +# tweak the xargs invocation. By default we invoke "xargs -n 1", and +# "xargs -n 9999" under SPATCH_BATCH_SIZE=0. +# +# Setting SPATCH_XARGS overrides SPATCH_BATCH_SIZE. To get concurrency +# when targeting a single contrib/coccinelle/%.patch use e.g. "-P" if +# your xargs(1) supports it: +# +# make contrib/coccinelle/strbuf.cocci.patch SPATCH_XARGS="xargs -P 8 -n 8" +# +# Or a combination -jN and "xargs -P": +# +# make -j4 coccicheck SPATCH_XARGS="xargs -P 2 -n 8" +ifeq (0,$(SPATCH_BATCH_SIZE)) +SPATCH_XARGS = xargs -n 9999 +else +SPATCH_XARGS = xargs -n $(SPATCH_BATCH_SIZE) +endif + include config.mak.uname -include config.mak.autogen -include config.mak @@ -2859,12 +2878,7 @@ COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES)) %.cocci.patch: %.cocci $(COCCI_SOURCES) $(QUIET_SPATCH) \ - if test $(SPATCH_BATCH_SIZE) = 0; then \ - limit=; \ - else \ - limit='-n $(SPATCH_BATCH_SIZE)'; \ - fi; \ - if ! echo $(COCCI_SOURCES) | xargs $$limit \ + if ! echo $(COCCI_SOURCES) | $(SPATCH_XARGS) \ $(SPATCH) --sp-file $< $(SPATCH_FLAGS) \ >$@+ 2>$@.log; \ then \ From patchwork Fri Mar 5 17:07:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12118921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 218E7C43331 for ; Fri, 5 Mar 2021 17:08:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DD4116509F for ; Fri, 5 Mar 2021 17:08:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231443AbhCERHz (ORCPT ); Fri, 5 Mar 2021 12:07:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231388AbhCERHr (ORCPT ); Fri, 5 Mar 2021 12:07:47 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54A56C061574 for ; Fri, 5 Mar 2021 09:07:47 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id m7so2092404wmq.0 for ; Fri, 05 Mar 2021 09:07:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=He+Wd5ZnP+7vfagJX6lyetcZiYpi2tHFw0QJOqeznsM=; b=uIFMfieD7n7NsiJIF27XrD7Xxnu+xYGEFViNa08VgRVG7EFMmzAUcnXmIHEnAgYGUg 60RAjK0uBpbLue+2EcIJRxFpU1VZX1Pnzo9VkRFqucUITdji7Cneu/OYPRn7DaKSm9kD APoYf9rOvEtqcOU6CtEl1CjkXvkRoAbQp5aGoOQLMk8IAg+UDoOmAHfpRbfHxtv0pIJA hdvFqYFQQBQdemtuxMiFbFTaBb320SzKKfRM383wi47/7SPx4BI7lA53YwjxPAxhU6QV FH9glUIC65ay5FLv8uwkLEdyGTs5uqWCWDD+xIFAYr+93nmusewszgw+5nq7L/U3viX0 y8fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=He+Wd5ZnP+7vfagJX6lyetcZiYpi2tHFw0QJOqeznsM=; b=YVQpSjE1B07Va9cuVYc3dlqKf0VkUtZOCENttDnfo4LNoYnpNNSSiwcNMLizvHKtih tseFETH8mToJUn0TRNsT48MwwL6QPXiPcS7FaXV07/ys7nYGblR0hXN5PMP8VQdjz1BZ Q5xbh1tlorWD/0tYH7kyc0aEDtXI/4qFWWLJGMYLZED0B1pa7XhTgY/zIkMK1Qso6sZF Hj9YjhsoKK9xcbG/i5ih6o4U7rXLFThRDzZ7rYxPzfGmE5k29viu3ppGozIeenpOHEu8 FH0J8X1pjiTgNDRkECrHKeUQE1PGcahZMz5FPLych5NDVyZkXpsz/74Zz1+JmYxdft5N Jsqw== X-Gm-Message-State: AOAM531m2pv2uoYYHHpz+QYp9cjRKFguHnBsEd1DSNdhY/AWKMkuwLze uNFrIb6gX8jl3FVBGJ7mDF5iAcMTPYO9Og== X-Google-Smtp-Source: ABdhPJzE7XkO7ExAUDnYTydJ8N6uW3lFsCap2SyJGEtSjk4fKodEs53JJYv0WiMF1d/lG23UZ4yzuA== X-Received: by 2002:a1c:2cc4:: with SMTP id s187mr10022080wms.4.1614964065888; Fri, 05 Mar 2021 09:07:45 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id i4sm5115328wmq.12.2021.03.05.09.07.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Mar 2021 09:07:45 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?q?Ren=C3=A9_Scharfe?= , Jeff King , =?utf-8?q?SZEDER_G=C3=A1bor?= , Eric Wong , Denton Liu , =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [PATCH v2 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Date: Fri, 5 Mar 2021 18:07:24 +0100 Message-Id: <20210305170724.23859-5-avarab@gmail.com> X-Mailer: git-send-email 2.31.0.rc0.126.g04f22c5b82 In-Reply-To: <20210302205103.12230-1-avarab@gmail.com> References: <20210302205103.12230-1-avarab@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Change the "xargs -n X" limit for running spatch from 1 to 8, as suggested by Jeff King[1]. Now that we're not using --all-includes anymore this is going to take much less memory, so bumping the limit should be OK. 1. https://lore.kernel.org/git/YEIGzXMDax83cwAx@coredump.intra.peff.net/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 9fa715e4037..9b4e350ed12 100644 --- a/Makefile +++ b/Makefile @@ -1201,7 +1201,7 @@ SPATCH_FLAGS = --no-includes --patch . # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will # usually result in less CPU usage at the cost of higher peak memory. # Setting it to 0 will feed all files in a single spatch invocation. -SPATCH_BATCH_SIZE = 1 +SPATCH_BATCH_SIZE = 8 # For the 'coccicheck' target; SPATCH_XARGS can be used to manually # tweak the xargs invocation. By default we invoke "xargs -n 1", and